MongoDB Large Generic Object Catalog. Is one large collection feasible? -

overview

we planning general object storage catalog our central data repository operations within our organization. have basic framework of system in place , point need decide on object storage implementation.

approach

our basic design approach have multiple collections contain metadata objects stored. each collection describe different "group" of data. then, have single gridfs store of objects. idea have single repository of objects each object can described 1 or more metadata record.

our concerns

we expect bursts of between 5-50 read , write operations per second. include 1 or more write metadata collections , 1 gridfs per write , same reads. there 25-50 million records in gridfs object catalog @ 1 time. index on gridfs key-value-pair index on array { object-metadata-identity: [ {record-identifier: uuid}, {...}, ...] } index on record-identifier. each record in gridfs have varying size of between <1kb - 9gb 90% of data closer 1kb range.

alternative considerations

we have considered several other approaches including combinations of following: separating objects multiple gridfs collections using hard drive larger files using regular collections files less 16mb

basic requirements

our goal have generic, extensible solution able grow our ever changing requirements. solution has fast because routing time sensitive data. solution has reliable (three 9's) once production ready.

summary

our overall question is, can mongodb support fast read/writes on gridfs reliability many records, or going need explore combination of other solutions supplement object storage.