Where would you keep 500 million documents? If you are Buffer, the social media management company, the answer is at MongoHQ. Buffer CTO Sunil Sadasivin talks about how it moved from early growth with third party metric tracking to building its own custom metrics and tracking system in an article on MongoDB's blog.
The company decided, last year, to build that analytics system on MongoDB and Amazon's SQS Queue and chose MongoHQ to host the data. "They've been super helpful with any challenges in scaling a db like ours" says Sadasivin, "Since our setup is write heavy, we've initially set up a 400GB SSD replica set. As of today (May 16) we have 90 collections and are storing over 500 million documents.".
Thats the data from over three million users, including tracking of page views along with context, every user API call logged and experiment data for feature measurement. Buffer pulls it all together and uses MongoDB's aggregation framework to build their metrics and understanding of how their systems are working.
There's lots of great information in there, from how you should plan for high disk I/O with SSD backed MongoDB with replica sets (as supplied by MongoHQ) to managing data structure and how to avoid the pitfalls like hitting the MongoDB document limit. Buffer plans to open source their platform later this year, so get an insight now into what you could be deploying with MongoHQ next year.
- How Buffer uses MongoDB to power its Growth Platform on the MongoDB blog