Continuously Synchronize Your MongoDB Data to Elasticsearch

Transporter, Compose's service for moving data from MongoDB to Elasticsearch, has already proven to be popular with our Elasticsearch and MongoDB users as it allows them to use both databases on the same dataset. But we can always make it better, and we just have with the new Continuous mode for Transporter.

With this new mode, a Transporter job attaches to the MongoDB database and, using the oplog, continuously updates the Elasticsearch destination database with any changes in MongoDB. This means that instead of snapshotting your data for analysis in Elasticsearch as you could do with the previous One-Time mode, you can now work with an always up-to-date copy of the data.

Up-to-date means that updating documents in Elasticsearch as they are updated in MongoDB and deleting them from Elasticsearch when they are deleted. The Transporter is unidirectional so any document changes you make in Elasticsearch will not push back into MongoDB making it safe to use aggressive analytics techniques.

Transporter Table

We've upgraded the user interface to support the new Continuous mode. The update now shows you a complete listing of recently run, or running, Transporter jobs, both One-Time and Continuous. You can see how many records have been processed and how many written to the destination database by each Transporter. You can also delete Transporters to remove completed One-Time jobs and to stop and removed running Continuous jobs. And of course, you can create new Transporters; the one change to the new Transporter page is you get to select Continuous or On-Time at the top; it defaults to Continuous.

In Continuous mode the Transporter remains running all the time so the cost of a continuous update stream is $9 a month during the beta. Beta One-Time Transporters remain billed by use at $0.0125/hr. The Transporter's ability to manipulate data in transit using its Transformer system is also retained in Continuous mode. Read about the Transformer system and how you can use it to control what passes through the Transporter.