We've Open Sourced the New Compose Transporter

We're really happy with how the Compose Transporter has been moving your data around on the Compose platform. That said, we always want to do better and with that in mind we've been building the next version of Transporter. We have now reached a point where we are pleased to release the code as is, as open source, while we continue development.

With the new Transporter, we have set out to create a flexible framework for all of the various data transportation and transformations that people will want to do. That's a big range of potential tasks, from simple bulk uploading to continuous synchronisation, taking data from different databases, data generators, pipes, sockets and files and delivering it back to a similar set of outputs.

To achieve this the new Transporter is more modular and flexible. It can be run as a self-contained command line tool, entirely configured using a combination of JavaScript and a YAML file, or it can be used as a Go library where the configuration and operation is controlled by your own code. Transporter can already work with MongoDB, Elasticsearch, RethinkDB, InfluxDB and raw files of JSON.

Devops should find the command line tool a compelling way of moving data around, developers will love the ability to create new, dedicated data transfer commands quickly and efficiently and both models should make system integrators lives easier. All these communities also have the potential to make the Transporter a better tool and the capability to make meaningful contributions to the code base. We want to add new and better adaptors, improve the configuration language, ensure the library is as simple as possible to consume and make the new Transporter as reliable and effective as possible.

That's where we would like you, the community, to join in. We've released Transporter as BSD licensed project and it's available in it's own Transporter Github repository, along with a Transporter Examples repository which currently has a custom Twitter to MongoDB Transporter.

This first release is an alpha and we'll be constructing our road map to version 1.0 in the coming weeks - on Github as an issue so you can take part in the planning. To help you using the new Transporter, there'll also be some articles here looking into how the new Transporter works as a command line tool and as a library. We look forward to you joining us in directing Transporter's evolution.