Transporter 0.1.0 Released

The latest release of Transporter, Compose's open source data transfer and replication application, brings a couple of new features, including one "big ticket" breaking change, and a number of smaller fixes as we move the development on, towards the still far off target of a version 1.0.

Multiple Namespaces

The "big ticket" item in this Transporter is the multiple-namespace support. Namespaces are how the Transporter identifies the collection, index or other structure in the source or destination for the data and since it's initial release up until now, the Transporter has only had one option, a statically specified namespace such as mydb.mycollection. It made life difficult though. If you wanted the data from more than one collection, you could have ended up running multiple Transporters.

With version 0.1.0, it's possible to create the messages – database records wrapped for traveling through the transporter – from multiple collections. It uses the namespace parameter and extends it to support regular expressions, a powerful matching mechanism. Using the same enhanced namespace parameter, you can also filter which transformers and database sinks work on those messages so they can follow the most appropriate path through your Transporter pipelines. There's a lot to take in with this new namespace system, so we'll have an article about that later this week.

Skipping

Not quite a new feature, but the return of an old one in two forms. Transformers in Transporter were able to set messages to be skipped by returning false in the JavaScript code. This was disabled for various reasons, but now it is back and, as well as the old technique for triggering it, there's a new more explicit way to skip a message – set the .op field of the message to noop.

Environmentally friendly

As Transporter has been deployed on more platforms, the Transporter has had to accommodate the best practices of those sites. For example, some services pass all user settings over as environment variables and the Transporter has had little in its toolkit to see those variables. In 0.1.0, we've added environment variable substitution into the config.yaml file parsing. Just use the syntax ${FOO} and the FOO environment variable will be inserted. Other environment variables are available.

Fixes and more

There are of course various fixes in this new release, improved instructions, better tests and small changes - like testing tailing Transporters have oplog access before they complete their time-consuming first pass of copying data over - which make life a little easier for all. It's all open source and we welcome pull requests, bug reports and any other ideas as we continue to build out the Transporter.

You'll find Transporter at https://github.com/compose/transporter. We'll have a binary build of the latest version available soon.

Photo Source: Stephen Tierney CC-BY-NC-2.0