Transporter 0.3.0 Released - Transporter Streamlined


TL;DR: Transporter 0.3.0 is out and it has a whole new way of working which will streamline its use on the mission to make moving data a joy, not a chore.

Transporter 0.3.0 has been tagged and released. This is the most tangible overhaul to how you work with the open-source Transporter to date, making it a more productive and elegant tool for moving your data between data sources, be they files, databases or, now, message queues.

In the process of releasing version 0.2.x of Transporter, it became clear that we really wanted to refine the process of how you configured the Transporter. We'd expected that addressing this would have taken longer, but Compose engineering has worked at a tremendous rate to bring us this new version.

What's new

Here's a brief summary of the changes:

The new Native Transformers are implemented in Go and allow messages to have common operations performed on them without the overhead of the JavaScript engine.

There's also "native" transformers to call up JavaScript:

Both take a filename as a parameter and use the contained JavaScript to process the message through a transformation function. We will have a detailed look at the new Native Transformers in a new article.

Getting going

You can download ready-built Transporter 0.3.0 binaries from the releases page or you can build your own with the available source.

We have also swept the Transporter issues and if you want to get involved we now have a great list of issues to get started on. We do Transporter development in the open and you'll also be able to track design and development through the Transporter issues, pull requests, commits and other activity.

Migrating to 0.3.0

If you are an existing Transporter user, be prepared for a lot of breaking changes for the better in 0.3.0. The first thing you may notice is that there are fewer commands. We've dropped the eval and list commands to simplify working with Transporter.

To show how the configuration has changed, if we look back to moving Transporter data from database to disk, when we ran transporter init mongodb file, we got a transporter.yaml file and a pipeline.js file.

If we run transporter init mongodb file with 0.3.0, we get just a pipeline.js file. The first stanzas of that file duplicate the intention of the old yaml file, but by creating JavaScript objects:

var source = mongodb({  
  "uri": "${MONGODB_URI}"
  // "timeout": "30s",
  // "tail": false,
  // "ssl": false,
  // "cacerts": ["/path/to/cert.pem"],
  // "wc": 1,
  // "fsync": false,
  // "bulk": false,
  // "collection_filters": "{}"

var sink = file({  
  "uri": "stdout://"

Now, each adapter is defined with var variablename=adaptorname(options) where the options are in the form of a JavaScript object. To get an example of the options for an adaptor, transporter about adaptorname will print out what options are available formatted as a JavaScript object and ready to cut and paste into your code. In our old tutorial, we set our MONGODB_URI environment variable and set the ssl option to true, and set the file to write to a file. With the new configuration that looks like this:

var source = mongodb({  
  "uri": "${MONGODB_URI}",
  "ssl": true

var sink = file({  
  "uri": "file://dump.json"

The init function also generated the pipeline script which looked like this:

Source({name:"source", namespace:"test./.*/"}).save({name:"sink", namespace:"test./.*/"})  

This needed modifying to select the database to read from, even though it was in the MONGODB_URI, and there were redundant elements in the namespace for the sink.

Now, with 0.3.0, the pipeline looks like this:

t.Source("source", source, "/.*/").Save("sink", sink, "/.*/")  

The t. is the "Transporter" variable; all pipelines are rooted with that now. The Source and Save functions take an (optional) name, adapter variable, and (optional) namespace. Notice also, that Save is upper-case now. The namespaces are simplified to just a regular expression. In the Source, the namespace is used to select the collections, in the Save, the namespace selects the messages that flow into the sink.

We don't need to edit this at all; the simplified namespaces mean these namespaces match everything. We can just do:

transporter run  

And our Transporter will run the pipeline.js file. That's another change; if no pipeline is specified, the Transporter looks for pipeline.js as a default.

As an example, if we wanted to add the pretty Native Transformer, that would look like this"

t.Source("source", source, "/.*/").Transform(pretty({}).Save("sink", sink, "/.*/")  

There's lots more to cover in Transporter 0.3.0 and we'll be doing that in the Wiki and here in Compose Articles. Look out for it and come and join us on where the development happens.

Dj Walker-Morgan
Dj Walker-Morgan was Compose's resident Content Curator, and has been both a developer and writer since Apples came in II flavors and Commodores had Pets. Love this article? Head over to Dj Walker-Morgan’s author page to keep reading.

Conquer the Data Layer

Spend your time developing apps, not managing databases.