Compose's new MongoDB and what you need to know


The release of our new MongoDB deployments has brought a lot of new questions about what you need to know and why and what has changed - everything from SSL, WiredTiger, Oplogs and Importing. So we're going to answer as many of those here.

How can I be SSL ready?

When you have SSL enabled on your MongoDB database, which is the default on the new MongoDB at Compose, you need to ensure that your drivers and tools are all SSL enabled too and that you are connecting telling those drivers and tools you want an SSL connection.

If you use the MongoDB command line tools, to check if SSL is enabled run mongo --help and if there's no --ssl flag mentioned in the help, you don't have SSL. For Windows and Linux, go to MongoDB's download site and get the latest tools – you'll want to anyway to get up to date – and they'll already have SSL built in. If you are on Mac OS X, we recommend that you setup the Homebrew package manager, then run brew install mongodb --with-openssl which will compile and install the latest MongoDB with OpenSSL configured.

Then, for the mongo command and most of the other MongoDB tools, add --ssl --sslAllowInvalidCertificates to existing commands to connect with only encryption enabled. If you want to validate you are connecting to a particular server, log into your account, select your MongoDB deployment and on the Overview page you'll see an SSL Public Key panel with a Show SSL Public Key button. Click that, enter your password and the page will refresh with the key details. Copy the contents of that key into a file on your local filesystem, say call it servercert.pem and then, when you run a MongoDB command include --ssl --sslCAFile servercert.pem in the command line.

You'll want to check your driver documentation for how it prefers to handle SSL connections, but as a rule of thumb, the addition of ?ssl=true to the connection URI, where the driver uses connection URIs, is enough to make them connect using SSL and the only thing you'll need to look up is how to pass the details of the server's public certificate.

Why does WiredTiger need more RAM?

The WiredTiger storage engine is designed to use more memory as part of an aggressive caching strategy. This, in combination with compression, is a strategy to reduce the number of I/O operations that the database has to carry out. This allows it to deliver better performance and use less disk space. The MongoDB default cache size starts at 1GB for WiredTiger and that doesn't include memory needed for file system cache or other runtime requirements like index storage. At Compose, we've tuned the cache usage and other RAM uses to provide what we think will be the optimal experience for most users.

Why use WiredTiger?

Think of WiredTiger MongoDB on Compose as a power up for your database. As well as better use of disk storage through compression, WiredTiger also delivers document level concurrency which can, under particular workloads, deliver better performance. Of course, your performance and storage use will depend on your data and your workload, so we can only deliver a general guidance, but for most users, your storage size on disk should be much less and there should be at least a marginal improvement in performance under load. For an example of the effect that compression can have, a readily compressible dataset like the Enron mail corpus uses around 2GB of storage with MongoDB's MMAPv1, while under WiredTiger, that comes down to just under 800MB. Remember that with Compose, you can easily start up a new WiredTiger deployment for testing and only be billed for the hours it exists for.

How do I get at the MongoDB oplog?

When we created the new MongoDB, our architecture changes did mean that we couldn't automatically offer the oplog to users as "just another database". The nature of the oplog means that it is not designed to be externally visible to client applications and the MongoDB sharded architecture ensures that is the case.

At Compose, we saw this as a challenge and one we've met. We've created an addon which is an access portal that cuts through that blocking and gives you access to your database's oplog. You'll find the Oplog Access addon with the rest of the addons. It has it's own MongoDB URI, username and password and can optionally have SSL enabled too.

Why the new architecture?

The older MongoDB deployments, now MongoDB Classic, is a replica set that has password protected ports visible on the internet. It's a simple and hard-working configuration but it does limit what you can do to expand and enhance the deployment.

Take SSL connections, a very common request. To make those work with a replica set we found that we were adding a lot of complexity, both in architecture and for users, for a somewhat limited payback. By the same token, our MongoDB Classic high-availability system keeps your data safe and available, but it does involve developers having to detect failures to initiate a switch to the available server in their code.

So, we took a higher view and looked at what kind of technology existed to make things work more effectively and we found it in MongoDB's sharded architecture. That's designed to take requests from clients, through routing nodes which work with a set of config servers, and get them to the right shard (or shards) and primary nodes within them.

We took our MongoDB Classic replica set and wrapped it within a shard and put that inside this architecture. This gave us the seamless high availability we wanted. We already secure your deployments on a private encrypted VLAN, so we were able to move SSL to the Internet facing edge of the deployment. That's the architecture we put into beta last year when we discussed it in detail. It's also the one we've been happy to make our new default MongoDB deployment. It's easier on developers who want high availability and more security.

Importing your data

We are still putting the finishing touches on our integrated import tool, so in the interim, we have two options depending on what you are more familiar with.

One is that people use MongoDB's mongodump and mongorestore tools to transfer their data from older MongoDB databases to our new MongoDB platform. The main thing to watch out for is to remember that when you restore, you will (probably) be restoring to an SSL enabled MongoDB, so you'll want to follow the instructions from the start of this article on making sure you have SSL enabled tools and that you use the --ssl... command line flags to turn on SSL support.

The other is using the Compose's open source Transporter. Transporter uses two files, config.yaml and application.js, to configure it's copying. Say you wish to copy the collections in a database called 'samples' on your now classic MongoDB, to your new MongoDB deployment, you would create a config.yaml something like this:

    type: mongo
    uri: mongodb://
    namespace: .
    type: mongo
    uri: mongodb://
    bulk: true
    namespace: .

This assumes that you'll substitute in your usernames and passwords, along with the hostname and port, for both the classic and new MongoDB databases. Note that we've set the namespace to . so that it should match all collections in the database. You can of course make it more specific so it only handles particular collections.

The nexts step is to create an application.js file to copy use these definitions. This file should look exactly like this:

pipeline = Source({name:"sourcemongo"}).save({name:"destmongo", ssl:{}})  

It's a simple pipeline to take the collection's documents from one database to another. The only thing that is really different is the , ssl:{} which is how you activate SSL connections in the Transporter. If you run this with transporter test application.js, the Transporter will test your configuration and if it does not give you an error, run transporter run application.js to start your copying.

You'll find binary releases of Transporter in its Github repository releases page.

So why am I waiting?

Good question! If you don't have an account, sign-up for a 30 day free trial and create a MongoDB deployment. If you do have an account, head to the Create Deployment page and select MongoDB to get going.

Dj Walker-Morgan
Dj Walker-Morgan is Compose's resident Content Curator, and has been both a developer and writer since Apples came in II flavors and Commodores had Pets. Love this article? Head over to Dj Walker-Morgan’s author page to keep reading.

Conquer the Data Layer

Spend your time developing apps, not managing databases.