One Missing Key & How it Broke Node.js & MongoDB+

Recently, we became aware that Node.js users were having trouble connecting to our new MongoDB+ service – MongoDB+ is the name of our currently in beta next generation MongoDB deployments. Now, as well as a much more expansive configuration, one of the top features of MongoDB+ was the ability to make SSL connections to it. And it was precisely there that people seemed to be having trouble, which was odd because we'd already tried numerous other SSL connecting drivers and we're pretty confident that our server-side implementation was fine.

But then looking at the Node.js code, it also looked fine... here's a connection snippet:

var MongoClient = require('mongodb').MongoClient,  
  fs = require('fs');

var ca = [fs.readFileSync(__dirname + "cacert.pem")];

MongoClient.connect("mongodb://dumper:dumper@example.1.dblayer.com:10900,example.0.dblayer.com:10901/dbname", {  
 mongos: {
    ssl: true,
    sslValidate: true,
    sslCA: ca,
    poolSize: 1,
    reconnectTries: 1
  },
}, function(err, db) { ...

For reference, the fs.readFileSync( reads the certificate for verifying the server into a variable. Anyway, we have all the elements we're supposed to have being passed in the mongos options map. There's ssl: being turned on, sslValidate: being requested and we're passing over the certificate by setting sslCA to an array of certificates. Yet when we run the code, it apparently gridlocks inside connect and never comes out.

It's time for some logging to see whats going on... Here's what we need to add:

var  Logger = require('mongodb').Logger,  
  assert = require('assert');
  Logger.setLevel('debug');

And as we run, there's a lot of log there and if you want to read it remember it's all coming in asynchronously so it's not going in any particular order. But working our way back this we see connections being opened and closed with a complaint that it is "unable to verify the first certificate". This is strange; we're presenting the right information... lets look at the docs, they tell us the options should match those for MongoS and when we looks there we see:

Ok. At a wild guess we're corrupting the certificate array, but its odd. Looking at the log it doesn't seem to be going up the wire. At this point the Compose developers took a look. And this is where we have to do some background. The Native MongoDB Node.js driver actually depends on another library, MongoDB-Core written by a MongoDB Inc employee. That library is where the SSL options eventually land. But, there's difference between the way options are done. The Native library uses a key sslCA for the certificate array but in MongoDB-Core the options key is ca. Inside the MongoDB Native driver, there's lots of code for shuffling around these options, copying them to new key/value maps. Specifically, when a Server connection is being made the code passes through this line:

if(clonedOptions.sslCA) clonedOptions.ca = clonedOptions.sslCA;  

That copies sslCA to ca so it will make it to the MongoDB-Core code. The problem for our connection is we are connecting to a MongoS server and the code for that doesn't have an equivelant line; it copies the option key/values en masse to the same names. So the certificate doesn't make it to the MongoDB-Core code and when we run it we find ourselves in an erroring loop.

So what can we do about it? Well, we're filing a bug on the issue of course, but that will take time to get out into the field and we need a work around now. Which is where is gets interesting. All those option maps don't get emptied. So why don't we intervene and set the ca key ourselves like this:

MongoClient.connect("mongodb://dumper:dumper@example.1.dblayer.com:10900,example.0.dblayer.com:10901/dbname", {  
 mongos: {
    ssl: true,
    sslValidate: true,
    sslCA: ca,
    ca: ca, 
    poolSize: 1,
    reconnectTries: 1
  },
}, function(err, db) { ...

And if we run that, we're connected.

So what have we learned.

All that and how to connect with Node.js to Compose's MongoDB+ with SSL enabled.