Mongoose 4 & Replica Sets

Some big changes have happened in some projects we've talked about before at the Compose blog and over the next few weeks, we'll be bringing you up to date with them. For our first refresher, we're going to look at Mongoose.

Mongoose 4

Mongoose is a ODM which we reintroduced readers to last Summer. The most recent stable version at that time was 3.8 - the developers operated a even-stable/odd-unstable versioning policy. At the end of March though, all that changed with the release of version 4.0.

This release switched to the MongoDB 2.0 JavaScript driver – Mongoose uses the MongoDB Node driver under the hood – and benefits from a number of performance and usability enhancements. This does mean that you need to specify the replicaset option in the connection string, but we'll get back to replica sets and Mongoose in a moment.

There are a lot of backwards breaking changes and anyone upgrading to 4.0 would be well advised to check the list of those changes in the release notes. More importantly, from 4.0 onwards, the odd/even rule is being dropped with all future releases being production ready and no backwards-breaking changes till version 5.0. But if you need more of an impetus to upgrade to 4.0, it's this – the 3.8 branch will be maintained up until August 1st this year (or September 1st according to the readme file). It's plenty of time to migrate and have one less thing to worry about over the summer.

There's also various enhancements to Mongoose too. Some you may only use as you prepare your code for the future such as the ES6 yield compatible promises. Others you may use when you are making your models stricter like the option to set defaults and validators on updates. There's middleware hooks added for queries so you can safely attach your own functionality before and after the execution of count, find, findOne, findOneAndUpdate and update – great for aspiring plugin builders. New ways to dynamically populate models, min and max validators for dates, write concerns for save and the ability to set full text search in the schema are also added to 4.0.

So now is probably a good time to think about testing and upgrading. Which brings us onto replica sets...

Replica Sets and Mongoose

There's seems to be an urban legend that Mongoose couldn't handle MongoDB replica sets, despite it being documented. Who knows where these tales come from, but we thought it would be a good opportunity, with 4.0, to show you a small script we put together for Node.js which can keep things busy so you can see what happens when a replica set steps down. In the process we discovered something about Mongoose and Node.

But first here's the script:

var mongoose = require("mongoose");

mongoose.connect("mongodb://user:pass@c999.lamppost.9.mongolayer.com:10000,lamppost.8.mongolayer.com:11000/tagging?replicaSet=set-999999999999999999999999");  

We pull in Mongoose and connect to a fully specified URI. This is how you'll get the replica set URI in the Compose Dashboard for your MongoDB database. Remember that 4.0 now requires that replicaSet parameter (which is included on the database settings too).

var Schema = mongoose.Schema;  
var tagSchema = new Schema({  
  _id: Number,
  name: String,
  value: Number
});

var Tag = mongoose.model("Tag", tagSchema);  

We then create a simple Mongoose Model for a "tag", with a string and number.

var cntr = 0;  

We clear a counter and then move onto the meat, a function to generate some activity:

function generateOrUpdate() {  
  if (cntr < 100 || Math.random() < 0.2) {
    // Generate
    var tag = new Tag({
      _id: cntr,
      name: "Tag Number " + cntr,
      value: 0
    });
    tag.save(function(err) {
      if (err != null) {
        console.log("Error Writing:" + err)
      } else {
        // console.log("Created:"+cntr)
      }
      cntr = cntr + 1;
      setTimeout(generateOrUpdate, Math.random() * 50);
    });

If it's the first 100 iterations of cntr or if a random number is less tan 2.0, we create a new tag, using the counter and save it. We print out the results if the operation failed (uncomment the console.log line for output on success) and then bump the counter up and set up a timer to rerun the function.

  } else {
    // Update
    var val = Math.round(Math.random() * (cntr - 1));
    Tag.findById(val, function(err, tag) {
      if (err != null) {
        console.log("Error Reading " + val + ": " + err)
        setTimeout(generateOrUpdate, Math.random() * 50);
      } else {
        tag.value = tag.value + 1;
        tag.save(function(err) {
          if (err != null) {
            console.log("Error Updating:" + err)
          } else {
            // console.log("Updated:"+val)
          }
          setTimeout(generateOrUpdate, Math.random() * 50);
        });
      }
    });
  }
}

Otherwise we randomly select one of the existing records, retrieve it, increment its value by one and write it back. Again we print out if the operation fails and set a timer to call the function again, both when we read the tag and when we write it back.

setTimeout(generateOrUpdate, Math.random() * 50);  
setInterval(function() {  
  console.log("Counter is " + cntr);
}, 5000);

Then we kick off that random timer to call the generateOrUpdate function and off it goes. If alls well, it should be silent, so we add in another timer which goes off every five seconds and prints the counters value. If somethings locked up or failing, that value likely wont go up.

Anyway, that's the script and we tested it with Node.js 0.12.2 and Mongoose 4.0.2. As we expected, it worked. When we manually stepped down the Compose deployment, there was a few seconds of pausing as the primary stepped down and a secondary took over and then things continued. We could repeat that step down and the script continued creating and updating.

Then we passed the script to our support engineers to check out and one of them found a problem. As soon as the step down happened, on his configuration the creation and updates stopped. You could tell because the counter stopped rising. It took some investigating to eliminate possible differences and, given the elastic deployments were practically identical, it pointed to a problem on his environment. In the end, the problem went away after upgrading his Node.js from 0.10 to 0.12.

That serves as a timely reminder that although Node.js has a rich and active development community and ecosystem, it's also still being refined and improved. That's why we create test scripts like this to ensure our expectations are met. If you want to use this script, you'll find it in the Compose examples Github repository and, as always, pull requests with enhancements, improvements and fixes are welcome.