Failing over yourself to succeed

Published

We are often asked how to do fully automatic, seamless failover on the various Compose databases. The answer is "practice, practice, practice" as we'll explain here.

There's only so much a database can do to provide high availability. We've talked about the general issue of being highly available in the past. Having multiple servers with automatic failover and recovery processes and multiple ingress routes for clients spread over regionally separated hosts helps.

It isn't the whole story though because your applications need to be aware of when things go wrong with the connection and choose the best course of action. If you expect failover to happen transparently, then you are also setting yourself up for some strange, hard to debug problems.

A Failover Example

Let's talk about some specifics. Let's say we have a MongoDB server on the Compose platform and we're using the Mongoose ORM and NodeJS to connect to it. If you go to your deployment's overview you'll find a connection string that looks something like this:

  "mongodb://user:password@aws-us-east-1-portal.25.dblayer.com:28102,aws-us-east-1-portal.26.dblayer.com:28102/compose?authSource=admin&ssl=true";

And we can start connecting like so:

var sslcerts = [fs.readFileSync("mongodb.cert")];

var options = {  
  ssl: true,
  sslValidate: true,
  sslCA: sslcerts
};

mongoose.connect(connstring, options);  

Now, Mongoose knows how to consume that connection string and it will make a connection which has the two hosts listed when asked. If it fails to connect, Mongoose will move to the next host in the list until it runs out. When does Mongoose connect? Well, it starts opening a connection fairly early in its lifecycle, but it will also reconnect if it's asked to perform an operation and the connection to the server it has been using is closed.

The connection can find itself being closed when Mongoose tries to perform an operation and there's some failure. So say we are writing a new Mongoose object to the database

    var newuuid = new UUIDS({ uuid: uuidv4() });
    newuuid
      .save()
      .then(id => {
        console.log("Added " + id.uuid);
      })
      .catch(err => {
        console.log("Oh um..."+err);
      });

This connection failure percolates up to the application as the operation failing. In the code above, that means we'll be catching an err. Performing another operation is enough to trip off the process of getting Mongoose to walk through its list of hosts to reconnect and we can carry on.

Loss in a time of failover

But, notice that in doing that we lost an operation, the one that tripped the error. Whether it was a write or a query or something else, that operation didn't happen. When we catch that error, we have to consider what we do with the operation that triggered it. It may be that we can safely discard it. Or it may be that all we need to do is log the loss. However, it may be that we have to ensure that it is written to the database. In the latter case, we can create code that pushes to reconnect and once reconnected, resubmits the operation it just failed on.

Mongoose is actually fairly good to work with here because it looks after the actual connections. With other database drivers and packages, that reconnection process may get more complex as you may be working with your own connection logic. You'll have to either abstract it out into a reusable function, wrap the database up in your own abstraction or come up with some other method that suits your code, style, and methodologies.

Deciding how to fail

And that's where you are also going to have to make a decision. You can go through the process of hardening all your database interactions. Or you can abstract them away to a common, smart, auto-reconnecting layer that works with your application code.

Or you can turn it off and on again as it may be easier to just restart the application and let it reconnect on its own. Then all you have to do is make sure that it has all the available host information at startup.

These aren't application design decisions your database host should be taking for you. We set out to give you as many options as possible to handle the variety of things that can get between you and your database application.


Read more articles about Compose databases - use our Curated Collections Guide for articles on each database type. If you have any feedback about this or any other Compose article, drop the Compose Articles team a line at articles@compose.com. We're happy to hear from you.

attribution John Philip

Dj Walker-Morgan
Dj Walker-Morgan is Compose's resident Content Curator, and has been both a developer and writer since Apples came in II flavors and Commodores had Pets. Love this article? Head over to Dj Walker-Morgan’s author page to keep reading.

Conquer the Data Layer

Spend your time developing apps, not managing databases.