Verify MongoDB Replica Set High Availability with the Step Down Button
PublishedMongoDB replica sets are engineered for high availability. Replica sets consist of replicated data-nodes, and are powerful and resilient to network interruptions. We have a new tool that will assist in verifying your application is equally ready for high availability.
Configuring your Application for High Availability
MongoDB drivers use auto-discovery when connecting to hosts. To begin auto-discovery, drivers require seed hosts to attempt a connection. These seed hosts are a list of MongoDB instances in the replica set specified by ‘host:port’. Proper driver configurations include every replica set member explicitly. Different drivers require different configurations, please see your driver’s configuration for Replica Sets. For a list of drivers, please see http://docs.mongodb.org/ecosystem/drivers/ and http://docs.mongodb.org/ecosystem/drivers/community-supported-drivers/.
On the MongoHQ database admin screen, the “Replica Set URI” contains the full list of seed hosts:
MongoDB Drivers' Role in High Availability
A ‘step down’ is an event that transitions a replica set’s Primary server from one MongoDB to another. For instance, if HostA is your Primary before a stepdown, then HostB will be the Primary afterwards.
When a driver adjusts to this stepdown, the driver will go through the process of:
- Receive 'not master' response from MongoDB server
- Raise an error
- Re-run auto-discovery of replica set members
- Return to a healthy state
Time to Test
Once you have connected to your application to your database database, (and hopefully before production), we have a new button that will assist with verifying your application for high availability. This is the “Step down button”:
You will find the button on your database’s “Admin” page. Clicking this button will send a command to the primary member of your replica set and instruct it to step down. What happens at this point is that the other members of your replica set will elect a new primary member. Typically, step downs takes a very short time, usually between 2 and 5 seconds; in extreme conditions, we have seen a replica set take up to 60 seconds to recover.
Testing your app before production is an important step in confirming your application is able to handle server failures and replica set step downs. Steps for a test are:
- Send an application server request (web request)
- Press the "Stepdown" button
- Send another application server request, and your application should work after stepdown
If your application responds to step 3, then you are ready to go. Because different drivers behave slightly differently in failover, give your application a few more attempts to recover. Should your application not recover, check your logs, and determine if any errors can be rescued, and connections reattempted. Check with your driver’s bugs to determine if any bugs exist for your driver version.
*Already running a production database? *If you are running a production database, just create another replica set for testing (we only charge by the day, so you will only be billed for your time used). Load some staging data, and run through the steps with a staging environment.
Need high availability MongoDB? If you haven't yet graduated to high-availability MongoDB, replica sets are one of the nicest features in MongoDB. Check out our Replica Set SSD MongoDB Databases for highly available, maximum performance.