Autossh for persistent database connectivity

Both SSL with whitelisting and SSH tunnelling into databases are powerful options when it comes to connecting to Compose database deployments. We recently introduced SSL for RethinkDB because it is the quickest and easiest way to get a secure connection, but there are still situations when you need an SSH tunnel, for example when your database drivers don't support SSL yet. When you need an SSH tunnel though you don't want to be starting it up by hand and you want it to stay running.

Consider a situation where your application servers are configured and restarted on the fly in the cloud and can appear on any of a number of ranges of IP addresses. It might be possible to white-list the IP addresses that the application servers will use, but these would also open up larger ranges which would expand the attack surface of your application, especially if its being allocated from shared ranges over which you have no control. So, how do you ensure that you can always make a verifiably secure connection?

The SSH tunnelling option offers a solution in that the database client, the application, isn't responsible for the security of the connection. The system the client is running on can create a port tunnel, protected by SSH, into the virtual private circuit (VPC) on which the database is running and all the application has to do is connect to one of the servers on the VPC to talk to the database. That port tunnel requires the deployment you are connecting to can validate the client through the use of PKI certificates. To create a tunnel you would look up the SSH connection string for the deployment which would looks something like this:

ssh -N compose@aws-eu-west-1-portal.1.dblayer.com -p 10664 \  
-L 127.0.0.2:28015:10.0.25.162:28015 \
-L 127.0.0.3:28015:10.0.25.163:28015 \
-L 127.0.0.4:28015:10.0.25.164:28015 

which would tunnel port 28015 from 127.0.0.2 to 10.0.25.162 at the other end of the tunnel. It also connects that port on 127.0.0.3 to 10.0.25.163 and 127.0.0.4 to 10.0.25.164. You can read more about the entire tunneling process in our guide to connecting to RethinkDB with SSH where we work through creating keys, adding them to a deployment and connecting with the SSH command.

What's so special about autossh?

Anyway, that command makes you an SSH tunnel and when everything is working well it's an effective and secure way to connect to the database. This being IT though means that things aren't always working and SSH's tunneling some issues about that. When an SSH connection is completely dropped, that is the server goes away, then the SSH process should quit. You can of course write a script to restart the SSH tunnel. But occasionally the tunnel can just choke up and stop moving traffic.

Situations like that are hard for SSH to detect unaided and thats where Autossh comes in. Autossh wraps SSH in an application which was designed to monitor the state of the connection. It will also restart SSH if it exits. The idea of the monitoring is that If it sees the packets aren't going through, it would also restart SSH. Originally, autossh's monitoring used a pair of ports to talk to the server at the other end of the connection, expecting them to be set up to echo the information sent to one port back on the other port.

Unfortunately, this process gets a bit fiddly to implement - configuring an echo style service on the server, ensuring you are using free ports. But, the developers of OpenSSH added some options – ServerAliveInterval and ServerAliveCountMax – which activate built in connection checking in OpenSSH. Together the options set checking at a set interval and exiting SSH if the count maximum is exceeded. And when SSH exits, autossh will restart it so it serves as much improved replacement as there's no extra ports needed.

Passphrase or not?

Before we go on, there is something you need to know. In our guide to connecting to RethinkDB with SSH we recommend that when you create your SSH key, you give it a strong passphrase. This passphrase is prompted for when you create a connection to ensure you are the owner of the certificate and authorised to present it.

But now, we're automating the connection we'll need a certificate with no passphrase or we need to figure out how to confirm identity when you are not present. The latter option is outside the scope of this article though; we're going to demonstrate autossh by having a key with no passphrase. The upshot of this is that you will have to take steps to ensure the security of your key files on the server so they cannot be read by unauthorized persons - ideally you should already have this level of security but do check.

Automatic for the SSH

Let's take out SSH example command from earlier:

ssh -N compose@aws-eu-west-1-portal.1.dblayer.com -p 10664 \  
-L 127.0.0.2:28015:10.0.25.162:28015 \
-L 127.0.0.3:28015:10.0.25.163:28015 \
-L 127.0.0.4:28015:10.0.25.164:28015 

And look at an autossh'd version:

autossh -M 0 -N compose@aws-eu-west-1-portal.1.dblayer.com -p 10664 \  
-o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" \
-L 127.0.0.2:28015:10.0.25.162:28015 \
-L 127.0.0.3:28015:10.0.25.163:28015 \
-L 127.0.0.4:28015:10.0.25.164:28015

You'll note they are remarkably similar. The differences start with autossh -M 0. This invokes autossh instead of ssh and sets the only autossh option we need here, disabling the native autossh monitoring. The rest of the first line is unchanged, but then we add a whole new line. With -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" \ we set a 60 second interval for checking for server life and set the number of times to fail this before exiting to 3 times. These are SSH options, not autossh options. The rest of the command's lines are unchanged.

When run, this command will establish the RethinkDB driver port over SSH and if anything happens to the connection, SSH will be restarted. If you've never connected with SSH to the server and use this, it will most probably ask for you to confirm the identity of the server you are connecting to. As is, the command will run in the foreground; which if you have a passphrase or haven't connected to the remote server before, you'll need. But once you know you won't be prompted, you can add -f to the options and it autossh will drop into the background.

Why autossh?

You may wonder why this is better than a looping script; well, autossh has the monitoring option to fallback on if OpenSSH's monitoring doesn't work for you. If you are using autossh with your own servers and want to create some custom monitoring, there's a number of options in the autossh documentation which let you customize its behaviour. So rather than everything in a script, you have one compact well-known command to deal with. If you want to bring your ssh tunnel up at system startup, add it to your rc.local script or whichever script your init system uses.

As a final note, if you KILL the SSH process autossh starts, then autossh will take that as a deliberate termination and exit itself, which means no battling with a looping script trying to kill it.

Autossh runs on Linux, OpenBSD and FreeBSD, Mac OS X and other Unix based systems. You'll find autossh available in most Linux distribution's package repositories - on Ubuntu, do sudo apt-get install autossh. Source can be found at on the autossh home page.

If you have a tool you use with Compose and recommend it, do let us know by dropping a mail to feedback@compose.io - or why not write about how you use it for the Write Stuff.