Etcd is a fine database for coordination and configuration data but, at least for backup and restore, there's some design assumptions made that aren't a good fit for a hosted service.

That's why we were rather pleased to come across the etcdtool utility. It isn't part of the etcd distribution but the work of Michael Persson and can be found at https://github.com/mickep76/etcdtool.

The prime motivation behind etcdtool was to create a version independent backup and restore utility for etcd, but it has many other capabilities which could make it an essential for etcd users who may want to automate and validate populating their etcd deployments. We'll start with looking at moving a tree from one etcd deployment on Compose to another using etcdtool.

Installing

Before we can do anything, we'll need to install it. The README file for it has details on how to build it from source and how to install on Mac OS X using Homebrew, and on the Releases page, there are builds for Linux, Mac OS X, an RPM and source code. As we're running Mac OS X and Homebrew we went for the simplest install:

brew tap mickep76/funk-gnarge  
brew install etcdtool  

Commanding the line

And we are ready to go. Before we backup, let's use etcdtool to take a look at our data:

$  etcdtool --ca ./etcdcert.crt --peers https://host.1.dblayer.com:10068,https://host.2.dblayer.com:10051 -u root tree /
Password:  
/
└── config/
    └── server-15/
        └── database

2 directories, 1 dirs  

Let's step through the command and its parameters at this point. The -ca ./etcdcert.crt tells the tool to use the self-signed certificate file Compose makes available on the etcd console - see this article for more about that. The next parameter, --peers https://host.1.dblayer.com:10068,https://host.2.dblayer.com:10051 tells etcdtool where to find the cluster's nodes in the same style as the etcdctl command does. Compose users will find that information on the Overview tab of their etcd deployments console. They'll also find the username and password for the etcd user on that page – usually root – which goes into the user parameter next - -u root. Ok, that's the boilerplate parameters for connecting covered. The last part of our command is the subcommand to run. In this case tree / generates a tree view of the etcd hierarchy of keys, but doesn't show values.

When we hit return, we're prompted for the password which goes with the "root" user which was displayed on the Compose console. Once entered we get our tree view. The command is pretty clumsy to work with so let's make things more readable and alias it.

$ alias etcdtool-s1="etcdtool --ca ./etcdcert.crt --peers https://host.1.dblayer.com:10068,https://host.2.dblayer.com:10051 -u root"

Now we can just say etcdtool-s1 for that server.

Rare exports

To export the contents of etcd, we use the export command and tell it what directory we want exported:

$ etcdtool-s1 export /                                                          
Password:  
{
  "config": {
    "server-15": {
      "database": "mongodb"
    }
  }
}
n  

Sending this to stdout this very useful, but it does show us what a version independent export looks like. Don't try and redirect the output; that password prompt will just confuse things. We need an option to send the output to a file. All the subcommands support --help to display flags and options and don't need a connection, so we can do this:

$ etcdtool export --help                                                        

NAME:  
   etcdtool export - export a directory

USAGE:  
   etcdtool export [command options] [arguments...]

OPTIONS:  
   --sort, -s       returns result in sorted order
   --format, -f "JSON"  Data serialization format YAML, TOML or JSON [$ETCDTOOL_FORMAT]
   --output, -o     Output file
$

And see we can specify an output file and format. We'll stick with JSON for now, but we'll send the output to export.json by running:

etcdtool-s1 export / -o export.json

As we have a JSON file exported, we can now look at importing the data into another etcd deployment. And here's one I prepared earlier.

alias etcdtool-s2="etcdtool --ca ./server2-etcdcert.crt --peers https://host.1.dblayer.com:10181,https://host.2.dblayer.com:101422 -u root"  

As is obvious this other server has a different certificate and ports. We’ll proceed by supplying the import command with the JSON data file and the destination in the new etcd hierarchy.

$ etcdtool-s2 import / export.json                                              
Password:  
Do you want to overwrite data in directory: / [yes/no]? yes  
$ etcdtool-s2 tree /                                                            
Password:  
/
└── config/
    └── server-15/
        └── database

2 directories, 1 dirs  
$

And we've rebuilt the tree in a new server.

Recovering Compose backups

Compose backups are snapshots of the data in etcd. Currently, from outside the cluster, they can't be restored directly. That's because the etcd developers expect restoration to be carried out by moving files around on the file system.

If you download an etcd backup, you will find all your data is there. We've just downloaded a backup and extracted its contents:

$ tar xvf 2016-04-17_10-52-03_utc.tar                                             
x etcd/  
x etcd/member/  
x etcd/member/snap/  
x etcd/member/snap/0000000000000003-00000000009e394f.snap  
x etcd/member/wal/  
x etcd/member/wal/0000000000000000-0000000000000000.wal  
$   

We can bring this data to life by running a local copy of etcd. Do ensure you are running the same or later version of etcd as is running in your Compose deployment. When we run etcd in this directory like so:

etcd --data-dir=./etcd --force-new-cluster

This starts a single instance of the database locally, which we can now talk to with etcdtool. No parameters are needed for etcdtool as this is a local instance:

$ etcdtool tree /                                                               
/
└── config/
    ├── server-15/
    │   └── database
    └── server15/
        └── test

3 directories, 2 dirs  

We can now export that data as JSON using etcdtool export / -o backupexport.json and use etcdtool import to load it into any other etcd server or deployment.

Editing with...

We've been focussed on the export and import functionality of etcdtool in this article but there's more to etcdtool than that. For example, to quickly edit your etcd data as JSON (or YAML or TOML, the two other supported formats), try etcdtool edit. This takes a directory in the store as a parameter. It then exports that to the required format and starts up an editor where we can make changes. On exiting the editor we are offered the chance to overwrite that directory with our changes. This is very handy for the trickier changes and a good alternative to firing off multiple etcdctl commands.

Finally, it's worth mentioning that etcdtool has the ability to validate JSON schemas on import using a configuration file to make schemas to the tree. You'll find examples of this features use in the Github repository, but, as yet, little documentation.

Wrapping up

etcdtool is simple but powerful and works well with Compose to solve the problem of converting snapshots into re-usable data.