A How-to for migrating from etcd 2 to etcd 3

Published

Thinking of moving from etcd 2 to etcd 3? We're here to give you the why to move, the how-to do it and the what changes to expect when you do move.

As an etcd 2 user, you already know why etcd is a great database for configuration and orchestration and you are as excited as us at Compose to be able to use etcd 3. That's great, but we thought we'd better guide you through the migration process. There's a lot of changes in etcd 3, compared to etcd 2.

Most of the changes are for the better, but if you have your own applications using etcd 2 then there is some engineering to do to get up and running on it. The great thing with Compose is that you can create an upgraded clone of your current etcd 2 database without impacting on your use of it. Let's talk about that process first.

Migrating your data

Go to your existing etcd 2 database deployment. Go to the Backups view. You'll want to make an up to date backup, so click on Back up now to make an on-demand backup. Wait for that to complete and then head back to the Backups view. There will now be a fresh on-demand backup. Go to the right of the row and select the circular arrow. This is the Restore to a new deployment button. Click on it and you'll go to a New etcd Deployment from Backup. Enter a new name. The Location will automatically match where your existing deployment is. Below that will be the new version number. Make sure it's 3.2.1 or later (if available). With all that in place, select Create Deployment. The new deployment will be created, your backup data will be loaded into it and then the etcd migrate process will be run.

The flattening of keys

This is where the first big difference between etcd 2 and 3 comes in. In etcd 2 you could have a hierarchy of keys and directories, in a very file-system-like way. It allowed for all sorts of novel tricks by grouping keys together. In etcd 3, the hierarchy is gone. Every key is now at the same level and as long as needed. This flattening avoids a lot of complexity and potential issues as things scale up.

It also means that to move from etcd 2 to 3, the keys for your existing data have to be transformed. So if you had an etcd 2 key ...

/services/servers/server1/enabled

... then that would become an etcd3 key ...

/services/servers/server1/enabled

The services, servers and server1 directory nodes which would exist on etcd 2 to support the key's path are discarded by the transformer as there's no such thing as a directory on etcd 3.

At Compose, we use the default key transformer (no relation to transformers in Compose's Transporter) which has these rules built in. It is possible, if you are running your own etcd server, to create a custom flattener which could do a more complex transformation of the key. For most people, this simple, default transformation should do.

This migration happens automatically when the etcd 2 database is imported into a Compose etcd 3 database.

Prefix Postscript

You may be concerned that with the keys being flattened that the directory structure has gone. The heirachy was useful in etcd 2, but in etcd 3 it is still possible to reduce your scope to a subset of keys.

For example, remember our flattened key; /services/servers/server1/enabled which became /services/servers/server1/enabled. Imagine the application that used it used to wait on the /services/servers/ directory for changes to it or any subdirectory in etcd 2. Well, in etcd 3, you'd wait with a prefix of /services/servers/.

Any key that changed which started with that would trigger a notification. It's simpler and leans in on etcd 3's key handling and ranges, making for a solid foundation to build out new key namespaces.

Porting your application

Ideally, you should start looking at this before you migrate your database; Compose has you covered though as your etcd 2 database carries on running till you turn it off.

If your application already supports etcd 3, then you are good to go and probably only need to change the configuration to tell the applications it is talking to the latest version of etcd. If your application doesn't support etcd 3 and it's not your own application, get in touch with your application vendor.

If it is your application, then you've got some work to do. In brief, directories have gone (as you can see), keys can now be attached to leases with a time to live, watchers are way more efficient and ACID compare-and-swap operations are replaced by mini-transactions. Most importantly, in etcd 3 the HTTP request based API is replaced by a gRPC-based API. This is way more efficient and capable, but also means it's likely your old libraries and scripts are out and you'll need a replacement.

For scripts, the good news is that the etcdctl application is much more capable than the etcd 2 version. It's got a whole new set of commands that reflect the gRPC API and can even run multi-operation transactions. If you are writing a script, using etcdctl is probably the best way to go when porting. Just remember to set the ETDCTL_API=3 environment variable to make etcdctl use the right API - yes, both APIs are baked into the one binary.

Finding libraries

For applications, you'll have to find a library for your chosen language or use the gRPC API of etcd 3 directly (etcd v3 API documentation).

First stop for Go users will be the clientv3 library from the etcd developers and the most official. Node.js developers have a number of packages to choose from. Most promising of them is Connor Peet's etcd3. Python users should check out python-etcd3, Ruby users should have a look at etcdv3-ruby, Nokia produce a C++ etcdv3 package and, finally, there's a CoreOS jetcd package for Java (and JVM-hosted languages by extension). Other libraries are steadily emerging for etcd, partially gated by the availability of gRPC support.

Switching modes

For more details of what you'll need to change, a previous Compose Article - etcd 2 to 3 - New APIs and New Possibilities - takes a deeper dive into the functional changes. You may also wish to step back and ensure you aren't porting etcd 2 patterns over to etcd 3 where they may be inappropriate. For example, setting up TTLs through Leases is a much simpler proposition now; etcd 2 encouraged a clumping of keys under a directory, but with leases in etcd 3 you can spread out your keys more appropriately. Watching, which was expensive, is now much more economical on resources. Consider what and why you watch things and see if etcd 3's lighter watching could let you watch more discretely distributed keys or just watch more. And finally, transactions will let you act on complex conditions and on many keys. Make sure you aren't working around etcd 2's simple ACID operations; you may already have transactions in your code.

Conclusion

Just migrating to etcd 3 is only the start. With easier TTL management, more "complex" transactions and more efficient watching, it offers a thoroughly enhanced version of its original v2 features. Compose makes it easy for you to explore how you can upgrade without disturbing your production databases.


If you have any feedback about this or any other Compose article, drop the Compose Articles team a line at articles@compose.com. We're happy to hear from you.

attribution Andrew Ruiz

Dj Walker-Morgan
Dj Walker-Morgan is Compose's resident Content Curator, and has been both a developer and writer since Apples came in II flavors and Commodores had Pets. Love this article? Head over to Dj Walker-Morgan’s author page and keep reading.