Utilizing Etcd3 with Go

Published

With etcd3 powering many cloud applications and infrastructures, it's time to learn how you can use it. In this Write Stuff article, Gigi Sayfan returns to etcd and brings us right up to date.

You've heard about the new etcd3 support by Compose and you're dying to try it out! You also prefer to use Go to access it. In this article, you'll learn about the storage model of etcd and how to use the Go client to interact efficiently with an etcd store. The API for some operations like getting multiple keys is a little unusual due to the nature of the Go language, but together we'll pull through.

Quick Introduction to etcd3

etcd3 is an open source distributed data store developed by CoreOS. It is very reliable and suitable for storing the most sensitive and critical data in a distributed system, and it is used to great effect by Kubernetes to store the entire cluster state. Etcd is implemented in Go and exposes a gRPC API.

Here are some of the properties of etcd3:

The Raft consensus algorithm is used by etcd to manage a highly-available replicated log.

Etcd had a command-line client called etcdctl for interactive use.

Here is a list of features Etcd provides:

For additional background check out Building a dynamic configuration service with etcd and Python.

Setup

You need access to etcd. You can install it locally or take advantage of Compose etcd3 beta offering, which runs on AWS in a clustered and high-availability configuration. This tutorial focuses on etcd3, so follow the instructions here:
https://coreos.com/etcd/docs/latest/getting-started-with-etcd.html

If you're on macOS the easiest way is brew install etcd.

Then just type etcd and you should see a wall of text that culminates in a warning about serving insecure client requests on 127.0.0.1:2379. This is fine for this tutorial.

To run the Go code you'll need to get the Go client package: go get "github.com/coreos/etcd/clientv3"

Sample Code

I wrote a little Go program that demonstrates the primary capabilities of etcd. Those include:

The full source code is available here: https://github.com/the-gigi/go-etcd3-demo

Note, that in the interest of conciseness I often ignore errors and don't check if operations succeeded or failed. In general, you should always check and respond properly if an operation fails.

Here is the main function that calls an individual function to demonstrate the various capabilities. The initial setup code creates a context (a Go feature that allows code across a goroutine to access shared information in a safe manner and cancel operations). Then the etcd client object is instantiated, configured with the dial time and the endpoint to the local etcd server (see instructions later for working with the Compose etcd cluster). The defer call is guaranteed to be used at the end of the function and ensures all etcd resources are released. Next, a KV object is created. At this point, the setup (and deferred cleanup) is complete and the various demo functions are invoked.

package main

import (  
    "fmt"
    "context"
    "log"
    "github.com/coreos/etcd/clientv3"
    "time"
    "strconv"
)

var (  
    dialTimeout    = 2 * time.Second
    requestTimeout = 10 * time.Second
)

func main() {  
    ctx, _ := context.WithTimeout(context.Background(), requestTimeout)
    cli, _ := clientv3.New(clientv3.Config{
        DialTimeout: dialTimeout,
        Endpoints: []string{"127.0.0.1:2379"},
    })
    defer cli.Close()
    kv := clientv3.NewKV(cli)

    GetSingleValueDemo(ctx, kv)
    GetMultipleValuesWithPaginationDemo(ctx, kv)
    WatchDemo(ctx, cli, kv)
    LeaseDemo(ctx, cli, kv)
}

There are additional advanced capabilities that I don't demonstrate in code due to space constraints, but I mention them in a later section.

Single Key Operations

The most basic operations for a KV store are Get, Put and Delete. The etcd store supports versioning and keeping history. The GetSingleValueDemo() function executes several variations. First, all the kv.Delete() method deletes all keys whose name starts with "key" due to the clientv3.WithPrefix() option. If it's not specified, then exactly the key whose name is "key" will be deleted if it exists.

Next, it inserts a key value pair "key", "444" via the kv.Put() method. Note that keys and values must be strings.

The Get() method is interesting. It returns a response object with a header that contains the revision and a "Kvs" (key values) field, which is a slice of key-value pairs. Here, it always contains a single key-value pair. When passing the WithRev() option, it returns the historical version of the value for the target key, but the header's revision will always contain the current revision of the value and NOT the requested revision.

func GetSingleValueDemo(ctx context.Context, kv clientv3.KV) {  
    fmt.Println("*** GetSingleValueDemo()")
    // Delete all keys
    kv.Delete(ctx, "key", clientv3.WithPrefix())

    // Insert a key value
    pr, _ := kv.Put(ctx, "key", "444")
    rev := pr.Header.Revision
    fmt.Println("Revision:", rev)

    gr, _ := kv.Get(ctx, "key")
    fmt.Println("Value: ", string(gr.Kvs[0].Value), "Revision: ", gr.Header.Revision)

    // Modify the value of an existing key (create new revision)
    kv.Put(ctx, "key", "555")

    gr, _ = kv.Get(ctx, "key")
    fmt.Println("Value: ", string(gr.Kvs[0].Value), "Revision: ", gr.Header.Revision)

    // Get the value of the previous revision
    gr, _ = kv.Get(ctx, "key", clientv3.WithRev(rev))
    fmt.Println("Value: ", string(gr.Kvs[0].Value), "Revision: ", gr.Header.Revision)
}

Output:

*** GetSingleValueDemo()
Revision: 1188  
Value:  444 Revision:  1188  
Value:  555 Revision:  1189  
Value:  444 Revision:  1189

Pagination

The kv.Get() method can return multiple key values and it allows you to paginate through them. The GetMultipleValuesWithPaginationDemo() function inserts 20 key-value pairs with a key prefix of "key", then it creates an option slice that means use prefix, sort ascended and limit the number of results to 3. It uses these options with the kv.Get() method (note the Go ellipsis operator that allows calling a variadic function with a slice of parameters of the same type). The first call to kv.Get() retries the first page of three elements. To retrieve the next page, I need to provide the WithFromKey() option that applies the same logic but starts from the provided key. I pass here the last key of the first page, so I skip it when printing the results and show only two items instead of three for the second page.

func GetMultipleValuesWithPaginationDemo(ctx context.Context, kv clientv3.KV) {  
    fmt.Println("*** GetMultipleValuesWithPaginationDemo()")
    // Delete all keys
    kv.Delete(ctx, "key", clientv3.WithPrefix())

    // Insert 20 keys
    for i := 0; i < 20; i++ {
        k := fmt.Sprintf("key_%02d", i)
        kv.Put(ctx, k, strconv.Itoa(i))
    }

    opts := []clientv3.OpOption{
        clientv3.WithPrefix(),
        clientv3.WithSort(clientv3.SortByKey, clientv3.SortAscend),
        clientv3.WithLimit(3),
    }

    gr, _ := kv.Get(ctx, "key", opts...)

    fmt.Println("--- First page ---")
    for _, item := range gr.Kvs {
        fmt.Println(string(item.Key), string(item.Value))
    }

    lastKey := string(gr.Kvs[len(gr.Kvs)-1].Key)

    fmt.Println("--- Second page ---")
    opts = append(opts, clientv3.WithFromKey())
    gr, _ = kv.Get(ctx, lastKey, opts...)

    // Skipping the first item, which the last item from from the previous Get
    for _, item := range gr.Kvs[1:] {
        fmt.Println(string(item.Key), string(item.Value))
    }
}

Output:

*** GetMultipleValuesWithPaginationDemo()
--- First page ---
key_00 0  
key_01 1  
key_02 2  
--- Second page ---
key_03 3  
key_04 4

There are other options to specify which items to fetch. For example, the Range option accepts an end key and fetches all the keys from the start key to the end key (not including the end key).

Working with Leases

Another important feature of etcd is leases and lifetimes for keys. When you put a key you can assign it a lease with time to live (TTL). When a lease expires the key is removed automatically.

The LeaseDemo() function deletes all keys with the "key" prefix and shows that there is no key called "key". Then it uses the Grant() function to create a lease with TTL of one second. It puts a key called "key" with the lease and shows that it is available in etcd. Then, it waits three seconds (must be greater than the dial timeout of two seconds) and shows that the key has disappeared.

func LeaseDemo(ctx context.Context, cli *clientv3.Client, kv clientv3.KV) {  
    fmt.Println("*** LeaseDemo()")
    // Delete all keys
    kv.Delete(ctx, "key", clientv3.WithPrefix())

    gr, _ := kv.Get(ctx, "key")
    if len(gr.Kvs) == 0 {
        fmt.Println("No 'key'")
    }


    lease, _ := cli.Grant(ctx, 1)

    // Insert key with a lease of 1 second TTL
    kv.Put(ctx, "key", "value", clientv3.WithLease(lease.ID))

    gr, _ = kv.Get(ctx, "key")
    if len(gr.Kvs) == 1 {
        fmt.Println("Found 'key'")
    }

    // Let the TTL expire
    time.Sleep(3 * time.Second)

    gr, _ = kv.Get(ctx, "key")
    if len(gr.Kvs) == 0 {
        fmt.Println("No more 'key'")
    }
}

There are a few other operations related to leases. You can extend a lease forever or just one second and you can revoke a lease, too.

Additional Features

The etcd store has a lot of additional features I didn't explore in this article.

Compose etcd3 Cluster

Compose offers a fully managed and highly available 3-node etcd3 cluster. This can offload the burden of configuring and maintaining it yourself. Check this out for further details: https://compose.com/databases/etcd

Conclusion

Etcd3 is a huge improvement since etcd2. It is a robust, performant and sophisticated distributed key-value store that is used by multiple large-scale enterprise systems (most prominently Kubernetes). It provides a great Go API and can be an excellent choice for managing critical data in your system.

Gigi Sayfan is the chief platform architect of VRVIU, a start-up developing cutting-edge hardware and software technology in the virtual reality space. Gigi has been developing software professionally for 21 years in domains as diverse as instant messaging, morphing, chip fabrication process control, embedded multi-media application for game consoles, brain-inspired machine learning, custom browser development, web services for 3D distributed game platform, IoT/sensors and most recently virtual reality.


This article has been published as part of Compose's Write Stuff program - if you want to write about databases or share your experiences with database technology, we invite you to participate.

attribution Raphael Koh

This article is licensed with CC-BY-NC-SA 4.0 by Compose.

Conquer the Data Layer

Spend your time developing apps, not managing databases.