The mystery of etcd's rationed requests

With etcd in beta at Compose, the complexity of workloads on the database has been steadily increasing. That increasing workload means that we get to understand how users' tasks can load our databases as synthetic testing can only do so much. Our understanding of etcd's behavior was already pretty extensive as we used it in Governor, the PostgreSQL high availability template that was developed at Compose to simplify making PostgreSQL always online.

So when reports started coming in that our performance was lower than expected, Compose engineers were on the case. The first port of call was to re-benchmark the clusters and what Josh in engineering came back with surprised us all; the current deployments were apparently not breaking ten queries per second. The first question had to be "are these benchmarks wrong?". Josh went through some variations of the benchmarking scenarios to see if there was some error that was making the numbers remarkably low.

A brief glimmer of light came when apparently a test run from the proxy node showed a 2x performance boost, but that turned out to be the benchmark miscounting authentication failures when basic-auth wasn't set up in the benchmark. Little did anyone know that this benchmarking error would be the key to understanding what was going on.

The next strategy was to test the networking fabric to ensure there wasn't some interaction with Compose's software-defined networks. Josh created new etcd deployments with more nodes to see if that theory would hold water - more nodes meant more inter-node communication and could amplify any underlying problem. That theory didn't hold up; the performance stayed at the same low level but didn't degrade any further. The next step was to eliminate the networking completely without etcd; running networking benchmarks showed 2Gbps rates between all nodes and 1.1ms latency. That removed the network from possible causes of the performance issue.

The next step was to iterate through various topologies of etcd running benchmarks inside the network to see if it was possible to iterate through them till we saw the performance issue. Josh set out on a marathon of configuration and benchmarking many topologies, but in the end, the performance issue stubbornly refused to appear.

At this point, Nick, an engineering lead at Compose, noted he was getting good performance on a deployment he'd set up as a user. A quick investigation into that revealed the same issue as the flawed benchmark earlier; basic-auth was not configured, so the benchmark was miscounting. But that second occurrence sent Josh on a new line of investigation. Was it something to do with the authentication? Digging into the code, and checking against the very latest code for etcd 2.x, Josh noted that the use of the bcrypt function to verify the password shouldn't incur that much cost.

And it doesn't. Except that every request on the cluster was going through bcrypt and it was that process which was beating the performance figures down. The solution offered was to switch to client certificate based authentication. That, though, is a solution that would disrupt existing users and could require new infrastructure to manage certificates. We have proxies already in place and they could take over the authentication load.

The authentication API would need to be disabled within etcd, but the proxy now manages access to the database, so that is not a problem. What won't be available are the granular access permissions for the etcd store, but our current understanding from the beta is this isn't used by most users. We will be working on bringing back granular user authorization and, if it's important to your use of etcd, please let us know how you want to use it so we can encompass as many use cases as possible in any future solution.

The big question is what happens now we have moved authentication to the proxy? Things go a lot faster. As in magnitudes faster. Repeating the earlier benchmarks saw the query time go from ten requests per second to forty thousand requests per second. The changes have also improved cluster stability under load. Those are all changes we know our users will appreciate. You should already be appreciating them as we have upgraded most etcd deployments to use proxy-terminated basic authentication.

With the mystery solved, you can try out beta etcd on Compose today, simply by selecting Create Deployment and picking it from the beta database list.


If you have any feedback about this or any other Compose article, drop the Compose Articles team a line at articles@compose.com. We're happy to hear from you.

Image by Mickey O'neil