Portal Scaling for smooth access: Noteworthy at Compose


This is your weekly summary of Compose news for those changes and updates which can make your life easier. In this August 13th 2018 edition, a look at portals and the why and the when of needing to scale them, what's missing in metrics, and Compose account's tax ids. And we also take a browse through the past week's Compose Articles.

Scaling Portals

Make sure that when you scale up, you scale up the right thing on Compose. There are, in fact, two different things you can scale on a typical Compose database deployment. One is the database itself; you can manually scale it up to preempt autoscaling or manually scale it down to reclaim space after removing data (and doing whatever the database needs to do to free up space). This is the most common variant of scaling.

The other is scaling portals.

Whats in a portal?

Portals are the software bastions that sit on the border between the internet and your databases, protecting them by only allowing particular traffic through, acting as an endpoint or authenticator for other connections. They are, by their nature, much smaller in terms of resource usage than the database nodes. If you are flooding your server with connections, they can also be the first to take the brunt of the overload problem.

A typical portal has 64MB of RAM allocated to it. Consume all that RAM by opening thousands of connections and the portal will start blocking and timing out new connections. Your applications may report database errors, but whats happening is the connections are being stalled in the portal.

Reading the metrics

How will you know this is happening? Head over to the Metrics page for the deployment and you'll see graphs with portal.n at the end of the name. These are the portal metrics; the database metrics are the graphs with data.n at the end of their names.

Anyway, if you consult these graphs, the chance are you'll find memory usage is running over the memory limit and/or there will be a regular presence of significant memory failcnts over time.

Scaling to size

If thats the case, then head to the Resources tab of the deployment and select the Scale button for the portals. By default, you can add a minimum of 64MB to your basic 64MB portal RAM allocation.

There's a cost involved in scaling up like this and your UI will tell you what that cost will be. You can go all the way up to 4GB of RAM in a portal but its very unlikely that you'll want to ever get too far from the lower end of the scale.

Pool powers

If, after scaling up the portals, your database is still flooded with connections, then you'll want to look at re-architecting your applications to make better use of connection pools and resource sharing.

If you have a web app and it creates a database connection for every session that logs into your application, things are not going to get better until you make your applications share one or two connections between multiple sessions.

Missing Metrics

The keen eyed among you may have noticed a slight change in the Metrics view on Compose.com. The "Memory Soft Limit" is now no longer displayed on the graphs. This is partly to simplify the graph and partly to to remove a metric which was, by policy, exceeded without any effect.

Compose and Tax IDs

We've been sharpening up our signup process to make it work around the world better. One effect of that is that new accounts for businesses around the world will now need to add their tax identification number, as is relevant to their location, when they create new accounts on Compose. This has the knock on effect that when you, as an existing Compose user, edit your payment information on Compose, you'll be asked for your tax identification number too.

Noteworthy articles

The lead article last week was How to keep your etcd lean and mean. This delved into to world of etcd's revisions and compactions. For the curious, etcd is different in that it keeps all revisions of the key/value store, sequentially numbering them. It keeps them until its told to dispose of them up to a particular number, leaving only the live keys. It's an interesting mechanism which gives etcd a lot of power but means you have to read your database statistics differently.

Also last week, Friday's NewsBits was packed with news about the PostgreSQL udates, Redis 4.0.11, JanusGraph 0.3.0, MongoDB 4.0.1, CouchDB 2.2, the 1.0 release of the Julia language and a giant 6502 processor.

And as a reminder, last week's Noteworthy on the retirement and promotion of databases so if you run Elasticsearch 2.4.6, do read.

That's it for this week's Noteworthy at Compose. Onwards to next week!

Dj Walker-Morgan
Dj Walker-Morgan was Compose's resident Content Curator, and has been both a developer and writer since Apples came in II flavors and Commodores had Pets. Love this article? Head over to Dj Walker-Morgan’s author page to keep reading.

Conquer the Data Layer

Spend your time developing apps, not managing databases.