Monitoring Redis: The small, hard way

Published

Ever wanted to get your Redis database statistics onto a tiny screen on your desk? I have, and it's a journey of discovery to see how small a device you can pack that functionality into.

In this article we'll show you one device, a prototype of Redismon, a desktop scaled Redis monitor. This one was a little challenging to build. The idea is to go beyond previous Compose Tinkertank projects and produce actionable graphs on your physical desktop. I want to track ops per second and the I/O flow of my Redis server.

The Hard Brains

For this version, I went with the Particle Photon. Its big attribute is that it's a small board with an Arduino-like toolchain, an ARM M3 processor, 128KB of RAM and WiFi integrated on chip. Even though there's plenty of memory, this project originated on a more constrained Arduino and I'm continuing to be a memory miser.

The project needs that Wifi to talk to the Redis server. The display is an Adafruit SSD1351, a 128x128 OLED panel. This is what the prototype looks like:

It also has a microSD slot which isn't supported at the moment, but if it were I could use it to store a copy of the statistics or configure the server details through files stored on MicroSD. Thats for another day though.

Connecting the OLED board to the Photon using the hardware SPI interface takes 5 wires plus two wires for voltage and ground. The details of how to connect are in the Readme file for the SSD1351 OLED driver for the Photon. That driver works with the Adafruit gfx library, so all our drawing needs are taken care of.

If you've not met the Particle Photon in the flesh, there's another attribute worth knowing about it. You can program it remotely. There's a web-based IDE for it that also has many libraries - including the SSD1351 driver - on tap. When you set up a Photon for the first time, you can use your phone to tell it which Wifi network to attach to. It then goes and logs into the Particle Cloud where you can claim it and send programs to it. It will reflash and restart itself. The USB cable you can see in the pictures is supplying power only (though if you aren't a cloud IDE fan you can program it locally over the USB). Anyway, software is what the Redismon needs now so...

The Software Brains

We also need to talk Redis protocol with the server. There's no driver for Arduino styled devices that I know of, but that's ok because we already know how to talk raw Redis from a previous article. If you've not read that article, go read it as it will show you how the Redis protocol is very simple, powerful and able to be used using no more than the echo and netcat command so you can experiment from the comfort of your command line.

Let's dive straight into some code. If you're following along at home, the full code is available on the Redismon repo on Github.

We're going to explain what happens in the main loop first; like the Arduino, Photon applications have a setup() routine which initializes it and a loop() routine which is repeatedly called forever. Let's assume everything is already setup and head to the start of the loop that runs on the Photon:

#define HOSTNAME "example.1.dblayer.com"
#define HOSTPORT 10030
#define HOSTAUTH "GNQVVEJSXMBGJQZD"
...
void loop() {  
    unsigned char buffer[BUFFLEN];
    uint8_t bytesread;

This gets us a buffer, ready for when we do read some data. There are three #defined constants you see here which actually appear earlier in the code: HOSTNAME - the name of the host running Redis, HOSTPORT - the port number to connect to and HOSTAUTH - the password/auth string for the Redis database. In the loop, we connect to this server:

    if(client.connect(HOSTNAME,HOSTPORT)) {

We need to log in to the server with our auth string so, as per the raw Redis article, we send an auth command. *2 says 2 strings are being sent:

        client.print("*2\r\n");
        sendString(client,"AUTH");
        sendString(client,HOSTAUTH);

sendString is a small function to make writing strings to Redis easier. They need to be preceded by a $ and the length of the string so this:

void sendString(TCPClient client,char *msg) {  
  client.print("$");
  client.println(strlen(msg));
  client.println(msg);
}

takes care of that task for us. Back to the loop. Now we've sent our auth string we need to check for a "+OK" to be returned. Anything else coming back is a good reason to close the connection and try again.

        bytesread=getLine(client,buffer);
        if(bytesread==4 && strcmp("+OK",(char *)buffer)==0) {

If we get our OK then we are ready to ask for our stats. We're using the INFO STATS command, so that's two words. Before sending the words we first send a count of the number of strings we are sending, and then each string...

          client.print("*2\r\n");
          sendString(client,"INFO");
          sendString(client,"STATS");

So we're going to be getting a fair wedge of text back now. The next command is a getBuffer command to read that...

          uint16_t nowread=getBuffer(client,buffer);

But that hides what was probably the hardest part of this project. You can check out the code in the repository if you want the detail but the short version is this. The command you use to check available characters in the buffer on the Particle Photon, getAvailable() is destructive. Call it and you have to read that many characters off because getAvailable() clears its counter after being called rather than counting down reads. It makes for an elaborate dance in the code and a simple lesson: when you're down at the embeddable device level, don't assume everything works as it does on a desktop system. Sometimes compromises get made for odd but legitimate reasons. What we end up with is a buffer full of something like this:

# Stats
total_connections_received:3533085  
total_commands_processed:113836687  
instantaneous_ops_per_sec:7  
total_net_input_bytes:5540092675  
total_net_output_bytes:4782042896439  
instantaneous_input_kbps:0.38  
instantaneous_output_kbps:1.40  
rejected_connections:0  
sync_full:5  
sync_partial_ok:0  
sync_partial_err:0  
expired_keys:0  
evicted_keys:0  
keyspace_hits:10000864  
keyspace_misses:8  
pubsub_channels:1  
pubsub_patterns:0  
latest_fork_usec:733  
migrate_cached_sockets:0  

There's a lot of values there so to efficiently strip them out we use a slightly larger version of the function below.

#define VALCNT 64

char *opspersecstr="instantaneous_ops_per_sec:";  
int opspersecstrlen=strlen(opspersecstr);  
int opspersec[VALCNT];  
int lastopspersec;

int off=0;

void update_stats(char *buffer) {  
    char *token;
    int state=0; // 0=label, 1=value
    char *label;

    token=strtok(buffer,"\n");
    while(token!=NULL) {
        if(token[0]!='#') {
            if(strncmp(token,opspersecstr,opspersecstrlen)==0) {
                lastopspersec=atoi((char *)(token+opspersecstrlen));
                opspersec[off]=lastopspersec;
            } 
        }
        token=strtok(NULL,"\n");
    }
}

This tokenizes the buffer into lines. If the start of the line matches the parameter we are looking for, then we copy it into an array which is being used as a ring buffer for values read over time. Ring buffers avoid copying memory around when inserting new values and on a small device, as every byte is precious. In the full version, we actually look for five different properties and parse integers or floating point as appropriate.

Once we've got those values, it's time to...

            render_stats();

We render three of the stats we collected in the actual code but here, we'll simplify it by only doing one of the stats as otherwise the code is overly repetitive. So here's the simplified code for the routine:

void render_stats() {

    int maxopspersec=0;

    for(int i=0;i<VALCNT;i++) {
        if(opspersec[i]>maxopspersec) {maxopspersec=opspersec[i]; }
    }

    float r=96.0;
    float sops=r/(float)maxopspersec;

The first part of rendering the stats is scaling things. We scan through the ring buffer looking for the maximum value. Once we have that, we work out a scaling factor to scale it to 96 pixels high. The display is only 128x128 and I want to use the top 32 rows for three lines of text. I'm lucky to be using a device with decent floating point handling. If I only had integer arithmetic available (as you can find), the values themselves would have to be scaled up to do this and later calculations.

    int i=off+1;
    if(i>=VALCNT) { i=0; }

    int loppt=0;

First we set ourselves a pointer into the ring buffer(s). When we render, we're assuming that the global offset is pointing at the current data point, the one that we just parsed. Add one to that and we're looking at the oldest data point in the buffer (with a quick check to make sure it didn't wrap around). Now we know where we are starting from, we'll also create a variable for the last point plotted.

Time to start looping through from 0 to the number of items in the ring buffer.

    for(int t=0;t<VALCNT;t++) {
        int oppt=(int)((float)opspersec[i]*sops);

        if(t==0) {
            loppt=oppt;
        }

Using i, our offset pointer, we multiply the data point by the scaling factor we calculated. If it's the first point we're are about to draw, we set out "last point" to this calculation too. Now, finally, we can draw something.

        tft.fillRect(t*2,0,2,127,BLACK);
        tft.drawLine(t*2,127-loppt,t*2+1,127-oppt,RED);

Well, not quite. First, we fill in the rectangular space where we're about to draw with black to clear old graph output. Why not call clear or something similar you may ask? Well, that would make the screen blink regularly. This way, we shuffle through the image updating it and more of an animated feel as the graph moves forward. To do that, we draw a line from the last point value to the new point value a pixel or so along. This produces a satisfactory graph; repeat three times for different data sets and you get your stats graph. We're not done yet though, we need to save the current point as the last point and bump that ring buffer offset on by one.

        loppt=oppt;

        i=i+1;
        if(i>=VALCNT) { i=0; };
    }

If it was just squiggly lines, our graph wouldn't be much use beyond impressionism. Those 32 rows at the top are plenty of space to print in some specific numbers. We print the current value and the maximum value rendered in the current view in the same color as its graph line.

    tft.setCursor(0,0);
    tft.setTextColor(RED);
    tft.printf("ops: %d/%d\n",lastopspersec,maxopspersec);
}  

And that's the rendering done. Back to the loop where we bump that ring buffer offset by one, stop the Redis TCP connection and go into a 5 second sleep before doing it all again.

          bump_off();
        } else {
          Serial.printf("Got %d bytes read, trying again\n",bytesread);
        }
        client.stop();
      }

    delay(5000);
}

More Brains?

This is the simplest iteration of the Redismon project and can happily show database activity in terms of operations per second and input/output data. In the full code, we do extract two other metrics so the option is there to add a toggle button to the device to switch between views of what data is rendered. As I mentioned, there's an MicroSD card reader there that could also be put to a number of uses. But what we've done here is the most important part - we've acquired Redis statistics and parsed them into buffers - the rest is left to the reader.

Beyond Redismon?

The next step for this version of the project is to 3D print a case for it and mount the Particle Photon behind the display. This is something that, you learn with experience, is easier with boards without pin headers. Pin headers make prototyping easy but packaging hard. There may also be some experimenting with LiPo batteries and boosters to see how long the device could run on a reasonably sized battery.

Hopefully, you'll feel inspired to make your own ambient database stats monitor. If you do, let us know at Compose Articles and we may feature it in a future article. In the meantime, look out for the next evolution of Redismon, the Pidb-Piper, coming relatively soon to these pages.

Dj Walker-Morgan
Dj Walker-Morgan is Compose's resident Content Curator, and has been both a developer and writer since Apples came in II flavors and Commodores had Pets. Love this article? Head over to Dj Walker-Morgan’s author page to keep reading.

Conquer the Data Layer

Spend your time developing apps, not managing databases.