To Scylla and RethinkDB - The Node.js Compose Grand Tour

Published

The Node.js segment of the Grand Tour of database connections continues with Scylla and RethinkDB.

It's time for Scylla and RethinkDB to be plugged into Node.js and the Compose Grand Tour's example web application. The Compose Grand Tour is a series of articles and examples which show how to connect to the nine databases of Compose with different languages. In previous tours, we've covered Go and Python and this is the third stage of the Node.js/JavaScript tour.

In the first stage of the Node.js tour we talked about the common example we use in Grand Tour articles and then covered MongoDB, Elasticsearch, and PostgreSQL.

The second stage moved on to Redis and RabbitMQ taking on two driver options for Redis and the messaging variant of the example for RabbitMQ. Now, we're onto Scylla, with all of its Cassandra-compatibility, and RethinkDB, the JSON database with real-time feeds. Let's start with Scylla.

Scylla

The code for the Scylla example is in the example-scylla directory of the Node.js Grand Tour Repository.

The Scylla Driver

For this example, we'll use the Datastax Cassandra driver, cassandra-driver. Its documentation lives on the Datastax developer site. We will also be using composeaddresstranslator, a package designed to make connecting to Scylla easier. We'll include these packages at the start:

const cassandra = require("cassandra-driver");  
const compose = require("composeaddresstranslator");  
const uuid = require("uuid");  

The Scylla Connection

Scylla is one of the more complex databases to connect to because it has three portals to connect to at the same time and it has to translate the cluster connections between inside.

For that to work, we have to set two environment variables outside of the code, COMPOSE_SCYLLA_URLS and COMPOSE_SCYLLA_MAPS. The first one is a comma-delimited list of the connection strings displayed in the Compose console overview. These should include username and password so that setting it would look something like this:

export COMPOSE_SCYLLA_URLS='https://scylla:PASSWORD@portal1156-7.cilla.compose-3.composedb.com:24836,https://scylla:PASSWORD@portal1085-8.cilla.compose-3.composedb.com:24836,https://scylla:PASSWORD@portal1085-6.cilla.compose-3.composedb.com:24836'  

The setting for COMPOSE_SCYLLA_MAPS is also in the overview and can be literally copied from the UI into an environment variable:

export COMPOSE_SCYLLA_MAPS='{  
  "10.150.130.5:9042": "portal1156-7.cilla.compose-3.composedb.com:24836",
  "10.150.130.6:9042": "portal1085-8.cilla.compose-3.composedb.com:24836",
  "10.150.130.7:9042": "portal1085-6.cilla.compose-3.composedb.com:24836"
}'

with these two environment variables in place, we can go back to the code and prepare to use them for our connection.

Preparing to Connect

We begin our preparations by retrieving the URLs from the environment.

let connectionString = process.env.COMPOSE_SCYLLA_URLS;

if (connectionString === undefined) {  
  console.error("Please set the COMPOSE_SCYLLA_URLS environment variable");
  process.exit(1);
}

Now, all we need to extract from these are the username and password and work out if we need to use TLS/SSL at all. To do that, we'll parse the URL for the authentication and, if the URLs begin with https:, we will set create a map for TLS/SSL options.

let myURL = require("url").parse(connectionString);  
let auth = myURL.auth;  
let splitAuth = auth.split(":");  
let username = splitAuth[0];  
let password = splitAuth[1];  
let sslopts = myURL.protocol === "https:" ? {} : null;  

We can then move on to processing our address translation map, retrieving it from the environment on the way:

let mapString = process.env.COMPOSE_SCYLLA_MAPS;

if (mapString === undefined) {  
    console.error("Please set the COMPOSE_SCYLLA_MAPS environment variable");
    process.exit(1);
}

let mapList = JSON.parse(mapString);  

We parse the environment variable from JSON into a hash.

Now we have enough to start building the connection. We'll start by turning that mapList into a ComposeAddressTranslator:

let translator = new compose.ComposeAddressTranslator();  
translator.setMap(mapList);  

Then we can create an authenticator using the username and password we parsed from the URLs:

let authProvider = new cassandra.auth.PlainTextAuthProvider(username, password);  

Time to Connect

And now we are ready to create our Scylla/Cassandra client. The cassandra.Client() constructor takes a map of various properties it uses to configure itself.

let client = new cassandra.Client({  
  contactPoints: translator.getContactPoints(),
  policies: {
    addressResolution: translator
  },
  authProvider: authProvider,
  sslOptions: sslopts
});

There's a lot to unpack here. The contactPoints are the internet facing addresses of the portals. Now, this information is available in the URLs but also available in the map and getting it from our freshly made translator with getContactPoints means we don't have to parse the URLs more.

The policies object has keys and values for alternative mechanisms for common actions. In this case, for resolving addresses, we will also use our translator. Now, when the client is given an IP address, it will first be checked against our map to convert the IP address into a hostname the client can "see".

The authProvider value comes from the PlainTextAuthProvider we created earlier. The sslOptions is either an empty object - enough to activate the TLS/SSL mechanisms - or null depending on whether the URLs were "https:" or "http:" when they were parsed.

Going live

We now need to make sure the keyspace and tables we need are in place before starting the web server in our application.

client  
  .execute(
    "CREATE KEYSPACE IF NOT EXISTS grand_tour WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3' };"
  )

We create our keyspace only if it does not exist. When that command has completed, we know the keyspace will be present so...

  .then(result => {
    return client.execute(
      "CREATE TABLE IF NOT EXISTS grand_tour.words (my_table_id uuid, word text, definition text, PRIMARY KEY(my_table_id));"
    );
  })

We can then, in the same way, create the needed table if it does not exist. Once thats completed, we are ready to start the web server:

  .then(result => {
    app.listen(port, function() {
      console.log("Server is listening on port " + port);
    });
  })
  .catch(err => {
    console.log(err);
    process.exit(1);
  });

For all those commands being executed, there's a simple catch to pick up any errors. It's time to do some reading and writing.

The Scylla Read

The read is a simple CQL query. As in other examples, we return a promise from getWords(). In this case, the SELECT command would return a promise that resolved to a cursor which isn't what we want. Instead, we'll wait for that promise to resolve with a then and that that result and extract the rows array from it. Finally, we'll wrap that rows in its own promise and return that.

function getWords() {  
  return client
    .execute("SELECT my_table_id, word, definition FROM grand_tour.words")
    .then(result => {
      return new Promise((resolve, reject) => {
        resolve(result.rows);
      });
    });
}

The Scylla Write

Unlike the read, we don't need to do anything to the promise returned by the addWord() function.

function addWord(word, definition) {  
  return client.execute(
    "INSERT INTO grand_tour.words(my_table_id, word, definition) VALUES(?,?,?)",
    [uuid.v4(), word, definition],
    {
      prepare: true
    }
  );
}

Here, field values are passed into the prepared statement through an array. That's about as simple as it gets. On to RethinkDB!

RethinkDB

The code for the RethinkDB example is in the example-rethinkdb directory of the Node Grand Tour Repository.

The RethinkDB Driver

RethinkDB has an official JavaScript driver, rethinkdb. It requires a simple npm install rethinkdb to install in an application. By convention, it's also included in code as r as in:

const r = require("rethinkdb");  
let parseRethinkdbUrl = require("parse-rethinkdb-url");  

There is another package we use with this, that's parse-rethinkdb-url. We'll explain why we need that when we make the connection.

The RethinkDB Connection

We get the connectionString and use parseRethinkdbURL to split it into an options map:

let connectionString = process.env.COMPOSE_RETHINKDB_URL;

if (connectionString === undefined) {  
  console.error("Please set the COMPOSE_RETHINKDB_URL environment variable");
  process.exit(1);
}

let options = parseRethinkdbUrl(connectionString);  

RethinkDB doesn't have a URL scheme as such. There's an informal scheme for one, but the RethinkDB drivers don't use it. That's why there's parse-rethinkdb-url to convert the informal URL scheme into the options map that RethinkDB drivers expect. Read more about those options in the connect ReQL reference page.

As the RethinkDB deployment has an associated self-signed certificate, we need to retrieve this and add it to the options map.

let caCert = fs.readFileSync(process.env.PATH_TO_RETHINKDB_CERT);

options.ssl = {  
  ca: caCert
};

With the options map complete, we can now go into the connection process. Skip towards the end of the code and you'll find that the actual act of connecting is simple enough:

let connection;

r  
  .connect(options)
  .then(conn => {
    connection = conn;
  })

But we also want to make sure that the "grand_tour" database exists. So, in a then() we do:

  .then(() => {
    return r
      .dbList()
      .contains("grand_tour")
      .do(function(exists) {
        return r.branch(
          exists,
          {
            dbs_created: 0
          },
          r.dbCreate("grand_tour")
        );
      })
      .run(connection);
  })

This is quite a ReQL query. It gets a list of databases and then runs a contains function to return true or false as to whether "grand_tour" is in the list.

Now, ReQL supports functions and conditional logic, so what's comes next is calling a function with the result of that contains as the variable exists. The ReQL branch function is called next, with exists as a parameter. If exists is true, it returns an object with a dbs_created key and a value of 0. If false though, it calls the dbCreate function and, that just happens to return an object with a dbs_created key and a value of 1.

This entire query is run against the server using .run(connection) which returns a promise. We can use that to pick up the result:

.then(result => {
    if (result.dbs_created > 0) {
      console.log("DB created");
    }

We need to go through a similar process to create a database table if needed.

  .then(result => {
    if (result.dbs_created > 0) {
      console.log("DB created");
    }
    return r
      .db("grand_tour")
      .tableList()
      .contains("words")
      .do(exists => {
        return r.branch(
          exists,
          {
            tables_created: 0
          },
          r.db("grand_tour").tableCreate("words", {
            replicas: 3
          })
        );
      })
      .run(connection);
  })
  .then(result => {
    if (result.tables_created > 0) {
      console.log("Table created");
    }
    // Listen for a connection.
    app.listen(port, () => {
      console.log("Server is listening on port " + port);
    });
  })
  .catch(err => {
    console.error(err);
  });

This checks for the existence of a table. If it doesn't exist the code runs r.db("grand_tour").tableCreate("words", { replicas: 3 }) to create it. The replicas setting is important when creating a table to avoid data loss later on. We grab the result of that creation, printing a message if we have created a table and start the web server as we've ensured the server is up.

The RethinkDB Read

We are now ready to read the word definitions from the database in the getWords() function:

function getWords() {  
  return r
    .db("grand_tour")
    .table("words")
    .orderBy("word")
    .run(connection)

Here we're going to return a promise that a query selecting all the rows of the "grand_tour" database's "words" table, and as a bonus, sorting the rows ordered by the "word" field. But this query as it is will return a cursor into the results, not the actual results that we want. So, when we have this cursor, we turn it into an array like this.

    .then(cursor => {
      return cursor.toArray();
    });
}

The ReQL cursor.toArray() function also returns a promise, which is neat and makes handling our data quick and simple.

The RethinkDB Write

Over at the addWord() function, things are equally simple.

function addWord(word, definition) {  
  return r
    .db("grand_tour")
    .table("words")
    .insert({
      word: word,
      definition: definition
    })
    .run(connection);
}

Like the read, this starts by constructing a query that selects the database and table we want. It then adds an insert() function to the query and embeds a small document with the word and definition in it as key/value pairs. Finally, it calls .run(connection) which returns a promise which the addWords() function, in turn, returns to its caller.

Parking up

That's the Scylla and RethinkDB stages completed. Now, at least for Node.js, the final stage approaches. We'll be looking at etcd and Compose for MySQL in the final part.


Read more articles about Compose databases - use our Curated Collections Guide for articles on each database type. If you have any feedback about this or any other Compose article, drop the Compose Articles team a line at articles@compose.com. We're happy to hear from you.

attribution Alexandra Nicolae

Dj Walker-Morgan
Dj Walker-Morgan was Compose's resident Content Curator, and has been both a developer and writer since Apples came in II flavors and Commodores had Pets. Love this article? Head over to Dj Walker-Morgan’s author page to keep reading.

Conquer the Data Layer

Spend your time developing apps, not managing databases.