Building JavaScript Microservices with SenecaJS and Compose

Published

Database-backed microservices are powerful and in this article we show how to use SenecaJS, NodeJS and Compose databases to create a virtual product catalog using them. Microservices are making a huge dent in the web development world, with companies like Netflix, Walmart, and IBM embracing microservice architectures for their mission critical applications.

Introduction to Microservices

First, in case you've been hiding in the dungeons of ancient enterprise application development for the past few years, let's start with a small introduction to microservices. If you're already comfortable with microservices, you can skip to the next section.

A typical software application consists of a single process that contains all of the instructions you'll need to execute the application. This monolithic architecture allows a process to be quickly and simply executed, but scaling up functionality requires executing multiple copies of the entire process even if you only need to scale up one small piece of functionality (for example, Netflix uses the streaming portion of its application far more than it uses the 'new user signup' portion).

One of the challenges that can arise from this "all-in-one" approach is that it's very difficult to scale your applications. Since monolithic applications are designed to be self-contained, the mechanisms for connecting multiple redundant copies of a monolithic process can be impractical or impossible if the application hasn't been designed to accommodate such tandem scaling. Errors in monolithic applications also tend to be catastrophic, requiring a reboot of the entire application and resulting in downtime for end users.

Modern distributed software, such as large web applications, need to be able to scale easily and fail robustly by running processes on multiple machines in multiple locations. Microservices architectures are designed to allow for exactly this type of behavior. Each microservice handles a small piece of functionality for the application, and multiple microservices connected together in a network constitutes a complete application. Designing applications as a "system of systems" means that developers can add new instances of a microservice to handle the extra load without impacting the existing system. Developers can also redirect traffic to redundant copies running on different servers in the event of server or application failure.

Microservices aren't without their faults - while a single microservice might be a small and simple piece of software, the way it interacts with other parts of the application is more complex than a typical monolithic application. Frameworks like SenecaJS provide tools to make these interactions easier.

Introduction to SenecaJS

SenecaJS is a NodeJS application framework that allows developers to define microservices and connect them together in different ways. The fundamental unit of communication in SenecaJS is a simple JSON message. By convention, that message will be an object with at least 2 keys: a role and a cmd. For example, a message that outputs hello, world might look like the following:

{ "role": "hello", "cmd": "sayHello" }

SenecaJS looks at the message and compares it to a set of commands that have been registered with it. When the message matches one of those commands, SenecaJS executes the function associated with that command.

Let's start with a basic SenecaJS application to see how commands can be registered. Create a new folder for your project on your system and initialize a new Node module using npm init. The npm init command will prompt you for options for your new module. You can use the defaults. Then, use npm install to install SenecaJS.

$ mkdir seneca-compose
$ cd seneca-compose
$ npm init
...
...
...
$ npm install --save seneca

Now, we'll create a simple ping application. This application will just respond with a message when we call into it letting us know that the application is Online and working correctly.

// Create a new SenecaJS application
var seneca = require('seneca')();  

First, we'll create a new SenecaJS application by calling the "seneca()" function. This provides us with a place to register new commands and functionality.

seneca.add({"role": "compose", "cmd": "ping"}, (args, done) => {  
  console.log(args);
  done(null, {result: "Hi there"});
});

Next, we'll call the seneca.add method. seneca.add is used to register a new command with the seneca system, and to define the function that should be executed when that command it sent. It takes two arguments: a command pattern and the function that executes when the pattern is matched. The function that executes also takes two arguments: an args object which contains the entire message that was sent (including the command pattern and any extra parameters) and a done function which is executed once the service is ready to send a response. The done function itself also takes two arguments in typical node.js fashion: an error as the first argument (or null if there's no error) and the results of any operations as the second argument.

seneca.listen({"type": "http", "port": 8080});  

Finally, we'll have the application to listen for connections using the seneca.listen method. Since SenecaJS uses simple JSON messages, there are many different ways you can communicate between microservices (called the transport in SenecaJS lingo). There are transport plugins for Redis, PubNub, TCP, and RabbitMQ among others. In our example, we'll use HTTP which has the added benefit of allowing us to test our microservices using the curl utility. The HTTP transport exposes a special route, /act, for this purpose and also useful for connecting other applications and external systems (for example, those designed in another programming language) to SenecaJS-designed microservices. If your platform has an HTTP library, it can communicate with a SenecaJS-based Microservice.

To test our microservice, you'll need to open two terminal windows. In the first window, you'll run your SenecaJS microservice.

$ node service.js

In the second window, we'll use the curl utility to send a message to our running service.

$ curl -d '{"role":"compose","cmd":"ping"}' http://localhost:8080/act

If all goes well you should see the following output:

{result: "Hi there"}

That's about it! All SenecaJS applications are designed in a similar fashion: use .add to register command patterns and tie that command to a function, and then execute that function by sending a message into SenecaJS matching the pattern.

Storing Data Using Seneca Entities

Now that you have a feel for creating a microservice in SenecaJS, let's do something a little more interesting: save data in a database. SenecaJS by itself is minimally functional, but there is an ecosystem of plugins (many developed by the SenecaJS team itself) that can be used to make SenecaJS far more powerful. We'll use a plugin called seneca-entity that makes storing and retrieving data in SenecaJS a snap. If you've ever used an ORM like ActiveRecord or an ODM like Mongoose, you'll feel right at home with Seneca entities.

As a demo, we'll create a catalog for a nerdy clothing store. Our clothing store contains products and we'll represent these products as entities. To create a new entity, you'll first need to add the seneca-entity and seneca-basic plugins.

npm install --save seneca-entity seneca-basic  

Then, you can tell SenecaJS to use those plugins with the .use command. If you have experience with NodeJS development with ExpressJS you'll recognize the syntax:

seneca  
  .use('basic')
  .use('entity');

The seneca-entity plugin decorates our SenecaJS application with a new method, .make$, which we'll use to create a new entity type. Call seneca.make$ and pass in the name of the entity you'd like to create. seneca.make$ will return a new instance of that entity that you can now modify as you would any other JavaScript object.

var product = seneca.make$("Product");  
product.name = "Star Wars Jacket";  
product.price = 100.00;  
product.description = "The force will be with you with this stellar Star Wars jacket!";  

Now, you can save the new product using the save$ function. save$ takes a callback with the function signature of (err, savedProduct) where err contains any errors encountered while saving (or null if there are no errors) and savedProduct contains the entity that was persisted to the database.

product.save$((err, savedProduct) => {  
  if (err) {
    // handle error in your application.
  } else {
    // handle saved product in your application
  }
});

Let's bring this all together with a new command we can add to SenecaJS. This new command will add a product to our virtual catalog. We'll also demonstrate how to take the name, price, and description for the product from arguments passed into the service.

// service.js

// Create a new SenecaJS application
var seneca = require('seneca')();

seneca  
  .use('basic')
  .use('entity');

seneca.add({"role": "product", "cmd": "create"}, (args, done) => {  
  var product = seneca.make$("Product");
  product.name = args.name;
  product.description = args.description;
  product.price = args.price;
  product.save$((err, savedProduct) => {
    done(err, savedProduct);
  });
});

// Listen for messages in the specified transport type and port.
seneca.listen({  
    "type": "http", 
    "port": 8080
});

Then, just like we did earlier, run your service in one terminal by calling the following:

$ node service.js

And, in a separate terminal, use this curl command to add a new product to the catalog:

$ curl -d \
  '{"role":"product","cmd":"create","name":"Star Wars Jacket","price":100.00,"description":"Awesome!"}' \
    http://localhost:8080/act

You'll notice that we've passed in far more data than we were listening for with the command. SenecaJS is only concerned with matching all the parts of it's command to your message (ie: role: product, cmd: create). Since those two fields are present in our message, it's considered a match. The extra data is sent into the function in the args object, which we used to send in extra information about our product in the catalog.

Since we sent back the savedProduct as our success response, we should see something like the following (notice the new entity$ and id fields added by the entity system):

{
   "entity$":"-/-/Product",
   "name":"Star Wars Jacket",
   "description":"Awesome!",
   "price":100,
   "id":"6esbkg"
}

Storing Entities In Memory

You might be wondering where the entities are currently being saved. By default, SenecaJS provides a memory store plugin that saves the entities using an in-memory database. This default behavior allows us to create some entities and test out operations on them during early development. This is also useful for creating test drivers that can validate entity logic without connecting to a persistent database.

Using this plugin concept, we can now swap out the in-memory database with a plugin that adds a persistent backing data store without having to change any of our application or entity logic.

Storing Entities in Mongo for Compose

To store our entities in MongoDB, we first need to spin up a new Mongo database in Compose.

Once you've created a new database, we'll use mongo-store to tell SenecaJS to use MongoDB as the persistent backing store for our application. First, we'll install the seneca-mongo-store node package:

npm install --save seneca-mongo-store  

Next, we'll modify our SenecaJS application to use the mongo-store plugin. Since SenecaJS plugins are added in the order your add them in your application, we'll .use the mongo-store plugin after we .use the entity plugin (otherwise the entity plugin will default back to memory store):

seneca  
  .use('basic')
  .use("entity")
  .use('mongo-store', {
      uri: 'mongodb://<youruser>:<password>@aws-us-east-1-portal.23.dblayer.com:16659,aws-us-east-1-portal.21.dblayer.com:16659/whatever?ssl=true' 
  })

Make sure you use your username and password in the connection URI.

The final program looks like the following (notice that the only thing we've done from the previous example is to add the extra .use('mongo-store', ...) after .use('entity')):

// service.js

// Create a new SenecaJS application
var seneca = require('seneca')();

seneca  
  .use('basic')
  .use("entity")
  .use('mongo-store', {
      uri: 'mongodb://testuser:secret@aws-us-east-1-portal.23.dblayer.com:16659,aws-us-east-1-portal.21.dblayer.com:16659/whatever?ssl=true'
  })

seneca.add({"role": "product", "cmd": "create"}, (args, done) => {  
  var product = seneca.make$("Product");
  product.name = args.name;
  product.description = args.description;
  product.price = args.price;
  product.save$((err, savedProduct) => {
    done(err, savedProduct);
  });
});

// Listen for messages in the specified transport type and port.
seneca.listen({  
    "type": "http", 
    "port": 8080
});

Our entities should now persist to our new Mongo database. To test this out, run your service like before in one terminal, and run the curl command to add a product in the other:

$ node service.js

And, in a separate terminal, use this curl command to add a new product to the catalog:

$ curl -d \
  '{"role":"product","cmd":"create","name":"Star Wars Jacket","price":100.00,"description":"Awesome!"}' \
    http://localhost:8080/act

You should get back the same response (something like the following):

{
  "entity$":"-/-/Product",
  "name":"Star Wars Jacket",
  "description":"Awesome!",
  "price":100,
  "id":"5873cec107d642a2e9216d73"
}

Now, when you navigate to the database browser in Compose, you'll see a new collection in your database called "Product" and a new item in that collection:

Seneca-REDIS-Store

Storing an item in REDIS uses the same process as storing an item in MongoDB. First, you'll need to spin up a new REDIS on Compose database.

Once you've created a new deployment, install the seneca-redis-store plugin:

$ npm install --save seneca-redis-store

Then, in your microservice, add the redis-store plugin and use your Redis connection string:

seneca.use('basic')  
    .use('entity')
    .use('redis-store', {
      'uri': 'redis://x:HFLFFYKEMELVLHKW@portal.tangy-redis-22.jwo.composedb.com:15933'
    }

If you've been following along, you'll notice that it's the same basic structure as the mongo-store configuration above. The final service looks like this:

// service.js

// Create a new SenecaJS application
var seneca = require('seneca')();

seneca  
  .use('basic')
  .use("entity")
  .use('redis-store', {
    'uri': 'redis://x:HFLFFYKEMELVLHKW@portal.tangy-redis-22.jwo.composedb.com:15933'
  });

seneca.add({"role": "product", "cmd": "create"}, (args, done) => {  
  var product = seneca.make$("Product");
  product.name = args.name;
  product.description = args.description;
  product.price = args.price;
  product.save$((err, savedProduct) => {
    done(err, savedProduct);
  });
});

// Listen for messages in the specified transport type and port.
seneca.listen({  
    "type": "http", 
    "port": 8080
});

Once again, run the service in one terminal:

$ node service.js

and run your CURL command in the other:

$ curl -d \
  '{"role":"product","cmd":"create","name":"Star Wars Jacket","price":100.00,"description":"Awesome!"}' \
    http://localhost:8080/act

You should get something like the following as output from the service:

{
  "entity$":"-/-/Product",
  "name":"Star Wars Jacket",
  "description":"Awesome!",
  "price":100,
  "id":"0c7b4cfa-ff6c-4329-97a6-9f58cc47c9f6"
}

And when you visit the data browser in your Compose for Redis database, you should see a new key that contains your newly saved product:

Wrap Up

Writing microservice applications in JavaScript doesn't have to be a hassle. With SenecaJS and Compose, you can quickly spin up new server-side applications and swap out databases as your needs change. SenecaJS also allows you to use multiple transport mechanisms, and in a future article, we'll discuss how to use RabbitMQ and Redis on Compose to create stable and secure distributed microservices applications.


If you have bits you think should be in NewsBits, or any feedback about any Compose articles, drop the Compose Articles team a line at articles@compose.com. We're happy to hear from you.

John O'Connor
John O'Connor is a code junky, educator, and amateur dad that loves letting the smoke out of gadgets, turning caffeine into code, and writing about it all. Love this article? Head over to John O'Connor’s author page to keep reading.

Conquer the Data Layer

Spend your time developing apps, not managing databases.