Website Engagement Tracking with Elasticsearch

Published

Website Engagement Tracking is a technique that allows businesses to see which parts of their website users are visiting, clicking on, and viewing. In this article, we'll take a look at tracking user engagement using Elasticsearch on Compose.

Tracking users’ focus on your website can be a great way to determine which content is engaging and which content is being ignored. There are many ways to track user engagement, but one of the easiest is to track where users’ are clicking on your website.

In this first of 2-parts, we’ll create a simple JavaScript snippet that tracks users' clicks on a website and stores them in Elasticsearch so we can find the most interesting pixel regions on our website. In part-2, we'll use the Kibana Add-on in Compose to visualize those clicks using a heat map overlayed directly onto our website.

Creating the Snippet

We’ll start out by tracking clicks on our website, which is relatively simple. We’ll use the addEventListener method from the DOM API to attach a click listener to our entire document object. This method will only work in HTML5-compliant browsers, so if you need to support earlier browsers check out the Mozilla Developer Docs for a few methods you can use.

First, let’s create a simple HTML page with some content we’d like to track. We’ll use some dummy content generated by the Lorem Ipsum Generator and some dummy images from Lorem Pixel.

<html>  
   <head>
   </head>
   <body>
      <div>
         <h1>This is a test page for some interesting content</h1>
        <a href="#">Click here!</a>
        <p>
          <!-- content goes here --
        </p>
        <img src="http://lorempixel.com/400/200" />
        <p>
            Integer non tortor ullamcorper, porta eros ac, dictum eros. Sed fermentum libero massa, sed egestas libero commodo non. Duis varius quam dignissim, luctus mi at, congue diam. Integer nec augue urna. Etiam ultrices sed justo vitae volutpat...
         <!-- fill this in with as much content as you'd like -->
        </p>
      </div>
   </body>
</html>  

Next, let’s put together some JavaScript that will listen to mouse events. For now, we'll log out the event directly and see what it gives us:

document.addEventListener('click', function(event) {  
  console.log(event);
});

Simple enough - a single event listener that listens for clicks throughout our entire document. Now, when we click on the screen, we should see a MouseEvent in the logs that looks something like this:

MouseEvent  
 altKey:false
 bubbles:true
 button:0
 ...
 returnValue:true
 screenX:189
 screenY:590
 ...
 pageX: 189
 pageY: 830
 ...
 toElement:div
 type:"click"
 view:Window
 which:1
 x:180
 y:97

We’ll ignore almost all of these fields, but there are a few that are interesting to us with our click tracker. There are a few x and y coordinates, but the ones we’re the most interested in are the pageX and pageY fields, which represent the location on the website that the click occurred, irrespective of scrolling and viewport size. This means that a click on the site at the pageX location will always occur at that exact pixel location no matter how the browser is sized or how far down the user is scrolled when they click. We’ll also want to track the timestamp and toElement methods so we can search for which element was under the mouse when it was clicked.

We'll want to associate all of the click events that occurred on each load of the site. We can do this by generating a random ID each time the user loads the page. The snippet of code can give us a workable random session ID:

var generateRandomSessionId = function() {  
   return 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g, function(c) {
      var r = Math.random() * 16 | 0,
         v = c == 'x' ? r : r & 0x3 | 0x8;
      return v.toString(16);
   });
}

We’ll also want to create a timestamp using the JavaScript Date object:

var timestamp = new Date.now();  

The resulting object that we’ll store to represent each click looks like the following:

{
   "x": 100,
   "y": 100,
   "timestamp": 150000020302,
   "sessionId": 'a53cdbe2-acd1-4231-a331-fc3280d42ef1'
}

Let’s put these all together into a snippet we can install on our site.

(function() {
   var generateRandomSessionId = function() {
      return 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g, function(c) {
         var r = Math.random() * 16 | 0,
            v = c == 'x' ? r : r & 0x3 | 0x8;
         return v.toString(16);
      });
   }
   var clicks = [];
   var clickApp = {
      trackClick: function(evt) {
          var click = {
              "x": evt.pageX,
              "y": evt.pageY,
              "sessionId": generateRandomSessionId(),
               "timestamp": Date.now()
          }
          clicks.push(click);
        console.log(click);
      }
   }
   document.addEventListener('click', function(event) {
    clickApp.trackClick(event);
   });
})();

Setting Up Elasticsearch

Now that we know what we want to store, let’s start pushing that data over to Elasticsearch. Elasticsearch uses a RESTFul API, but we don’t want to push our clicks directly from the browser since our URL includes our Elasticsearch credentials. To fix both of these, we’ll create a simple Node.JS application, following along from an earlier article on using Elasticsearch from Node.JS and use Express to handle our own API.

Let’s start by creating a new Compose Elasticsearch deployment that we can push our click data out to. Then, create your Elasticsearch user and find your connection string on the Deployment Overview page. Once you have a valid connection string, we can start sending our RESTful API calls over to Elasticsearch.

You'll need npm and the following Node modules to get this working:

Install the modules using npm:

npm install express body-parser elasticsearch get-json  

We’ll use the technique presented in the Getting Started guide to create a client.js and info.js, which we can use to make connections to our Elasticsearch deployment and get info about the deployment. They should look like the following:

// client.js
var elasticsearch=require('elasticsearch');

var client = new elasticsearch.Client( {  
  hosts: [
    'https://[username]:[password]@[server]:[port]/',
    'https://[username]:[password]@[server]:[port]/'
  ]
});

module.exports = client;  
// info.js
var client = require('./client.js');

client.cluster.health({},function(err,resp,status) {  
  console.log("-- Client Health --",resp);
});

Let’s use our new info.js file to do a quick check before we move forward. Type the following into the terminal:

node info.js

You should see a response that looks like this:

-- Client Health -- { cluster_name: 'el-petitions',
  status: 'green',
  timed_out: false,
  number_of_nodes: 3,
  number_of_data_nodes: 3,
  active_primary_shards: 0,
  active_shards: 0,
  relocating_shards: 0,
  initializing_shards: 0,
  unassigned_shards: 0,
  delayed_unassigned_shards: 0,
  number_of_pending_tasks: 0,
  number_of_in_flight_fetch: 0 }

If you get an error message (usually in HTML format) then double-check your connection credentials and make sure you’ve added a user / password for your deployment.

Creating an Index

An Index in Elasticsearch is different than you might be expecting - it’s more analogous to a Table in relational databases or a Collection in MongoDB. We can create the index a number of different ways, but here we’ll follow the Getting Started guide and do this in NodeJS.

Create a new file called “init.js” and add the following:

// init.js
var client = require('./client');

client.indices.create({  
  index: 'clicks'
},function(err,resp,status) {
  if(err) {
    console.log(err);
  } else {
    console.log("create",resp);
  }
});

Run your new init.js file:

node init.js  

And you should get the following response:

create { acknowledged: true }  

Finally, let’s create the expressjs app that our frontend will call to save clicks to our database. We’ll use the Elasticsearch index call to add clicks to our index.

// app.js
var express = require('express'),  
    app = express(),
    bodyParser = require('body-parser'),
    client = require('./client'),
    path = require('path');

app.use(bodyParser.json());

app.post('/registerClick', function(req, res) {  
    client.index({  
      index: 'clicks',
      id: '1',
      type: 'click',
      body: req.body
    },function(err,resp,status) {
        res.send(resp);
    });
});

app.get('/', function(req, res) {  
    res.sendFile(path.join(__dirname, 'index.html'));
});

app.listen(process.env.PORT || 8080);  

Our app creates two routes, a /registerClick route where we’ll send our clicks to, and a / route which renders our HTML. You can access the site by running the following:

node app.js

And then opening http://localhost:8080 in your web browser.
Right now anyone can send click events to our app, so when we’re ready to take this live we’ll probably want to add some security measures to make sure that only requests from the same server are allowed (ie: so someone can’t send bad click data into our app), but we won’t cover that for now.

Connecting the Click Tracker to Node

Now that you have your backend set up, let’s send our clicks back to our server so it can relay them on to Elasticsearch. For this article, we’ll include the JQuery library and it’s .ajax method to make our RESTful a little more readable. Add JQuery to the <head> of your HTML file:

<html>  
<head>  
...
   <script src="https://code.jquery.com/jquery-3.2.1.min.js" integrity="sha256-hwg4gsxgFZhOsEEamdOYGBf13FyQuiTwlAQgxVSNgt4=" crossorigin="anonymous"></script>
...

Then, let’s update our snippet so that an ajax call is made every time a click is detected:

(function($) {
   ...
   var clickApp = {
      trackClick: function(evt) {
         var click = {
            "x": evt.pageX,
            "y": evt.pageY,
            "sessionId": generateRandomSessionId(),
            "timestamp": Date.now()
         }
         $.post("/registerClick", click).then(function(response) {                        
            console.log(response);
         });          
      }
   }
   document.addEventListener('click', function(event) { 
      clickApp.trackClick(event);
   });

...
})(jQuery);        

This snippet generates an AJAX POST request and sends the click data directly over to our NodeJS web application. We’re also logging out the response we get back, so we should be able to determine whether our click tracker is working.

Finally, run your application again using node app.js, navigate to http://localhost:8080 in your browser and start clicking around. In the developer console of your browser, you should see something like the following:

created:true  
_id:"AV3NW08fEajW3QBwsZU2"  
_index:"clicks"  
_shards:Object  
_type:"click"  
_version:1  
__proto__:Object  

The created: true is what you’re looking for - this means that your click was created successfully. You can head back to the Elasticsearch browser and click on your index to confirm:

Wrapping Up

Now that you have your website clicks being tracked, you can add the Kibana plugin and start looking at which regions of your website are being clicked on the most often. In our next article, we’ll look at how to use this click data to generate a heat map of clicks and overlay them onto an image of our website.

John O'Connor
John O'Connor is a code junky, educator, and amateur dad that loves letting the smoke out of gadgets, turning caffeine into code, and writing about it all. Love this article? Head over to John O'Connor’s author page to keep reading.

Conquer the Data Layer

Spend your time developing apps, not managing databases.