Messaging, AMQP and RabbitMQ - A Speed Guide

With RabbitMQ being Compose's first messaging product, it's a good time to explain the concepts behind messaging and how they relate to Compose RabbitMQ.

What is messaging?

Messaging, or Enterprise Messaging as it is often referred to, is about moving away from a model where the connections between systems, servers and processes are rigid and hard coded into every application. The approach becomes one of a model where those interconnections which make up the glue of a modern distributed application can be software defined, routable, flexible, reliable and managed outside the scope of any one application component.

This means, practically, that rather than application A sending data to application B, application A produces messages containing that data to a messaging system and application B listens for those messages on the same system and processes them. This may seem like complexity for the sake of complexity, but now consider if you needed application C to take data from A. Without messaging, the chances are you'd have to modify application A and hard code in knowledge of application C and how to deliver data to it. With messaging you do pretty much nothing, just tell application C to listen to the messages from A.

It's not just simplicity of configuration that makes messaging so useful. A good messaging architecture can provide the components you need to ensure reliability. Without messaging, if our application A was producing data so quickly that application B couldn't cope with it, a developer might find relief in writing a data buffer process to take up the slack, sitting between A and B, but that's going to be a new moving part in the architecture.

With messaging, the connection between A and B is typically a queue which - by design - stores messages until they are ready to be processed, or consumed, by the receiving application. If there are multiple instances of application B running, then the load can be distributed between those instances using the messaging system's queues. There's also publish/subscribe mechanisms which can distribute messages to many consumers and various pattern matching techniques for ensuring messages only get delivered to interested parties.

All of this amplifies the "loose coupling" between the applications and services using the messaging system. This makes scaling and rearchitecting simpler, isolates the complexity of communications to a manageable service and reduces the need to redevelop applications when other applications come online.

What is AMQP?

As you might be able to tell, messaging is a great idea and lots of people had similar ideas such that there were a number of enterprise messaging systems. Many of those were closed proprietary specifications, but among them were some open designs. AMQP was one such open design coming out of the finance industry. Rather than define the infrastructure, the Advanced Message Queuing Protocol focussed predominantly on the protocol that AMQP implementations would use to communicate.

There were two landmark points in the development of AMQP. The first was the version 0.9.1 specification (PDF) in 2008 which saw many companies and developers come together in general agreement over not just the protocol but also the language of exchanges, queues and bindings. The other was the 2011 announcement of AMQP 1.0 by the OASIS standards group – this was a significantly different beast to 0.9.1 because of a change of scope to how interoperability was defined. This meant that some vendors stuck with the 0.9.1 specification of AMQP or other intermediate versions. One of the implementations that took its own path after AMQP 0.9.1 was RabbitMQ.

What is RabbitMQ?

RabbitMQ is an Erlang based message broker that natively speaks AMQP 0.9.1. It first appeared in 2007 and has been steadily and regularly maturing since then. It implements all the AMQP 0.9.1 concepts of messages, queues, exchanges, bindings, virtual hosts, producers and consumers within the broker application.

In short, a broker can be divided up into virtual hosts (mainly for administrative simplicity). Each one of these virtual hosts can have a number of queues and exchanges associated with it and bindings can be used to attach an exchange to a queue or queues. Those exchanges can be direct exchanges (passing a message on to a bound queue determined by a routing key), fanout exchanges (sending all messages onto all bound queues) or header exchanges (sending messages onwards based on matching fields in the header).

The developers have added their own set of extensions to the RabbitMQ mix though. Exchanges can be bound to other exchanges to create richer routing, improved mechanisms for tracking message delivery, time-to-live settings for queues and messages and more, with a focus on being an efficient, balanced messaging platform. Add to that the RabbitMQ implementation which gives you control of the performance/reliability ratio, clustering with high availability queues, a web-based Admin UI and a plugin extension system for protocols and other functionality.

What is Compose RabbitMQ?

We've taken RabbitMQ and created a Compose deployment of it which delivers the best of RabbitMQ for most users. Using three nodes, clustered together and operating in a high availability mode as a single logical broker; each node mirrors the cluster's configuration. Access to the cluster is, as with all Compose offerings, managed through a pair of haproxy TCP portals which are whitelisting and SSL supported. With Compose RabbitMQ you can move your interapplication connection infrastructure to the cloud where your applications live without having to add it to the services you directly manage and maintain.

Compose RabbitMQ is currently in beta. If you want to check it out now and don't have a Compose account just sign up for a Compose account and select RabbitMQ as your first service. If you already have a Compose account, go to the Compose Console and select "Create Deployment" and RabbitMQ to get your own RabbitMQ on Compose.