Configuring RabbitMQ Exchanges, Queues and Bindings: Part 1

Just starting with RabbitMQ and want to understand the configuration options better? In this article, we'll dive into configuring exchanges, queues and bindings, discussing different setup choices and reviewing a variety of configuration settings.

RabbitMQ is a robust and popular message broker, a reliable solution for routing messages asynchronously between systems and apps. Configuring the core modules of RabbitMQ — exchanges, queues and bindings — is extremely easy, but understanding how they fit together and when to use one setting over another can seem somewhat esoteric. To help you feel more confident in your setup, we'll use a hypothetical business called Content Corp to expose the key configurations in RabbitMQ and look at how and when to apply various settings.

Contextual Use Case: Customer Notification App

Our hypothetical business Content Corp runs a content website that publishes new content frequently throughout the day. They are developing an app that will notify their customers of new content the customers will be interested in whenever they're online and something new gets published. When customers are not logged in, they want to route messages either to a database to be retrieved later when the customer logs in or, if the customer has indicated they want to receive email notifications, they will push those messages to an email service.

On the website their customers will be able to set content preferences to be notified when content gets published in a topic area they specify. Content Corp also has in mind a variety of other uses for the notification feature, including notifying customers when their privacy policy has been updated, when terms of service have been altered, when site maintenance is scheduled or even when the customer's credit card has expired.

RabbitMQ is the perfect component for Content Corp's app to broker the messages to be delivered. Let's look at how to configure RabbitMQ to handle these different types of messages.

Preliminary Setup

Using RabbitMQ on Compose is easy. In our guide to Getting Started with RabbitMQ we introduce how to create a deployment, how to connect to your cluster, how to use the Compose administrative console to monitor and scale your deployment, how to create users with specified permissions for different roles, and how to access the RabbitMQ console.

For our purposes in configuring RabbitMQ for Content Corp, we'll create a user for all of the permission levels: read, write, and configure. In our example, we're allowing access to all of the servers in our vhost by indicating ".*", but if you have more than one vhost, you may want to further limit user access to only a specific vhost by using a regular expression for the vhost.

To create our user, we'll use the Compose administrative console to create a user called configureRabbit and grant full permissions for "configure", "write", and "read":

Now that we have a user with configuration permissions, we can access the RabbitMQ management console by going to the URL provided in the deployment overview. Note that all created users, regardless of permissions, can view the RabbitMQ console, but only those with configuration permissions can actually make changes.

Message Basics

Let's start with the most important part — the messages — and get some basics under our belts before we move on to the configurations.

Messages have two key pieces: the attributes and the payload. The attributes are metadata about the message. Attributes can include things like the content type and encoding, the routing key that is used by the exchange and the binding to route the message to the right queue, whether the message will be persistent, and whether it has a priority level, among other information. The payload is the message itself, which usually consists of keys and pointers to data to keep the message lightweight, that is sent from one system or app to another. RabbitMQ is oblivious to the actual message, but it does use the message attributes for routing and processing.

Messages are constructed by an external application called a message "producer". The producer figures out what messages need to be sent, how the attributes should be configured for routing and processing by RabbitMQ, which exchanges the messages should start from when they enter RabbitMQ, and what the actual payload is that is being sent. Once a message is constructed by the producer and pushed to the appropriate exchanges in RabbitMQ, then RabbitMQ reads the message attributes to figure out what happens to the message next.

Messages may go through more than one RabbitMQ exchange, depending on the message attributes and how RabbitMQ is configured, before landing in the right queue. When a message lands in the right queue, another external application called a message "consumer" picks up the message and determines what to do with it. The consumer can subscribe to one or more queues, listening to be alerted whenever a message shows up, or it can poll the queue at regular intervals to see which messages were delivered since the last time it made the request. As the consumer cycles through each message, it can send acknowledgements back to RabbitMQ that the message has been received (known as "acks") or rejected ("nacks" for negative acknowledgements). Additionally, depending on how it was developed for the production environment, the consumer application could deconstruct the messages and process the payloads or it could just push the messages on to other apps for deconstruction and processing elsewhere down the line.

If you'd like to get more of an overview of the concepts and some of the history behind messaging, take a look at our speed guide to messaging, AMQP and RabbitMQ.

Now that we have a basic understanding about the messages themselves and how they are produced and consumed, let's configure the exchanges that Content Corp will need for their notification app.

Exchanges

Let's login to the RabbitMQ console with the user we created that has "configure" permissions. On the "Exchanges" tab, we see several exchanges have already been created as part of the deployment. These are available for use, but we'll want to create new exchanges for Content Corp, according to their use case.

There are four kinds of exchanges:

In addition, there is the default exchange. The default exchange is special because it will route messages directly to queues using the queue name as the routing key. All queues are automatically bound to it. If a message does not have an exchange name set, it will go through the default exchange.

For Content Corp's notification app, we'll use Direct, Fanout and Topic exchanges. Header exchanges could alternatively be used if the messages are constructed with that in mind, but we won't be focusing on that type of exchange for their use case.

All of our exchanges are going to be durable because we expect to be continuously using them (the Content Corp website should be always up and have some customers logged in at all times). Also, that way, when our queues and bindings are created (which you'll learn more about in Part 2 of this series), they will always have the exchanges available to them. Each exchange is actually just metadata in the in-memory Mnesia database internal to RabbitMQ so they use very little system resources.

Now, a different use case than Content Corp's may require transient instead of durable exchanges, where exchanges are only created when they are needed. Additionally, an auto-delete setting may be used for exchanges so that they will be automatically deleted when no queues or other exchanges remain connected to them. Neither of these settings is favorable to Content Corp's use case so we'll leave our exchanges as "Durable" and auto-delete at "No".

Also, note that the setting called "Internal" is for exchanges that do not have an external client connecting to them. In the case of the three main exchanges we're configuring, the message producer will need to connect to them so we'll leave the internal setting at "No".

Direct exchange

A Direct exchange will be used to route messages intended for a specific customer directly to that customer. For example, if a customer's credit card has expired, we'll want to notify only that customer. Our routing key will use the customer's website user ID so the message goes directly to the queue for that customer. We'll also set an alternate exchange argument (alternate-exchange) to "customers_alt.fanout". This will be for messages aimed for queues that don't currently exist because the customer is not currently online (we'll go over the alternate exchange in detail in Part 2, but for now we'll go ahead and use this setting to get ahead of the game).

Because Content Corp wants to send specific messages to specific customers, messages that will be unique to each customer, we'll configure a Direct exchange to handle those messages, like this:

Note that RabbitMQ provides many different client options for configuring and interacting with the system. Besides the management console UI, we can also use the command line, the REST API, or a variety of client libraries in different languages.

Here's an example of using the rabbitmqadmin CLI tool to setup the Direct exchange using the command line:

$ rabbitmqadmin declare exchange name=customers.direct type=direct durable=true auto-delete=false internal=false \
     'arguments={"alternate-exchange":"customers_alt.fanout"}'

Here's an example using the REST API:

$ curl -i -u configureRabbit:myPassword -H "content-type:application/json" \
    -XPUT -d'{"name":"customers.direct","type":"direct","durable":true,"auto_delete":false,"internal":false,"arguments":{"alternate-exchange":"customers_alt.fanout"}}' \
    https://aws-us-east-1-portal8.dblayer.com:10235/api/exchanges/heroic-rabbitmq-62/customers.direct

Or, we could use one of the many client libraries available. In our recent article about RabbitMQ secure connections, we demonstrate how to connect securely and declare exchanges in Java, Ruby, Python and Go to get you started.

Have a look at this conceptual diagram that shows how the Direct exchange will work when we're all done. It sends a specific message to a specific customer queue, in this case the customer whose website user ID is an exact match to our routing key 440099:

Using the same approach, we'll configure our remaining exchanges.

Fanout exchange

We'll use a Fanout exchange for messages intended for all customers, such as a notification for a privacy policy update, an alteration to the website terms of service, or scheduled maintenance on the website.

Because Content Corp will have some messages that are not unique that need to go to all customers, let's configure a Fanout exchange to handle them:

Our Fanout exchange will work like this:

Topic exchange

A Topic exchange will be used to route messages that might apply to some customers but not others. For Content Corp's app, this is going to be the primary exchange type for routing messages based on customer preferences for types of content they're interested in. The message routing key will be compared to words in the binding specification to pattern match for the right customer queues (we'll talk more about bindings in Part 2).

Here we'll configure a Topic exchange to handle messages intended for some customers but not others based on their topic:

Content Corp's topic routing keys will be hierarchical in nature, following the format <topic>.<sub-topic>. By pattern matching on the routing key, the Topic exchange can route the message to the customer queues interested in that subject matter. Here's how the Topic exchange will work for customers interested in databases:

Now we've got Content Corp's exchanges set up, but as we can see from our conceptual diagrams, exchanges need a place to route messages to (the queues) and to know how to route the messages to the right places (the bindings between the exchanges and the queues that hold the instructions for matching on the routing keys).

In Part 2 of this series, we'll take a deeper look at queues and bindings and we'll also look at what happens to messages that don't have a place to go. At the end, we'll put the whole thing together so you can see how it all fits.