Configuring RabbitMQ Exchanges, Queues and Bindings: Part 2

We continue our look at configuring RabbitMQ in this article as we move on to queues and bindings. We'll also look at what happens to messages that don't have anywhere to go. In Part 1 we learned about message basics, including producers and consumers, and we configured a Direct exchange, a Fanout exchange and a Topic exchange for our customer notification app use case.

The Customer Notification App

The contextual premise for our configurations is a customer notification app being developed by Content Corp. While their customers are online, they'll receive a variety of notifications, primarily about newly published content they'll be interested in. When they're offline, messages intended for them will be sent either to a database to be queried and displayed when they login again or to an email service if they've elected to receive individual notification emails.

Message Consumers

In order to provide notifications to their customers, Content Corp will need to develop a message consumer component in their app. Message consumers, which we reviewed in Part 1, can either subscribe to a queue to pick up messages as they arrive or they can poll the queue at regular intervals to see what messages have arrived since the last time. Content Corp has determined they're going to subscribe to each customer queue to pick up messages as soon as they come in. Also, their message consumer will be responsible for creating the customer queue and the bindings programmatically when the customer logs in. There will be one consumer connection for each customer queue.

Let's hop right in!

Queues

Content Corp's app will require a temporary queue for each of their customers while they're logged in. You might be concerned at how many concurrent queues that may be, but not to worry. RabbitMQ is extremely good at using resources efficiently and was developed with an awareness around this concern. And, it's not only queues, but exchanges and bindings as well. All are intentionally very lightweight. Queues, for example, will hibernate when they are idle for more than 10 seconds and also will automatically write messages to disk if memory usage is high. Note, too, that if a message is duplicated in more than one queue, which might happen via our Fanout and Topic exchange routing, it is only held in memory once. Also, besides the fact that binaries are shared across processes by virtue of Erlang (the development language for RabbitMQ), your Compose deployment runs RabbitMQ on a 3 node cluster further distributing the resource needs and can be scaled as your usage grows. As always, though, use common sense in your configurations. If what you're doing feels like it's going to overly tax the server, then it probably will. You should test your configuration plan while monitoring server connections, processes and resource allocation, then make changes accordingly.

You may want to look at the RabbitMQ blog post about understanding memory use as well as a very useful article about topology considerations for RabbitMQ performance and scaling by Spring.io to get familiar with how RabbitMQ allocates resources. If you find yourself wanting more on this topic (and others), RabbitMQ has done a great job collecting how-to resources on their site for developers.

Now that we have a better understanding of resource allocation in RabbitMQ, let's look at how configuring a queue works. We'll demonstrate this by going through the RabbitMQ management console, which helps us illustrate the different configuration options easily.

With the user created in Part 1 that has "configure" permissions, login to the RabbitMQ management console. On the "Queues" tab click "Add a new queue" and populate the queue details:

In this example, we're creating a queue for a specific customer (with a website user ID of 440099). Remember, each customer will have their own queue when they're logged in. The queue will not be durable since we only want it to be available as long as the customer is logged into the website. Once the user is logged off (voluntarily or after the website idleness limit is reached), the message consumer will stop monitoring the queue and the queue will auto-delete. Furthermore, in our example we're going to set an auto-expiration limit for the queue of 24 hours to ensure that no one queue remains alive beyond a reasonable amount of time for the customer to be actively online.

Note that, because we're using the RabbitMQ UI, we're assigning the queue manually to one of the member nodes in our server cluster; however, Content Corp's message consumer will be creating queues programmatically. They could use a round-robin approach in assigning new queues to member nodes in order to spread the love or they could poll the nodes to see how many queues are already assigned and choose the one with the least queues.

When creating a queue, additional arguments may apply. In this case, we are setting a time-to-live of 24 hours (in milliseconds) for messages in the queue, an auto-expire for the queue also of 24 hours (in milliseconds) which we mentioned above, and a dead letter exchange (DLX) where dead messages will go (we'll review that in more detail in the "Message Exceptions" section below, but for now we'll just set it to "deadletters.fanout").

The full list of available arguments includes a variety of options:

Note that rather than provide arguments for each queue as it is created, we could instead set one or more queue policies. All new queues and existing queues are subject to the policies that are enabled for them. Examples of policies we might consider are setting a dead letter exchange or time-to-live for all queues. Here's a quick example of setting a dead letter policy on all queues in our cluster using the rabbitmqctl broker command line tool:

rabbitmqctl set_policy DLX ".*" '{"dead-letter-exchange":"deadletters.fanout"}' --apply-to queues  

Now, just as we demonstrated previously with exchanges, there are many clients that can be used to create queues. The management-cli plugin rabbitmqadmin can be used to declare our customer queue using the command line:

$ rabbitmqadmin declare queue name=uid-440099 durable=false auto-delete=true \
     'arguments={"x-message-ttl":86400000,"x-expires":86400000,"x-dead-letter-exchange":"deadletters.fanout"}' \

Or, we can use the HTTP REST API:

$ curl -i -u configureRabbit:myPassword -H "content-type:application/json" \
    -XPUT -d'{"name":"uid-440099","durable":false,"auto_delete":true,"arguments":{"x-message-ttl":86400000,"x-expires":86400000,"x-dead-letter-exchange":"deadletters.fanout"},"node":"10.43.128.3"}' \
    https://aws-us-east-1-portal8.dblayer.com:10235/api/queues/heroic-rabbitmq-62/uid-440099

Language client libraries can also be used for declaring queues, which is the likely approach Content Corp would take for their message consumer. Check out the clients and developer tools documentation provided by RabbitMQ and have a look at some RabbitMQ connection examples we've developed in Java, Ruby, Python and Go to get you started.

So, now we have exchanges and we have queues. Let's bind them together.

Bindings

Bindings contain the instructions for the exchanges to know which queues to send messages to.

In the RabbitMQ management console we can configure bindings either from the exchange detail view or from the queue detail view.

For our example, we'll configure a binding from the queue detail view. Remember that since our queues are temporary, the same will be true for our bindings. For the Content Corp use case, the bindings, just like the queues, will need to be created programmatically by the consumer component of the app when the customer logs in.

In Part 1 we configured our primary exchanges: a Direct exchange, a Fanout exchange and a Topic exchange. Our Direct exchange will be responsible for routing specific messages to specific customers based on the routing key. In our case the routing key for unique messages will hold the customer's website user ID to ensure that only that customer receives that message. With that in mind then, let's configure our binding from the customers.direct exchange to the customer queue uid-440099:

Again, we can use a variety of clients to create bindings. Here's an example of using the management-cli plugin rabbitmqadmin to bind our customer queue (the destination) with our Direct exchange (the source) on the command line:

$ rabbitmqadmin declare binding source="customers.direct" destination_type="queue" destination="uid-440099" routing_key="440099"

Or, using the REST API:

$ curl -i -u configureRabbit:myPassword -H "content-type:application/json" \
    -XPOST -d'{"routing_key":"440099"}' \
    https://aws-us-east-1-portal8.dblayer.com:10235/api/bindings/heroic-rabbitmq-62/e/customers.direct/q/uid-440099

Or, we could use the client library of our choice.

The Fanout exchange we previously created will be used to route messages relevant to all customers to each of the queues. For example, these could be messages letting customers know about a scheduled website maintenance or a corporate event. Here is how we'll configure our binding for the customers.fanout exchange (note that it has no routing key because Fanout exchanges ignore routing keys):

Along with the Direct and Fanout exchanges, we also previously configured a Topic exchange. The Topic exchange will be used to route messages based on pattern matching to a topic-oriented routing key. In our example, we have a known topic hierarchy that follows this pattern in the routing key: <topic>.<sub-topic>. Because our customer is interested in the "databases" topic and all sub-topics in that area, we can use the "#" wildcard sign for zero or more words (since we don't know how many words the sub-topics might include). Our binding configuration for the customers.topic exchange to our customer queue will look like this:

Besides being able to use a "#" for matching zero or more words, we can also use a "*" for a single word replacement. If we had another customer who was interested in the "sql" sub-topic, which we knew fell under the "programming" or "databases" topics, we could indicate that like "*.sql". In this example, the "*" indicates the single word topic preceding the ".sql" sub-topic. So, if messages came through with routing keys specifying either "databases.sql" or "programming.sql", those messages would go to customers with a binding routing key of "*.sql".

When a customer sets or updates their content preferences through the Content Corp website, the bindings associated with that customer can then be programmatically created or deleted accordingly by the message consumer.

Message Exceptions

Now, we've foreshadowed a bit in our exchange and queue configurations, but you may be wondering what happens to messages that have nowhere to go. For Content Corp, we'll be using two different message exception handling methods: an alternate exchange that will handle messages aimed for a queue that is not currently available and a dead letter exchange to handle messages that die in their queues. Both of these exchanges will be durable and not auto-delete because we want message exceptions to be handled continuously.

Alternate exchange

If the customer is not online when the message is generated by the producer component of the app and pushed to our exchanges (we covered message producers in Part 1), then we'll use an alternate exchange to route those messages to an alternate queue. A message consumer that subscribes to that queue, and is listening all the time, can pick them up to add them to a database or send them to an email service (according to customer preference). That way offline messages have a place to go and don't loiter around in the broker. We'll set this as a Fanout exchange since we don't need to be concerned with any of the routing keys; all messages will be going to the one alternate queue connected to our alternate exchange. Additionally, we'll set this as an internal exchange since no clients will be connecting directly to it, only other exchanges. Let's set it up:

We'll need a queue and binding for our alternate exchange. Here's how we're setting ours:

Our alternate queue:

Our alternate binding:

The way Content Corp's app could work is that messages consumed and stored in a database can get queried and displayed to the customer the next time the customer logs in or regularly queried (like once a week, for example) and then emailed to the customer in aggregate as a "what you missed while you were offline" email. If the customer prefers to have notifications sent individually by email each time one is generated while they are offline, then the message consumer can post them through to an email service each time instead of storing them in a database. These are only a couple options that Content Corp is considering for their app.

This diagram shows how our alternate exchange will work. Messages from any of our exchanges that don't have a queue to go to will be routed instead to the alternate exchange, which in turn, will route them to the alternate queue for special handling:

Dead letter exchange (DLX)

Now, let's also set up a dead letter exchange to handle messages that just die. Messages die for three possible reasons:

We are setting our dead letter exchange as a Fanout exchange so that we don't need to mess with any routing keys, though we could use the dead letter routing key (described in more detail in the "Queues" section above) if we chose. For our scenario, all dead letters go to the same place so we'll just stick with the Fanout exchange and a single queue connected. Just like for the alternate exchange queue, there will be a consumer listening to the dead letter queue all the time and then pushing messages to a log for later review or possibly sending them to an email service so that developers can be alerted. Let's configure our dead letter exchange, queue and binding:

Our dead letter exchange:

Our dead letter queue:

Our dead letter binding:

This diagram shows how the dead letter exchange collects messages from all of our customer queues and routes them to the dead letter queue which is continually monitored by a message consumer:

Putting it all together

Now that we've got all our configurations ready, our setup will look like this for Content Corp:

Configuring exchanges, queues and bindings should be less mysterious now, though we recommend that you read the official RabbitMQ documentation for more information. We also provide RabbitMQ help docs on Compose and we've written a RabbitMQ speed guide to familiarize you with the underlying concepts.

And, if you find yourself in need of a database while building your app, check out the many database options from Compose to complement your RabbitMQ setup.

Go forth and message!