Message Oriented Middleware — RabbitMQ — Part 2

This article is originally published at https://www.learncsdesign.com

Messaging that just works — RabbitMQ

In the previous post, we discussed different ways services can communicate with one another. RabbitMQ was discussed along with message concepts and attributes of messages, queues, and exchanges. Let’s continue and discuss Exchanges next.

Exchanges

RabbitMQ uses exchanges as message routers. A producer does not send a message directly to a queue, rather they send it to an exchange. With a binding definition, queues are bound to one or more exchanges. Messages are received by exchanges and routed to zero or more queues that are bound to them. Messages can only be routed to queues that are bound to exchanges. The following four types of exchanges exist:

1. Fanout
2. Direct
3. Topic
4. Headers

A RabbitMQ system has at least one exchange. This predefined exchange is called default exchange and its type is direct. All newly created queues are implicitly bound to this exchange.

RabbitMQ Message Broker

Types of Exchanges

Let’s discuss in more detail the internal of these four exchange types along with default exchange.

1. Fanout Exchange — The simplest exchange type, it sends all the messages it receives to all queues that are bound to it. It makes no attempt to filter the messages, and it disregards routing information. It is like a newspaper deliveryman who has copies of the same newspaper and delivers one copy to each subscriber.

Fanout Exchange

2. Direct Exchange — By using direct exchange, messages are routed to queues according to the routing key defined in the binding definition. To send a message to a queue, the routing key on the message and the routing key of the queue must match.

Direct Exchange

3. Topic Exchange — In topic exchange, the routing key is used to route a message, but it does not require a full match, it checks for patterns instead. It determines whether the routing key pattern of a queue matches the routing key of the message received. A routing key can consist of more than one word separated by dots. Queue routing keys can contain wildcards to match a message’s routing key.

Available wild cards are * and #

Asterisk (*) matches exactly one word like *.png will match flower.png but not red.flower.png

Hash(#) matches zero or more words like #.png will match flower.png, red.flower.png and png

Topic Exchange

4. Headers Exchange — The headers of a message are used to route a message to the bound queues. The routing key value is ignored. There may be multiple headers with different values per message. Upon binding to a queue of this type, every queue specifies which headers a message may include as well as whether it requires any or all of them to exist. In the matching process, x-match is a header key whose value is either all or any, indicating whether to match all or match any.

Header Exchange

5. Default Exchange — New queues in RabbitMQ are implicitly bound to a system exchange called a default exchange with a routing key that is the same as the queue name. The default exchange has no name and is a direct exchange. If the exchange name is left empty when sending a message, it will be handled by the default exchange.

Default Exchange

Exchange to Exchange Binding

Binding an exchange to another exchange is similar to binding a queue to an exchange. The rules for binding and routing messages are the same. By binding an exchange to another exchange, messages from the source exchange are routed to the destination exchange. The destination exchange routes these messages to its bound queues.

Exchange to Exchange Binding

Alternate Exchange

There may be some messages published to exchange that are not suitable for any of the bound queues. The exchange discards the unrouted messages, so they are lost. For any exchange, it is possible to define an alternate exchange in order to collect these messages. Messages that cannot be routed are sent to this alternate exchange. An existing exchange can be set as an alternate exchange. Fanout exchanges that do not filter the data make a good alternative exchange.

Alternate Exchange

Pull vs Push Messages from the Queue

Mainly two approaches are there to fetch messages from a queue

1. Push Method — The consumer service subscribes to the queue and waits for messages. The subscriber is automatically notified if there is already a message in the queue or if a new message arrives. This is the recommended method of getting messages from the queue.

Push Messages

2. Pull Method — The consumer service does not subscribe to the queue, but it constantly monitors the queue for new messages. Consumer service manually pulls messages from the queue if they are available. While the pull method is not recommended, it is the only solution when there is no live connection between a message broker and consumer services.

Pull Messages

Competing Consumers Pattern

Multiple workers are assigned messages/tasks via work queues. Tasks are added to a queue by producers and distributed to multiple worker services.

Competing Consumers Pattern

The following considerations should be taken into account when using this pattern.

1. If the broker application crashes or restarts, the message must not be lost.
2. Each message must be delivered and processed exactly once.
3. In the event that the worker service fails to process a task, it must be readded to the queue and later delivered.
4. The number of retries for failing tasks must be limited, otherwise, the queue may become overfilled with constantly failing tasks.

Publish and Subscribe Pattern

In a publish and subscribe pattern, the same message is delivered to all subscribers. There can be one or more subscribers and each subscriber has a separate queue. Each subscriber receives a copy of each message delivered to each of these queues. This pattern is mostly used for publishing event notifications.

Publish and Subscribe Pattern

Request-Reply Pattern

A request-reply or request-response pattern is used when the publisher of the message, called the requestor, needs to get a response to its message. Request messages usually contain a query or command. Request-response patterns can also be used to implement remote procedure calls (RPCs). In request-reply patterns, there are at least two queues, one for requests and another for replies, this second queue is also called callback queue in RPC scenarios.

Request-Reply Pattern

Priority Queues

Messages may not all have the same urgency level. There may be urgent messages, while others may be processed if there is no other message.

RabbitMQ queues and channels can be configured for message priority.

1. You can set the Max Priority of the queue by setting the x-max-priority argument. Values range from 0 to 255, with 0 meaning the queue does not support priorities. It is recommended to use values between 1 and 10.
2. Publishers can then publish priority messages by using the priority field in basic.properties.
3. To set the Prefetch Count to 1, use channel.BasicQoS(). A worker’s channel will not send a new message until the worker acknowledges the previous one.
4. When subscribing to a queue, set Auto Ack to false. If we don’t configure the channel to not send a new message until the worker acknowledges the last one, every message will be sent immediately. There will be no time to reorder them.

I hope you enjoyed learning RabbitMQ message broker messaging concepts. Microservices rely heavily on message brokers to decouple asynchronous communication between services.

If you like the post, don’t forget to clap. If you’d like to connect, you can find me on LinkedIn.