Messaging Reliability and Persistence with the MQTT Protocol

May 13th, 2016 5:59am by Janakiram MSV

Feature image via Pixabay.

MQTT is the most preferred protocol for machine-to-machine (M2M) and Internet of Things applications. Based on the publish/subscribe pattern, it simplifies the connectivity between devices. One of our previous articles introduced the basics of MQTT. In this part, we will cover how the MQTT architecture ensures messages get delivered and what options it offers when things don’t work so smoothly.

MQTT is a lightweight protocol that uses the pub/sub pattern to connect interested parties with each other. It does this by decoupling the sender (publisher) with the receiver (subscriber). The publisher sends a message to a central topic which has multiple subscribers waiting to receive the message. The publishers and subscribers are autonomous, which means that they do not need to know the presence of each other. If you are interested in seeing MQTT in action, go ahead and set up the broker along with the clients on your machine. The last article on MQTT has detailed steps involved in installing and configuring the environment.

Understanding Quality of Service (QoS)

MQTT may be a lightweight protocol, but it is used in some of the complex scenarios that demand reliable delivery of messages. Clients can configure different levels of Quality of Service (QoS) to ensure reliable message delivery. Since it is a part of the specification, MQTT brokers are expected to implement this feature.

Similar to other service-oriented environments, QoS in MQTT is a level of agreement or contract between the broker and clients. When both of them agree on a specific QoS level, it implicitly means that the broker honors the contract.

There are three levels of QoS in MQTT:

QoS 0: At most once delivery.
QoS 1: At least once delivery.
QoS 2: Exactly once delivery.

QoS 0: At most once delivery

This is the default level of quality of service used by the clients and the broker, which doesn’t guarantee the delivery of messages. In MQTT, the message delivery is dependent on the network capabilities and connectivity. The subscriber is expected to acknowledge the receipt, and the publisher is not expected to retry the delivery. The message may or may not arrive at the subscriber at all. QoS 0 is preferred in environments where the network is reliable and losing messages occasionally is acceptable. For example, when multiple temperature sensors are publishing messages every 10 seconds, losing a few will not make much difference to the system.

QoS 1: At least once delivery

This quality of service ensures that the message arrives at the receiver at least once. Subscribers may also receive the same message multiple times. They need to implement the logic that deals with duplicate messages. Subscribers acknowledge the receipt of messages to the broker, which will stop retrying the delivery. QoS 1 is used in scenarios where guaranteed message delivery is important. It’s acceptable to receive the same message many times, but it should be delivered at least once. This QoS level is configured in remote monitoring scenarios that periodically report the state of critical devices or systems. This is the most commonly used QoS level.

QoS 2: Exactly once delivery

QoS 2 delivers the highest quality of service where neither the loss nor duplication of messages is acceptable. It comes with an increased overhead associated with reliable quality of service. While this may be the most reliable QoS, it’s also the slowest. QoS 2 should be used sparingly with specific clients that demand guaranteed delivery of messages. It is used in mission-critical scenarios that cannot afford to lose messages or receiving duplicate messages. Architects and developers should be aware of the performance tradeoff while using QoS 2. For most of the deployments, QoS 1 would suffice.

It’s important to understand that each client subscribing to a topic can request a different level of QoS. The MQTT broker can maintain different QoS levels for each client subscribed to the same topic. Since the clients (publishers and subscribers) are autonomous in MQTT, they can use different QoS levels independent of each other. Publishers may use a completely different QoS level than the subscribers.

If you are using MQTT.fx tool as the client, you can configure QoS for each message that’s sent by a publisher.

MQTT_QoS

Persistence

Even though MQTT is not a full-blown message-oriented-middleware, it supports persistence of messages. This feature becomes essential when dealing with clients running in a constrained environment.

Each time a client connects to the MQTT broker, it starts a new session for subscribing or publishing to topics. If the connection is lost, the process starts all over again with the client establishing a new session. This process may interfere with the performance of a system especially with clients that have low processing power and intermittent connectivity. This is where the persistence feature of MQTT broker becomes helpful.

When a client connects to the MQTT broker, it can set the ‘Clean Session’ flag to false, indicating that the broker should retain session information even after the client is disconnected. The broker then starts persisting the session for that client. Each session contains details such as the topics subscribed or published by the client, QoS 1 and QoS 2 messages that were sent while the client was offline, and QoS 2 messages that are yet to be acknowledged by the client.

With the clean session flag to is set to false, the client starts from where it left instead of starting with a blank slate. But, how does the broker know that it is the same client? Each client that connects to the broker is expected to have a unique identifier. When the client id matches with an available persistent session, the broker immediately reassigns that session with the client. So, it’s important to make sure that every client participating in an MQTT network has a unique identifier.

Persistence also plays an important role in ensuring that QoS 1 and QoS 2 are met by the broker. This feature must be used sparingly as it will interfere with the performance of the broker.

In MQTT.fx, you can set the clean session flag to true by clicking the checkbox in the options.

MQTT_Clean_Session

Retained Messages

Remember that in MQTT, clients are autonomous which don’t acknowledge the presence of each other. This also means that when a subscriber connects to the broker, it has no clue of when it is going to receive a message. There many not be a publisher sending messages to that topic or the publisher may be offline for a long period.

In certain scenarios, it makes sense for the publisher to share the last message with all the new subscribers immediately after they are connected to the broker. For example, when a mobile application (publisher) sends a message with the brightness and color values to multiple connected bulbs (subscriber), it expects that the new bulbs are set to the same values. So, when a new bulb connects to the MQTT broker, it receives the last message that all other bulbs would have received. This ensures that the state of existing subscribers and new subscribers is the same.

Publishers can instruct the broker to deliver the last sent message to all the new subscribers. This is done by setting the retained flag to true. When the broker gets a retained message, it knows that the message has to be pushed to new subscribers. This feature is very useful for bringing the new subscribers up to speed with the same state as other subscribers.

When using MQTT.fx, you can click on the ‘Retained’ button before sending a message from the publisher.

MQTT_Retained

Last Will and Testament (LWT)

It’s common for clients to get abruptly disconnected from the broker. The device may lose power or network connectivity forcing it to disconnect. It many scenarios, it may be helpful for other clients to know that a specific client abruptly ended the session.

MQTT brokers offer a powerful feature called last will and testament (LWT), which enables a client to choose proactively a topic and message that will be published when it gets disconnected. Interested clients can subscribe to the LWT topic, and they will immediately get notified by the broker when the client is disconnected ungracefully.

For example, when a mobile app (publisher) loses connectivity, all the devices that are controlled by the app will receive a message that the app is offline. When the mobile app gains connectivity, it can publish a retained message to the same topic informing the clients that it is online. It’s important to know that LWT topic and message are not special or reserved. The client can publish messages to it like any other topic.

The LWT topic and message provide a mechanism to let all interested clients know about the state of a specific client.

Click on the LWT option in MQTT.fx to configure the topic and message.

MQTT_LWT

AWS IoT has a useful feature called Device Shadow that effectively utilizes the concept of quality of service, retained messages, and LWT to set the state of offline devices.