84Codes Provides Hosted Messaging Queues from Kafka, RabbitMQ, MQTT
The lightweight nature of Internet of Things (IoT) devices means keeping power and CPU overhead as low as possible. Still the devices need to be able to send and receive messages and possibly communicate with each other. Increasingly projects are designed for computing at the edge to provide more real-time analytics.
That makes message queues — and which ones are best suited for IoT projects — a hot topic. Message queues can help architecture flexible and enable applications written in different languages to communicate, noted Lovisa Johansson, a developer for 84Codes, a Norwegian provider of cloud services with a focus on supporting IoT services.
A message queue also provides a temporary storage for messages when its target program is busy or not connected, and can ensure data will be persisted and processed eventually when traffic spikes occur.
It all depends on the type of communication required. As she points out, a greenhouse temperature monitor that reports every 15 minutes can reduce network bandwidth by simply reporting that data to another service to process rather than being always connected.
However, Forrester predicts an increase in IoT computing and automation at the edge during 2018, such as automation to regulate temperature should it vary from set parameters. Apache Kafka’s capabilities figure to play into such scenarios, driving discussions of Kafka versus protocols such as RabbitMQ and MQTT.
Hosted Options
The hosting provider 84codes seemingly has its bases covered. With the notion that developers should never have to set up or maintain servers, 84codes offers hosted versions of four open source projects:
-
- RabbitMQ, which it calls CloudAMQP
- Apache Kafka, which it calls CloudKarafka
- PostgreSQL, which it calls Elephant SQL
- MQTT, which it calls CloudMQTT
CEO Carl Hörberg founded 84 Codes in 2012 when he saw the need for a hosted version of RabbitMQ when he was trying to use RabbitMQ in combination with Heroku and AppHarbor.
The Swedish vendor markets the services separately so far rather than branding the company as a whole.
“When we started, no one else was offering RabbitMQ as a service and when we started with Kafka, no one else was doing that, so we’ve been really fast with these new services. Before that, [companies] had to have people in-house to set up these services, which can be really daunting, especially with clustering and you have to set up SSL certificates and all that,” Hörberg said. “We like these products, but they can be very hard to set up and use.”
The company touts having more than 20,000 customers, including Datek, Shutterstock, HP, KLM and Docker.
RabbitMQ
RabbitMQ is a distributed publish-subscribe message queue system usually run as a cluster of nodes where queues are spread across the nodes and optionally replicated for fault tolerance and high availability. It uses AMQP natively, which provides cross-language flexibility, and other protocols such as AMQP, MQTT, HTTPS, STOMP, and WebSockets (Web-Stomp) as plug-ins.
AMQP is a secure and reliable protocol, with relatively low overhead, but still more than MQTT. It provides durable and persistent queues, clustering, federation and high availability queues.
Communication in RabbitMQ can be either synchronous or asynchronous as needed. Its highly flexible routing capability is its defining feature.
RabbitMQ guarantees “at least once delivery” but not “exactly once delivery.”
RabbitMQ is a fast message broker written in Erlang. Its rich routing capabilities and ability to offer per message acknowledgments are strong reasons to use it.
Kafka
Kafka, meanwhile, is designed to ingest massive amounts of data in pub-sub messages and streams. A single Kafka instance can handle 100,000 messages per second, versus closer to 20,000 messages per second for RabbitMQ.
It’s designed to be durable, fast and scalable. It acts as a gateway to the data processing pipeline for processing, querying, and analysis using technologies such as Storm, Spark, and Hadoop.
Message-oriented middleware such as RabbitMQ is not specifically designed for firehose-style data streams originating from thousands of publishers, Janakiram MSV explained in an analysis for The New Stack.
With Kafka, data is retained for processing either in real-time or in offline mode, or “hot” or “cold” paths.
The company 84Codes cites its use as a centralized gateway in home automation, which includes automated security, lighting, home appliances, as well as heating, ventilation and heating/air conditioning:
“Every bit of data generated by the system is organized as sets. These sets correspond to various metrics, like light intensity, light usage, ambient temperature, load consumption etc. As these are mostly system-on-chip IoT devices, they use lightweight protocols for communication like Bluetooth LE (BLE), Z-Wave and ZigBee. Using these protocols, the data is collected and passed on to the data-processing pipeline via [this] centralized gateway.”
The fundamental difference between message-oriented middleware and Kafka, Janakiram MSV explained, is that the clients will never receive messages automatically. They have to explicitly ask for a message they are ready to handle.
Kafka scales much better than RabbitMQ — all you need to do is add more partitions. It should also be noted that RabbitMQ clusters do not tolerate network partitions.
MQTT
Message Queue Telemetry Transport (MQTT) is a lightweight protocol designed for connecting power-constrained devices over low-bandwidth networks. Its lightweight packet structure conserves memory and power in high-latency environments, making it the preferred protocol for machine-to-machine communication.
A connected device subscribes to a topic hosted on an MQTT broker. Every time another device or service publishes data to a topic, all the subscribers automatically get the new information. The broker can also “push” similar data to the client.
Its design goals are different from those of Kafka, MSV explained. It cannot handle high-volume ingestion, though it’s better suited to machine data.
Disadvantages include lack of encryption, which would significantly add to its overhead, and that it does not require authentication between devices and servers, issues security teams are rallying around.
The PubNub Data Stream Network last month announced support for MQTT devices.
“MQTT seems to be winning the IoT protocol wars. It’s simple for low-power IoT sensors to chirp data to a topic and then have processes pick up the data for processing,” said Chris Mattieu, CEO of Computes, a company building a mesh supercomputer from excess compute capacity from laptops, phones and even IoT devices.
“RabbitMQ or Kafka would be a better fit for environments that may have different vendors’ IoT devices running different protocols. You could combine all of their messaging queues into a single MQ platform rather than deploying a different MQ platform for each protocol.”
There are some interesting use cases involving combinations:
- Abilisense, which offers an IoT messaging platform for people with hearing difficulties, is using IBM Watson, the serverless OpenWhisk platform, Kafka and MQTT to send alerts to users of sounds such as a smoke alarm or a window broken.
https://www.youtube.com/watch?v=z4qGil4ZjWE
And other options as well:
- Google announced Cloud IoT Core last May, consisting of two parts: a MQTT bridge that ingests the data from devices in the field and delivers them to Google’s pub/sub service. Once the data has been processed, the second new part — Device Manager — sends the data back to the connected device.
- In December, Amazon has released Amazon MQ, a managed message broker service for Apache ActiveMQ.
Feature Image: “Blue vivid image of globe and space tin can” by Patrick Bombaert, licensed under CC BY-SA 2.0.
