Off-The-Shelf Hacker: Machine Vision Meets MQTT Messaging
In the past few weeks, we’ve covered recognizing objects with the JeVois sensor and sending the data to text-to-speech Processing scripts. I started thinking that passing the recognized object name and location data to other machines and systems, using the MQTT protocol, opens up a lot of additional automation possibilities. MQTT has become a common message-exchanging standard in the Internet of Things (IoT) and Industrial Internet of Things (IIoT) worlds.
Integrating MQTT messaging into our Processing scripts is fairly straightforward with a few additional lines of code. Posting MQTT messages to a networked broker makes the data readily available for logging purposes, announcements, controlling physical actions and other uses.
Machine Vision Plus MQTT
We’ll need an MQTT broker to be able to send messages back and forth. JakeMakes has a pretty good installation procedure. I also covered MQTT installation in a past wearables story.
I installed the Mosquitto MQTT broker and clients on my Raspberry Pi-based talking skull. It runs automatically at boot-up, so is available whenever it needs to handle messages to or from clients. I also installed the MQTT contributed library through the Processing IDE. The modded text-to-speech Processing script takes the recognized object name, passes it into an MQTT publish function and makes it available through the broker. I also installed the standard MQTT broker/client package using the Synaptic application manager on my ASUS Linux notebook to be able to test the new Processing script on another networked machine. I could examine the messages (recognized name and location data) using the following command line, once software changes were made to the Processing script.
Here’s the command line I used on the Linux notebook to see the MQTT messages from the JeVois sensor.
|
1 |
pi% mosquitto_sub -h 192.168.1.111 -t mqtt5 |
The following procedure adds the MQTT library through the Processing IDE. Start Processing on the Raspberry Pi. Under the “Sketch” tab, click “Import Library,” then “Add Library.” Type “MQTT” into the Contributing Manager search box and then hit “Enter.” Click the “MQTT library for Processing based on the Eclipse Paho project” from Joel Gaehwiler. Push the “Install” button at the lower right-hand corner to install the library and then close the window upon installation completion. I’m using version 3.5.3 of Processing, both on the ASUS Linux notebook and Hedley’s Raspberry Pi. The MQTT Processing library version on both machines was 1.7.1. The Mosquitto broker and clients on the Linux notebook are at version 1.4.15-2ubuntu0.18.04.3.
Modding Last Week’s Code
One of the coolest things about prototyping physical computing projects today is that you can plug hardware modules together, then radically change a gadget’s behavior by changing some code. Point to a newly installed library here, add a few lines of code there and poof…new functionality. Processing has a lot of interesting libraries you might find useful. Cousin, Arduino IDE has similar libraries.
Here’s the Processing code.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 |
import processing.serial.*; import mqtt.*; MQTTClient client; int lf = 10; // Linefeed in ASCII String myString = null; Serial myPort; // The serial port void setup() { // Open the port you are using at the rate you want: myPort = new Serial(this, "/dev/ttyACM0", 115200); // myPort = new Serial(this, "/dev/ttyUSB0", 115200); myPort.write("streamoff\n"); myPort.write("setpar serout All\n"); // myPort.write("setmapping2 YUYV 320 240 15.0 JeVois TensorFlowEasy\n"); // myPort.write("setmapping2 YUYV 640 480 15.0 JeVois DetectionDNN\n"); myPort.write("setmapping2 YUYV 640 480 15.0 JeVois DarknetYOLO\n"); myPort.write("streamon\n"); myPort.write("setpar serstyle Normal\n"); myPort.clear(); client = new MQTTClient(this); client.connect("mqtt://192.168.1.111", "nothing"); // client.connect("mqtt://test.mosquitto.org", "nothing"); } void draw() { if (myPort.available() > 0) { myString = myPort.readStringUntil(lf); if (myString != null) { println(myString); String[] q = splitTokens(myString, " :"); if (q[0].equals("N2") == true) { // exec("espeak", "I see a " + q[1]); client.publish("mqtt5", q[1] + " at x= " + q[2] + " y = " + q[3]); } } delay(3000); myPort.clear(); } } void messageReceived(String topic, byte[] payload) { println("new message: " + topic + " - " + new String(payload)); } |
The MQTT library and client references are initialized at the beginning of the script. The client and connection are then set up with the IP address of the broker. Be sure to include the “mqtt://” part of the address. You can also use an external MQTT broker. Mosquitto has a public test broker that is free to use. Insert “mqtt://test.mosquitto.org” in place of the IP address, as shown in the commented outline.
The actual message going to the broker is handled by the client.publish() function. In our case, “mqtt5” is the topic. The variable q[1] refers to the recognized name. q[2] and q[3] correspond to the x and y coordinates of the recognized object, within Hedley’s field of view. Could you use object location data, in your machine vision/MQTT project?
One odd situation I had was a “RuntimeException: MQTT] Callback not found” error. Not sure about the cause. The “messageReceived()” function appeared in one of the example programs under the MQTT library, so I just included it in the code and it got rid of the error.
Next Steps
There are lots of possible use cases for the JeVois sensor data and MQTT.
Maybe we could connect the JeVois sensor to a NodeMCU board and use MQTT messaging to notify us when someone is at the front door. The Arduino IDE has a number of MQTT libraries. Or, how about using a Processing script on the Linux notebook for remote text-to-speech audio object recognition? Or maybe a networked lamp, using a NodeMCU board would turn on when the front door sensor detects a car pulling up in the driveway? Hey, how about turning on a sprinkler valve, using MQTT, to discourage stray animals in the yard? Making data available on a network is simple and fast with MQTT.
What else could we do with this tech?
Catch Dr. Torq’s Off-The-Shelf Hacker column, each Saturday, only on The New Stack! Contact him directly for consulting, speaking appearances and commissioned projects at doc@drtorq.com or 407-718-3274.
