Modal Title
IoT Edge Computing

Off-The-Shelf Hacker: Build a Hand Held Object Recognizer

Off-The-Shelf Hacker offers the lasest development tips from the land of the Makers. Check basck each Saturday for future editions.
Jun 22nd, 2019 6:00am by
Featued image for: Off-The-Shelf Hacker: Build a Hand Held Object Recognizer

The JeVois machine vision sensor can recognize a wide variety of objects and symbols. My own project, Hedley the Robotic Skull, uses one to track me as I walk around in his field of view. The sensor communicates with an Arduino microcontroller, which moves the pan servo to keep me centered in his visual frame. Actually, he tracks an Aruco symbol attached to my shirt. It’s a pretty interesting effect. He could track my face too.

I’d like to branch out and look at a few new applications of the JeVois technology. For that, I surgically removed the sensor from Hedley’s noggin and temporarily taped it and an Arduino Nano to the big Li-Po battery. The idea is to build a simple “hand-held object recognizer.”

This particular prototype is minimalist. Point the JeVois at something and if it is the specified object, an LED on the Arduino will light up. An awful lot of the hard work is done inside the off-the-shelf JeVois device. This project will grok the nuances of communicating with an Arduino and how to take some action based on the information coming from the JeVois sensor. More advanced functionality will come in the near future. The important thing is to start doing something, somewhere, now and build your knowledge base from there.

On the Arduino Nano Side

The hookup is not too complex. The JeVois USB connectors plug into the big lithium-ion polymer (Li-Po) battery. The Arduino USB cable also plugs into the battery. Finally, the four JeVois hardware serial wires go to the Vin (red), ground (black), TX (yellow) and RX (white) pins on the Arduino. I added a connector to the Arduino so the hardware serial wire pins could be easily detached and plugged back into Hedley’s skull. You also have to unplug the TX and RX lines before uploading new firmware to the Arduino, otherwise, you’ll get a communication error.

Top view of the JeVois sensor, the Arduino Nano, big Li-Po battery and the power supply

Arduino programming is pretty light for all the work it does. We begin by setting up the serial line and a few parameters. Next, we send several commands to the JeVois to configure it’s AI algorithms and analyze the strings coming back to determine if it is the target object. If yes, the LED turns on. Otherwise, it stays off. The code is an example I pulled from the JeVois site with a few added minor tweaks.

Notice the delay of 8,000 milliseconds at the beginning of the setup section? You have to allow enough time for the JeVois to boot up and get ready to accept commands over the hardware serial line. This takes 3 or 4 seconds. This is very important when switching on power to the JeVois, an Arduino and other attached peripherals simultaneously. Here, everything is plugged into the big Li-Po (15000 mAh) battery. I arrived at the 8000 ms (8 second) delay time through trial and error.

I also inserted several LED write statements in the code to show that we are ready to recognize objects. After boot up, it’ll blink twice when ready. As the JeVois recognizes the target object, in this case the screwdriver, the Arduino will turn on the LED. The response is pretty fast, taking roughly 25 ms for each recognized object.

The JeVois can recognize 1,000 different objects, read bar codes and Aruco symbols, locate color blobs and even pick out the dots on a pair of dice. It does this at a wide range of resolutions and speeds.

Setting configuration parameters, using the JeVois interfaces are a bit strange, to the new user. Here’s how it works.

Unraveling JeVois Video Mappings

We interact with the JeVois through the concept of “video mappings.” Different artificial intelligence algorithms, neural network schemes and video filters use different video mappings.

For example, the TensorFlow Easy algorithm, which can recognize 1,000 different objects has four video mappings. Each mapping roughly applies to a different screen resolution. Typically, the higher the resolution, the wider the field of view but also a slower response (recognition) time.

JeVois TensorFlow Easy video mapping page

There are two ways to change the video mappings. If you are viewing a video feed from the JeVois (over the USB cable) with a desktop program like guvcview, just flip to the screen settings page and change the video resolution there. The video mapping will change on the JeVois and the new algorithm will become active. You’ll see the change take effect in the video feed.

guvcview video resolution configuration page

The other way to change mappings is by sending commands over the hardware serial line. The JeVois hardware serial line is connected to the Arduino serial port. We can send data either via a serial terminal or from another device using a string of characters. Hardware serial communication works in both directions, which makes sense if you are working with a “hand-held object recognition device.” In our case, there is no video feed to look at so we’d just send and receive text strings, to interact with the JeVois. A robot car wouldn’t use a video feed either. We interpret the data coming back from the JeVois to take some action, like turning on the LED attached to the Arduino. In a robot car, we might analyze the data from the JeVois to intelligently turn in a certain direction.

The TensorFlow Easy mapping uses the 320 x 308 resolution, as shown in the first part of the mapping line when controlling the JeVois from the guvcview viewer. We use the second half of the mapping line to send the resolution to the JeVois over the hardware serial line. That shows up as the 320 x 240 notation in the Arduino code. I know it is a bit weird, just keep it in mind and use the correct one according to your interface.

It’s possible to send text, back and forth over the USB serial port, but we won’t go into that technique right now.

Flowing object recognition, location, orientation and other data streaming across the hardware serial line from the JeVois is great. Just parse each data line, analyze and interpret the data, then make something happen. This project just turns on an LED when it sees a screwdriver. What could we do with that data streaming into a Raspberry Pi that is connected to a network?

Here’s an example of the data stream from the JeVois when pointing at the screwdriver.

Raw serial data from the JeVois as viewed on the Arduino IDE serial monitor

Notice that it has the text string “TO” at the beginning of each line, then a space followed by the object detected. This is in a short data format. Data can be set to various levels of detail. Each valid line ends with a newline (‘\n’) character.

Next Steps

Some of the things I’d like to dig deeper into include having the recognizer identify humans, cars and a few other objects. I think it would be cool to have a conference badge that can tell when a human is in front of it and post an appropriate picture or message on the LCD screen. Maybe couple the JeVois with the ultrasonic range finder.

Another thing I’d like to try is being able to recognize a small list of objects that can be selected using controls on the Arduino, like switches or a potentiometer. Perhaps, we could use different three-color LED combinations for object identification. A small LCD or LED text display might be fun.

I think it would also be slick to write a Processing script that shows pictures of things that the JeVois detects. Or, maybe map the locations of various objects within the JeVois sensor’s field of view, on a little radar screen. An ultrasonic range finder might tie the objects to the distance measurement.

Feel free to email me ideas for JeVois sensor projects we can try.

Catch Dr. Torq’s Off-The-Shelf Hacker column, each Saturday, only on The New Stack! Contact him directly for consulting, speaking appearances and commissioned projects at or 407-718-3274.

Group Created with Sketch.
TNS owner Insight Partners is an investor in: Shelf, Torq, The New Stack.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.