Off-The-Shelf Hacker: Conversations with Hedley, the Robotic Skull
One of my dreams has always been to carry on a conversation with Hedley, the robotic skull, during shows and small-scale demos. Speaking to a skull and actually having it answer in any meaningful, real-time way is pretty complex. Alexa-styled responses are possible, although it requires a solid connection to the internet. That’s not always available, sitting outdoors at a patio table during a vendor-sponsored local-pub demo gig. Standalone hardware artificial intelligence (AI) machine vision is getting pretty mature. Hedley’s JeVois machine vision sensor is a good example. AI speech, not quite so much.
Like others in the talking robot niche, my solution is to “simulate” a conversation using scripted replies from Hedley. He and I become actors who say our lines at the right time.
We’ve covered Hedley’s Arduino and jaw servo combination along with the recently added audio amplifier and speaker installed in the roof of his mouth.
The last major piece is the software used to control Hedley’s utterances during an exchange. I covered a baseline Processing program back in September but didn’t have any of the audio file list stepping or push-button code developed at the time.
Making Scripts
In this latest version, the program opens a list of pre-recorded WAV sound files and steps through them as Hedley and I talk to each other. Additionally, the sound is analyzed on-the-fly, producing a digital stream produced that feeds the Arduino-jaw servo sub-system. The sound files can be either recorded using a microphone and Audacity or by saving the audio output of a text-to-speech command line tool like eSpeak. You can develop much of code the on the Linux notebook and then move everything over to the Raspberry Pi to finish out the push button code. Alternatively, you might use the “Upload to Pi” function under the Processing version 3.4 IDE tool tab. This lets you develop on the notebook and run it on the Pi over the network. Be sure to put the audio list text and WAV files in the appropriate directory on the Pi.
Here’s the code, that runs on Hedley’s Raspberry Pi.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 |
import processing.sound.*; import processing.serial.*; import processing.io.*; Serial myPort; SoundFile sample; Amplitude rms; float scale=1; float smooth_factor=.2; float sum; int valpos; int sendpos; int i = 0; int playfile = 0; int sendout = 0; String[] lines; int noframes = 0; int playall = 0; int lineno = -1; int a = 0; float dur = 0; int incfile = 0; public void setup() { size(100,100); GPIO.pinMode(16, GPIO.INPUT_PULLUP); myPort = new Serial(this, "/dev/ttyUSB0", 115200); // read control text file for audio files and servo positional data lines = loadStrings("/home/rob/hedleytheskull/roblist.txt"); println("there are " + lines.length + " lines"); for (int i = 0 ; i < lines.length; i++) { println(lines[i]); } } public void draw() { if (GPIO.digitalRead(16) == GPIO.LOW) { fill(204); println("button pushed"); delay(300); if (lineno < lines.length - 1) { ++lineno; println(lines[lineno]); } else { exit(); } } else { fill(155); playfile = -1; } stroke(255); ellipse(width/2, height/2, width*0.75, height*0.75); if (playfile == 0) { sample = new SoundFile(this, lines[lineno]); sample.rate(.45); sample.play(); dur = sample.duration(); rms = new Amplitude(this); rms.input(sample); playfile = 1; sendout = 1; } if (sendout == 1) { sum = sum + (rms.analyze() - sum) * smooth_factor; myPort.write(str(int(map((sum * 700), 30, 85, 8, 3)))); myPort.write('\n'); } if (a < int((dur * 70))) { a++; // print(", "); // println(int(dur * 70)); } else { // println("done"); playfile = 0; a = 0; dur = 0; // ++lineno; } } |
As always, we start out initializing variables, constants and libraries. Next, we count the number of lines in the audio file list to use as an index for each phrase as we step through the list.
Going into the main program (draw), we need to check the button status. Not noticing any, it will just play the first the audio file using the SoundFile function. A button push will increment to the next audio file. You can vary the playback rate and other parameters, to get the sound you want. The Amplitude function gives us a digital record of the audio file that we can then reference in the rms.analyze function, in the next statement block. Rms.analyze returns a digital stream of data that we then send to the Hedley’s Arduino Nano microcontroller over the USB line. The Nano interprets the stream and moves the servo based on the data while it simultaneously tracks through the audio.
The whole process is pretty quick. There is an ever-so-slight lag between the audio coming out of Hedley’s mouth and his synchronized jaw movement. Most audience members probably won’t see it.
Going Further
While the Processing program works well, there is plenty of room for improvement. For example, I’ve thought about eliminating the Arduino Nano controlling the jaw servo and actuate the jaw right from the Raspberry Pi. I don’t think it would take much more than adding a reference to the servo library at the beginning and rewriting the code to send servo angles (from 0 to 30 degrees) out to the servo over a general purpose (GPIO) pin on the Pi. Nixing the Nano would simplify overall power management and maybe even remove the tiny lag between the audio and jaw movement.
Another area I want to explore is near real-time text-to-speech and speech recognition. Wouldn’t it be cool to say a word or a phrase to Hedley and then have him somehow respond with his own appropriate phrase. We could start out simple and have him say certain phrases based on a keyword or two that I say.
I’m not sure if Processing is up to the job of this near real-time stuff. We may have to investigate other programming languages, like C. I’ll keep you posted.
