Prototype: A Smartphone Universal Remote for the Internet of Things

Many of us have heard the siren call of so-called “smart homes” outfitted with intelligent appliances like smartphone-controlled thermostats that let you turn on your air conditioning on the way home from work, or app-adjusted LED lighting that can turn on or off according to your daily schedule, helping you to save money and energy. It’s estimated that there will be a mind-boggling 34 billion smart devices of all kinds on the Internet of Things by 2020.

But despite the hype, the current reality is that the user experience in controlling all these devices is a fragmented one. Having to swipe through to find the correct app for the appliance in question can be frustrating and time-consuming. Also, it’s not cost-effective for manufacturers to include full-featured touchscreens on many of these devices — meaning that an app may be the only way to go. One can only imagine how tedious this will become when our homes will be filled with such appliances — scrolling through pages of apps just to find the right one — if you could only remember the name!

So it’s little wonder that we are intrigued by this prototype for a smartphone that acts like a master remote-control of sorts. Built by the Future Interfaces Group at Carnegie Mellon University, it’s capable of sensing which kind of appliances are in close proximity, and will automatically pull up the correct app to allow you to control it, without you having to manually swipe around to find it.

Presented at last month’s Association for Computing Machinery’s human-computer interaction conference, ACM CHI 2017, the EM-Sensing Phone uses a special sensor that can detect the various electromagnetic signals that each smart appliance emits when the user taps the phone on the appliance. The phone’s microchip uses machine learning to distinguish between these signals and matches these electromagnetic signatures to the associated app installed on the phone. Watch it in action:

“Everything acts like it’s own tiny radio station–only it’s broadcasting centimeters,” explained assistant professor of human-computer interaction and paper co-author Chris Harrison to Fast.Co.Design.

For instance, users can tap their smartphone on a thermostat to launch its associated app, which will appear on a new full screen, no matter what current application the phone is running.

If the appliance that has been touched has no app associated with it but has an open API, then the phone can also call up a “contextual charm” — or a small widget that enables the current smartphone app to perform device-specific tasks. One could be reading a PDF on the phone, then tap the phone to the printer to conveniently bring up a “print” button on the screen, or tap the phone on a computer screen with text to call up a “copy and paste” button. The charm application framework has several predefined actions but is expandable.

EM-sensing phone can call up manufacturers’ dedicated apps for controlling lights, television, temperature, door locks, WiFi and more.

Contextual charms used to print a PDF document from a mobile phone (top row); contextual charm used to ‘Send Document’ from phone to desktop computer (bottom row).

The CMU researchers are calling these handy features “on-touch contextual functionality,” and the goal here is to bypass the current “interaction bottleneck” users now face, using a more efficient and streamlined solution that can easily scale up as the number of IoT devices grows.

“Instead of having 50 touch screens in your house for every appliance, you could use the smartphone as this gateway,” Harrison said. “That’s a really powerful notion. It’s not a magic bullet, but it gets us closer to the smartphone knowing the context around me. That’s a much more magical, powerful experience than what we’re seeing right now.”

Final prototype of the EM-sensing phone, with integrated PCB.

Other solutions with a similar aim have included utilizing RFID tags, near-field communication (NFC), laser pointers, fiducial tags, handheld projectors and personal area networks. But the drawback to these approaches is that they require appliances to be retrofitted with additional special tags and sensors.

In contrast, the EM-sensing concept requires much less intervention, allowing users to benefit from a “full-stack” approach that offers near-universal control and cross-device interaction using public IoT APIs.

The phone’s microchip can distinguish between these signals, and matches these electromagnetic signatures to the associated app installed on the phone.

On top of that, the prototype is surprisingly accurate in identifying appliances, as the researchers discovered an impressive 98.8 percent accuracy rate in recognizing the different electromagnetic signatures of over a dozen appliances. The only potential issue would be the phone not being able to distinguish the EM signals of two or more identical appliances — which might be the case in an office environment with more than one of the same printer, for example. It’s something that the researchers are now working to address.

For the concept to work, yet another important requirement would be for manufacturers to use open APIs from the get-go in order to allow for widespread adoption in the future. In any case, the idea of having a seamless and fluidly interactive experience with one’s smart appliances is pretty alluring, and having one single, universal way to control it all would be at least one step forward in a better-managed Internet of Things.

Images: Carnegie Mellon University