Here’s a novel fix for the headache of interacting with all sorts of connected devices: researchers at Carnegie Mellon University have devised a system that lets smartphone users tap their phone against an IoT device in order to have a contextual menu automatically loaded on screen — thereby saving them from having to scramble around looking for the correct app to control each device. Or fiddling with actual physical buttons and trying to navigate less consumer friendly menus.
So, in other words, “no more scrolling through endless pages of apps on your phone to control with your supposed ‘smart home'”, as CMU researcher Chris Harrison puts it.
The system, called Deus EM-Machina (see what they did there?), leverages the fact that electromagnetic noise is emitted from everyday electrical objects to power a device classifier — they’re using a smartphone kitted out with an EM-sensor that can detect what IoT device it’s resting on — enabling contextual functionality to be pushed to the smartphone screen so it can be a dynamic control device.
“There’s no reason this couldn’t be integrated into smartphones in the next few years,” says Harrison, anticipating miniaturization of the EM sensing tech to pave the way for future commercial deployment.
And while researchers at CMU’s Future Interfaces Group have previously shown a similar electromagnetic sensing system running on a wearable device — also for powering contextual awareness of other devices — the use of a smartphone as the control device in this latest research scenario means richer menus can be made available to users, allowing more control functions to be supported.
Harrison confirms the latest EM sensing research is an extension of the earlier wearables project, telling TechCrunch it has improved accuracy, as well as being “much smaller and self contained”. But the big difference is the exploration of a “very different form factor” — and how that can be utilized at the application layer.
“It isn’t that one project/technology is better than the other, it’s more that switching form factors opened up different interactive opportunities,” he says. “In particular, unlike a smartwatch, you perform distinct activities on a smartphone. It’s more of a tool than an accessory.”
In a research paper on the new research, the team writes:
We propose an approach where users simply tap a smartphone to an appliance to discover and rapidly utilize contextual functionality. To achieve this, our prototype smartphone recognizes physical contact with uninstrumented appliances, and summons appliance-specific interfaces. Our user study suggests high accuracy – 98.8% recognition accuracy among 17 appliances. Finally, to underscore the immediate feasibility and utility of our system, we built twelve example applications, including six fully functional end-to-end demonstrations
Examples of the apps the researchers built to demonstrate the sensing system are shown in the below video — including controlling a thermostat; configuring a router; printing a document that’s on screen on the mobile device with a single print button push; sending text from a mobile to a desktop computer; and more.
The researchers created a background Android service running alongside their IoT device classifier that pushes so-called “contextual charms” onto the screen for certain applications — aka small floating buttons that appear at the right edge of an app when the phone touches a supported device, and which can execute commands (such as a “cast charm” to stream video content to a smart TV, or a print button to print what’s on screen).
“We envision that future smart appliance applications would register their device’s EM signature and a set of verbs with the charm system service upon installation, which would enable existing apps to immediately take advantage of appliances and devices in a user’s environment. This is analogous to the current paradigm of applications registering Android “share” handlers to support system-wide sharing of content to e.g., social media,” they write on this.
Harrison adds that the “contextual charms” idea was developed for where there are co-occurrences of two things — such as reading a PDF on your phone, then tapping it on a printer. “This all of a sudden opens up a much richer set of synergistic activities between phones and devices — whereas right now, your phone doesn’t really have any clue what is going on around it or what functionality it should reveal),” he adds.
Discussing limitations of the system in general in their paper, the researchers emphasize the need for IoT devices to have open APIs, noting: “We initially set out to produce full-stack implementations for all of the network-connected devices on our list. However, we were stymied by the lack of public APIs on several of them. Furthermore, even when APIs were available, some were vendor-locked (e.g., the Apple TV casting APIs were only open to Apple devices). In order for the future Internet of Things to have true impact, open APIs are a strong requirement. Until then, our system will be limited by the inability to talk to all smart devices.”
Other limitations include difficulties recognizing multiple instances of the same device (e.g. more than one connected thermostat); and external interference from powerline noise which can confuse the device classifier. The sensing system also cannot work if a device is truly powered off — although the researchers note that low power or sleep modes might still render an IoT object detectable.
The research is being presented this week at the ACM CHI Conference in Denver. CMU is also presenting another interesting bit of interface research at the conference — which involves using a conductive spray paint and an array of electrodes to turn any surface into a touch-sensitive surface.