One of the main limitations on touchscreen interfaces these days is that all you can do is poke at them. We do all kinds of things with our hands, but when it comes to screens, we just poke at them all day. UIs are doing all right, since our phone OSes still mimic mouse-based desktop OSes to some extent, but Microsoft is looking to ways to integrate more natural hand gestures incorporating more than just a “click” derived from a fingertip.
Hrvoje Benko, a researcher at Microsoft, is working on methods of recognizing shapes formed by hands and equating those with spatially-consistent gestures. Putting the side of your hand down like a wall forms a straight line that could be used for a boundary, cropping, or “pushing” objects. Forming an O with your hand could automatically call up the magnification loupe, and so on.
It’s a very natural extension of how you interact with a surface, since really, that’s how you interact with most surfaces. Right now “Rock and Rails” only includes support for three gestures: a fist, which holds things down, a straight hand, which sets a line to which UI elements can move perpendicular, and a “curved rail,” which sets a pivot point. I’m sure you can imagine a few more, but at the same time, limiting the “vocabulary” of a UI is part of making it usable. Nobody wants to remember a dozen different gestures.
Speaking of surfaces, the video shows the tech being demonstrated on a Surface, which uses a different detection method than your average smartphone or tablet — it can detect shapes far more easily. And the new Surface units, as we learned at CES, have thousands of pixel-sized cameras that can even detect patterns and text. So don’t expect this kind of special recognition to come to iPads any time soon, although it would be similarly unwise to underestimate developers who might want to make it happen.
The whole paper is being published later, so unfortunately this video is pretty much all there is right now, but if you head over to Hrvoje’s page, you can keep track of this and his other projects.