Apple Patents Hint At Multi-Part Gestures, Touchable OS X

Designing a user interface for touch isn’t an easy thing to do. At least, it isn’t easy to do well. The great number and variety of gestures possible when four fingers and a thumb hit a touchscreen may well cause development and design paralysis. Yet the gestures we see implemented often seem so simple and intuitive that as soon as we perform them once, we wonder how anyone would have trouble coming up with it.

Apple’s early success with the simple and intuitive gestures on the iPhone has actually worked against it in a way, as adding gestures over-complicates a UI known for accessibility. On the other hand, it has prevented them from providing richer gestures like drawing shapes, creating spontaneous UI items, and so on. But I doubt they ever stopped looking into it. Some newly published patent applications, while questionable as far as patentable ideas go, is chock-full of interesting ideas and promising new UI concepts.

Patent 20110197153 was found by IP sniffers Patently Apple. The patent, filed in February 2011, covers a number of multi-part gestures in which a gesture is done to create or invoke a certain UI element, and then a second action using that element is enabled.

For instance, “digging a hole.” In this case, you’d tap or “scratch” the screen a few times to create a “hole,” which you could then drag files into, either to delete, hide, or what have you. The hole would presumably fill itself in, or a second gesture could be used to do this.

Another example: opening a portal, trap door, or window by drawing a rectangle. This is a similar “drag and drop” concept, but as the gesture and the graphic would be different, it can fulfill a different purpose: creating a folder or archive, or adding an attachment to an email.

A more involved gesture described in a second patent application (20110193788) has the user drawing a circle or blob around a set of files, which would then be enclosed in a graphical bubble. The device could then be physically tipped and the bubbled data “poured” into a second device.

I know what you’re thinking. How can Apple expect to patent gestures that have been used in games, applications, and other UIs (using a mouse, for instance), in similar fashions, for years? I have to imagine that somewhere in the hundred thousand games there are for iOS, there’s one where you have to dig a hole by scratching the screen. If not on iOS, then on the 3DS, a platform saturated with interesting touch-based UIs. And we even had a demo of data being “poured” from one device to another at Disrupt in New York.

To be honest, I really don’t know. The patent really does claim the “digging a hole and then dragging an item over it to be processed” method, as well as the others, and it doesn’t restrict it to anything as specific as managing file representations or app icons. Apple seems to want a patent on digging virtual holes and putting things in them.

There are examples of these UI elements going back a long, long time, and even with the permissive patent system we have in place, I can’t imagine Apple’s approach will be deemed sufficiently different just because it’s done on a touchscreen. Drawing an X on something to delete it? Is it even possible that they could let that fly? I certainly hope not.


Patent worries aside, the gestures are very interesting from a user point of view. I’ve always advocated rich touch interaction in tablets, and have been disappointed by the refusal of tablet OS makers to implement things that are only possible with a touch interface. Games have done a much better job of exploring the possibilities than OS developers. It’s nice to see someone going beyond clicking and dragging. Given Apple’s focus on the trackpad and touchscreen, it was always a good bet that they’d be among the first to integrate rich gestures, even if they aren’t even close to the first to create them.

Interestingly, the patent application uses very OS X-esque images in its tablet illustrations. This doesn’t have to mean anything; the patent writer might have just felt the explanation worked better with a more traditional desktop interface. But there’s more to it than that. Lassoing files, transferring them to nearby devices, hiding and deleting files and windows — these aren’t things you do on Apple tablets. They’re things you do in OS X.

The portal? Sending a window to your other desktop — on the other “side” of the tablet, perhaps, where iOS lives? The “trap door” is a great metaphor for passing things between two distinct areas. Or is it for saving things for later — like the reading list in Safari?

What about X-ing out items? There isn’t room for that in many places on iOS — you delete listed items by swiping, and icons by long-pressing and bringing up their little x boxes. Besides, how would you make an X without swiping over to another screen? A quick flick is interpreted as a directional swipe, not a stroke on a touch canvas. The X gesture itself demands a single-screen desktop, where there is dead space that isn’t interacted with. The same can be said for the “pinch” gesture, which requires negative space on all sides of an item. There is very little such space on iOS — but your OS X desktop is covered with it.

Sharing files from one device to another close by? Sounds familiar — oh right, it’s a marquee feature of Lion. Creating a bubble on an unlocked device is a tacit approval for a file transfer, without going into any menus or selecting a network. Bubble up some files on the other device, the devices pair, and when you pour, it transfers.

These are desktop concepts, not iOS concepts. Or are they both? Lion brought iOS-like interface elements to OS X, and half the world thought we’d have convertible, touchscreen MacBooks right now. No — Apple is bringing OS X to the tablet, though not completely, of course. But I can imagine a second “face” to the iPad, for file management and sharing, with a more familiar desktop metaphor, updated with gestures like these. Apple seems to be imagining something along these lines as well, and while there’s still a good chance this patent is more a “just to be safe” collection of interesting ideas not actually being implemented, you have to admit it’s compelling to think about how they might apply if they were actually executed.