Monday, June 25, 2012

Are 3D Gestures the Future of UI?

A recent CNET report that a startup, LeapMotion has cracked high accuracy 3D gesture detection, got me thinking of this being another instance of technology out of sync with user behavior. The technology, of course is very cool as can be seen in their demo video:

However, I argue that major advances in UI will come from a close examination of inter-related factors such as physiology, technology and behavior.

The question of arm fatigue

One of the insights shared by Steve Jobs was the idea that touch screens on iMacs made no sense as arm fatigue would set in when constantly manipulating a vertical screen. Surprisingly LeapMotion attempts similar screen manipulation in a virtual vertical plan, with the added disadvantage of no tactile feedback when a gesture has registered. How do you know when a gesture begins and ends? This certainly does not look like an improvement in UI.

Others perhaps have been paying closer attention. Note that the new Microsoft Surface has a touchpad with its keyboard even though the Surface has a full touch screen.

The Kinect model

Coming back to 3D gestures, the Microsoft Kinect has a more natural model of gestures, one that may not be hyper-accurate, but gets the job done. One can tell it is a winner with the rabid excitement in the developer community of hacking the system and doing cool things with it. Not one to lose an opportunity, Microsoft has posted an SDK for it.  Once Kinect is embedded in laptops and mobile phones the possibilities of combining it with AR (augmented reality) get quite interesting.

Another company with a similar concept is SoftKinetic which is experimenting with a range of use cases from flipping a presentation to AR like interactions with large displays.

"No wine before its time"

As is often the case, technology is far ahead of the use cases and shifts in behavior. Just as it took us several years to adapt to simple swiping gestures, it will take at least that much before we settle on the next level of gesture interaction.

Sunday, June 10, 2012

Time and Space

For some years I have been watching QR (Quick Response) codes trying to make the leap to mainstream use, but despite their considerable success I have never been convinced of their usability. With the emergence of novel ways to use low cost, embedded NFC (near field communication) chips, the days of QR seem to be numbered.

So what's wrong with QR codes and why is NFC a disruptive technology?

It is About Images and Gestures

QR codes are really an instance of a gesture that uses an optical image and a series of button presses to connect the real world to a web page. However, with much fewer steps (both physical and mental) one can easily imagine embedded NFC causing the same connection to happen. Unique signatures can even be created without NFC, such as the Bump app which captures vibrations from bumping two phones and turns them into unique signatures. But what distinguishes NFC is the ease with which an embedded chip can change how we interact with the physical word.

The Virtual Supermarket

Tesco did a cool experiment in South Korea about a year ago by creating full-size images of the actual store on the walls of a subway station. You capture the QR codes of the things you want to buy and they are delivered when you get home.

What a great idea! But does it really work? Lets consider two variables:

Space: Although the video announcer claims that the items are life size and placed exactly where they are in the real store, this is not exactly true as one long wall is not the same as a two-dimensional shelf layout. I am not sure how you find something on the subway wall when you have the real store layout memorized.

Time: Since the amount of waiting time for a subway rider is random (while the shopping list is not), what do you do when the train approaches and you are not done?

I am sure the campaign achieved its goal of increasing registered members of the online store and even offline store traffic, but in practical terms one will buy a few things that are on the nearby wall.

However, it is an interesting discovery mechanism and I think wall scanning is much better suited for impulse purchases of digital music and apps.

Barnes & Noble and the definition of a hardback

As the CEO of B&N recently explained, a hardback book will no longer be a book to lovingly hold in your hand and browse the pages while in the store. Rather books will have embedded NFC tags and by simply waving your NFC enabled Nook at a  book sitting on the shelf, a vast array of information will be at your fingertips.

This type of gesture connecting the online and the offline is far more compelling. Looking at the same two variables:

Space: A typical  customer who enters the store is committing their time to shopping in a bricks-and-mortar store. Since shelves are a natural way to browse for books (especial the ones with their front covers in view), why not just re-imagine shelves as simply a spatial organization of books? The odds are that the book you are interested in will be purchased as an eBook anyway!

Time: Here too, having NFC enabled books greatly improves the normal pattern of looking at reviews and making a decision. A bit creepy, but B&N would also know your physical (as in 3D) browsing pattern as you waved your Nook around from shelf to shelf. Of course it goes without saying that a waving gesture is much more satisfying than trying to focus a QR code.

Nintendo and Toys

And finally the news comes from E3 that the WiiU Gamepad will have NFC built in. Undoubtedly NFC embedded toys and figurines will quickly follow the game cards. It is not at all surprising that Nintendo president Iwata-san specifically mentions MIT professor Sherry Turkle and her book Alone Together in the launch video. Nintendo is indeed giving considerable thought to social dynamics and how computers (and by implication game consoles) are not the social tools we thought they were.

Floodgates for Gestures

Although there is considerable user behavior that will need to adapt to bring any of the NFC scenarios to the mainstream, the floodgates may well open when (it is not the question of "if") Apple embeds NFC and creates finely honed and intuitive gestures.

Let the games begin.

UPDATE: Samsung announces Tectiles, NFC tags that can be programmed to trigger a series of settings in a phone. Although expensive and clunky at present, they show how QR Codes will eventually be replaced by NFC technologies that are even easier to use.