Rich interfaces

We are just about to enter, for the first time, an age of rich human/computer interfaces. It is true that advanced techniques, beyond the impoverished model of Windows/Icons/Mouse/Pointer, have existed for years. What has not been true is that millions of ordinary folks at home have had access to them.

Video cameras haven’t done the trick. If you try to use a computer figure out what your fingers are doing by pointing a video camera down at your hands, you run into all kinds of problems. Skin color variations, lighting changes and depth ambiguity all work against you. A technique that works just fine in the morning might fail miserably once the afternoon sun starts streaming through your window.

But now that the Kinect provides a cheap 3D camera available to millions (and soon to get both cheaper and better, especially when competitors start jumping in), it’s easy to write software that tracks fingers accurately and reliably.

The big question is now not how, but what. Will we end up using the full richness of our hands and fingers when we use computers, or will we (collectively) cop-out, and end up with some boring variant on the pinch gesture?

My hope is that we will take seriously the language building powers of small children. There is strong linguistic evidence that natural language is actually created by small children (children younger than 6 y.o.), rather than adults. Among evidence to support this, is work by Ann Senghas on the creation by small children of Nicaraguan Sign Language in a very short amount of time.

Also, there has been work by Derek Bickerton and others on how small children spontaneously created Hawaiian Creole in only a few years, a language as complex as any other natural language.

The possibility of tapping into this capability of small children makes Kinect and similar technologies particularly exciting as potential platforms for human/computer interfaces far richer than any we have ever seen.

Leave a Reply

Your email address will not be published. Required fields are marked *