This evening I finally got around to implementing a simple paint program with the Microsoft Kinect. I’m not using it the way you’re “supposed” to. For one thing, I’m pointing it down at a table from above, rather than forward toward me. For another, I’m ignoring all of Microsoft’s cool body recognition software, and instead using it just as a 3D camera (that is, a camera that gives you both color and distance).
After an evening of hacking, my paint program (well, it’s really more of a finger painting program) is still pretty crude. There’s no color yet, just black and white, like drawing with charcoal. But it does detect pressure, and it has my three favorite operations: add white, add black, and smudge. And of course it’s multitouch. 🙂
You can see from the first image below that the program does a pretty good job of detecting pressure (note the progression from the “1” to the “4”):
One cool thing about using the Kinect is that I’m getting an enormous amount of hand-shape information from that 3D camera. If I change the way I hold my hand, then instead of drawing white, fingers that are touching the surface will draw black:
If I hold my hand yet a third way, fingers that are touching the surface will smudge the drawing:
It’s not yet clear to me what are good hand positions for each of these things. It’s a very rich space. For example, I can use the 3D camera image to figure out the difference between straight and curled fingers, knuckle up or knuckle flat, wrist up or down, or hand leaning to one side. There is such a wealth of possibility to choose from. And I haven’t even gotten started on gestures that use both hands, which will allow a far greater wealth of expressive possibilities.
This painting program is really just a way to get started on “pointing a Kinect down at a table” interactions. What I really want to do this summer is create ways to puppeteer interactive virtual worlds via hand gestures — from music to character animation to lighting and camera cues.
Looks like it’s going to be a fun summer. 🙂