This evening I finally got around to implementing a simple paint program with the Microsoft Kinect. I’m not using it the way you’re “supposed” to. For one thing, I’m pointing it down at a table from above, rather than forward toward me. For another, I’m ignoring all of Microsoft’s cool body recognition software, and instead using it just as a 3D camera (that is, a camera that gives you both color and distance).
After an evening of hacking, my paint program (well, it’s really more of a finger painting program) is still pretty crude. There’s no color yet, just black and white, like drawing with charcoal. But it does detect pressure, and it has my three favorite operations: add white, add black, and smudge. And of course it’s multitouch. 🙂
You can see from the first image below that the program does a pretty good job of detecting pressure (note the progression from the “1” to the “4”):
One cool thing about using the Kinect is that I’m getting an enormous amount of hand-shape information from that 3D camera. If I change the way I hold my hand, then instead of drawing white, fingers that are touching the surface will draw black:
If I hold my hand yet a third way, fingers that are touching the surface will smudge the drawing:
It’s not yet clear to me what are good hand positions for each of these things. It’s a very rich space. For example, I can use the 3D camera image to figure out the difference between straight and curled fingers, knuckle up or knuckle flat, wrist up or down, or hand leaning to one side. There is such a wealth of possibility to choose from. And I haven’t even gotten started on gestures that use both hands, which will allow a far greater wealth of expressive possibilities.
This painting program is really just a way to get started on “pointing a Kinect down at a table” interactions. What I really want to do this summer is create ways to puppeteer interactive virtual worlds via hand gestures — from music to character animation to lighting and camera cues.
Looks like it’s going to be a fun summer. 🙂
very very cool! I have a 10yr old beta tester if you need one 🙂
From just glancing at the photos I was kind of hoping this would be more about “writing in the dark”. Kinect finger painting is cool too though. Maybe there’ll be some convergence down the road … 🙂
How are you detecting pressure?
When I press my finger down hard onto the table, I see that
1) my finger flattens, so the outermost joint is depressed a few millimeters.
2) my finger becomes a bit discolored, the area around the joint becomes much whiter, and the skin under the perimeter becomes whiter, and everything else becomes more pink.
But seem subtle and hard to detect with a camera that is a few feet away.
Ah, that’s the beauty of using Kinect (or any range camera, actually). The 3D camera is allowing me to see the changes in the shape of your finger joints as you press down harder, and I can then easily map that to pressure. I am not using the camera’s color image at all, only its range image.
At the same time, I can also use tricks like variable rolling of the finger between “touch with finger tip” and “touch with finger pad” to create an additional pseudo-pressure effect, which is easy to do when you have good quality range data.
Also, the Kinect is suspended only about 30 inches above the table (rather than several feet away), which improves effective resolution.
Make the thumb smudge, the pinky does black, all other fingers white.
Ta da.
Yes, that’s one very reasonable possibility. Maybe the best thing to do is to keep the assignment from fingers to function reconfigurable for now, and then let people try it out with different finger assignments. One particular arrangement might quickly emerge as better than the others, once we actually compare them against each other in use.
can you tell me how to detect fingertips and track them
You ever do any more with this?