Ken’s Blog

Vi Hart came around the lab today and tried out the Kinect-based finger painting program I made a few days ago.

She started drawing shapes, and I soon joined in (it’s not just multitouch, it’s also multi-user). Within about a minute the two of us had created the following evocative image:

Several people told us it looks a lot like sand painting. It’s hard to disagree.

Several people commented that it would be good to have a fast thumb-typing for SmartPhones that can work with a single hand, in case you don’t have two hands free.

The following variant seems to do the trick: The 14 fast characters are the same, but the other characters are “typed” not by chording, but rather by sliding off the original virtual key in one of the principal four directions.

Click on the image below to link to a java applet that shows the idea:

Using this approach, you can pick up a lot of extra speed by using both thumbs, but everything still works (albeit more slowly) with a single thumb.

If you start from scratch, aiming to create the most efficient possible thumb-chording that does not require looking at your SmartPhone, you can do a lot better than Braille. For example, let’s start with the fact that over 75% of all characters typed are one of SPACE,E,T,A,O,N,R,I,S,H,D,L,C,U (roughly in descending order of frequency).

We can use the entire SmartPhone screen to lay out a 5×3 thumb keyboard with the following arrangement:

E	U	C	L	T
→	O	H	S	→
A	I	D	R	N

This keyboard has already taken care of more than 3/4 of all the characters you’ll ever type (we include two ways to type SPACE because it’s by far the most frequently typed character).

We can then get an additional 75 additional characters by “chording” with both thumbs at once, with the left thumb on one of the three left-most columns, and the right thumb on one of the three right-most columns. 75 characters is far more than needed to type all the letters, digits, punctuation and special characters on a keyboard.

I think this keyboard is going to be very fast to type on. Since all the other characters after those first 14 are used much less frequently, having them as two-thumb chords won’t slow you down very much.

Not only does this arrangement allow non-sighted people to type quickly, but it can also be used by anyone in a situation where you need to type quickly into your cellphone without looking at it (eg: during a meeting).

As is usual for keyboards for the non-sighted, there would be a tutorial mode that allows you to drag your thumbs over the keyboard to sound out each character.

About two years ago I wrote a series of posts here about using Braille on SmartPhones. Then just today I had a conversation with a visually impaired colleague, and discovered that there are still no fast text entry methods on SmartPhones if you are blind. Yes, there are text entry methods, but no fast ones. So I proposed the following approach, which I am going to try to implement on my handy dandy new Android phone.

Any Braille character consists of some combination of three dots on a left column, and some combination of three dots on a right column. For example, below are the 26 letters of the alphabet. Other six-dot combinations represent punctuation, numbers, and special characters:

I propose a thumb-chording method where your left thumb touches an on-screen button showing the dot combination for the left column, while your right thumb does the same for the right column. For characters where only one column has any dots, just use one thumb.

You can arrange these on-screen buttons as follows (in the image below, red represents buttons for the left thumb, blue for the right thumb):

This paragraph is for non-sighted people reading this: There are three rows, with five buttons in each row. Top row: L100 L010 L001 R111 R101. Middle row: L110 L011 unused R011 R110. Bottom row: L101 L111 R001 R010 R100.

Because there are so few buttons, each button is easy to find with your thumb, once you practice a bit (and that unused middle space could be used for meta-commands). In the actual app, you’ll be able to learn the button positions by dragging a thumb around, and the phone will sound out which button that thumb is over (and then you tap with your thumbs to actually type characters).

This evening I finally got around to implementing a simple paint program with the Microsoft Kinect. I’m not using it the way you’re “supposed” to. For one thing, I’m pointing it down at a table from above, rather than forward toward me. For another, I’m ignoring all of Microsoft’s cool body recognition software, and instead using it just as a 3D camera (that is, a camera that gives you both color and distance).

After an evening of hacking, my paint program (well, it’s really more of a finger painting program) is still pretty crude. There’s no color yet, just black and white, like drawing with charcoal. But it does detect pressure, and it has my three favorite operations: add white, add black, and smudge. And of course it’s multitouch. 🙂

You can see from the first image below that the program does a pretty good job of detecting pressure (note the progression from the “1” to the “4”):

One cool thing about using the Kinect is that I’m getting an enormous amount of hand-shape information from that 3D camera. If I change the way I hold my hand, then instead of drawing white, fingers that are touching the surface will draw black:

If I hold my hand yet a third way, fingers that are touching the surface will smudge the drawing:

It’s not yet clear to me what are good hand positions for each of these things. It’s a very rich space. For example, I can use the 3D camera image to figure out the difference between straight and curled fingers, knuckle up or knuckle flat, wrist up or down, or hand leaning to one side. There is such a wealth of possibility to choose from. And I haven’t even gotten started on gestures that use both hands, which will allow a far greater wealth of expressive possibilities.

This painting program is really just a way to get started on “pointing a Kinect down at a table” interactions. What I really want to do this summer is create ways to puppeteer interactive virtual worlds via hand gestures — from music to character animation to lighting and camera cues.

Looks like it’s going to be a fun summer. 🙂

The world didn’t come to an end today after all, thereby sparing most of us a whole lot of inconvenient fire and brimstone.

In celebration of not having to go to hell, it seems like a good day to revisit an earlier post, in which I had talked about a more hopeful sort of fire. In particular, Sharon recently wrote the following comment on my post about education entitled Lighting a Fire:

“As someone who studies games for learning, what would you say is the fire triangle for gaming? Does the concept of the fire triangle help relate gaming to learning? I got to thinking about this after reading this post because when I was a kid I used to love playing games (especially board games). I couldn’t get enough of them. As an adult I find that I don’t usually have a lot of patience for them. Now I get the kind of intellectual satisfaction that I used to get from playing games from constructing and debugging computer programs (for a similar amount of intellectual effort), and the payoff in relevance and meaning is a lot higher. I’d be interested to hear your thoughts.”

It’s a very thoughtful comment. In my post I had related the three ingredients for lighting a fire to the three preconditions for learning as follows:

	Fuel:		Intellectual curiosity
	Oxygen:		A sense that what is learned is relevant or meaningful
	Heat:		The excitement that comes from true learning

It seems to me that this all translates directly into how games relate to learning, and why that relationship can change as one grows up. My experience with games is quite similar to Sharon’s. When I was a kid I played a lot of games, and now most of that energy has shifted to creating things on my computer. In fact, when I try to play computer games, I usually get an overwhelming urge to stop playing and write a computer program instead.

I suspect that this is because playing a game is active (as opposed to seeing a movie or a play, which is passive). If I’m going to be doing something that involves actively making choices, I generally prefer to be making choices that come from me — that really represent who I am — as opposed to just the illusion that they come from me (which is what a commercial computer game generally offers).

I would argue that the fire triangle of learning describes this situation perfectly. When we are children, we have a greater ability to learn from games — because children are veritable learning machines, and they are able to learn from any activity that involves making choices.

But as we get older, we generally lose much of this childlike ability to learn from such externally guided interactions. Yet many of us still want that excitement that comes from true learning. So we turn to a kind of game that is uniquely relevant to who we are. In the case of Sharon or myself, that game can be programming.

When we grownups feel that hunger to learn, we still require the same three ingredients for learning: (i) intellectual curiosity, (ii) a sense that what is learned is relevant or meaningful, and (iii) the excitement that comes from true learning. But since we can no longer get that spark from externally defined games, we create our own games.

Programming, like any of the creative arts — from sculpture to songwriting to keeping a daily blog — is at its core a kind of game for learning.

I was at a wonderful concert this evening, in which I wanted to write down my thoughts about what I was experiencing. But I wanted to do this in such a way that I wouldn’t distract the people around me.

I realized that what I really wanted was some sort of technology that would allow me to write things down in a way that was completely invisible to the people around me.

In such a situation, writing on a standard SmartPhone or similar PDA is completely inappropriate. No thoughts you have during a shared experience are so important that they justify that annoying glowing rectangle which distracts other audience members and takes them out of their own experience of what is happening onstage.

I am coming up with some interesting ideas for how to do this right, which I will describe here when they are fully developed. 🙂

Having just experienced a particularly whirlwind week — a very nice week, but a whirlwind week nonetheless — one that was split up into multiple experiences with multiple casts of characters, I realize more acutely than usual how much peace of my mind depends upon grabbing those few moments (sometimes those few precious moments), when there is nobody around but myself.

Just as food needs to be digested, so do experiences. We need those moments of quiet reflection to sort it all through, to assess what has happened, what dramatic changes have just taken place in these latest scenes of one’s life. Without the little pauses between the action, our moments can tend to run together into one undifferentiated blur.

The next time you are tempted to fit in that one extra meeting, to go to one more place, squeeze in just one more activity, it might be good remember that the soul, like the body, also needs its rest.

It is certainly something that I would do well to keep in mind more often than I do. 🙂

A random moment
In a busy day, I stopped
And took it all in

This odd, unlikely
Accident of existence.
This one universe

Out of the billions
Where you and where I, my friends
Know this gift of life

Yesterday I saw a presentation by enthusiastic improvisational theatre troupe that calls itself “Story Pirates”. They work with little kids, helping those kids to build confidence in their creativity. The particular way that they work is to convert stories from kids into theatre pieces, which they then act out, with verve, humor and fun.

In yesterday’s presentation, they worked with the kids to a really fun little seven minute epic, which ended up as something like this:

“Shiny Pants was sad because he was incomplete — ne needed shiny shoes. One day he meets a friend, Shiny Shirt. They decide to go together to find Shiny Shoes. The run into Toby the Witch, who tells them that to find the shiny shoes, they will need to journey to Shiny Island. On the way they run into the dreaded Toilet Paper Mummy, but they overcome this obstacle, make it to the island and find the Shiny Shoes. In the end they realize that they are happy, not because they found the Shiny Shoes, but because they found each other — they are now a family.”

Although when I saw a demonstration of the troupe in action, what I saw was more like a kind of Mad Libs game — the actors would ask the kids such questions as “what’s the name of the hero,” and then make a story around that. Although the kids thought they were creating a story, everything but the specific names actually came from the improv troupe.

As I thought about what this underlying story was, I realized it was the classic Hero’s Journey, as described by Joseph Campbell in The Hero with a Thousand Faces. Here is my diagramming of this underlying story:

What the kids provided was just a naming of the pieces of this diagram:

Hopefully, in their extended workshops the Story Pirates bring kids in on this underlying process. Because that’s what it will take to really help kids learn how to be great storytellers.