Ken's Blog

The foregoing gesture examples may all seem plausible, but that doesn’t make them correct. “Correctness” in this case means whatever is naturally learnable.

Linguists have a very specific definition for the phrase “naturally learnable”. It doesn’t mean something that can be learned through conscious practice and study. Rather, it means something that one learns even without conscious practice or study.

For example, one’s native spoken language is naturally learnable. We didn’t need to go to school to learn our first spoken language — we began to gradually speak it when we were still young children, simply by being exposed to it.

In contrast, written language is not naturally learnable. Most people need to put in the effort required to consciously study and practice before they can read or write effectively.

Attempts to create a synthetic “natural language” generally fail, in the sense that children will not learn them. For example, when children are exposed to Esperanto, they will spontaneously try to alter it, because its rules violate their innate instinct for natural language.

There is now a general consensus amongst evolutionary linguists that natural language and children below around the age of seven are a co-evolution: Natural language evolved to be learnable (and modifiable over time) by little children, while simultaneously little children evolved to learn and modify natural language.

Tomorrow we will discuss what this means for our topic of future language.

In addition to relying on speech to fill in the meaning of iconic gestures, it will also be useful to provide each conversant the option to sketch specific shapes in the air while speaking. This would be helpful in situations where a physical shape contributes strongly to the intended meaning of the speech act.

For example, saying the word “time” while drawing a rectangle might result in a calendar, whereas saying the same word while drawing a circle might result in a clock. In each case, the drawn shape acts as a modifier on the spoken word, lending it a more context-specific meaning.

It will also be useful to distinguish between three ways to gesture with the hands: one-handed, symmetric two handed, or asymmetric two-handed.

An example of a one-handed gesture would be: Close the fist on a visual icon, move the hand, and then release the fist, which could be a way to indicate: “I move this to over there.”

An example of a symmetric two-handed gesture would be: Hold the two hands open with palms facing each other so that an icon is positioned between them, then spread the hands further apart, which could be a way to indicate “Let’s zoom in on this to see more detail.”

An example of an asymmetric two-handed gesture would be: Close the fist of one hand on an object, then pull the other hand, with fist closed, away from the first hand, which could be a way to indicate: “Let’s make a copy of this.”

More tomorrow.

We’re talking about using gesture to create visible representations, as a way to augment conversational speech. To break this down into different types of gesture, we should refer back to the reasons we already use various types of gesture.

Pointing is easy. Anything we have visually created can be pointed to, while saying words like “this” or “that”. Also, we can point at one item and then at another to establish a relationship between the two objects.

If we have drawn anything that has a process component, beat gestures are a natural way to iterate through that process. Beat gestures are essentially a way saying “here is the next thing.”

There is an interesting relationship between pointing and beat gestures when it comes to describing time-varying processes: To go back to our cooking recipe example, we can use pointing to refer to a particular place in the recipe. Then we use beat gestures to advance step by step through the recipe instructions.

When used to augment speech, symbols essentially act as adverbs. For example, we can use symbolic gestures to make it clear that things are happening fast or slow, calmly or with agitation, definitively or with confusion, or in a friendly or hostile manner.

Lastly, icons, particularly when used in tandem with spoken words, can be used to create visual representations of actual topics of conversation — a chair, a tree, a calendar, the Sun. Because we are speaking while gesturing, we don’t actually need to draw the objects under discussion. Rather, we can use iconic gestures to indicate a location and size for the visual representation of each object or concept under discussion.

More tomorrow.

So the gestural tools we have to work with are symbols, pointing, beats and icons. In addition, we will have the ability for all participants in a conversation to see the results of their gestures, perhaps as glowing lines floating in the air.

As we think about how to use our gestural tools, it is important to remember that we are not trying to replace verbal speech, but rather to augment it. The situation is somewhat analogous to having a conversation at a whiteboard. As each participant speaks, they draw pictures that help clarify and expand upon the meaning of their words.

One key difference is that if you have a computer in the loop, then the lines you draw can spring to life, animating or changing shape as needed. You are no longer stuck with the static pictures that a whiteboard can provide.

For example, if you are trying to convey “this goes there, then that goes there”, you can do better than just draw arrows — you can actually show the items in question traveling from one place to another. Which means that if you are trying to describe a process that involves asynchronous operations (for example, a cooking recipe), your visualization can act out the process, providing an animated representation of the meaning that you are trying to convey.

So how do we use symbols, pointing, beats and icons to make that happen? That’s a topic for tomorrow.

We will get back to the series about future language tomorrow.

Meanwhile, today I finally sat down and wrote the “origin story” for the big virtual reality theater piece that we will be doing at the SIGGRAPH conference two weeks from now.

Sometimes it is not sufficient simply to bring something out into the world. It is also important to provide some context — to state your intention, so that people understand where the work is coming from.

I think I did a pretty good job of that in this post on our Future Reality Lab blog.

Technology continues to evolve. But for the near future we are still stuck with the brains we have, which have not changed in any fundamental way for the last 30,000 years.

So when we look at using our hands, in combination with any forthcoming mixed reality technology, to “create things in the air”, we should look at how humans gesture naturally. We are going to focus specifically on gestures made with the hands (as opposed to, say, nodding, shrugging the shoulders, etc).

There are four basic kinds of meaning people usually create with hand gestures: symbols, pointing, beats and icons. Symbols are culturally determined. Some examples are waving hello, fist bumping, crossing fingers, or shaking hands.

We usually point at things while saying deictic words like “this” or “that”. Beats are gestures we make while talking, usually done without really thinking, like chopping hand motions. Beats come so naturally that we even use them when talking on the phone.

Finally, icons are movements we make during speech which have a correlation to the physical world. Examples are holding the hands apart while saying “this big”, rubbing the hands together while talking about feeling cold, or holding out one hand palm down to indicate height.

Some of these types of gestures are going to be more useful than others in adding a computer-mediated visual component to speech. More tomorrow.

Back in February 2014 I wrote a post on Future Language. What I meant by that is how language itself will evolve in a future where ubiquitous mixed and augmented reality will be an everyday part of life.

Children growing up in such a world will create shared visual representations of thought by gesturing in the air with their hands. To children born into that reality, this will simply be taken for granted, the way we now take for granted the ability to text or speak on the telephone.

Such forms of visual communication will not replace verbal speech. Rather, they will augment it, allowing speech itself to be used in new ways — much as phone and text have not replaced speech, but rather have extended its reach, allowing it to be used and shared in ways that have altered the way we communicate.

Since my initial post, we are four years nearer to that reality. So this seems like an auspicious time to delve more deeply into the topic.

In the coming days I will go into more detail about how visually augmented speech will evolve, and what that change will mean.

Since today is the 27th day of the month, I find my thoughts drifting toward mathematical patterns. That’s because 27 happens to be 3 raised to the power of 3.

Which suggests the idea of raising a number to the power of itself. If we do this with integers, we get a series that starts: 1, 4, 27, 256, 3125, 46656, 823543, 16777216, 387420489, 10000000000 …

But we don’t need to do this with integers only. We might just as well raise 1.5 to the power of 1.5 (in which case we get a result between 1.8 and 1.9).

If we try it with negative numbers, for example -1.5, things start to get more complex. And what if we start with complex numbers?

If we consider the entire complex number plane, this operation gets very interesting. If you are mathematically inclined, you might want to explore the question: What is the shape formed by raising every complex number to the power of itself?

Sometimes when I’m programming I look back at old code that I wrote long ago and I am surprised. I say to myself “I wonder what was going on in that guy’s mind.”

There are times when I think “Would, he really didn’t have a clue, did he? I’m just going to have to fix this now.”

Then there are other times when I look at code that I wrote some time ago and I think “This guy is so much smarter than I am. I have no idea how he figured out how to do that.”

I’m not sure what it all means. Is what I am describing some failure of long term memory? Or is it just the fact that we use multiple parts of our mind when we do something like programming a computer?

I know that technically I’m talking about a single-person activity. But sometimes it sure feels a hell of a lot like collaborating with somebody I don’t quite know.

There are tasks we never get around to doing because they seem overwhelming. Then there are other tasks we break down into little pieces, doing a little bit every day.

I freely admit that there are quite a few tasks that for me fall squarely into the first category. In fact I may never get around to doing them. I look up at that mountain and all I see is insurmountable height.

On the other hand I have practices that fall very much into the second category. For example, most mornings I wake up very early and head to the lab. Before anybody else shows up I have already put in a solid two hours of programming.

If you were to up all of the time I spend programming every year, it comes to quite a lot. And yet it doesn’t seem like a lot, because I divide the work into those manageable little chunks.

And it doesn’t even seem like work, because I love programming. Perhaps one definition of what we love is whatever we make sure to do a little bit every day.