Ken's Blog

This evening the wonderful students in my graduate computer graphics class showed their final class projects. The amazing work they produced exceeded my expectations, and then some.

I am deeply inspired by what happens when you give brilliant young people the tools they need to create something new. Especially when you also realize that the best way to support them is to stand back and get out of their way.

If you are curious, and want to read a bit more about it, feel free to drop in on our Future Reality Lab blog.

As I’ve said here before, I’m always looking for apt metaphors for the future that our lab’s research is enabling. There are so many metaphors to choose from: Faerian Drama, the Holodeck, The Matrix, the Jedi Council and many others beside.

The other day, as I described our research to a colleague, he said “What you are creating is The Magic Schoolbus!” And I realized he is right.

We are giving people a way to physically gather, and then be whisked off together on a guided tour of fantastical worlds. The possibilities are limited only (as my friend Lance Williams used to say) by the imagination.

This evening I watched an episode of The Magic Schoolbus, and it was completely delightful. I won’t tell you which episode I watched, but I can say that my favorite line was “We’re up Ralphie’s nose???”

Normally I would say that the future is nothing to sneeze at. But having just seen that classic episode, I’m not so sure.

This evening I am thinking of going with a friend to attend a solo concert. The musician plays cello and sings.

Until today I had never heard of such a thing. The idea of somebody singing while playing the cello seems so exotic to me.

Yet I find myself wondering why I think this way. After all, I have attended many concerts where somebody sings while playing the guitar, or the piano.

What is it about the cello that makes the situation seem so different? I have been pondering this, and I have yet to come up with a good answer.

Any ideas?

Perhaps the most important element of the language of cinema is montage. When a filmmaker creates a cut, and the camera suddenly moves from one point of view to another, the viewer is effectively being taken inside the mind of either the storyteller or a character in the movie.

We don’t respond so much to what happens in any one shot, but rather to the relationship between shots. Cutting to an actor’s face at a key moment can create a profound identification between the audience and the character the actor is playing.

But what will happen when shared cinematic experiences become completely immersive? When we can simply walk into the world of a narrative, will montage even be necessary?

It’s exciting to be working on completely immersive storytelling media at a time when the answers to such questions are not yet clear. These are early days, and the language of immersive media has not yet been worked out.

Perhaps we, like Méliès and Eisenstein before us, will get to create a new language for visual storytelling. Exciting times!

Every time I give a talk about our research and our mission, I make some kind of change. So in some sense the talk is a kind of evolving creature, growing a tentacle here, losing one there.

To do this, I generally go by the energy I get from the audience. Sometimes I feel that I have glossed over something important, and need to add a bit of connecting tissue. At other times I feel that I’ve belabored something inessential, which distracts — and therefore detracts — from the main points.

It’s a constant process of editing, and it feels very productive. Not just because it’s good to get out our message, but because for me it’s an essential part of understanding our message.

I suspect this is true for all of us: You don’t fully understand your own mission until you are able to clearly explain it to others. And at the end of the day, there’s nothing more satisfying than understanding your own mission.

If you want to assess the appeal
Of making the virtual real
The thing you must follow
In a novel or Holo
Is the way that it makes people feel

Today at Electronic Arts
They made me sign an NDA
I still have yet to figure out
Just what it is I shouldn’t say

The students in my graduate computer graphics class will show their shared metaroom final projects one week from today — the evening of Monday December 16. Today several groups have been at our lab working away to get their “one week to go” preliminary demos ready.

I can feel a really wonderful excitement as they work through creating these alternate worlds. Every once in a while, a student will pull me aside, hand me an Oculus Quest headset, and say “here, try this.”

And suddenly I am in another world. But a world in which we can all walk around together with our own feet, no wires attached.

I can’t wait to see their visions of the future come to life. Just one week to go.

I’ve been thinking about what to call our platform that allows multiple people to roam around a room together while seeing and hearing an alternate version of their shared physical reality. It occurs to me that it could be interesting to tie it into the seminal 1947 essay by Tolkien that I blogged about several months ago On Fairy Stories.

After all, Tolkien’s description of “Faërian Drama” within that essay is a very nice summary of the possibilities of fictional narrative using the tools we are developing. So why not look to it for inspiration when naming our project?

In that spirit, I think we might want to describe what we are doing in a way that produces an apt acronym: Shared Highly Immersive Reality Emulation

As our lab’s research explores shared worlds, we find it useful to look at movies for useful diegetic prototypes. For example, TRON, The Matrix and Ready Player One all tell essentially the same kind of story: People are beamed into an entirely virtual world, where they become completely separate from their actual physical bodies.

In contrast, the Jedi Council in the Star Wars universe tells the opposite story: Members of the Council always remain connected to their actual physical bodies. The people who are beaming in are simply enacting a full-body holographic version of Skype or Google Hangouts.

The essential point is that the Jedi Council does not posit any separate virtual world. The only world that exists is the one people inhabit with their physical bodies. They just happen to have a better version of Skype. I think our own lab’s research is much closer to this vision than it is to the vision within TRON and its descendants.

The Star Trek Holodeck is somewhere in between. There is a suggestion that people in the Holodeck remain attached to their physical bodies, but it is only a suggestion.

A different approach is found in the Visual Instrument and Sight Organ Replacement (VISOR) worn by Geordi La Forge in Star Trek the Next Generation. Geordi inhabits the same physical reality as everyone else, but he perceives that reality quite differently.

It is not entirely certain what Geordi sees when he looks at his fellow crew members. Perhaps he sees them as computer graphic reconstructions of themselves.

But it is clear that he can perceive things that are invisible to others. For example, Geordi can look at a wall and see the electrical wiring within it. I particularly like this diegetic prototype — partly because I suspect that something like it will become a part of our ordinary everyday reality within the next decade or so.