Ken’s Blog

The World Wide Web is soon going to move decisively off of screens and into the world around us. It is just waiting for one key consumer platform — a well designed wearable. There are strong indications that this platform will be arriving in the next few years.

Which leads to the question: What, essentially, is the Web? I don’t think it is, at heart, a particular set of technologies. Rather it is a powerful idea that is independent of any given technology.

The Web is our shared electronic space. It is where we can all publish and read and look at and listen to information posted by one another. The Web is the world’s bulletin board.

This distinguishes it from, say, Apps, which are meant to serve specific functions. Because the Web is ubiquitous and freely accessible to vast numbers of people, it is essentially a shared place — although not a place in our physical world.

Once the wearables arrive, this “shared place” will start to have more of a presence in our own physical world. It will become accessible as part of our face to face conversations, our navigation through time and physical geography, our very concept of reality itself.

Yes, you will still be able to choose not to use it, just as you can still choose to turn off your SmartPhone (or, for that matter, choose not to talk to other people). But for more and more of the social contract, living life among our fellow humans, we won’t want to turn it off, because its ubiquitous will seep into our very concept of reality itself.

I don’t think that will be a bad thing. Like the screen-based Web of today, the future Web will quickly simply be thought of as normal reality. In fact, people will wonder how on earth anybody managed to get along without it.

A wonderful feature of this year’s Siggraph conference has been something called the “Geek Bar”. A large number of bean bag chairs face an enormous super-high resolution screen that simultaneously shows eight conference presentations.

You pick up a portable audio receiver/switcher and a pair of headphones from the nice people at the desk, nestle down into one of the comfy chairs and put on your headphones. Then you just pick one of eight audio channels to listen in on.

Each time you choose an audio channel, you are effectively transported instantly from one lecture room to another. The whole thing is so wonderful that I want to set up something similar as a permanent fixture of our lab at NYU.

I don’t like the feeling of earphones covering my ears, but fortunately I didn’t need them. To see the screen more clearly, I wear a pair of glasses, which in my case are Bose Frames. They look just like a pair of glasses but they also provide high quality stereo audio (that other people can’t hear) — without covering your hears. I’ve had my Frames fitted with prescription lenses, since I am slightly nearsighted.

The Frames are a wireless Bluetooth device, but that’s not a problem. I recently purchased a TaoTronics Bluetooth 5.0 Transmitter/Receiver, Wireless 3.5mm Audio Adapter on Amazon. It’s a tiny gizmo that you can plug into any audio jack to effectively turn your audio device into a Bluetooth transmitter or receiver.

I just plugged my TaoTronics into the audio jack of the receiver/switcher, and voila, I was connected. The experience felt much more pleasant than having to wear a stupid pair of headphones, and the sound quality was better too.

On a meta-level the experience was also immensely satisfying, and kind of poetic. I was geeking out at the Geek Bar.

Last year we presented a work that allowed 30 people to share a VR “virtual theater” experience, all in the same room. People could see each other, but as avatars of themselves.

This year (this week in fact) we presented a work that allowed 6 people to share an AR “virtual theater” experience, again all in the same room. The experience itself was not as overwhelming — it all took place in miniature on a tabletop — but people could literally see each other.

It’s been fascinating to observe the tradeoffs between these two modes of immersive storytelling. The shared VR experience is all around you — you and your friends are transported together into another world entirely.

The AR experience simply affords you a view into another world, without actually taking you there. Yet you and your friends have a greater sense of each others’ presence and response to the work.

Ideally we will find a third way — one that transports you together into another world, yet maintains the richness of connection that you feel with one another in real life. Maybe we will show that next year. 🙂

I have spent quite a few years giving presentations about technical papers. But presenting an art paper is something that is relatively new for me.

Today as we were presenting our paper, I started to see some distinct differences. When you present a technical paper, the focus is generally on the work, not your relationship to the work.

The rhetoric around a technical paper is focused on objectivity. Rather than focusing on “I did this,” one focuses on “This is true because.” Ideally the author fades away, and the truth reveals itself in all of its unbiased glory.

An art paper is quite different. You are expected to speak from passion, from faith in your work. Your opinion matters very much, as well as your belief in what would make for a better world.

This idealistic focus seems to be the very heart of an art paper presentation. Fortunately, I found myself embracing it wholeheartedly.

After years of presentations where I needed to try to create a distance between myself and my work, it was wonderful to be relieved of that burden. Scientists and artists are both extremely passionate types of people, but they express that passion in very different ways.

Today I published a post on our Future Reality Lab blog about our winning the ACM/SIGGRAPH 2019 Best Art Paper Award. In this post I focus on a particular aspect of that paper.

Our work on that paper aimed to change the tone of the conversation about virtual reality experiences. With that goal in mind, the closing thought of our paper was inspired by that of another paper written many years earlier.

The last sentence of that paper by Watson and Crick described the double helix structure of DNA. It was the scientific equivalent of a mike drop:

“It has not escaped our notice that the specific pairing we have postulated immediately suggests a possible copying mechanism for the genetic material.”

Here, by comparison, is the last sentence of our paper:

“It has not escaped our notice that the specific pairing of co-located audiences and physically shared immersive virtual narrative immediately suggests a possible future path for the motion picture industry.”

In a very strange and hallowed ritual, authors of technical or art papers published at the Siggraph conference get 30 seconds per paper to describe their work. This is done on the very first evening of the conference (ie: this evening), which means I need to do it in about two hours.

There is no room for slipping up. While you are talking, videos are played in succession on a big screen, and the whole system is automated. So if you stumble over your words, you don’t get a second chance.

I’m mentally preparing for my 30 seconds. Of course the best way to prepare for this would probably be to not think about it too much, which is the opposite of what I am doing.

On other hand, one way or another, it’s going to feel great afterward. That’s because then I will no longer need to worry about whether I will screw up. 🙂

Tomorrow is the start of the big annual SIGGRAPH conference — the center of the computer graphics universe. I am writing this from our booth in the Immersive Pavilion (booth VC 101, if you happen to be here), where people come to see the very latest in VR and AR.

As it happens, the piece we will be showing is both. When people put on Magic Leap headsets, they will have an AR experience in which they will be able to look down at a table to see and hear a miniature virtual theater performance of an original play we have created. The characters and sets will appear to be at 1/10 scale.

Our play tells origin story of Mary Shelley’s Frankenstein. We chose this story in particular because it is, in a sense, the origin story for science fiction itself.

But that’s not all. Other participants wearing an Oculus Quest will have a corresponding VR experience. As they walk around the room, they will see and hear the same play as though they are immersed in a full size theater, with life size actors and sets.

The best part is how these two experiences intersect: AR audience members will be able to see the VR audience members as characters walking around in the miniature theater. One virtual world will be nested within another.

It is going to be awesome!!!

On a long flight today from New York to California, I started wondering whether I could write a sentence in which each word begins with a letter from the backwards alphabet. I decided to make a game of it.

Beyond that constraint, the rules of the game are simple: Your sentence doesn’t need to be profound, it just needs to make grammatical sense.

I don’t think I did too badly. But you might want to see if you can do even better:

Zestful young Xerox workers vaulted up the stairs, rushing quickly past old newspaper men, like kids joined in happy giggling, for every day can be amazing!

In “The Matrix”, Morpheus offers Neo a choice:

“This is your last chance. After this, there is no turning back. You take the blue pill—the story ends, you wake up in your bed and believe whatever you want to believe. You take the red pill—you stay in Wonderland, and I show you how deep the rabbit hole goes. Remember: all I’m offering is the truth. Nothing more.”

In popular culture, the “blue pill” has come to be associated with choosing to live within a happy and tranquil illusion. The “red pill” has come to be associated with making the braver but more difficult choice to live in reality, no matter how harsh that reality may turn out to be.

So here we have a comparison between two opposite approaches to life. Should we optimize for avoidance of pain or for truth?

Speaking of truth, in truth there are three primary colors, not two. So suppose there were a yellow pill? What would it signify?

Other than reality or illusion, what else is there? What could that third thing be?

When you put on a wireless virtual reality headset and walk around, your mind is in two physical places at once. Of course you know you are in an actual physical room. You realize that if you were to keep walking forward you would eventually hit a solid wall, even if you can’t see it.

But your mind is in another physical place as well — the artificial one that you are perceiving with your senses. This other physical place can seem very real indeed.

Eventually, as VR becomes more integrated into how we work and play together, people will start to get used to hopping between different artificial physical worlds. Such transitions will come to seem second nature. One moment I might need to be in a place where I am picking up some supplies, the next I might be in a room where a meeting is happening, and sometime after that I might visit a beach and just chill out a bit.

Meanwhile I am also always in a physical room, and part of my mind will never forget that. This is just a more visceral version of the double consciousness you feel when you watch a play: Even as you cry at some on-stage tragedy, part of your mind knows that you are actually sitting in a theater watching a fictional story.

Yet I wonder whether there is also a difference. Because we will be chosing when we enter these artificial physical worlds, we will feel a sense of greater ownership in our relationship to those fictional worlds.

Just as we always feel that we belong in the physical world, we might start to feel that we also belong in these other worlds as well — that in a sense we are actually living our life in multiple physical universes.

When that happens, the physical world may simply become one world among many, and I don’t necessarily think that’s a bad thing. It might simply be a natural evolution of the human condition.

Because we are creatures who communicate with one another through language, we have always been quite aware of the feeling of being in two places at once. In the late Paleolithic age when a tribal elder stood up to tell of some great victory in battle, the members of the tribe were transported into the world of that story, even as they remained seated around the fire.

We have always lived in multiple worlds. That is simply what we are.