VR, AR and beyond

Last year we presented a work that allowed 30 people to share a VR “virtual theater” experience, all in the same room. People could see each other, but as avatars of themselves.

This year (this week in fact) we presented a work that allowed 6 people to share an AR “virtual theater” experience, again all in the same room. The experience itself was not as overwhelming — it all took place in miniature on a tabletop — but people could literally see each other.

It’s been fascinating to observe the tradeoffs between these two modes of immersive storytelling. The shared VR experience is all around you — you and your friends are transported together into another world entirely.

The AR experience simply affords you a view into another world, without actually taking you there. Yet you and your friends have a greater sense of each others’ presence and response to the work.

Ideally we will find a third way — one that transports you together into another world, yet maintains the richness of connection that you feel with one another in real life. Maybe we will show that next year. 🙂

Art paper presentation

I have spent quite a few years giving presentations about technical papers. But presenting an art paper is something that is relatively new for me.

Today as we were presenting our paper, I started to see some distinct differences. When you present a technical paper, the focus is generally on the work, not your relationship to the work.

The rhetoric around a technical paper is focused on objectivity. Rather than focusing on “I did this,” one focuses on “This is true because.” Ideally the author fades away, and the truth reveals itself in all of its unbiased glory.

An art paper is quite different. You are expected to speak from passion, from faith in your work. Your opinion matters very much, as well as your belief in what would make for a better world.

This idealistic focus seems to be the very heart of an art paper presentation. Fortunately, I found myself embracing it wholeheartedly.

After years of presentations where I needed to try to create a distance between myself and my work, it was wonderful to be relieved of that burden. Scientists and artists are both extremely passionate types of people, but they express that passion in very different ways.

Closing thought

Today I published a post on our Future Reality Lab blog about our winning the ACM/SIGGRAPH 2019 Best Art Paper Award. In this post I focus on a particular aspect of that paper.

Our work on that paper aimed to change the tone of the conversation about virtual reality experiences. With that goal in mind, the closing thought of our paper was inspired by that of another paper written many years earlier.

The last sentence of that paper by Watson and Crick described the double helix structure of DNA. It was the scientific equivalent of a mike drop:

“It has not escaped our notice that the specific pairing we have postulated immediately suggests a possible copying mechanism for the genetic material.”

Here, by comparison, is the last sentence of our paper:

“It has not escaped our notice that the specific pairing of co-located audiences and physically shared immersive virtual narrative immediately suggests a possible future path for the motion picture industry.”

Fast forward 2019

In a very strange and hallowed ritual, authors of technical or art papers published at the Siggraph conference get 30 seconds per paper to describe their work. This is done on the very first evening of the conference (ie: this evening), which means I need to do it in about two hours.

There is no room for slipping up. While you are talking, videos are played in succession on a big screen, and the whole system is automated. So if you stumble over your words, you don’t get a second chance.

I’m mentally preparing for my 30 seconds. Of course the best way to prepare for this would probably be to not think about it too much, which is the opposite of what I am doing.

On other hand, one way or another, it’s going to feel great afterward. That’s because then I will no longer need to worry about whether I will screw up. 🙂

Day before SIGGRAPH

Tomorrow is the start of the big annual SIGGRAPH conference — the center of the computer graphics universe. I am writing this from our booth in the Immersive Pavilion (booth VC 101, if you happen to be here), where people come to see the very latest in VR and AR.

As it happens, the piece we will be showing is both. When people put on Magic Leap headsets, they will have an AR experience in which they will be able to look down at a table to see and hear a miniature virtual theater performance of an original play we have created. The characters and sets will appear to be at 1/10 scale.

Our play tells origin story of Mary Shelley’s Frankenstein. We chose this story in particular because it is, in a sense, the origin story for science fiction itself.

But that’s not all. Other participants wearing an Oculus Quest will have a corresponding VR experience. As they walk around the room, they will see and hear the same play as though they are immersed in a full size theater, with life size actors and sets.

The best part is how these two experiences intersect: AR audience members will be able to see the VR audience members as characters walking around in the miniature theater. One virtual world will be nested within another.

It is going to be awesome!!!

From Z to A

On a long flight today from New York to California, I started wondering whether I could write a sentence in which each word begins with a letter from the backwards alphabet. I decided to make a game of it.

Beyond that constraint, the rules of the game are simple: Your sentence doesn’t need to be profound, it just needs to make grammatical sense.

I don’t think I did too badly. But you might want to see if you can do even better:

Zestful young Xerox workers vaulted up the stairs, rushing quickly past old newspaper men, like kids joined in happy giggling, for every day can be amazing!

Yellow pill

In “The Matrix”, Morpheus offers Neo a choice:

“This is your last chance. After this, there is no turning back. You take the blue pill—the story ends, you wake up in your bed and believe whatever you want to believe. You take the red pill—you stay in Wonderland, and I show you how deep the rabbit hole goes. Remember: all I’m offering is the truth. Nothing more.”

In popular culture, the “blue pill” has come to be associated with choosing to live within a happy and tranquil illusion. The “red pill” has come to be associated with making the braver but more difficult choice to live in reality, no matter how harsh that reality may turn out to be.

So here we have a comparison between two opposite approaches to life. Should we optimize for avoidance of pain or for truth?

Speaking of truth, in truth there are three primary colors, not two. So suppose there were a yellow pill? What would it signify?

Other than reality or illusion, what else is there? What could that third thing be?

We have always lived in multiple worlds

When you put on a wireless virtual reality headset and walk around, your mind is in two physical places at once. Of course you know you are in an actual physical room. You realize that if you were to keep walking forward you would eventually hit a solid wall, even if you can’t see it.

But your mind is in another physical place as well — the artificial one that you are perceiving with your senses. This other physical place can seem very real indeed.

Eventually, as VR becomes more integrated into how we work and play together, people will start to get used to hopping between different artificial physical worlds. Such transitions will come to seem second nature. One moment I might need to be in a place where I am picking up some supplies, the next I might be in a room where a meeting is happening, and sometime after that I might visit a beach and just chill out a bit.

Meanwhile I am also always in a physical room, and part of my mind will never forget that. This is just a more visceral version of the double consciousness you feel when you watch a play: Even as you cry at some on-stage tragedy, part of your mind knows that you are actually sitting in a theater watching a fictional story.

Yet I wonder whether there is also a difference. Because we will be chosing when we enter these artificial physical worlds, we will feel a sense of greater ownership in our relationship to those fictional worlds.

Just as we always feel that we belong in the physical world, we might start to feel that we also belong in these other worlds as well — that in a sense we are actually living our life in multiple physical universes.

When that happens, the physical world may simply become one world among many, and I don’t necessarily think that’s a bad thing. It might simply be a natural evolution of the human condition.

Because we are creatures who communicate with one another through language, we have always been quite aware of the feeling of being in two places at once. In the late Paleolithic age when a tribal elder stood up to tell of some great victory in battle, the members of the tribe were transported into the world of that story, even as they remained seated around the fire.

We have always lived in multiple worlds. That is simply what we are.


Someone I know and greatly respect who is a devout Christian recently explained to me the difference between “Christian” and “Christiany”. The first is a sincere feeling of faith. The second is a kind of provincial tribalism.

I completely get this, because it parallels a similar phenomenon in my own Jewish subculture. True adherence to a higher code and mere tribalism often get tossed together, but they are most definitely not the same.

I was thinking of this recently when I started to research the family tree of Ayanna Pressley, the U.S. Representative for the 7th congressional district in Massachusetts. Since our president recently suggested that she go back to the country she came from, I wanted to understand just what country that was.

Apparently, it is the country of Cincinatti Ohio. According to our president, this exotic country of Cincinatti Ohio is a “totally broken and crime infested place”.

I also learned that Ayanna Pressley comes from a deeply religious Christian family, and that her family members’ faith has continually sustained them through great adversity. I can’t help but contrast this with our president, who says things calculated to appeal to “Christiany” tribalism.

Ironically, there is quite literally nothing about this man that Jesus would have recognized as virtuous. In fact, I suspect that the tears Jesus would have wept upon learning of such a person would easily have filled the Sea of Galilee.

If you take the time to read the New Testament, you realize that the arguments advanced by the various parties fall rather neatly into two camps. One camp we would today recognize as “Christian” and the other we would recognize as “Christiany”. The former were articulated by Jesus and his disciples, the latter by the Pharisees.

When applied to current U.S politics, the words of Jesus of Nazareth — of inclusiveness, compassion, of the idea that we are here upon this Earth to care for one another — are very well aligned with the politics of Ayanna Pressley. In short, Jesus was a Democrat.

In contrast, the political philosophy of the Pharisees was unapologetically tribal, oligarchic and brutally nationalist. Today we would call them Republicans.

Global cooling

In future news, this just in:

In 2040 global warming reaches the point of no return. Scientists realize that by the year 2100, the Earth will be uninhabitable to humans.

Fortunately, the invention in 2038 of the Zickbocker-Schlumbeck treatment offers a chance of survival. All around the world, in a rare example of full international cooperation, expectant mothers receive the treatment. The slim hope of saving humanity hangs in the balance.

Nine months later, the first thermihumans are born. They seem identical to their predecessors in every way, except for their preference for hot climates. Really hot climates. Also a great resistance to ultraviolet light and an aversion to lettuce.

To a thermihuman, any temperature below 80 degrees fahrenheit (about 27C) is unbearable. In contrast, a thermihuman finds 120 degrees fahrenheit (about 49C) to be rather pleasant.

Global warming continues unabated, yet the new generation thrives. By the time all of the Earth’s hydrocarbon based fossil fuels have been spent, the ozone layer is nearly gone. Thermihumans are in heaven.

But then, with no more hydrocarbon to burn, the planet gradually starts to cool down. Most people shrug it off, except for a few eccentrics who rant about “global cooling”.

Nobody believes them, until at last plummeting temperatures make the end seem inevitable. Wars start to break out around the planet.

A group of top scientists gathers. They cannot restore balance to the earth, but that doesn’t mean they are ready to give up.

At last a lone scientist comes up with a solution. “Why,” she asks, don’t we just engineer human beings that are resistant to cooler temperatures?”

Humanity is saved.