Pragmatic perspective

Today my mom called to wish me a happy new year. We commiserated together over what a terrible year 2016 has been.

On that general topic, I brought up the sad news about the recent passing of Debbie Reynolds, since she and my mom were pretty much contemporaries. And in return I learned some personal history that my mom had never told me until today.

It seems that my dad (who passed away some years back) always had a crush on Debbie Reynolds.

“Really??” I asked. “How much of a crush?”

My mom said that he used to tell her “You have only one rival.”

But my mother then told me that the rivalry had never really bothered her. As always, she had a very pragmatic perspective on the situation. “Fortunately,” she said, “Debbie was not available.”

Interactive projection in the theater

Today I saw the off-Broadway show Gorey. It was essentially an eccentric romp through the eccentric psyche of the artist/writer Edward Gorey.

The production employed projection to bring some of his creations to life. In one scene, Gorey interacted with an animated version of the main character from his story The Doubtful Guest.

Like most such theatrical moments, the projected images were animated beforehand. That pre-built animation was then projected onto a wall of the stage, while the actor precisely timed his movements to make it seem as though he and the animated character were having a moment together.

There was even a moment when “Gorey” fed an apple to the animated figure of the doubtful guest, an eerie recreation of the work of Windsor McCay, who had done the same thing in 1914 — feeding an apple to a projection of his animated character Gertie the Dinosaur within a live theatrical performance.

We spoke with the director and projection designer after the show, and were surprised to discover that they had never heard of either Windsor McCay or Gertie. Perhaps some ideas are just so good that they are destined to be invented over and over again.

Of course one of the great things about live theater is that it is never exactly the same twice. Unfortunately, interacting with a previously created animation removes much of that quality. The live performer must do everything at exactly the right moment, or the illusion is broken.

Having seen this show, I am now more motivated than ever to user our interactive Chalktalk animated drawing program in theatrical performance. Rather than “performing” with a previously created animation, it would be so much cooler to perform with an animated character that is truly responsive in the moment, and responsive to variations in the performance of the live actor. Every performance could then truly reflect the unique emotional interaction with that night’s particular live audience.

Windsor McCay had no choice but to interact with a character that does exactly the same thing every time. But now we have a choice. A century after Windsor McCay and Gertie, we might finally be ready to truly bring animated characters into the glorious world of live theater.

Dark matters

How strange that Carrie Fisher and Vera Rubin passed away on the same day. There is a karmic connection between them.

Carrie Fisher has long been a hero of mind, mostly because I have been a fan of her enormously witty and funny writing, and I am very saddened that she passed away too soon. Of course in the popular culture she is best known for embodying Princess Leia, a character she first played at the tender age of nineteen.

A long time ago in a galaxy far, far away, that character devoted her life to battling the dark side of the Universe. Along the way, she showed the world that there is no problem too large if you have a ready wit, a fearless attitude and a blaster pistol.

Vera Rubin never quite entered the popular culture. But pretty much everyone is aware of her greatest accomplishment: In the 1980s her research provided the first strong empirical case that the Universe actually has a dark side.

Along the way she fought and overcame other sorts of dark forces. Alas, those forces were (and still are) considerably stupider and less glamorous than anything you can defeat with a blaster gun.

The pioneering work by Rubin and her colleagues has held up spectacularly well in recent decades. In fact, empirical measurements currently suggest that there is more than five times as much dark matter in the Universe as light matter.

Come to think of it, if you’ve devoted your life to fighting the dark side, those can seem like pretty daunting odds. It may be just as well that Leia Organa did not know about Vera Rubin’s research.


I saw “The Encounter” this evening. This is a one-man show that relies heavily on binaural audio. Every seat in the theater has a pair of headphones, and the sensation while you are watching the show is that there is a rich sonic landscape all around you.

The effect is very powerful, and quite emotionally engaging. If you’ve never tried binaural audio before, just put on a pair of headphones (even cheap headphones will do) and listen to this classic:


Seeing this show has gotten me much more focused on binaural audio for our own work in socially shared VR. Audio can convey a visceral emotional impact that can be far greater than the impact of visuals.

And this makes sense, in evolutionary terms. After all, 30,000 years ago, when predators were sneaking up on our ancestors, it was our ears, not our eyes, that warned us in time that something dangerous was behind us.

I look forward to incorporating binaural audio into the virtual worlds we are creating. I am sure it will greatly increase the emotional impact of those experiences. After all, where’s the fun in having a virtual saber-tooth tiger sneak up behind you, if it’s just a silent movie? 😉

New, but not too new

From time to time our lab research gets spun off into various start-up companies. We’ve done it a few times now, so we know the general terrain.

But one issue that never changes is the difference between what’s valuable in University research and what’s valuable in a start-up company. The values differ because each is optimizing for something different.

University research optimizes for a general sense of expanding knowledge. From that perspective, it doesn’t matter who makes money from that knowledge.

It doesn’t even matter if the knowledge won’t be of practical use to anyone for another twenty years. The important thing is the overall arc of long-lasting impact.

But start-up companies are first and foremost about bringing into the world something that is economically self-sustaining. And it has to be economically self-sustaining the relatively short term.

You need to create something useful fairly soon, but that in itself is not sufficient. You also need to make sure that the world knows to beat a path to your door.

In other words, you need to create something new, but not too new. If your idea is so far ahead of the curve that nobody else can understand why it’s important, then you’ll just run out of money before you manage to get your product or service out into the world.

It can be difficult for people from Academia to make the psychological transition from “great research idea” to “great start-up idea”. When you’ve been spending years optimizing for one value system, it can be hard to switch gears and optimize for a different value system. There are many ways to get it wrong.

Yet sometimes we manage to get it right. And when we do, it feels great. 🙂

Another kind of virtual reality

These days many movie theaters play commercials before the previews for coming attractions. I guess it’s just one of those sad things we all have to put up with.

But this weekend a friend and I had an even sadder experience. My friend was talking to me during the interminable (and very loud) commercials, when a woman in front of us, who seemed to be in her late 70s or early 80s, turned around and told my friend to stop her loud talking.

I recognized the woman, because I had held the door open for her and her husband when they had first entered the theater (her husband walked with a cane). I am not sure whether she recognized me.

To put the woman’s complaint in context, all the while she was saying this to us, pretty much all the other people around us in the theater were also talking to each other. It’s not as though most people actually watch those stupid commercials. 🙂

We tried to reason with her, but the woman was adamant. She was convinced my friend was being inappropriate by talking in a movie theater during the pre-preview commercials.

I found the encounter disturbing mainly because of my suspicion that the woman may have been suffering from some form of dementia. How awful it must be, I found myself thinking, to live in a world where you are convinced that all the people around you are being rude all the time.

I wanted to help her, to find a way ease her evident suffering, but I could think of no way to bridge the seemingly insurmountable gap between her reality and the reality of all the rest of us.


Last night with a friend I saw Denis Villeneuve’s new film Arrival. I am a huge fan of the story it is based on, Ted Chiang’s novella Story of your Life. I have very fond memories of reading it when it first came out in 1998, and I’ve reread it several times since then.

So I went into the movie theater with more than a little trepidation. After all, I have seen how the meaning of wonderful written works can be betrayed by Hollywood adaptations.

Think, for example, of what Hollywood has done to the work of Alan Moore. Zach Snyder’s adaptation of Watchmen completely reversed the meaning of the work — turning what had been a thoughtful criticism of violence in popular culture into a celebration of it. Even worse, the Hollywood adaptation of The League of Extraordinary Gentlemen just tossed out all of the wonderful ideas in the graphic novel, and the resulting film was just excruciatingly stupid and incoherent.

So I was delighted to find that Arrival was true to the original. Of course it was, in a literal sense, very different from the story, because film is a visual medium. But the deep sense of wonder, the intelligence and humanity of the central character and the poignancy of her psychological journey, the brilliant and exciting conceptual twist at the heart of the story, all made it to the screen intact.

Great science fiction stories so rarely turn into great science fiction movies. Which makes those rare occasions when it happens even more wonderful and satisfying.

Artificial Intelligence in your face

When I think about the potential of those future cyberglasses that will be showing up sometime in the next fivew years, it’s not the graphics that excite me the most. It’s not even the visual registration of those graphics with the physical world. It’s the A.I.

Sure, those glasses will provide a wide angle view of the augmented world, with high resolution, accurate motion tracking and correct stereo. Sort of like a Hololens, but with wider field of view and smaller form factor.

But what’s really interesting is that you will essentially have the equivalent of real-time Google search operating behind the scenes wherever you look. If you look at a person, your glasses will know who that person is. If you look at a bus stop, the glasses will understand, and will be able to tell you how long until your bus will arrive.

Unlike a Google search, which is something you do while you are not doing other things, all of this will be happening while you are walking around the world and talking with people, and that creates a key difference: It can be happening without you needing to be giving it your attention.

Which means that there is an opportunity for the machine learning algorithms to start figuring things out for you behind the scenes. The experience will start to feel more like having a personal secretary who can figure things out for you.

By analogy, some of us remember a time when you needed to memorize telephone numbers. To millenials, those old ways now seem strange and archaic.

Similarly, suppose you and a friend are on your way to the airport. You won’t need to think about your route if your glasses can just tell you “go one block and take the bus that will be arriving a minute later”.

Or more likely, your glasses will simply indicate those directions visually as an overlay on the world around you — you won’t even have to think about it. The process will be so seamless that you won’t need to take your attention away from the conversation you are having with your traveling companion.

After that world arrives, our old ways will seem strange and archaic.