Month: September 2014

We have a demo now working at our lab at NYU in which two people sitting at different computeers, and each wearing an Oculus Rift DK2 virtual reality headset, can “see” the other’s head as a box with a face painted on it. To get to this point required of a lot of hard work by students here — mainly Zhu Wang.

We are deliberately keeping it simple: No fancy graphics or high polygon count, just head movement. And the results are spectacular.

When you are in the experience, even though all you can see is a silly box floating in space, you can really “see” the other person via their movement. And the longer you look at them, the more real and vivid they seem — as though your mind is relearning how to see that person from only motion cues.

For example, after the obvious things, like looking at something, or nodding yes or shaking the head no, we tried doing an “I don’t know” gesture, which mainly consists of shrugging the shoulders while doing a subtle little tilt movement with the head.

We could each plainly see that the other person was shrugging their shoulders, even though no shoulders were visible. It seems we were effectively transmitting the “shrug” gesture just from the subtle movement and timing of our head movement.

Of course we will continue to add things like upper body movement, hands and fingers, eye gaze, mouth position, movement of objects, and more, to the information transmitted. Yet I wonder whether we’ve already hit a kind of perceptual sweet spot just through the movement of heads.

Sometimes the reviewers get it right. On the strength of a rave review in the New York Times, I got tix for Nico Underground, a woozily surreal and dead-on recap, written by and starring Tammy Faye Starlite, of the life and times of the iconic 60’s singer and object d’art Nico.

The entire experience was spectacular. Starting from the fact that we truly were underground — in a cute little downstairs performance space within the bowels of the famed Theater for the New City — the show hit every eccentric and culturally resonant note with pitch perfect accuracy.

When I wasn’t being utterly entranced by the proceedings onstage, a part of my mind was thinking “This is it. This is what New York downtown theater is supposed to be like, and so rarely is.”

And for those of you who were also fortunate enough to see this wonderfully weird and appealing production, I must confess that I may never again be able think of Bob Dylan without also thinking of Joseph Goebbels.

Today I gave a talk where I showed some recent research to a really great group of students, followed by a lively discussion. Because my talk touched on the topic of virtual and augmented reality, a number of the questions were about the long term societal impact of advanced VR and AR.

I was surprised that questions kept returning, repeatedly, to variations on the worry that replacing the real with the virtual would result in people becoming disconnected from reality.

I kept giving variations on the same answer: Humans, by our very nature, live in a virtual reality. Century after century, we develop new technologies, then we convince ourselves that those artificial technologies are reality.

And those technologies are, indeed, reality, to the extent that we use them to communicate with each other. For in one way we are the same as any other species: Our day to day reality is comprised mainly of interaction with other members of our species. Everything else is just detail.

Therefore, any technology that supports our interaction with other humans seems like reality.

Think of all of the highly artificial inventions that we have somehow convinced ourselves are “real”: Agriculture, clothing, written language, moveable type, pianos, telephones, ball point pens, radio, washing machines, automobiles, cinema, airplanes, television, air conditioning, the Internet.

None of these things actually exist in any meaningfully objective sense, other than as interfaces that help people in their interaction with other people. Without human minds around, they are all just utterly meaningless collections of atoms and bits.

They seem “real” to us only because we find them useful for connecting with other human minds. Should it turn out that some kind of virtual or artificial reality ends up being equally useful for connecting with other human minds, then we will eventually cease to label it as technology.

Like the houses we live in, the shoes on our feet, the glass in our windows, or the stores where we purchase our groceries, they will become invisible to us, just another part of our reality.

But temporarily, while any of these things are still new — whether they be books, phones, movies, comic books or whatever — we can become convinced that *this* time will be different, *this* technology could be the end of civilization as we know it.

I understand the argument, but I don’t agree with it. If we didn’t manage to destroy ourselves with nuclear weapons, we’re not going to do it with pixels.

My cousin is visiting now, staying in NY for a few days, crashing at my place. It’s a real treat for me, because he lives far away, so we don’t get to see each other very often.

We’ve been chatting up a storm, having intense discussions about all sorts of ideas, people, politics, inventions, movies, pretty much everything under the sun. Just like we always have.

He and I are pretty much the same age, so our memories go way back. When we were both five years old there was a big old tree in the place out in the country where our families would go each summer. My memories of us trying to climb that tree — and then eventually succeeding in climbing that tree — are as vivid today as when it all happened.

There are people you are connected to in life, and then there are people you are really connected to, on a whole other level. For me there are very few people on earth I can say that about.

When you have the pleasure of getting to spend true quality time with somebody like that, it can really make you appreciate, all over again, the wonderfulness of life.

This evening I was invited to two simultaneous events. One celebrated the opening of a new entrepreneurial center for helping to launch hi-tech start-ups at NYU. The other was the launch party for a new book on ethical values in computer games.

Oddly — and conveniently — the two parties happened to be directly across the street from each other. As I went from the first party to the second, I could see several colleagues passing me, on their way from the second party to the first.

The two events represent two very different parts of our University. One is concerned with capitalism, with commercial exploitation of ideas, with maximizing economic return on investment.

The other is engaged with Truth, and with an ongoing conversation about how we can lead ethical lives. The issues raised touch on a highly philosophical question of great practical import: How can we be better versions of ourselves?

I was struck by the contrast between the two events, by their different tone and focus — and dress code. The entrepreneurial event was clearly geared toward making money. The “Values at Play” event was focused on higher goals and aspirations.

One thing I love about being in the University is there where was no contradiction at all between the two mandates. These two events were just complementary aspects of the same diverse intellectual community.

I wouldn’t want it any other way.

I met a friend this evening at Flannery’s bar on West 14th Street.

Manhattan, where people define themselves by where they go, is full of incredibly cool night spots. Our little borough is rife with fancy restaurants, bars, dance clubs, coffee shops and book stores — places where people go to see and to be seen.

But Flannery’s is different. It’s become one of my favorite places because people here simply don’t care. The bartender serves cheap beer, people hang out and order a side of fries while cheering their football team on the TV, and old guys spend all evening in the back room playing darts.

There is absolutely nothing cool about any of it — and that’s what makes it such a cool place. I think it might very well be my favorite place to go.

God, I hope somebody sees me here…

Last weekend I spent some time with Marc Ten Bosch, and got to see him demo his 4D game Miegakure several times. It’s a wonderful game, with a simple premise: Your character lives in a four dimensional world — one vertical dimension plus three horizontal dimensions.

At any given time you can see only two of the three horizontal dimensions. However, you can rotate between two of those dimensions at will.

So you are always within some three dimensional “slice” of the game’s 4D world. But because you can rotate that slice, you have tremendous freedom of motion within the full four dimensional space.

Today, thinking back on the game, I thought that it might be fun to make a movie where the hero has a similar ability. Even in our plain-old 3D world, this would be an interesting superpower to have.

After all, if I rotate my 3D body so that part of me sticks out into a fourth dimension, then I can make myself perfectly flat within our own world — like an infinitely thin piece of paper.

This would give me the superpower of sliding between cracks, or fitting into impossibly thin places. For example, I could hide behind a bookshelf, or slip through a sliding door.

It might be fun to frame each shot so that the character with these superpowers is always fully dimensional to the audience, becoming “thinner” only in the dimension perpendicular to the screen — the one that we cannot see.

That would allow us to share our hero’s point of view. It would also mean that all the special effects could be done through simple image compositing.

Now all we need is a good script…

Today I was given a wonderful tour of the giant mansions in Newport, Rhode Island. The sheer size of some of these houses is almost beyond comprehension.

Some are best described not so much as houses, but as palaces. Huge, sprawling and impossibly opulent, some contain dozens of rooms and nearly all are set within vast estates. The Vanderbilt Mansion alone, with its 70 rooms, is practically a species unto itself.

No human-made artifact is completely random — everything made by our species is an attempt to solve some sort of problem. Sometimes it may not be obvious what that problem is, but sooner or later you can always figure it out.

But what problem is being solved by these houses? Clearly they are not designed merely for housing and shelter. It is hard to argue that such crazily outsized structures serve any real utilitarian purpose.

I think the problem being solved here is primarily one of social signaling. If you are an obscenely wealthy robber baron, you still look like other humans. You have two arms and two legs, a face with two eyes and one mouth, hair, teeth, fingers, and all the rest. You are a little bipedal creature upon this planet who doesn’t look any different from your fellow little bipeds.

So to assert the superiority conferred by your vastly greater wealth, you need to create a virtual projection of yourself — some sort of avatar that is beyond ordinary human experience. And if you lived in the late 19th century, the best way to do this was through a vastly over-built house.

As evolving technology begins to permit us to “build” more and more within a socially shared augmented and virtual reality, what will the Vanderbilt Mansions of the future look like? They may not be physical buildings at all, but virtual habitations that allow us to make manifest our social status.

If you had all the money in the world, and you wanted everybody to know it, what sort of virtual mansion would you build, in the shared augmented reality of the future?

This evening I am visiting a couple who have a lovely house out in the country. They are also somewhat older — the husband is eighty four.

We got to talking about their neighbors, and my host told me that he is in a running feud with the man who owns the house on an adjoining property. They got into a fight ten years ago, and since then they have not spoken one word to each other.

He went on to say that the only form of communication they now have is a sort of grim unspoken competition — whoever stays alive the longest wins.

At this last bit of news I perked up. “Do you realize,” I said, “that your neighbor is doing you an enormous favor? After all, because of him, you might end up staying alive and healthy for years to come.”

This afternoon I finally realized something on a conscious level that I have sort of known for a very long time: Familiar places are haunted.

In particular, I entered my office at NYU today — something I do pretty much every day — and realized that every time I walk through that door, somewhere in the back of my mind I flash back to people I used to know.

These ghostly people are quite a diverse bunch, and the memories span many years. What they all have in common is that at some point they were in my office, and they made an impression on me.

Don’t get me wrong — I am not assaulted by these memories. Many of them are very subtle, just tendrils of memory, impressions of ideas and relationships, that come to mind when I look at a particular book on my shelf, or at some little gift given to me by a former student.

But other memories are very strong indeed, and can cause me to replay entire snatches of conversations in the back of my mind, the way you can suddenly realize that you’ve been hearing a song in your head for the last half hour.

One strange aspect to all this is that these are not memories of these people as they are, but as they were. The actual people may be long gone from my life — or even gone from this earth.

But their ghosts remain, flickering just at the edge of my view, as if waiting for me to notice them.