Ken’s Blog

If you’re going to create virtual objects that appear to float in the air between people, one way to do it is to actually put a virtual object in the air between people. This seems to be the principle of the Holodeck from Star Trek, the Next Generation. Such approaches have the disadvantage that you need some kind of site-specific projection device, so they are most likely not going to scale up to inhabit the entire shared world around us.

As I mentioned in an earlier post, an early fictional version of this projection-based approach is the one developed by the Krel in the 1957 film Forbidden Planet, clearly a direct inspiration for what George Lucas put on the screen twenty years later:

Art imitates art

Around 2002 Jeff Han and I collaborated on a project to try to make something for real, which we called Holodust.

Our basic approach was to draw the virtual object with a laser beam directly onto a cloud of dust. Of course you don’t know the exact position of each particle in a cloud of dust, which is why our plan was to use two lasers: An infrared laser would sweep through the cloud. Whenever it happened to hit a dust particle that it would be useful to illuminate, a second — visible — laser would flash, thereby lighting up just that one dust particle.

And here is a link to some Java applets simulating a Holodust display:

Here is a visualization of what a Holodust display might look like, together with a photo from an experiment that Jeff built to test the principle:

Note that there is a big distinction between such displays and, say, the Heliodisplay by IO2 Technology, which projects a flat image seemingly in thin air (actually into a thin sheet of water mist). The Heliodisplay is not eccescopic, since you don’t see a different image when you walk around it.

In contrast, the wonderful 360^o Lightfield Display at USC is indeed more eccescopic than Holodust, because even the shading of the virtual object can change as you look at it from different directions. Unfortunately, it relies on a slanted metal mirror rotating at very high speed, so it you tried to touch it you would most likely destroy both the display and your hand.

Sometimes art imitates life. When I see the little sparkly dustlike particles in the air within the floating display of a virtual brain from Joss Whedon’s recent TV series Dollhouse, I definitely get the impression that it’s supposed to operate through some kind of Holodust:

While creating visions from a computer that appear directly in the world around us is a wonderful thing, it’s not the be all and end all of eccescopy. It was arguably George Lucas who whet our appetite for the real killer app: Using that information as a way for people to communicate with each other — without disrupting the sense of shared physical space.

The original Star Wars film provided many distinct examples of this, of which here are just three:

Princess Leia in a beam of light

Playing Dejarik aboard the Millenium Falcon (“Let the Wookie win”)

Rebels watching a visualization of the Death Star circling Endor

 

What’s most striking to me about these scenes, and others like them, is the clear sense they convey that in the world of Star Wars there is no need for computer screens, since a technology exists that allows information to simply appear in mid-air.

And yet the world of Star Wars is filled with computer displays. Perhaps George Lucas hadn’t really thought this through. But more likely, he probably realized that a sci-fi world without traditional computer screens would be completely incomprehensible to contemporary audiences.

Forgetting for the moment how we would technically realize a vision of computer information simply coexisting with our physical world, it’s fun to see the fantasy versions of this vision that people have created — many of which are available on YouTube and Vimeo.

One that is particularly nice from a technical/aesthetic perspective is a video entitled “What Matters to Me”, in which Christopher Harrell describes his ideas by pulling them out of the air and arranging them in front of him in space:

I especially like the way some of those ideas perch atop his fingers, until he is ready to wave them away.

Another work that seems to get at some of these ideas (although it isn’t nearly as elegant) is the augmented office scene from the recent computer game “Heavy Rain”:

Then there is the lovely video by Bruce Branit, in which a young man constructs an entire world out of the air, using only his hands, for the woman he loves:

Closer to current technical possibility — and a great example of street theatre — is the delightful demonstration system in which Marco Tempest turns a piece of cardboard into a magical interactive space:

Mr. Tempest only manages to turn that one piece of cardboard eccescopic, but this is clearly a step in the right direction.

What these visions all have in common is the idea that there is no “computer” — there is only us. Information appears not on some disembodied screen, but rather right here in the physical world we share.

But how can we do this for real — not just on pieces of cardboard, but everywhere?

Some years ago I was visiting Will Wright at Maxis, while they were still working on “The SIMS 2”. He showed me a box, exactly the size of a computer game CD box, with nice artwork, text, system requirements, everything you’d expect. Except that it was labeled “The SIMS 4”, and the release date was sometime around 2012. I looked more closely at the system requirements, and they were far beyond anything available at the time.

Will explained to me that this was always the way he and his colleagues plan new game releases. Right up front they design the box, the artwork, that characters, the nice little blurb that goes on the back of the box. Then they set about making it possible for you to open the box (which might take a few years). In Will’s own words: “The game is in there. You just can’t open the box yet.”

And so I’ve decided to expand on yesterday’s post with a series of descriptions of the emerging field of “eccescopy”. My techno-geek side likes to think that “ecce” stands for “eye centered computed environment”. An eccescope is simply a device to let everyone see an alternate world created within the computer cloud, thereby allowing that world to appear before our eyes, right alongside our own physical world. It’s the ultimate extension of what is currently called “augmented reality”.

I chose that word because neither “eccescope” nor “eccescopy” appear even once in a Google search (although after today’s post, that will presumably change). I also chose “ecce” in order to rescue that perfectly respectable Latin word, which means “to see”, from its ignominious association with a certain unfortunate fable involving an ancient Roman prefect of Judaea. 🙂

When you put “ecce” together with the Latin word “scopus”, which means “to look”, you get the idea — in order to see, all you need to do is look. Well, that’s that’s the basic idea anyway. In follow-on posts I will describe what an eccescopic future might be like, and how we might get there from here.

One day, in not so many years, we will no longer need computer screens. One way or another, it will become cheap and easy for people to see objects that are not there, superimposed onto the physical world around us. Not only that, but each of us will be able to see our own personal view of this augmented reality, customized for where we happen to be at the moment.

Only a few years ago I thought that the level of technology required for this would not arrive for perhaps half a century, but now I’ve come to see that it will probably be here well within the next decade. Which means it’s time to think seriously about how to make the most of things.

Will it be a good thing or a bad thing when virtual objects will inhabit the physical space between us all — when the collective ideas of humans burst forth, free from mere books and screens, becoming free to roam the world?

Will this new way of seeing information alter our fundamental relationship with our physical selves? Or will it have the opposite effect — freeing us once and for all from the harsh limitations of a screen-bound information world, so that we can return once more to the world of mind joined to body, for which our evolution has so well prepared us?

For most of my life it was generally received wisdom within the United States of America that New York City was a terrible place to live. That is, of course, unless you lived in New York City. New Yorkers have always loved their city with a fierce pride, a zealous passion akin to that which possessed the ancient citizens of Imperial Rome.

But if you didn’t actually live in the Big Apple, you knew it only from movies and TV shows, which invariably showed it to be a crazy scary brutal place where catching a cab is hard, but getting shot by a drug dealer is easy. Or if you did visit, you probably did what all tourists did, and headed straight for ridiculous places like Times Square (a part of the city actual New Yorkers try to avoid).

But then, nine years ago, New York City got attacked. Yes, I know — technically everyone in the U.S. got attacked — but believe me, it was different if you were there, and people you were actually connected to died, and for months after you had to breath in that sick smell of death and rubble. For us it was a violation more personal than political. And of course it was difficult to stay calm and gracious when so many well meaning visitors wanted to “check out Ground Zero and then head up to see Les Miz”.

A number of my friends found it all too depressing, and moved away from NY during those years. But the flip side was that people all around the United States suddenly liked New Yorkers. We were warmly embraced by our fellow Americans, supported, even honored, and all of that snide anti-New York attitude seemed to fade away.

Until recently. Lately I’ve noticed that the anti-New York sentiment seems to be coming back. Somehow we have once more come to represent, in some circles, everything that is wrong with this country. Yet as it happens, a recent poll has shown that New Yorkers are quite happy — that we are in fact more content and satisfied and enthused about our city than we have been in years.

And so it seems there is some law of conservation at work: The more depressed New Yorkers get, the more they are embraced and celebrated by their fellow Americans. Whereas the happier people are to live in New York, the more convinced the rest of the country is that it’s a terrible place to live.

I wonder, why is that?

This evening, returning from a lovely dinner out with some friends, I realized — at the very moment I reached my door — that I had been replaying the same song over and over in my head for the previous twenty minutes. During those twenty minutes I had not been conscious of doing any such thing, but as soon as I caught myself, awareness and memory came flooding back all at once.

I’ve found myself doing this sort of autopilot song-playing many times, with a surprisingly large variety of musical genres. This time it was Neil Diamond’s “Song Sung Blue”. One recent evening I realized my mind had been endlessly replaying a version of Leonard Cohen’s “Hallelujah” (in particular, the Sad Kermit version of the Jeff Buckley cover).

My unconscious mind does not seem to favor any particular type of music. At various times I have found myself endlessly replaying everything from Beethoven’s “Ode to Joy” to Lady Gaga’s “Bad Romance”. Apparently deep down my mind has highly catholic tastes.

Several years ago some well meaning friends pitched in and bought me an iPod Touch. Dutifully I tried carrying it around the city with a pair of earphones, listening to my favorite songs. But it never really worked. No matter how much I liked any given song, hearing it through earphones seemed vaguely annoying, as though some intruder were trying to barge in on my brain uninvited.

I realize only now what the problem was: The song in my ears was most likely interfering with whatever song was already playing in my head.

Mari’s comment about the difference between American and Japanese styles of email writing got me thinking that it would be interesting to compose equivalent hypothetical emails from New York to Kyoto using the respective conventions of these two contrasting cultures.

First, here is the American version:

Hey Kazuo,

Dude, can’t believe it’s October! Long time no chat. I’ve got news – Skype me.

Later,
Ken

Now, if this letter were written in accordance with Japanese convention:

Fond greetings from New York.

The lingering summer weather has now passed. As the cold of autumn gently emerges, the leaves have turned at last to gold and to crimson.

I earnestly hope your health is well. My own health has been much improved in recent days, no doubt from thinking upon your warm friendship.

It was so kind of you to allow us to look after your little dog Aki. He has brought much delight to family and friends here with his funny antics and zest for life.

I wish you all the best for continued health, and please convey my sincere regards to your family.

I am praying from my heart for your happiness,
21 October, 2010
Ken Perlin
Mr. Kazuo Tanaka

Also, please accept my apologies for accidentally running over your dog, who is now dead.

There was a time — roughly from the 1950s through the 1990s — when TV was pretty much the thing for American families who wanted to share time together by consuming media. Yes, you could play games together, but it was hard to find games that parents and children would enjoy equally well. And movies were a “once a week” thing — something you went out to the cinema to see on a Friday night — not something you did every day.

But every night, without fail, most families would get together, gather around their TV after dinner, and watch whatever was that night’s prime time hit show. And then mom and dad (and maybe the older kids too) would stay up to see the Tonight Show. This pre-packaged way of spending time with your family was such a given, that after a while people stopped thinking about it.

Until, that is, it started to go away. The rise of the Web, and more compelling computer games, and even TV on demand, has, over the course of the last decade, gradually dismantled the entire concept of a family getting together at one place and time to have a shared media-consuming experience. Now when we see images of mom and dad and the kids all sitting around the living room watching TV, it feels like a window into another time, a time that is gradually receding into history.

And now I think the same thing is starting to happen with email. Once email was unassailable — the great connector, mighty cybernetic unifier, bringer together of worlds, killer of postage stamps. But now that we have Facebook, Twitter, micro-blogging, MMORPGs and various forms of on-line chat, it seems that the once mighty email — the electronic version of the long-form letter — has started to go the way of its tree-killing forebear. In just a few short years, the very idea that one would take the time to compose a fully formed personal communication with a beginning, a middle and an end has started to look like a relic of another age.

And I feel sad about it.

Two flowers grew in a garden one morn
One flower perfect, one tattered and torn
The beautiful flower was cheerful and glad
The other was woeful, forlorn and sad

“Why don’t you cheer up?” the first flower said
“Enjoy life! It’s not all discomfort and dread.”
The drab flower answered “Well maybe for you,
“But my life is dreary, and I’m feeling blue.”

The sun rose up high and then settled back down
And a young girl came back from her day in the town
She saw in the garden two flowers were there
And delighted at one, which was lovely and fair

That one she plucked from the garden with glee
And brought it inside for her mother to see
The fair flower proudly within their house shone
While the drab flower stood in the garden alone

The next day the plucked flower, no longer fair
Was dried out and gray and as dead as the air
Whilst the one in the garden was no longer sad
“Maybe,” it thought, “my life isn’t so bad.”