Ken's Blog

I talked to my class yesterday about this project of making it easier to learn how to visualize things in four dimensions.

We went over the principle of the 4D virtual trackball, then everybody got to try out the Oculus Rift, to experience for themselves what an immersive graphical world is like.

Even before we demo’d the Oculus Rift, I asked the class how many of them would be interested in exploring the fourth dimension. A lot of hands went up.

I consider this a very good sign.

Today it was raining in NY City as I walked to work. It was a rather light and pleasant rain. Except for one oddity.

As you undoubtedly know if you live in a big city, fellow pedestrians carrying umbrellas can be hazardous to your health. If people are not mindful, those ungainly objects, with their sharp and spiky metallic parts, can poke you in the eye.

Unfortunately, there is now a class of pedestrian that is very much not mindful. These are the people who hold their umbrella in one hand and their SmartPhone in the other.

It’s not so much that they are holding the SmartPhone — it’s that they are staring intently into it, oblivious to the world around them. When I see somebody carrying both an umbrella and a SmartPhone, I steer a very clear path.

I suspect that if one of these multi-taskers did poke you in the eye with their umbrella, they wouldn’t even notice. After all, they are not really present on that rainy street. Rather, they are living in the world of whatever or whoever is currently on the little screen in their hand.

It makes for an odd rain experience, and one can only live in hope that something will come along soon to wean people off this strange way of being.

Hmm, come to think of it, “odd rain” is an apt anagram for the situation. As is “in hope”, for that matter.

Just what they are anagrams for, I will leave as an exercise for the reader. 🙂

Many years ago, when I was a lot younger, I saw Star Wars for the first time. Like a lot of people, I was entranced by the adorable robots. I thought “wouldn’t it be cool if we actually had robots like that in our own lives?”

Being a practical young man, I set about thinking what might be a good way to do this. It occurred to me that the best way might be to cheat — to create the theatrical experience of a robot, without trying to really make the robot as smart as a human. I called my concept “House with robot”.

I’ve come to realize that all through my career I’ve embraced this sort of cheating as a general theme — from my work on procedural textures to interactive animated characters to robot swarms to zooming interfaces: The trick is to create the illusion that something impossible is going on, while pulling strings behind the scenes to support that illusion.

The basic idea of “house with robot” was that your house itself would actually contain the real smarts — via computers responding to sensors embedded in the walls and floors, microphones, heat detectors, and cameras that would watch you to figure out what you were doing. Your robot servants, which to you would appear to be really smart, would actually just be puppets, tele-operated by a behind-the-scenes computer array to create the illusion of an embodied intelligence.

The robot would be the face that you see, but the brains and much of the sensing smarts would really be somewhere else in the house.

This might seem weird, but it’s actually what your iPhone already does. Without the “cloud”, your pretty SmartPhone would be little more than a useless hunk of metal, plastic and glass. A distributed server located somewhere else is actually computing all those map routes, looking up facts for you, feeding answers to Siri, and doing most of the cute and amazing things that you associate with the magical electronic object in your hand.

Today I was talking with a friend who is working on writing a book about the future of robots in our lives. I described my idea from long ago, and he said that it still sounds like a good approach, even after all these years.

So if you want to have R2-D2 and C-3PO in your life, maybe you need to build a house, from the ground up, that does all the heavy lifting to make your robot seem intelligent. What you really need is a “house with robot”.

If you recall, when you look head-on at a virtual trackball, you can’t actually see that your movements to “spin the ball up and down” and “spin the ball left and right” are creating curved motion. In the picture I first showed (below), the red and green arrows look straight to us, even though they are describing curves:

That’s because we were looking edge-on at the third direction. It’s like looking at a spinning bicycle wheel when the bicycle is heading toward you.

Similarly, if you are looking at a sphere floating in space in front of you, and you are looking edge-on into a fourth dimension, then your view of any spin that rotates the ball into that fourth dimension is going to have the same problem: You are basically looking head-on at a spinning bicycle wheel.

To turn a floating sphere into a four dimensional trackball, you just need to count how many ways there are to spin a bicycle wheel edge-on. It turns out there are three, since each of our familiar three dimensions can spin into that fourth dimension.

If you add the three ways of spinning a sphere that only involve our familiar three dimensions, you get six spin dimensions all together:

It should be pretty clear from the picture what’s going on: You can rotate the ball in three directions that don’t involve that fourth dimension at all, just like a regular track ball (the red, green and blue arrows).

Or, you can rotate it in a way that swaps any of our three dimensions for that fourth dimension (the cyan, magenta and yellow arrows).

Recall that some rotations on a regular virtual trackball require moving your finger in a straight line on your iPad screen (as in the first picture above). Something similar is happening here.

Namely, on a 4D virtual trackball you make those last three rotations (cyan, magenta and yellow) by moving your hand in a straight line inside the sphere that is floating in front of you.

I find it thrilling that you can manipulate a 4D object while staying safely inside our little 3D world. We’re working on building this stuff right now at NYU, using the Oculus Rift and our own hand tracking software. I will let you know when we have something cool to show.

Changing up the theme for a day — for the sake of those who are not interested in 4D — I had a wonderful thought today.

Imagine that a hotel owner got a hold of the names of various rock and roll groups, and decided to create a theme hotel, where each room honors one of the names on the list.

But here’s the kicker: The hotel owner doesn’t know anything about rock and roll, and in fact doesn’t realize that these are the names of music groups.

I imagine such a hotel would be a very interesting. One room would be filled with little stones that roll around, and another would contain giant sculptures of beetles. Yet another would feature crickets. Down the hall would be a room where all the furniture was made of lead, and shaped like little zeppelins.

I had some trouble with “Radiohead”, until I realized that a room could be filled with portaits of top executives from the radio industry. These would be the radio heads, of course.

I’m sure I’m missing lots more. Does anybody else have any suggestions?

One piece of advice: I’d stay away from the Heavy Metals. After all, who’d want to stay in a hotel room build around the theme of Anthrax or Poison?

I suppose I could just tell you what sort of things in four dimensions correspond to yesterday’s post. But that wouldn’t be much fun, would it?

Instead, let’s think about this a bit. Here we had a virtual trackball that looked like a circle. It had some straight lines on it, but when we looked at it from the side, it turns out those lines were actually curves. That’s because those straight lines were, in fact, painted onto the front of a sphere.

So what looked at first like a flat disk was actually an image of the outside surface of a sphere.

What if we bring this whole discussion up to one higher dimension? At first we see something that looks like a sphere floating in space. But in fact it is something more.

The sphere we are looking at is the “front” side of a four dimensional sphere. Anything that is drawn inside our sphere is actually being drawn onto the outside of this four dimensional sphere.

If any of this seems mysterious, just go back to the three dimensional case: Something that looks like a flat disk is actually the front face of a sphere.

We’re just taking the same argument to one higher dimension: Something that looks like a sphere is actually the front “face” of a four dimensional sphere.

That’s a pretty weird idea, so I’m going to give it a day to sink in before going on.

In order to understand what’s going on with a 4D virtual trackball, let’s take a more careful look at a 3D virtual trackball.

What you see is something like this:

You can move your finger up and down (ie: in the direction of the red arrow), or from side to side (ie: in the direction of the green arrow), or around the circle (ie: in the direction of the blue arrow).

In each case, you are causing a sphere to rotate. This might seem mysterious to a denizen of Flatland, who does not have any direct perception of a third dimension. After all, the first two gestures look like straight line movement — very different from the circular movement of the third gesture.

Fortunately we are, in fact, 3D creatures. Which means we can employ a great 3D power-up to see what’s going on: We can rotate our view of the object:

Now we can see, viewing from an oblique angle, that all three of the movements are actually circular arcs. The only reason the red and green gestures look straight is that the user cannot directly perceive the third dimension, since the z direction is perpendicular to our view. The red and green paths actually curve toward us, around the front of the sphere.

Which means that for a 4D trackball, some things that look like straight line gestures are actually going to be curved — in particular they will curve around the “front” of the 4D trackball.

Tomorrow we’ll see just how many such curves are needed for this.

It’s very cool that we can rotate things in three dimensions just by playing with a circular disk. After all, a disk is flat!

What’s also cool is that we can rotate things in four dimensions just by playing with a sphere. It’s not even that difficult to do — but it can be a little difficult to understand.

Just as you can rotate 3D objects by putting a finger down on a disk on a computer touch-screen, then moving the finger around inside the disk, there’s an equivalent action for a spherical region floating in space in front of you.

Namely, you can put your hand inside the sphere, pinch your fingers together to “grab” the sphere, then move your hand around inside the sphere (then unpinch your fingers to “ungrab”).

One way to think about this operation is that the volume inside the sphere is kind of like the area inside a circle, except in one higher dimension. As you move your pinched fingers around inside the sphere, you’re essentially rotating a 4D trackball.

But how do your movements inside that floating sphere translate to rotations in four dimensions? We’ll get to that tomorrow.

Continuing yesterday’s thought … if you were looking at a 4D object in front of you — holographically, the way Tera presented it to Trey — how would you rotate it, to see it from different points of view?

First let’s look at how people rotate 3D things on computers. One common way uses something called a “virtual trackball”. It’s basically just an image of a trackball, seen from above. When you drag your finger over the image, it does what a real trackball would do.

As you can see in the above image, a virtual trackball “rotates” around the vertical axis when you drag left and right (the red arrow), about the horizontal axis when you drag up and down (the green arrow), and about the axis that comes toward you when you spin around its edge (the blue arrow).

So here we have an example of a completely two dimensional thing that lets you rotate stuff in 3D. Perhaps we can do something similar for rotating stuff in 4D…

Sorry to switch the topic, but this evening I spent a delightful forty minutes with a friend, lost in conversation while walking through the rain in Paris.

If you’re going to walk through the rain anywhere, I highly recommend Paris. 🙂

Going back to the question of four dimensions… If you were a two dimensional being in Flatland, and you really wanted to understand 3D, you would want the most high quality visualization you could get of that mysterious higher dimensional world.

In that spirit, it seems to me that it would make sense to create the most vivid possible representation of our four dimensional object — even if it is impossible for such an object to exist in our world.

However we choose to simulate this four dimensional object, our simulation should appear before us with the same sense of “presence” as a real object. And if we reach out our hand, we should be able to touch it.

Sounds like a great opportunity to build a really good virtual reality experience.