Ken’s Blog

When you work in developing new forms of media, you can’t help but keep one eye on the future. Many of the things you work on will amount to nothing, but every once in a while you will get it right.

It’s interesting to look back through history to identify times when people “got it right”. There are so many examples: movable type, the gramophone, radio transmission, projected cinema, binary digital encoding. The list goes on and on.

I wonder how many of the people who developed those innovations realized that they were shifting the very course of history. I wonder how many people today know when they are doing so.

Every once in a while someone is fortunate enough to come upon a new way of communicating that continues to echo down through the ages. What better privilege can there be, than to be able to bestow a gift to our children’s children’s children?

I wonder how many filmmakers in the silent era thought about the idea of talking movies. From a design perspective, we now know that the addition of synchronized sound transforms cinema into a radically different medium.

But we know this only because people actually went through the process of making talkies. Intrepid filmmakers experimented, tried out different things, failed and then tried again.

The technology of sync sound is very different from the design of movies created for sync sound. Knowing such a thing is possible, but having no experience of it, is like being on the threshold of a dream, but not having the dream itself.

We are now in a similar place with shared augmented reality. We understand the technology, and we have some theories about what people will do with it. We’ve even seen some cool demos.

Yet the real work of designing AR experiences for ordinary people has not yet really started. We are standing on the very threshold of a dream, but the dream itself has not yet begun.

When we try to describe the new immersive cinematic medium that our Future Reality Lab is creating, exemplified by CAVE at SIGGRAPH 2018, it’s difficult to properly convey the idea to people who haven’t seen it.

We can’t just say something like this: “It’s like going to the movies crossed with theater crossed with being in a video game.” If we do that, people usually get the wrong idea about what we’re up to. They think that it’s a video game, or that it’s like Facebook Spaces or High Fidelity, when it’s really very different from any of those things.

I find myself imagining a futurist in 1926, in days of future past, trying to explain the soon to be released film The Jazz Singer. Maybe this visionary could say: “It’s like a movie, but with sound.”

But that wouldn’t work, because movies already had sound. Pretty much every movie you went to back then had a score, and a very talented organist playing along.

Maybe the futurist could say: “It’s like a movie, but people are talking.” But that wouldn’t work, because actors already talked in movies. And the intertitles were right up there on the screen, to tell you what they were saying.

Maybe it would have worked best if the futurist had said: “It’s like a movie crossed with radio.” That might have sounded completely crazy to people in 1926, but they probably would have gotten the idea.

We need to find our equivalent, for what we are doing in 2019, to “It’s like a movie crossed with radio.”

Come Saturday morning, I find myself sadly reading about the death of Andre Previn. Which makes me think of Dory Previn, which makes me think of the song Come Saturday Morning, for which she wrote the lyrics, which makes me think of the movie The Sterile Cuckoo, for which that was the theme song, which makes me think of Liza Minelli, who starred in that movie.

The film is a tender coming of age story in which Liza plays a young woman who falls in love with a man who seems also to be in love with her. But then it turns out he’s more attracted to men than to women.

Liza then went on to superstardom in Cabaret, a musical drama set in Germany between the Wars in which she plays a young woman who falls in love with a man who seems also to be in love with her. But then it turns out he’s more attracted to men than to women.

During the years when both these films were produced and released, Liza Minelli was married to singer/songwriter Peter Allen. He seemed also to be in love with her. But then it turned out he was more attracted to men than to women.

Once, in my younger days, I took a girl I liked to a Peter Allen concert. At some point in the middle of the concert the young man sitting on the other side of me, to my great surprise, made a pass at me.

I thought it was very rude of him. Regardless of his theories about my preferences, it was obvious I was there with someone else.

Now, many years later, as I look back on the semiotics of all this, a different theory comes to mind. Perhaps that young man wasn’t insulting my date, but subtly complimenting her.

Perhaps that was just his roundabout way of suggesting that the young woman beside me was the Liza Minelli of our generation.

Our final step adds the real magic sauce: Simulating both wind and coriolis effects. Both of these will be done via shameless cheats.

In fact, we are not going to simulate the physics at all. Instead, we are just going to simulate the appearance of wind and coriolis effects.

We do this by adding just two lines to our shader code:

   T.z += .1 * uTime;    // MOVING TEXTURE FAKES WIND.
   T.y += noise(.5 * P); // UP/DOWN MOTION FAKES CORIOLIS.

The first line of code simply slides the entire texture perpendicular to the screen. This has the effect of creating smooth yet unpredictable changes over time — exactly the visual impression we would get from wind forces across the planet surface.

The second line of code makes different parts of the texture travel up and down in latitude as the texture slides past the planet. The wispy cloud texture flows upward in some places, while flowing downward in other places — exactly the visual impression we would get from coriolis forces across the planet surface.

We have now rebuilt the complete cloud-roiled planet texture, step by step. The code looks slightly different from the original code, but it does exactly the same thing. You can see the live animation, together with the editable code that generates it, by CLICKING HERE.

A cloudy planet where the clouds don’t move is boring. After all, one of the fun things about clouds is that they move and shift around and change shape over time.

The next step doesn’t look very different from the previous one, except that now it is starting to move:

So far the movement is very simple. The cloudy planet is just spinning.

But after all, as we know from real life, a spinning planet leads to coriolis forces, and those forces result in far more interesting cloud movements.

If you look at the code, you can see that all I’ve done is use sine and cosine functions to make the x (left-to-right) direction and the z (back-to-front) direction spin around each other. You can see all of this running live, together with the code that generated it, by CLICKING HERE.

Today we deal with the problem of the cloud cover not being wispy. The trick we do that is to combine textures together.

Rather than just use a turbulence texture, we use that turbulence texture to highly distort a stripe pattern (like the way things look wavy when seen through a mirror that isn’t flat). The result doesn’t look like stripes, because those stripes are so highly distorted, but the result ends up having the wispier look we were aiming for:

In the code, the stripes are created by a sin function. I put the new code right after the old code, so that you can compare them to see the difference. You can see the live version, with the code that generates it, by CLICKING HERE.

Fortunately, we can do better than simple noise texture. Back when I came up with noise, I also used it to build some other interesting functions. One of them was the turbulence function, which creates a kind of fractal texture.

Here you can see I’ve replaced noise with turbulence:

Now the texture looks a lot more like cloud cover. But it doesn’t yet have that delicate wind-blown wispy quality, and it doesn’t yet animate. You can see it live, together with the code that generates it, by CLICKING HERE.

The first thing we can do is throw on a simple noise texture to use as our cloud cover. As you can see, that by itself is pretty boring.

It doesn’t really look anything like cloud cover. For one thing, it’s lacking that wonderful wispy quality that clouds have. For another, it’s not animated.

In order to do that, we are going to be a lot more subtle in our use of procedural textures. Meanwhile, you can see this example live, together with the editable code that generates it, by CLICKING HERE.

I thought it would be interesting, for those who are interested in these sorts of things, to do a step by step breakdown of how I made that animated cloud texture for the cloudy planet I showed the other day. I plan to do that over the next several days.

Taking things step by step, first we start with a planet that has no cloud cover at all. All the lights are there, and it’s got the right watery surface color — just no texture.

This is where we will begin. To see this running live together with the code that generates it, CLICK HERE.