Ken's Blog

Today a friend – and reader of this blog – told me that when I describe how we did the Wild Things test, I should go into more detail about the technology. I objected that many people who read this are not technical in that way. But he pointed out that it would be a shame to water it down, when there are a number of people out there who really want to know the techniques. So I’m going to go for it, but before I do that I’m going to make sure you get the background, so that everything is sufficiently motivated.

First, a little history – let’s go back in time a bit, to the making of TRON. There were a lot of brilliant people behind the visual ideas in TRON. One of them was the art director, Richard Taylor. Richard was coming to TRON fresh from having worked for the legendary production company Robert Abel and Associates. While at Abel, Richard had perfected a technique he called the “candy apple glow” – which became a kind of signature look for the many award winning commercial spots created by Abel and Associates through the years.

The basic idea of the candy apple glow was to take a white silhouette image of an object, blur the hell out of it, and then slap the image of the original object on top. The result looked like a kind of corona surrounding the object. Edges remained crisp and well defined, but the entire object would seem bathed in an unearthly angelic halo.

It was a very successful look, much sought after by ad agencies, but rather difficult and expensive to achieve. In those days, the only way to composite images together was to run them through an optical printer – a big, cumbersome and expensive machine that reprinted film from one reel onto another, allowing you to apply a simple special effect each time you ran the film through the printer. In order to make the candy apple glow, quite a few steps through the optical printer were required, each one requiring another run of film through the optical printer.

First you needed to make a silhouette image of the object, white against black, then you needed to print a blurred version of that white against black silhouette. Then you needed to subtract the original silhouette from the blurry one – which required another run through the optical printer. Then you had to print the original image, adding it to the glowing white outline – yet another run through the optical printer.

The results looked great, but all those multiple passes through the optical printer were slow and expensive, and film costs ended up being very high – especially if you made a mistake anywhere in the process and had to do it all over.

So one day Richard said to those of us at MAGI, “Could you create the candy apple glow look in computer software?”

That turned out to be a fateful question…

3D computer graphics is a great way to make things look real. After all, the techniques under the hood are basically simulating the physics of how real cameras capture real life: the shapes of objects, how a camera moves around a scene, the way light shines on surfaces. But there can be such a thing as being too realistic.

That’s one of the problems we were fighting in our Wild Things test. Here we had all of this fancy computer software that we had painstakingly tuned to convince people they were looking at reality, and now what we wanted was to convince people they were looking into a magical storybook. Not the same thing at all.

Fortunately, we had learned while making TRON that when you’re combining computer graphics with other things (like Jeff Bridges in a weirdly glowing spandex unitard), the trick is to modify the look of everything, so that it all meets in the middle. In the case of TRON the live action footage of the actors was deliberately given an eerie grainy look, and made to look hand-tinted – like something out of Fritz Lang’s “Metropolis” – and this same processing was done to the computer graphics backgrounds.

That, together with the “everything has red or blue glowing lines” motif, was very effective in marrying the foreground actors to the background computer graphics. The cranked-up graininess and stylized color palette masked any differences between computer graphics and physical props, and made you believe you were looking into some sort of consistently strange alternate universe – very different from the brightly lit, almost clinical, look of the scenes in TRON that were located in the “real” world outside the computer.

For Wild Things we were going for a very different look, but the principle was the same: marry foreground and background by visually stylizing both the computer graphics and the hand-drawn animated characters in a mutually consistent way. Today I’m going to talk about how we made the computer graphics backgrounds look like something out of a storybook. Then tomorrow I’ll talk about how we made the animated characters look like they were three dimensional.

When you look at a real object lit by a light source, you immediately see that the light only directly affects the portion of the object that faces the light. The portion of the object that faces away from the light source remains dark. So generally cinematic lighting revolves around using a bright light in the front (the “key light”) to highlight the roundness of each shape, and a softer backlight (the “fill light”) to give definition to the shape’s silhouette. Using such a technique, the bedknobs on Max’s bed might have looked something like this:

That looks very nice, but it doesn’t look particularly magical. When you look at Maurice Sendak’s drawing style, you eventually realize that he had a wonderful trick of keeping you a bit confused about where the light comes from. In his drawings, light seems to come from everywhere and nowhere all at the same time. So to make this test, I modified our shading system in several ways. For one thing, I changed it so that the highlight on an object didn’t need to be in the right place. You could make an object brighter on the left, while having the highlight seem to come from the right.

I also added an option to change the way objects in our 3D system react to light. Instead of only the part facing the light source getting brighter, I rigged it so that the light could seem to “wrap around” the object, bathing it in a softer light. This made the objects seem to glow a bit, like they do in Sendak’s illustrations. Finally, to capture the deliciously ominous feeling of the deep shadows Sendak adds to his illustrations, I added a feature that’s really impossible in reality – lights that can make things darker, instead of brighter. Using all of these techniques together, you could literally paint objects with light. So the bedknob on Max’s bed ended up looking more like this:

The difference is subtle but essential. The first bedknob looks like a simple but plausible 3D object. The second bedknob looks both real and not real, all at the same time. When an entire room is lit this way, the result looks not so much like a real place, but rather like an imagined idea of a real place. In other words, like the storybook version.

Here is a frame from the final test. With these lighting and shading tricks in place, the 3D scene seems to be slightly surreal, with a sense of drama and otherworldliness that would be hard to achieve using “physically correct” lighting:

The question still remained of how to take Glen Keane’s wonderful drawings of Max and the dog – which were really just pencil lines on paper – and make them appear to be rounded and three dimensional, so that they would look like they belonged together with the 3D backgrounds. And of course there was the little problem of how to make these pencil drawings cast believable shadows into a computer graphic scene.

Good topics for tomorrow!

The process we came up with to do the Wild Things test ended up being called Synthamation. In concept it was very simple – but God was in the details.

It all started with an animatic provided by the Disney team – a shot by shot view of the 3D background, created as a series of hand-drawn sketches. Then our artists at MAGI (Chris Wedge and Jan Carlée) worked from this animatic to create 3D models of everything – the room, hallway, bed, staircase, the little table with the lamp on it seen in the last shot.

Chris and Jan then computer animated a camera path through their 3D scene that matched the successive viewpoints of the hand drawn animatics. The result was rendered out frame by frame, not as a fully shaded background scene, but rather as a series of computer generated line drawings (called a “pencil test”). These pencil test frames of the 3D animated scene were printed out on big sheets of paper, so that the Disney artists could use them as a guide to rotoscope the frame by frame animation of the characters of Max and his dog.

Rotoscoping in those days was a process whereby an animator draws on paper while looking through a big piece of slanted glass. The animator sees the paper containing his own drawing through the glass, while simultaneously seeing another piece of paper containing a pencil test frame reflected by the glass. To the animator, it looks as though the two pieces of paper are superimposed on each other. So he can always see the computer generated pencil test of the background image, but the drawing he makes actually goes onto a clean white sheet of paper.

Here is where our team came up with a clever trick. When we rendered the pencil test, we added dummy versions of Max and his dog. These were only to guide the animator – they would not appear in the final animation. These dummy versions were really simple – each was just composed of a few simple shapes, as you can see in the image below. But that was enough to show the animator (in this case, Glen Keane) where to draw the characters for each successive frame of the animation.

The dummy characters also served another, more subtle purpose. In the final animation the hand-drawn characters are moving all around the scene – getting closer or further away, or running behind or in front of the staircase. The dummy characters allowed our programmers to know how far away to place the animated characters when they were finally composited into the scene. For example, in the final animation Max’s dog runs behind the staircase when he runs down the stairs – just like the dummy version of the dog.

When the animator was done, he had created various big stacks of drawings. Some of these drawings were of Max and some were of Max’s dog. We digitized each drawing into the computer at high resolution, using an Eikonix flatbed scanner. Once we had all the pieces scanned into the computer, all that remained to do was combine those pieces together in software, and add shading and lighting.

That last bit is not as simple as it might seem. We wanted the hand-drawn animated characters to appear not flat, but rather rounded and three dimensional – as though they were being lit along with the rest of the 3D scene. We also wanted them to cast shadows onto the floors and walls of the 3D computer generated background. And the background itself needed to have a kind of magical storybook appearance. To do all that, we needed to invent a few new techniques.

But that’s a topic for tomorrow.

Seeing Spike Jonze’s excellent film version of Maurice Sendak’s “Where the Wild Things Are” got me thinking back to the time that I helped get John Lasseter started in computer graphics.

I know that sounds completely weird, little old me helping to get the director of “Luxo Jr” and “Toy Story” into computer graphics, but it turns out to be true. It all happened in the months after Disney’s release of TRON, which unfortunately had not been a raging box office success. Nonetheless, since I was the young “coming up with crazy new ways to do things” guy at MAGI SynthaVision – the Westchester based computer graphics production house where we did the cool light cycles, game grid and more for TRON – I was flown out to Buena Vista, California to meet with Disney’s brilliant young animation director John Lasseter, where the two of us brainstormed ideas.

I was itching to try out some new techniques for making flat-shaded characters look rounded and 3D, and other techniques for making 3D graphics look more like hand illustrated storybooks. Meanwhile John wanted to do something that combined hand-drawn characters with the sorts of 3D worlds he’d seen in TRON. Together we came up with a suitably harebrained scheme.

I would lead a team at MAGI back in New York, where we would create a shaded 3D background animation, while John supervised a team over in California – with the great Glen Keane doing the character animation – to create hand-drawn animated characters that the MAGI team would magically integrate into the 3D backgrounds, with matching lighting, shadows and camera moves. None of the commercial 3D graphics software everyone now takes for granted existed back then, so we pretty much had to come up with new and sometimes unexpected ways to do everything.

Fortunately, I was working with an incredible group of fellow young turks, including Josh Pines, Christine Chang, Chris Wedge, Jan Carlée and Carl Ludwig. Every one of these people went on to amazing careers. Josh ended up going to ILM, where he revolutionized the process whereby film and computer graphics are combined together, so that you can’t tell which is which (for which he won a Technical Academy Award). Christine ended up at Don Bluth Studios in Ireland, and Chris, Jan and Carl went on to co-found Blue Sky Productions, makers of “Ice Age”, “Robots” and other visually stunning films.

Our Wild Things test was delightfully successful, and John became completely smitten with the possibilities of computer graphics. There is a video of our little test up on YouTube. Unfortunately, soon after the test was finished there was a political regime change at Disney headquarters, and John Lasseter – the fair-haired boy of the outgoing regime – ended up getting fired, for the crime of continuing to push this weird new computer graphics stuff (hard to believe now but true). Fortunately, John was able to take our little “Where the Wild Things Are” test over to Ed Catmull at LucasFilm as a calling card, where Ed (who may be the smartest guy I know) immediately brought him on as creative director of what would soon become Pixar. The rest, as they say, is – well, you know.

Some of the techniques we came up with in doing that test were extremely cool, and I don’t think they’ve been properly described anywhere. So I’m going to take the next few days to describe them here. Think of it as a rare window into the old “wild west” days of computer graphics, when computers ran slow, programmers ran fast, and we all just made it up as we went along, with nothing to go on but some pixels and a dream.

Erwin Schrödinger introduced his famous “Schrödinger’s cat” thought experiment to illustrate the apparent absurdity of one of the key implications of quantum theory. Namely, its implication that something could simultaneously exist and also not exist. Basically, down at the quantum level, a particle can remain in a quasi-state of existence, both existing and also not existing. The particle stays this way until it is observed by an outside system. At that moment it instantly “snaps” to one of the two states.

Schrödinger’s complaint was that this can lead to absurd outcomes, since you could easily tie a macroscopic object – say a house cat – to the fate of a single quantum particle (recipe: place the cat in a sealed box with a geiger counter; when the counter detects a single random quantum event, kill the cat). Quantum theory states that the cat is literally both alive and dead at the same time. Until, that is, somebody opens the box, at which point the cat instantly snaps to one of its two quasi-states: It becomes either a fully alive cat or a fully dead cat.

Yes, this sounds absurd, and people not familiar with quantum theory often respond by saying that we’re just describing the probability that the cat is alive or dead at any given moment. In fact, they say, the cat must always be completely alive or completely dead. But that turns out not to be the case. Strange as it seems, it turns out that Schrödinger’s objection was wrong – quantum theory’s prediction has been experimentally verified to be true. If you run various experiments with actual particles, the “cat is either completely alive or dead” assumption gives you the wrong answer. If you assume this crazy sounding “quasi-state” of an object both existing and not existing at the same time, the results you get match the experimental data perfectly.

Joel and Ethan Coen’s recent film “A Serious Man” is actually a treatise on this very subject, in disguised form. It starts with a reference to Schrödinger’s famous thought experiment, and then proceeds to show – in a very elegant fashion – that even in the domain of human actions, an object can be in a quasi-state of simultaneously both existing and not existing, up until the moment an observer forces the question of whether the object exists or not. At which point the object instantly snaps to one of these two states, as though it had been in that single state all the time.

I won’t spoil the movie for you by saying any more (many have not seen it yet – and I suspect it has not yet been released in various parts of Europe), but I wanted to pay tribute to a moment of cinematic genius: A moral fable that transposes one of the most difficult concepts of quantum theory into human terms, with perfect clarity.

When you see the film, see if you can spot what the “quasi-existing” object is.

The other day, for the first time ever, I encountered a wrong-way Oreo. For those of you who don’t know, that’s an Oreo cookie that has one of its two dark chocolate wafers somehow turned around, so that its engraved outer side ends up on the inside, pressing inward to form a tell-tale impression, in perfect mirror-reverse, upon the snowy white cream filling.

I hadn’t been expecting it. In fact, I hadn’t even been aware that such a thing exists. Perhaps there are people who go around and speak of wrong-way Oreos, swapping tales of this arcane mystery in the same hushed and knowing tones they use when speaking of Bigfoot sightings or the alligators that dwell in the sewers of New York. Not that I have ever been in such a conversational group. Until now.

Today I asked various people if they had ever seen a wrong-way Oreo. My friend Charles said he saw one once, a few years back. Several other people reported having seen one as well. Charles has the theory that some part of the manufacturing process involves the chocolate wafer dropping downward, and that every once in great while a wafer lands the wrong way. He may very well be right.

But as I contemplated my oddball Oreo, I couldn’t help thinking there might be some deeper meaning here. Was this perhaps some sort of sign or omen? And if so, why was I chosen to get this cookie on this particular day? Would it still have counted if I had just eaten the cookie without ever looking at it? Or would fate then have conspired to place another wrong-way Oreo in my path?

And if fate were to deliver more wrong-way Oreos to me, what would happen if I were so oblivious that I just kept eating the darned things without ever noticing? Would fate then need to keep feeding me cookie after cookie, hoping against hope that one day I would become less oblivious? Would I one day find myself mysteriously eating entire boxes of Oreos, consuming vast quantities of the things until I became as round as – well – as an Oreo cookie?

These are metaphysical questions, far out of my league I am afraid. My feeble brain can contemplate only one wrong-way Oreo at a time. But even one cookie can have significance. Am I, perhaps, one of the few lucky humans, chosen by alien invaders, set apart by this secret sign from billions of less fortunate earthlings? I can envision a day dawning, after our planet’s ignominious defeat at the hands of the Lepusian space invasion force, perhaps sometime after the dust has settled, when the broken slag heaps of what had once been great earth cities lie smoking in ruins, and the once mighty suburbs of New Jersey have been reduced to desolate wastelands by beams of phase disruptor particles from the Lepusian imperial mothership. The few dazed remnants of a defeated human race slowly emerge, stunned, from out their hiding places, only to be picked off by precision laser fire from the dreaded roving lepudroids. On that day I shall stand triumphant, proud and free, ready to take my rightful place as a citizen of the galactic empire, holding my wrong-way Oreo cookie high for all to see, my ticket to a new world.

On the other hand, there is a chance that might not happen.

Weighing the promise of one day living a life of fabulous adventure roaming the galaxy far and wide in search of new civilizations, against the prospect of eating an Oreo cookie now, my internal struggle was brief.

Reader, I ate it.

Idle thoughts in random moments
Drift upon the vacant air
They flit about in lazy circles
Floating here, alighting there

Nothing’s ever truly lost
All the thoughts we’ve had remain
To fill an evening with regret
Or echo some forgotten pain

In woven mists of tender dark
They haunt the hollows of your sleep
In dreams they whisper soft and low
Of all the secrets that you keep

But in the mornings, then they gather
Memories of smoke and lace
Forming haloes ’round your head
To fill your day with light and grace

In “Jurassic Park” the lawyer character was eaten by a Tyrannosaurus Rex while sitting on a toilet in an out house. In the theatre where I saw the movie, everybody cheered.

Isn’t there something about this scenario that bothers you? In our hearts we find room for so many different ideologies, ethnicities, ways of thinking and being. Hell, last year Tom Cruise played a sympathetic Nazi, ferchristsakes. But lawyers? No, not lawyers. Those folks be dinosaur bait. When bad things happen to them we laugh, we cheer, we run around the room and do the antler dance. We wait with barely restrained glee for terrible events to befall lawyers everywhere they may appear in pop culture, whether it be movies, books, theatre, comic books or bubble gum cards.

But what exactly is their crime? Why the intensely focused cultural hatred toward our advocationary class?

I submit that we are actually engaged in a collective act of deflected self-hatred. We use lawyers to do our dirty work, and then we blame them. Heaven forbid we should blame ourselves. Particularly in America, where lawsuits are only slightly more common than bathroom breaks.

We sue each other in our courts of law, and then go out for drinks together afterward. When questioned, we shrug our shoulders ruefully and say “well, you know, those lawyers.” It’s no wonder they are paid so well. They are our ethical buffers, cleaning up the Stygian stables of our collective litigious excess and then conveniently taking the blame.

Not to put too fine a point on it, lawyers are our moral sanitation workers.

The other day I was invited to a party where almost everyone was a philosopher. I don’t mean amateur philosopher, armchair philosopher, or reflective soul with a philosophical bent. I mean they were professional philosophers – people who do this for a living. Many of them were connected with the NYU Philosophy Department (one of the top philosophy departments in the world, as it turns out) and others were colleagues and collaborators of these folks from other institutions of higher learning around the world.

I found out, in the course of conversation, that a rather high percentage of these people focus on questions surrounding “theory of mind” – in which one looks at questions on the order of what is a human mind, what is consciousness, what is thought, what is self?

The friend/colleague who invited me to the party is something else – a psychologist. Therefore he looks at theory of mind questions from a different angle, one more related to the sorts of questions we ask in computer science: How the mind operates from a somewhat cybernetic perspective, as an extremely advanced sort of computational device. If I understand correctly, it seems that an essential difference between the philosophical and psychological views of humanity come down to the question of “can we build one?”

I don’t mean can we build one now. Enough is already known about how the human brain functions to make it clear that in 2009 there is simply not enough computational power in all the world’s silicon chips to replicate the functioning of even a single brain. But of course that might not always be true. So psychologists are tempted to look at a time in the future – perhaps 50 years from now, perhaps 500 years from now – when something on the order of the brain’s level of functional complexity can be replicated in silico.

Philosophers, unlike psychologists, are not exactly interested in the mechanism itself, but rather in what that would mean. Would we be replicating the essential nature of the brain, the aspect that we think of as humanity, and if so, would that mean we can codify humanity the way we currently codify computer software?

I also found that that both psychologists and philosophers ponder the future implications of this question in a very specific way: If human brain functioning – “thought”, if you will – could one day be replicated in computer circuitry, then could those future electronic humans make their own cyber-progeny, second generation artificial thought machines? And would their progeny then go on to make third, fourth, fifth generation machines, ad infinitum?

And if so, at what point would the descendents no longer be recognizably human? At what point would such creatures cease to feel any need to keep us silly humans around, even as quaint biological specimens of an outdated ancestral brain?

Here’s the kicker: On the above subject, it seems that there are “optimists” and “pessimists”. The optimists believe that it is indeed possible to create such generative species of artificially intelligent creatures. The pessimists believe that it is highly unlikely such a thing will happen in the foreseeable future.

The friend who invited me to the party is an optimist, and so he is quite morose on the subject. He believes it may be only a matter of time before our human species is replaced by an uncaring cyber-progeny that has evolved beyond our limited powers of recognition, a meta-species that will ultimately cast us aside altogether, once we no longer serve its unfathomable purposes.

I, on the other hand, find that I am a pessimist on the subject. And so I remain quite happy and carefree, fascinated as I may be by the gloomy and dire predictions of my sad friends, the optimists.

Recently in a conversation with a group of colleagues, I complimented one colleague on his ingenious way of putting together simple things to make remarkably new and innovative discoveries. Graciously he deflected attention from himself by talking about people who had been tinkerers as kid. He pointed out that most individuals who grow up to be inventors started out in childhood, and probably had some experience performing dangerous experiments with chemistry sets or some equivalent.

We all mused that perhaps there would have been more such people in the world, but that some of the more daring young would-be inventors had actually succeeded in blowing themselves up at an early age, and had therefore never made it out of childhood alive.

At this point the conversation took a curious turn, as each person related something they had done in their experimentally inclined youth that might have put them at risk.

When it was my turn, I talked of the day – I think when I was somewhere around seven years old – that I became curious about the electrical outlet, and wanted to find a more “hands on” way of exploring its properties. I did this by taking a wire coat hanger from my parents’ closet, bending it into a U shape, sticking one end of the hanger into one terminus of the 120V wall socket, and then gingerly poking the free end of the wire into the other hole, to see what would happen.

At this point in the story my colleagues were all looking at me with concern. Possibly they were wondering why I was even now alive to tell the tale. “Well?” one of them asked, “What happened? Did all the lights in the house go out?”

I explained that the lights had managed to say on, but that a spray of very impressive sparks had immediately shot out of the wall outlet, creating black scars on the wooden floor of my bedroom. As soon as the sparks started to fly, I had pulled the wire back out, discretion having finally overcome curiosity within my young brain.

I don’t think I understood back then just what kind of fire I was playing with. I realized only when I was older, looking back on that experience, that what had saved me was the fact that my body was never actually in the path of the high voltage electricity. The short circuit had gone entirely through the wire – an excellent conductor – rather than through me. Had I used two coat hangers – one in each hand – instead of the one, I would have been very efficiently electrocuted, and that day would have marked my final experiment.

I’ve never told my parents about this little escapade – I think it would only have worried them. The floor in that room of their house has long since been covered by carpeting, beneath which I suspect one would still find the tell-tale burn marks on the floor near one electrical outlet, evidence of the early career of a very lucky young scientist.