Fate, part 3

They say that fate is written in the stars
But all that have I seen are cloudy skies

We lock our dreams away in plastic jars
That dance on window sills, like fireflies

In half-remembered thoughts we lose ourselves
On hazy restless early winter nights

The long arc of causality

Today is, appropriately enough, Steve Jobs’ birthday. And so it is a fine day to talk about the long arc of technological causality.

Jobs would have appreciated how long the journey has been from my first shaders back in 1984 to Nvidia’s ever faster line of graphics processors to the use of those processors to train AI models — an exciting development which probably won’t reach maturity until at least 2034.

Which leads to an interesting question: What algorithms might somebody be developing right now, in 2024, which could eventually lead to a new kind of computer processor, which in turn could lead to a surprising and world-changing new use case? I wonder whether we would recognize the significance of such an algorithm if we were to see it today.

The Aha moment, part 10

A.I. is having quite a moment. Convolutional Neural Nets have led to large data models based on General Purpose Transformers, and the results seem astonishing.

A mere eight days ago, OpenAI unveiled Sora, and we are now starting to see text prompts turn into richly detailed and nearly realistic 3D animations. And in the coming years the technology will only grow more impressive.

Within another decade, talented creators will be able to create compelling feature length animations simply by properly describing them. It will still take great human talent and visual judgement to make a great movie, but the work of applying that human talent and judgement will have shifted from manual labor to higher level cognitive and linguistic skill.

By 2034 there will be another animated movie as compelling and groundbreaking in its own way as Fantasia was in 1940. But unlike the original, the brilliant and highly skilled people who make this new movie will do so by talking to a computer.

To recap:

In 1974 I first saw Fantasia. Inspired by that, in 1984 I created the first procedural shader language, with user-specified matrix operations at every pixel. By 1994, animated feature films were incorporating procedural shaders as standard practice.

By 2004 hardware accelerated shaders powered by graphics processors from Nvidia had become standard in computer games. Around 2014, those Nvidia processors began to be repurposed to train the convolutional neural nets of A.I.

Now in 2024, general purpose transformers are starting to create the first believable short A.I. movies. In another ten years from now, in 2034, it will be possible to create new A.I.-enabled versions of Fantasia simply by talking to a computer.

And we will have come full circle.

The Aha moment, part 9

Over the years, the ability of graphics processors to multiply matrices continued to grow at an exponential rate. This development allowed game designers to create ever more realistic scenes, with better animation, larger numbers of polygons and more realistic textures and lighting effects.

It would already have been interesting if this is all that ended up happening. But meanwhile, something else was happening. There was another application for all of those matrix operations which would become even more important than graphics.

Because in order to train the Convolutional Neural Nets that had been invented by pioneers such as Yann LeCun, as well as successive generations of A.I., you needed matrix math — lots and lots of matrix math. So that Nvidia hardware ended up finding a new purpose.

You have only to look at the history of Nvidia stock prices to see what happened next. A little less than a decade ago, A.I. began to undergo a revolution. It turned out that all of those graphics processors turned out to be the perfect devices for training ever more complex A.I. models.

And that leads us to where we are today. But the story doesn’t end there. More tomorrow.

The Aha moment, part 8

What happened next was very exciting. Driven mainly by the popularity of computer games, an entire industry sprung up around the challenge of making inexpensive graphics chips that could quickly perform the massive numbers of matrix and vector operations needed for shader programs.

By the late 1990s there were many such companies, but most of them failed, due to bad quality control. If you get too ambitious and ship even one generation of graphics chips that don’t work properly, your company goes out of business.

The one company that stuck around — mainly because it was more sensible and cautious in releasing new features — was Nvidia. As other companies fell by the wayside, Nvidia quickly cornered much of the market for providing hardware that could do fast 3D rendering for computer games.

As the years went on, and graphics chips became ever faster and more capable, real time shaders grew progressively more sophisticated, and computer games continued to gain ever new levels of visual realism. The principles that I had introduced in that first shader language were still being applied, but now with enormous speed and power.

But the most interesting part hadn’t even happened yet. More tomorrow.

The Aha moment, part 7

Interestingly, my idea of running a complete computer program at every pixel of an image ended up having some surprising consequences. This was also because I had introduced a particular kind of computer program to run at every pixel.

The programming language I implemented in 1984 made it very easy to do vector math and matrix math. I deliberately made no distinction between color vectors (with or without an opacity channel) and position vectors (with or without a homogeneous coordinate).

And any of those vectors could be transformed by matrices. Instead of the cumbersome notation of calling library functions to multiply matrices or to transform vectors, the programmer could just combine simple built-in operators.

As shader languages like RenderMan started to adopt my programming model, people became interested in using those shader programs to render images as quickly as possible. Eventually, special hardware was developed to do all that per-pixel matrix and vector math with great speed.

More tomorrow.

The Aha moment, part 6

To me one of the oddest things about all this is that a lot of people think about the noise function, and nobody really thinks about where shaders come from.

I am glad that I came up with the noise function, and it’s incredibly useful. I use it every day.

But my real contribution to the field from that time was introducing procedural shaders. When I presented that work during a session at Siggraph 1984, some people from Pixar were in the audience.

After the Pixar folks saw what a full shader language could do, they pivoted from Shade Trees (essentially a fixed expression parser) to what eventually became Renderman — a fully functional shader language that became an industry standard.

But the ripples spread even wider than that.

More tomorrow.

The Aha moment, part 5

Back then everything was expensive — not only computation, but also computer memory. So when I wanted to look at just a rectangle within an image, it wasn’t so easy, because in order to save space, images were all run length encoded.

So to get to the right set of pixels within each scan line of an image, I needed to do some work. First I had to find the right place in the run length encoding where my rectangle started, then I needed to chop something up in the middle, all of which was kind of a pain.

I remember staring at my screen and wishing that I could just put an “if statement” at every pixel. I could then just check each pixel to see whether it was inside the rectangle. That would be so much easier.

And that’s the moment when it hit me: If I could write a custom computer program that would run at every pixel that would be part of the content of the image — not just as part of the underlying system — then I could have as many if statements as I wanted. And I could also have loops, and variables, and procedures, and all of the other power that you get from a programming language.

That was the Aha moment — the moment when I invented the concept of a shader language, although I didn’t call it that — I called it a pixel editor. Of all of the things I’ve contributed to the field of computer graphics, coming up with what we now call shaders was by far the most important.

I spent the next month or two implementing my own interpreted language on top of Fortran, specifically to run at every pixel. It had many of the features we now associate with shader languages, like having vectors and matrices built right into the language, as well as lots of built in operations between vectors and matrices.

I then spent several happy months wearing my artist’s hat, creating the world’s first procedural shader based images (I’ve pasted one of them below). And in the years since then, shaders have become a mainstay of computer graphics, and have led to the rise of multi-billion dollar companies.

But it all started in that single moment, when I realized that I wanted to write an “if statement” at every pixel.