Actually, the future I described yesterday already exists for millions of people. But not for their eyes — for their ears.
Consumer audio equipment such as the Apple Earbuds do something that could be called “audio passthrough”. They take reality (in the form of sound waves entering the ear), digitize it, modify that digital signal to taste, combine it with a synthetic digital signal (eg: recorded songs), and then convert back to sound waves for the user to hear.
This lets those audio devices do some pretty impressive things. For example, they can selectively filter out or enhance sound in the world around you, let you hear only sound in front of you but not from other directions (like when you are talking with a friend in a crowded restaurant), or block out sudden loud sounds that might damage your ears.
The key is those four steps: (1) digitizing sound, (2) modifying that digital signal to taste, (3) mixing with synthetic sound, and finally (4) turning that mixture back into sound waves. This is exactly is exactly what you would want (but cannot yet have) in visual passthrough.
So why is that capability available for audio, but not for video? It’s because of Moore’s Law.
Moore’s Law states that computers get approximately 100 times faster every decade. And it turns out that the computer power needed to interactively process an audio signal is about 100 times less than the computer power needed to interactively process a video signal.
I realized back in the 1980s, when I was developing the first procedural shaders for computer graphics, that some of my colleagues in the field of computer music had gotten there a decade earlier. In the 1970s, they had already been adding synthetic noise to audio signals, modifying frequencies, applying filters that turned one musical instrument into another, and much more — all in real time.
As I learned more about computer music synthesis, I gradually came to understand that I was following in their footsteps. And I think that principle is just as valid today. If you want to understand future video passthrough, study present-day audio passthrough.
More tomorrow.