Ken's Blog

This afternoon I plugged a midi keyboard into my MacBook. I connected it to a Web-based audio interface, used some samples of piano notes that I had lying around, and was soon able to play music in my Web browser.

Then I added that code to my WebXR system, so I could use my music keyboard to create graphics in XR. At some point I realized that I could just plug the midi keyboard directly into my Meta Quest 3, using its Web browser based midi interface.

All of a sudden I can play the piano while making 3D graphics animate in the air above the keyboard — not in VR, but in video-passthrough XR. So I can still see my room, and I can see my real hands playing music on the keyboard.

But now I can also create magical 3D graphics in mid-air just by playing the piano. Like Harry Potter, if he were a musician.

As humans, we have a collective belief in a single shared reality. We trust the evidence of our senses — the Sun moves across the sky, a kitchen table seems firm and solid.

But since we are all human, we all share the same perceptual limitations. If I can’t perceive something, for example a four dimensional object, then for the most part you can’t perceive it either, since we were all born with pretty much the same perceptual equipment.

Which means that there is very likely a large universe out there — or perhaps right here with us — that is outside of our shared understanding of what the universe is. Which leads me to two questions:

(1) How can we all collectively perceive things, perhaps with the use of the right technology, that are outside of the range of our natural human perception?

(2) If we manage to do that, will be be able to use that enhanced perception to build a more true and accurate model of the Universe around us?

I had a meeting today with some NYU colleagues in which we talked about the future of extended reality (XR), and what our research can do to help make it happen. Afterward, we all went out to a nice dinner at a nearby restaurant.

At some point during the dinner I turned to the people next to me and said “I think I know how we will be able to tell when XR has finally arrived.”

“How?” they asked.

“We will know,” I said, “because we will all be wearing our XR glasses while eating a nice meal together at a restaurant like this one. And it will all be perfectly normal.”

I am pretty sure that I am effectively a different person in the mornings and in the evenings. In the early morning, I can get an entire day’s work done in an hour or so, and I really enjoy it.

By the evening, all I want to do is chill out. I can manage to get work done, but it doesn’t feel like much fun. And the results are not that great either, because my heart just isn’t in it.

So you could say that morning me is supporting evening me. Both of me are just fine with that.

The other day, I posted a particular teaching challenge:

“Suppose you are giving a lecture on a favorite topic of your choice. You and everybody else in the room are wearing XR glasses, so that you can make anything at all magically appear for everybody to see and hear, as though it were actually in the room.

“What would you choose as your lecture topic? And what would your audio-visuals be?”

And I realized that I needed to ask myself this question, and discover what my own answer would be. After a few days thought, it came to me.

I would show people how to make my noise function. My visuals would evolve gradually from the algorithm to a 3D model of the resulting noise itself.

Today I started building this on my handy dandy Quest 3. I don’t know how it’s going to turn out, but I am having fun!

I notice that for me there are two distinct steps to writing useful software. The first step is to just get it working. If it doesn’t work, then it’s not going to be useful to anybody.

But then after that, there is the question of making it useful to people who are not me. Not only should it be useful for the particular problem that I am trying to solve, but it should also be able to help other people to solve different problems.

To make things useful for other people, I need to pull out all of the independently useful bits, give each one an interface that is clean and easy to use, and all the while make sure that I haven’t broken anything. When all that is done, then I can go on to working on other projects.

So every time I solve a problem for myself, I am also creating new tools that can help other people to get things done. Most other programmers also think and work this way.

Which means that for most programmers, the act of coding is inherently generous. The ability to continually build useful tools for the community is one of the wonderful things about programming.

Today, October 21, is the day in history when Thomas Edison applied for a patent for his version of the electric light bulb. Which was, in many ways, the canonical invention. In fact, the light bulb is the most often used symbol for the idea of invention itself.

Today is also the day in history, in fact exactly 40 years ago, when a length of one meter was first officially defined as the distance that light travels through a vacuum in a precisely defined period of time.

I can’t help but feel that those two events are connected. Each of these bright ideas, in its way, illuminates the other.

Suppose you are giving a lecture on a favorite topic of your choice. You and everybody else in the room are wearing XR glasses, so that you can make anything at all magically appear for everybody to see and hear, as though it were actually in the room.

What would you choose as your lecture topic? And what would your audio-visuals be?

I am in the lobby of a theater right now, waiting to see a play. Fortunately, it doesn’t bother the ushers that I am sitting in a corner typing away on my computer while I wait for the house to open.

My recent experience with the Meta Quest 3 has gotten me thinking: Imagine live theater with every audience member wearing XR glasses. The computer would not replace the live performance, but would rather provide scenery and lighting cues and special effects that complement the actors’ performances.

What would that be like? I guess the best way to find out would be like is to try it.

Maybe I will do just that. 🙂

Most of the time these days, the only way I can get out of endless meetings and rounds of emails is to have a plausible reason to not be near my phone. This is not a good thing.

There is something freeing about leaving my phone at home. All of a sudden I have gone dark, cybernetically invisible, off the grid.

At such times, nobody can reach me except the person I am physically with, and therefore I can be completely in the moment. It’s odd to realize that until a mere three decades ago, this was our usual state of being.

As smart glasses gradually replace smart phones, there will be times when we will wish we didn’t have them. Yes, our smart glasses will let us experience infinite worlds. But something that can liberate you can also bind you.

Sometimes I will simply want to be away from all those endless ways to connect. If anybody later asks me why they couldn’t find me, I might just say “I forgot my glasses.”