This afternoon I plugged a midi keyboard into my MacBook. I connected it to a Web-based audio interface, used some samples of piano notes that I had lying around, and was soon able to play music in my Web browser.
Then I added that code to my WebXR system, so I could use my music keyboard to create graphics in XR. At some point I realized that I could just plug the midi keyboard directly into my Meta Quest 3, using its Web browser based midi interface.
All of a sudden I can play the piano while making 3D graphics animate in the air above the keyboard — not in VR, but in video-passthrough XR. So I can still see my room, and I can see my real hands playing music on the keyboard.
But now I can also create magical 3D graphics in mid-air just by playing the piano. Like Harry Potter, if he were a musician.