Musical puppetry

December 17th, 2014

This week I sat in on some really wonderful thesis presentations by Masters students in the NYU Music Technology program. The sheer amount of intellectual energy and inventiveness on display was very inspiring.

One trend I noticed was that some students would start with a traditional acoustic instrument they really know and love, such as a trumpet or a classical guitar, and convert it into a musical controller. The result would not make any direct use of the sounds that the instrument produces naturally.

Rather, the student would use those sounds as data, to be input into a computer synthesizer. A new sound would then be computed — perhaps one that could only be created with computer assistance. The result, however radically different, would retain a subtlety and expressiveness that is characteristic of the original musical instrument.

This general approach reminds me of other recent trends in computer mediated performance. For example, there are similarities to the way Andy Serkis “performed” his own body in the Lord of the Rings films, to create a digital Gollum. You never saw the actor himself, but only the computer-transformed result of his performance. Essentially, he was using technology to puppeteer his own body.

In a sense, an artist’s use of any tool — from the paintbrush to the piano — is a kind of puppetry. And as computers continue to become more powerful, new kinds of puppetry will continue to emerge, allowing us to use our brains and bodies to create ever more powerful forms of aesthetic expression.

After all, what is the piano, but a complex industrial innovation, enabled by advanced technology, which interposes itself artificially between musician and nature, in order to give the performer a greater power of expression?

Frozen as Buddhist manifesto

December 16th, 2014

The lyrics to the wildly popular theme song “Let It Go” from the recent Disney film Frozen have puzzled many people. What exactly is the song trying to say?

It occurs to me that one valid way to interpret it is as an avowal of Buddhist tathagatagarbha. Specifically, the song expresses a mindful path to contentment based upon acceptance of what is, rather than on a craving for what is not.

Excerpting lyrics from the song, we uncover something that sounds very much like a succession of Zen koans:

snow glows white on the mountain
a kingdom of isolation
makes everything seem small
and the fears that once controlled me
can’t get to me at all
no right, no wrong
the past is in the past
let it go, let it go
i am one with the wind and sky
here i stand
in the light of day
let the storm rage on
the cold never bothered me anyway

That’s pretty cool. Also the fact that this is the very first Disney song in history with lyrics that include the word “fractal”.

I’m trying to imagine

December 15th, 2014

The tragic stories keep piling on, one atop the other. One that has recently cycled back to public consciousness was the teenage boy shot to death by a cop — in the kid’s mom’s bathroom.

This was in 2012 right here in New York City. The police department says they thought the young man might have a gun, and that he was running from the police. But surveillance video shows the teenager calmly walking to and entering his mother’s apartment building.

A group of cops then ran into the building, guns drawn, without a warrant. When they burst into his mother’s bathroom, they found the teen trying to dump some marijuana down the toilet.

One cop, thinking the teenager might be reaching for a gun, shot him dead in the chest where he stood, while his six year old brother and grandmother looked on. Afterward, the cops took his grandmother down to the precinct for seven hours of questioning, not letting her lawyer talk to her for about two hours.

No gun was ever found.

What I have been trying to imagine is how this scene would have played out if the kid had been white. Would the police officer have shot the blue eyed youngster right in front of his tousle headed six year old brother, and his sweet old white grandma? Would the police department then have interrogated the old lady, who wasn’t charged with any crime, for seven hours straight, right after she’d just seen her grandson shot dead?

I’m trying to imagine whether that policeman would have pulled the trigger — whether he would have been emotionally capable of pulling that trigger. I’m trying to imagine whether, if the kid had been white, such a scenario — such a now horribly familiar scenario — could ever ever happen, even once, in these United States of America.

And I cannot.


December 14th, 2014

A few days ago I had the pleasure of attending the NYU ITP New Instruments for Musical Expression event, in which students perform original music on novel electronic musical instruments that they themselves have invented.

While there was a tremendous variety among the various instruments, one theme in particular seemed to emerge: In the hallowed tradition of the Theremin — perhaps the very first electronic musical instrument, invented in the 1920s by Léon Theremin — many of the students designed their instruments around the idea of a single large sliding control.

In Theremin’s original instrument, the performer’s hand forms part of an oscillating capacitive circuit. As the performer makes subtle adjustments to her hand position, the frequency of oscillation changes over time. The Theremin then transforms this human-modulated signal into an audible musical tone.

The Theremin can make beautiful sounds when properly played, but it is notoriously difficult to operate. Yet its basic mode of operation is easy to adopt: Move your hand position to modulate an electronically generated musical tone.

On the night of the show, I saw students doing this in various ways. One instrument used a single long resistive strip, another contained an infrared optical distance measuring circuit, yet another relied on an ultrasonic distance detector.

In every case, the fundamental act of playing the instrument consisted of moving one’s hand through space. To make music, the musician/inventor would modulate pitch by changing the position of his/her hand over time.

Despite the fact that they used different underlying mechanisms, all of these instruments reminded me a bit of the Theremin. Each was built around the same fundamental idea of forming a melody by moving one’s hand in space to vary pitch. Consequently, the music they made all shared a certain lovely simplicity that I found quite charming.

At some point I turned to the person next to me, and said “I think we are hearing a new musical genre.”

“What’s that?” he asked.

I was ready with my shiny new word. “Thereminimalism!”

Face time

December 13th, 2014

Fast forward to that hypothetical time in the not-too-distant future when everyone is “wearing”. So rather than seeing the world through our naked eyes, we will all be able to see, through our future contact lenses or implants, some computer-mediated transformation of that world.

The extent to which this visually transformed world differs from the literal world will ultimately not be a technological question, but rather a question that centers on individual and collective values, as we have discussed here in earlier posts.

When such transformations become possible, you will be able to “dial in” a preferred age to show the world. For example, someone in their forties can choose to appear as their twenty-something self in a party situation, and then revert back to a truer appearance to take a business meeting, if that is desired.

You should also be able to project forward, running plausible simulations of what you might look like in ten or twenty years, and then choose, at times, to show that face to the world.

It will also work the other way: When you talk to a person in their seventies, you might opt, for whatever reason, to see them as they looked when they were twenty or thirty.

It’s not clear to me what this capability implies from a social, cultural or ethical perspective. But it might be worth thinking about.


December 12th, 2014

Our various human senses do a very good job of complementing each other. Our sense of sight provides enormous bandwidth, while our hearing allows us to detect and locate events all around us. Touch lets us accurately assess the texture and solidity of objects, and permits us to use our amazing fingers and hands to manipulate tools.

One difference between our respective senses is in the scale of time at which each operates. For example, to simulate our sense of vision accurately, you need to flash about 100 different images in front of the eyes every second. Of course, that’s just a rough approximation, but it’s reasonably accurate. For example, 30 images per second isn’t quite enough, whereas 300 images per second would be overkill.

In contrast, to simulate our sense of touch you need to provide about 1000 changes in haptic sensation per second. If you only provide 300, then the things you are simulating will always feel spongy and soft, whereas 3000 haptic sensations per second feels pretty much the same as 1000.

To create a perceptually perfect simulation of sound, you need to go up to about 100,000 vibrations per second. If you try to get away with 30,000, you end up losing phase information at the high frequencies, which makes some objects sound like they aren’t quite coming from the right location. On the other hand, anything above 100,000 vibrations per second would pretty much be pointless.

I find it interesting that we’ve got these three nice round numbers: 100, 1000 and 100,000, for visual, haptic and auditory “samples per second”, respectively.

So what happened to 10000? We seem to have skipped right over that one. I’m pretty sure we won’t find it in our sense of smell, since smell works at a very slow time scale, compared with most of our other senses.

Maybe in the future, after we have figured out a practical way to interface directly to the brain, we will discover that some high-tech version of mind reading requires 10000 neural samples per second.

But it might be a while before we know whether that is true or not. By my reckoning, we will probably need to wait about seven trillion neural samples, more or less, before we find out.

Identity politics

December 11th, 2014

It was not too long ago in our nation’s history when something like the following would have been inconceivable:

For reasons that I have not yet fully worked out, being in a nation where stuff like this can happen makes me incredibly happy.

Three is a crowd

December 10th, 2014

It occurs to me that the “Three famous people” game that I described yesterday can be played the other way. Rather than the challenge being to find the answer, the challenge can be to come up with a good question.

It can be surprisingly difficult to find two famous people with exactly *one* other famous person obviously connecting them.

For example, yesterday Sharon suggested “Tom Hanks” and “Neil Young”. I thought she meant Jonathan Demme, since he connects them through the film “Philadelphia”. But it turns out she meant Daryl Hannah, which definitely works as a good answer at the moment.

So maybe we can design a kind of crowd sourcing game: Contestants post two famous names, and various people out there on the internet try to guess who is the third famous person clearly connected to both of them.

Your goal as a contestant is to post two names that will result in nearly everybody guessing the same third name. The greater the unanimity of response, the higher your score.

By the way, we should probably ban the use of famous people who strongly evoke one person in particular (e.g.: Stan Laurel).

Come to think of it, I wonder whether we can use this general method for turning around any guessing game: Given any given guessing game, use the Crowd to create another game, one which measures the quality not of the answers, but of the questions.

Three famous people

December 9th, 2014

There are some pairs of people who are so indelibly tied together that if I name one, you’ll probably think of the other. We could even make a game of it. For example, if I say “Spencer Tracy” — assuming you know who Spencer Tracy was — you would immediately say “Katherine Hepburn”.

But some very famous people are not clearly tied to just one person. For example, if I say “John Lennon”, do you say “Paul McCartney” or “Yoko Ono”?

So it might be interesting to make the following variant of this game: I name two famous people, and you need to figure out what famous third person is associated with both of them.

Let’s give some examples. In each case, you need to name a third famous person that both have in common:

(1) Bill Clinton and Tommy Lee Jones
(2) Peter Bogdanovich and Bruce Willis
(3) Heath Ledger and Taylor Swift

I’m guessing that many people will be solve one or another of the above, but most people won’t be able to solve all three.

I suspect that if you design this game properly, you can learn quite a lot about a person by which answers they get right and which they don’t.

Can you come up with interesting examples for the “Three famous people” game?

† To clarify, by “solve” I mean that you don’t need to check your answer with a search engine, because you know for sure. :-)

Psycho-genetic testing

December 8th, 2014

You can send a sample of your DNA to or one of its competitors, and they will tell you where your forebears came from, your ethnic mix going back many generations, and all sorts of things about your family tree that might end up surprising you. Clearly the people who subscribe to such services would like to know the truth about themselves and where their genes came from.

Yet when it comes to other things, we might not be so eager to know the truth about ourselves. I’m thinking in particular about our attitudes about race, gender and other groups of people that society likes to lump together.

As I watch the various responses to the tragedies in Ferguson and on Staten Island, I think that one of the tricky aspects of trying to discuss all of this is that most people don’t realize that they themselves harbor prejudices. Almost anybody you talk to will assure you that they themselves are not prejudice. Yet they are perfectly comfortable with the thought that a very large number of other people are indeed prejudiced.

Suppose we had the equivalent of genetic testing, but one that gives you a map of your prejudices, racial or otherwise, that you yourself never knew you were harboring.

I would argue that a conversation in the U.S. about race could only be helped by such self-knowledge. After all, there is no shame in harboring prejudices. All of us have irrational emotional responses to things, and many of those responses stem from times in our early childhood when we had no control over whatever nutty ideas our all-powerful parents may have been feeding us.

The real question is what you actually do and say, how you treat and speak about others, whatever your inner demons may be. It can only help to learn about prejudices in your own soul that you never knew you had. To speak truly about anything, you must first know the truth about yourself.