Ken's Blog

Suppose we apply the structure of composition → researsal → performance to computer programming? This has been a topic of conversation recently between Murphy Stein, Vi Hart and myself.

Currently, computer programming is generally thought of as a strictly compositional activity. A programmer generally creates a program in relative isolation, over a period of time, iterating until the program’s behavior is ready to be experienced by others. The programming languages and environments that have evolved to support this activity very much privilege this way of working.

But what if the semantic constructs of programming — sequences of variable assignment, iteration, conditional execution, defining procedures, building hierarchies of objects — were part of some sort of live performance for an audience?

Suppose we were to design a form of programming from the ground up, specifically to be a performative medium? It wouldn’t replace programming as composition, but rather would complement it. Much of the program would still be pre-written, as a compositional stage. In such a paradigm, one would also expect a rehearsal stage.

This splitting of computer programming into two such different modalities is not as odd as it might sound. After all, this is what happens with plays and music. While a playwright shares the medium of words — the semantic level — with the actor, their modes of expression are wildly different. The playwright expresses through typed or written words on paper, whereas the actor uses voice, facial expression and body movement. What they share is the underlying meaning.

Similarly, a musical composer writes down notes on a staff, whereas a musician plays a physical instrument. What they have in common is, again, the underlying meaning.

A kind of performative programming is done now in the avant guarde music community. Performers manipulate programs written in Max/MSP (or its open-source counterpart PD) to create live variations in the procedure controlling an algorithmically generated or modified composition.

Personally, I’ve never found the experience of attending such concerts to be entirely satisfying, since Max/MSP was never written for this purpose. Max/MSP programming on stage usually remains relatively opaque to the audience, even if audience members can see the computer screen.

It would be fun to design a programming language and environment specifically to be a medium for performance in front of an audience — with the understanding that much prior composition has already been done to scaffold what the audience is seeing.

A programming modality that privileges performance would be useful not just for music, but also for dance or theatre. One could imagine an evolving performance acted out by robot actors or dancers. As the performance progresses, the human performer modifies the procedural behavior of the actors. The audience experience consists not only of observing the actions of the robot actors or dancers, but also of understanding the rules those “performers” are following.

Theatre, dance and musical performance generally share the following structure: There is an initial stage where the work is composed, then a period when it is rehearsed, and finally the work is performed one or more times in front of an audience.

Each of these three stages has its own distinct properties. Composition is often (although not always) done by one individual, generally over a long period of time. For example, a work that an audience will experience for ninety minutes might require months or even years to compose.

Researsal is interesting, in that it takes place in an intermediate time-frame — generally performative, and yet not at all optimized for an audience. For example, a group researsing a work of theatre or music might repeat the same passage many times.

The audience on some level senses all this prior effort at the time of actual performance, but those efforts cannot subsume the performance itself. In the implied contract with one’s audience, the considerable sweat and toil of composition and rehearsal need to remain offstage.

One of the exciting qualities of of this three part structure lies in the rich opportunities it affords for feedback and iteration. Either researsal or performance can lead to modification of the composition itself. Unlike, say, a film, a theatrical, dance or musical performance is never entirely fixed — each performance is unique, and is subtly influenced by that evening’s audience.

What if we apply this structure of composition/rehearsal/performance to other domains? That’s a topic for tomorrow.

My ISP had technical problems the day this post was created, so apologies if you are only first seeing it the following day.

This evening I was walking along the street in Greenwich Village, minding my own business, when suddenly my eyes were pleasantly assaulted by an exhibit in a nearby window — a retrospective of drawings by Al Hirschfeld.

Now, this next part will either make perfect sense to you, or it will seem completely nuts. For the next half hour, like a kid in a candy store, I stood on that sidewalk drifting happily from drawing to drawing, in a state of utter bliss, counting Ninas.

I remember exactly the day, when I was twelve years old, that I was first initiated into the mystery of Nina. On that fateful Sunday morning, somebody pointed out to me that in every brilliant Hirschfeld caricature on the cover of the New York Times Arts and Leisure section, there was a little number to the right of his signature. And they explained just what that number meant.

From that day forward my teenage soul was hooked. Each week I would wait faithfully for that weekend’s paper. When it arrived, the very first thing I did, before doing the crossword, before checking out movie reviews, even before reading letters to the editor in the NY Times Magazine section, was worship at the church of Nina.

I could explain more about this, but that’s what Google is for, right?

I have been reading The King in Yellow, Robert Chambers’ masterful collection of gothic short stories from 1895. It is odd to finally read something that was such a powerful root influence upon authors whose works I have known for years, from Lovecraft to Heinlein and Silverberg to Neil Gaiman and Stephen King. Also, quite evidently, Harlan Ellison.

A unifying theme behind some of the stories — the existence of a text that makes its reader go mad, simply through the sheer seductiveness of its ideas — seems to post deep questions about the nature of literature, communication and human interconnectedness. Could mere words and expressed thoughts indeed have the power to drive us insane, to separate us from whatever hold we may have upon objective (or at least collective) reality?

Whatever side you come down on in answering that question, the implications are profound. I suspect Jung would have responded in the affirmative, whereas most thinkers today would say no.

Could such a text in fact exist? I find the question interesting, but I’m not going to, you know, go crazy trying to figure it out. 🙂

It always amazes me when people say, upon hearing that I work with computers and technology: “oh, that must be so complicated!”

I can’t help wondering, when I think about how complicated we humans are, and how complex the relationships between us, just what these people are talking about.

A computer is, in fact, eminently learnable. If you put in the time to study a computer language, technology or interface protocol, you will indeed master the instrument. No matter how complex it might seem, technology is predictable — if you poke it in a particular way, it will respond the same thing every time.

People, in contrast, are utterly unpredictable, capable of immense generosity and equally immense venality, often in the space of a single breath. Just when you think you know what’s going on, human minds will charge off in some completely unexpected direction. It is both our tragedy and our glory.

To paraphrase a grand old quote: Technology is easy. People are hard.

I find it heartening that so many people weighed in on the question of children learning a second language through the aid of on-line communities, and related questions.

It’s becoming clear that this is not one question but a web of related questions. For example, experiences for teenagers (like the one John Nordlinger started around W.O.W.) are clearly going to be vastly different from experiences for six year olds, in almost every conceivable way.

Also, as Manooh points out, there might be much opportunity simply in making the right resource available to kids, and letting them build from there, as a community of peers. The wonderful thing about that, if we can get it to work, is that we might be able to achieve a transformative level of change out of an affordable investment in resources.

Personally I find myself drawn to creating the fun exploratory environment that promotes multilingual capability for children who are still in their language acquisition age range (seven and younger). There is something so cool about tapping into that powerful potential. Besides, it gives me more excuses to play with cute little fish characters. 🙂

By the way, people I know at Google have told me that they are not actually phasing out the experiments in Google Labs, but rather just reorganizing how they roll them out. As long as these wonderful experiments keep coming, I’m happy!

Steven Pinker once said that “Language is essentially a creation of the mind of children.” He also referenced, in his book “The Language Instinct”, a study that showed that when you try to teach children Esperanto (an early artificially constructed attempt at a “universal language”), the children spontaneously start to fix it — they immediately start to change the language itself rather than learning to speak it. Esperanto does not conform to the meta-grammar common to all natural languages. It is not naturally learnable by children.

Yesterday J. Peterson commented on my post about using on-line games as a way to get kids around the world to each other their respective languages. The comment focused on the idea of introducing a shared invented language — perhaps Esperanto, or Klingon, or Elvish. It’s not prohibitively difficult to create such a language. There are even books out there to help you along, such as “In the Land of Invented Languages” by Arika Okrent and “The Language Construction Kit” by Mark Rosenfelder.

Yet I see a problem with this approach: Such languages, unlike the hundreds of naturally evolved languages, are not “naturally learnable” by young minds. Children don’t just naturally learn an arbitrary language — they will only naturally learn languages that have the features found in natural languages like English, French, Chinese, Swahili, etc.

I could see getting a group of small children to devise a new shared natural language on-lline, much as a community of children spontaneously created Hawaiian Creole, or Nicaraguan Sign Language. Such a project could be very interesting, although it’s not immediately clear how one would go about doing that. I’m just not sure what advantage it would have over simply getting children to play shared games in which they learn each other’s natural language.

In the last several days I have participated in several discussions that touch on the question of how to increase children’s knowledge of other languages by getting kids from different countries to communicate with each other on-line.

This topic is in some ways a response to the problematic nature of school-based teaching of a second language, which usually comes too late to take advantage of the “sweet spot” for proficiency in language acquisition (up to around age seven), while often devolving to teachers who are ill equipped for the task.

Putting aside for the moment the issues of security and provenance, what constitutes an effective way to use on-line social engagement between kids to promote mutual language acquisition? It came up during one of these discussions that mere conversation, such as Skype chat, seems not to work very well, whereas anecdotal evidence suggests that interacting with others by playing an on-line strategy game, such as “Worlds of Warcraft”, might be a much more effective way to learn elements of a second language.

This suggests that it is not merely engagement in conversation which leads to second language learning, but invested engagement — there needs to be something at stake.

Which leads to the question — could we construct an international on-line multi-player game that specifically promotes multilingual skills through invested engagement? And if so, how would we operationalize the question of how effective such a game is?

One strategy might be to build in design variations along various dimensions, instrument the game to measure effectiveness for language acquisition, and then gradually tune our design variables, in response to gathered data, so as to optimize the game for maximum language acquisition.

Another strategy might be to reward actions in the game that require multi-lingual proficiency — such as the ability to say a “magic spell” in two different languages, as in the recitation of the two spells: “place the magical cup upon the rock” and “coloque o copo mágico sobre a rocha”.

It might also be most effective to target such games toward kids seven years of age and younger, when facility for language acquisition is at its peak. This suggests, among other things, using spoken words and pictures in the game mechanic, rather than relying on written language.

There are so many possibilities — as well as unique challenges, such as the difficulty of building any computer software that must understand the speech of younger children.

Wouldn’t it be wonderful if such a project were successful, and on-line game play could lead to greater understanding between people around the world?

I’m having dinner this evening with my old friend Bill Buxton. Bill is the only person with whom I ever rode an elephant. He also told me the other day that I am the only person with whom he ever rode an elephant.

Since then my relationship to individuals of other species has evolved, to the point where I would never again ride an elephant — since, in retrospect, it is clearly not an experience that the elephant enjoys.

It is therefore quite likely that Bill and I will remain true to each other — that we will each remain the only person with whom the other has ever ridden an elephant.

Which, I believe, makes us both monopachydermous.

How many people can say that about each other?

Last week, during a workshop at Google, I was asked to do a verbal report about our break-out session on “the future of gesture on mobile devices”. The Google V.P. presiding over the day, Alfred Spector, whom I quite like, said we could do these reports in any fun way we like — “even in iambic pentameter”.

If you know me at all, you’ll know that of course this was music to my ears. I composed and on the next day presented a faithful report entirely in iambic pentameter rhyming couplets. Which our hosts seemed to like, because today, on the Google blog, they linked to my little poem.

Alas, you can’t get the full effect of my presentation from reading the poem. There was a whole level of physical comedy in the performance that doesn’t show up in the words (with some of my more expressive comic gestures unsuitable for publication on Google’s blog). I’m thinking now that I might need to create a procedural character to act out the performance here.

OK, that was the good news. The bad news was that the very next posting on Google’s blog — also today — was an announcement that they are terminating Google Labs. As many of you know, Google Labs is where they put all the cool edgy crazy delightful stuff that may or may not turn into a product.

I love the brilliant randomness of Google Labs, and will be sad to see it go. There is something admirable about a company that has the courage to float brilliant innovations out in public, before those innovations have been neatly wrapped into a specific product. Hopefully Google will find some other way to put their researchers’ wonderful creative ideas out there for us to try out, without feeling they need to wait for official product cycles.