Month: July 2014

Today I’ve been editing a video for on-line viewing, and I realize that I don’t really want to produce a single video. Rather, I want to produce what might be called a fractal video.

If a person has only three minutes to watch my video, I want her to be able to see just the highlights, a kind of trailer for the whole thing. But she should then be easily be able to “open up” the video to, say, a 10 minute version.

This accordion-like unfolding should be able to continue, recursively, until the full content — perhaps an hour or so all together in length — is revealed.

Doing things this way would allow people to get the sense of what I am trying to say, drawing them in and giving them the basic narrative flow, without requiring the commitment up front to sit through an hour long video.

As far as I can tell, this is not how video editing programs and and on-line streaming formats work today. Which seems like a shame.

In many ways an architect is like a novelist. Both need to balance the needs of structure, balance and robust design against the practicalities of construction and technique, with an eye toward the eventual finalities of decoration and detail.

Each creates an immersive space that is completely artificial, and yet must seem real and natural to its visitors. Reconciling all of these simultaneous challenges is difficult, and requires not only inspiration, but also technique and discipline.

I think of this when I think about the emerging field of interactive narrative. It’s not a field that gets much wide recognition, because our culture misclassifies it as “computer game” rather than “literature”. Alas, you are probably not going to see a work of interactive fiction reviewed in the New York Times book review any time soon.

But the issues run deeper than that. After all, a work of narrative fiction can be modified by the very act of experiencing it.

If the analogy between architect and novelist is allowing someone to walk through one’s beautifully architected building, then the analogy between architect and author of interactive fiction can easily look like something else.

The experience might be more akin to telling a reader: “Here is some plaster and a chain saw. Go ahead and knock holes in the walls, cut out new windows and doors, or rebuild the basement.”

And yet the result must still be a viable place to inhabit. It’s not always clear, when an author is faced with such a daunting challenge, what good architecture would even look like.

A longstanding trick in computer graphics is to add fog to a scene. Not only does fog help create a sense of atmosphere, but it also helps to avoid needing to render huge scenes, since everything further than a certain distance away becomes invisible.

The math behind fog isn’t all that complicated, which is why I was surprised yesterday to realize that a scene I was rendering had the fog exactly wrong: Everything near the camera was obscured by misty white fog, whereas objects far away remained perfectly visible.

It wasn’t right, but it certainly looked very cool. I could see entire vistas, mountains, streams and villages, with crystal clarity, going far off into the distance, even as nearby objects remained shrouded in mist.

I pored over my code, trying to understand where I had gone wrong, but couldn’t seem to find the bug. I didn’t actually understand what was going on until later that night, when I woke up in bed, and realized that the entire episode had been a dream.

So it seems that I program when I dream. I wonder whether that is a good thing. For all I know, I’m a better programmer when asleep.

But if last night was any indication, I am lousy at debugging in my dreams.

I’ve given several talks about my research in the last few weeks, and I’m seeing a consistent rhythm. In the days before the talk I add new capabilities, build cool and sexy demos, work through the narrative and timing, practice making everything flow.

There is a particular kind of development that happens in those times — cool, bold new features, sure to catch the eye of onlookers. There’s a lot of focus on textures, movement, all the flash and sizzle.

But the days after one of these talks are just as interesting, only in a different way. I have no pressing deadline, but my head has become filled with all sorts of questions raised by the act of presenting my work — both from the things that worked well and from the things that failed to work.

I find myself enormously productive in the days after one of these talks, but it’s a very different sort of productivity. It is a time for gradually building toward new directions, for making good on promises I had sort of made in my presentation, for moving things from mere “cool demo” to actual working system.

It’s funny — people are always appreciative when I give a talk about new work and I show demos of things they’ve never seen before. But another way of looking at it is that those people are doing me a favor. In a way, they are my research collaborators.

There is a saying in evolutionary biology that “Every genotype requires a viable phenotype.” In other words, each genetic mutation needs to result in creatures that can survive — otherwise the mutation dies out.

Even if, say, four particular steps of evolution would have led to a super-being, that path of evolution will never happen should any of the first three steps produce a fatal vulnerability.

It occurred to me today that this concept has something to teach us about education. Very often an educator will introduce a lesson in the following general way: “Later this semester we’re going to get to the fun stuff, but first you need to learn these basics.”

What often follows is a grueling and demoralizing exercise — equations are written, theorems proved, complex formulas worked through. Students become bored, disenchanted, distracted. They surreptitiously check their iPhones for incoming texts. Nobody wins.

Even if the stuff that comes later in the semester turns out to be fun, the damage has been done. Students who lost the narrative during the earlier lesson are almost certainly not going to be able to pick it up later. They will be unprepared for what comes next.

I would argue that every lesson needs to be fun and exciting on its own terms. I know that in practice this can be difficult.

But it is necessary.

There is a wonderful scene in Woody Allen’s “Annie Hall” where the two main characters, who have just met, are provided with subtitles. They are both attracted, but have no idea how to talk to each other. As they natter on awkwardly, trying and mostly failing to hold a meaningful conversation, the subtitles say what each is really thinking, to hilarious effect.

One day, perhaps far in the future, the technology may exist to make those subtitles. Your wearable device will analyze your partner’s body language, facial expression, eye saccades, tone of voice, slight pauses, and any other available clues.

This data will be correlated in real time with a vast database in the Cloud, then run through various machine learning algorithms, together with techniques for rapid language parsing and synthesis.

The net effect will be an interpretive stream of words that appears to float in space between you and your friend, telling you what the other person is really thinking and feeling.

And here’s the weirdest part: The people who grow up with this technology won’t have the faintest idea why anybody would have a problem with it.

In any real-time performative medium, you walk in with something already prepared. If you are a pianist, the piano has already been built — the audience generally does not wish to see you construct a piano there and then on-stage.

In theatre there are sets, costumes, lights, stage directions, usually a script, all of which have been carefully prepared before the live event itself.

Even improvisational performances require much preparation. Nobody goes into a jazz improv session without lots of practice and grounding in the rules of the genre. The same goes for improv comedy or dance. In fact, the more “improvised” genres tend to have very well understood boundaries — that’s what lets them work without dissolving into chaos.

Everything you come in with, every asset available to you to use in your live performance, can be thought of as an instrument. Your trained voice and body, your guitar, the spot light from stage left, these are all instruments.

It might be interesting to study genres of live performance entirely from the perspective of nature of the instruments they require. There are many overlapping taxonomies of instruments, and the more one thinks about it, the more complex and intricate the questions get.

I wonder whether anybody has ever looked at understanding varieties live performance entirely from this formal perspective — through an epistemology of instruments.

I suppose the more standard and traditional route, upon getting a new musical instrument, would be to learn the repertoire — standard chord progressions, popular songs, whatever is out there.

But I find that when I pick up my blue Ukelele I don’t want to do any of that. Instead, my fingers want to explore, to try random things, to figure out sounds and chord progressions from scratch.

I’ll figure out an F chord, then an A7, then maybe a D followed by an E minor, and go from there, working out picking patterns as I go. It really doesn’t matter, as long as the next chord feels right.

It seems that what I really want is to explore the ukelele as a kind of hack space — a place to work things out on my own, and see where it all will lead.

Which is turning out to be, I am happy to report, a lot of fun. 🙂

Ideas are brilliant
Alas, the world really needs
Deliverables

Walking along Commercial Street this morning in East Vancouver, on my rounds to shop for various necessaries, I found myself drawn to a little guitar shop. Walking in, I got a sudden urge to buy a ukelele.

Since I don’t play the ukelele, it didn’t seem like a good idea to get a fancy one. It turns out that there is a little beginner model for around $40 that comes in all sorts of vibrant colors. The store owner told me that his girlfriend, who is a professional musician, uses the pink one in her stage performances.

Somehow the phrase “blue hawaii” rang in my head, and I realized I just had to get the blue one. Which I did.

You can see my shiny new toy in the photo below, in the arms of its proud owner. My blue ukelele has a delightful sound, and I’m having tremendous fun learning all the chords I can make with just those four G,C,E and A strings.

And unlike my computer, it will never need batteries!