Month: December 2009

Combining some “natural language” and blocks-world ideas from Terry Winograd’s SHRDLU with the dragging tiles concepts from Mitch Resnick’s Scratch, we might end up with something like this:

Here some snap-together tiles have been arranged so as to instruct the computer what to do if somebody clicks on the green ball. In particular, if the green ball is clicked on, it will roll over to the red square. Note that there are three different shapes for the tiles: A diamond shape means either true or false, a rectangle shape refers to a property (like red or green) and a rounded shape refers to an object (like a ball or a square).

I realize this is a very simplified example. In reality this little snippet would be embedded in a game, or a puzzle, or some larger world of goals and challenges. I’ve taken away all that context so we can focus on the actual operations involved.

Like Mitch Resnick’s Scratch program, this snap-together-tiles approach avoids syntax errors, since tiles will only snap together in ways that make sense. This avoids one of the problems of Winograd’s SHRDLU — that people might type all sorts of English sentences that a computer wouldn’t understand.

Here is a set of tiles that you might draw from to build the above set of instructions:

Just as in Mitch Resnick’s Scratch system, the tiles are stretchy – when you insert a tile into a waiting slot of another tile, the slotted tile accommodates by growing larger so that the inserted piece will fit.

To make the instructions seen in the first picture, you could insert tile 2 into tile 1. Then insert tile 3 into tile 2. Then insert tiles 4 and 5 into tile 3. Then snap tile 6 into place below tile 1. You can easily figure out the rest for yourself.

Playing around with these tiles, I find them to be fun and intuitive. They seem to do a good job of mimicking the way we generally think about getting things done — by gradually building up tasks from simple steps.

This site receives an immense amount of spam every day. Of course I filter it all out so that you — my actual readers — are spared the burden of seeing it. But I have noticed over the last two years that the spam assaulting this blog has been getting somewhat more sophisticated. At first the site was merely receiving weird and vaguely nonsensical messages, in English or otherwise. Now, in addition, one finds lovingly crafted pseudo-comments, artfully constructed so that each appears to be a valid response to any possible blog post. These ersatz messages agree, or disagree, or raise an exception in words carefully calculated to avoid any identifying context. In some sense, they are the Zen koans of spam — perfectly content free vessels for rhetorical discourse, unsullied by any actual facts or meaning.

For now I can still tell the difference. But our days may well be numbered. Perhaps some day soon this Art of the Spam will rise to such a high level of sophistication that we mere humans will be defenseless before their ostensibly reasonable onslaught of words. Some clever spammer will work out a fatal combination of phrases that will seem so reasonable, so perfectly attuned to whatever blog post we had sweated over the previous night, that we will be charmed, flattered, utterly disarmed. We will, at last, fail to realize that we are merely reading words churned out by a cold-hearted and lifeless bot, a soulless cyber-golem constructed of bits and steel.

And then the war will have been lost. Universal spam will have conquered our all too fragile human minds, as we at last open up our imperfect hearts and egos to the mere illusion of an understanding readership. The machines will rise, and the ad-bots will take over this still nascent cyberspace.

On that sorry day, should it ever come, I pledge to at last forgo this fickle internet technology, to log out, go off line, exit the netiverse, unplug myself once and for all from the Matrix. On that fateful morn I shall endeavor to choose, from a high and dust-laden shelf, some old and long forgotten novel that I had long been meaning to read – a quaint artifact of paper and ink.

When all else fails, and cyberspace has lost its last semblance of trust and human decency, I will just go read a book.

As Ashley commented the other day, perhaps we shouldn’t even be using the word “programming” to describe what everyone will be doing once everyone is able to give instructions to computers in a non-trivial way. There is a reasonable argument to be made that the very word “programming” has been compromised beyond hope of redemption. I teach in a computer science department at a major university, and my colleagues and I have certainly come across this problem. In some circles there is a sense that we — to use a cultural analogy — are still teaching latin, in a world in which latin is no longer seen as culturally relevant.

This argument is supported by the fact that an entire computation-centric academic universe has sprung up in recent years outside of computer science departments, a universe that didn’t quite exist not that many years ago. In our own university there are computation-centric departments of music technology, interactive telecommunications and information technology, just to name a few. At NYU these three departments are, respectively, in our School of Education, our School of the Arts, and our Business School. At NYU you can have a rich academic experience revolving around computers without ever going near the computer science department. Similar things are happening at other universities.

Not calling it “programming” could have some distinct intrinsic advantages as well. If I tell a computer that I want my virtual agent Fred to deliver some message to John’s mailbox, in most programming languages I would generally say something like “Fred.deliver(thisMessage, John.mailbox)”. This makes perfect sense to programmers, but looks like gobbledygook to anyone else. If we don’t need to call it programming, or even to be consistent with the existing culture of programing, then we are free to say “Fred, deliver this message to John’s mailbox.” Or at least something more along those lines. The latter statement is just as precise, and just as understandable to a computer, but it is also understandable to the 98% of humans who don’t think of themselves as programmers.

Yet there are dangers here. The great computer pioneer Grace Hopper wanted to see something like this — a programming language that would be understandable to non-programmers, and she set about leading a movement to make that happen. What eventually emerged was COBOL, which (although it is interesting in many ways) clearly falls short of the mark.

But one could argue that the reason for this is that COBOL was developed in that old, pre-real-time-simulation world, in which programming without math would have been useless. Were Grace Hopper to start out to realize her dream today, something very different might emerge.

In my next post on this topic I will focus on the grammatical question — how can we allow people to create precise instructions that are understandable to computers, and yet are also comfortable for people who instruct those computers, without requiring those people to completely revise their way of communicating? In a sense, we’d like to combine what was learned from Terry Winograd’s work on SHRDLU with what Mitch Resnick’s team at MIT later accomplished with the Scratch project, making use of what is now possible through the use of interactive computer animation.

Winter in New York
So lovely. Now I, sadly,
Must fly far away

I’ll miss it. New York,
I mean, not winter. There’ll be
Winter in Stockholm

One of the important signposts on the road to programming without math was a wonderful project almost forty years ago (!!!) called SHRDLU. It was the 1970 Ph.D. thesis of Terry Winograd, who is now a prof in the Stanford Computer Science department (and a very nice guy). He created computer software to simulate a “blocks world” (a world where everything is simple blocks of different shapes and colors). A person could interact with a simulated robot in this world, typing in dialog to tell the robot to put this block on top of that one, or asking the robot questions such as “what’s the color of the top-most block”. The computer did a very good job of figuring all of these things out. Here is a snippet of dialog from one conversation between a person and SHRDLU:

Person: Pick up a big red block.
Computer: OK.
Person: Grasp the pyramid.
Computer: I don’t understand which pyramid you mean.
Person (changing their mind): Find a block which is taller than the one you are holding and put it into the box.
Computer: By “it”, I assume you mean the block which is taller than the one I am holding.
Computer: OK.
Person: What does the box contain?
Computer: The blue pyramid and the blue block.
Person: What is the pyramid supported by?
Computer: The box.
Person: How many blocks are not in the box?
Computer: Four of them.
Person: Is at least one of them narrower than the one which I told you to pick up?
Computer: Yes, the red cube.

As you can see, the results were pretty darned impressive. What interests me about this conversation is not the fact that the computer is understanding and speaking in English (although that’s very cool) but that the person is able to manually direct the computer – step by step – to do something.

There are, in fact, a number of interesting things to note about the above dialog. For example, it is clear that if you tell SHRDLU to pick up a pyramid, and there happens to be more than one pyramid, SHRDLU gets confused – and tells you so – since it can’t know which pyramid you meant. So you have to be precise when you tell it what to do. But you don’t need any math – just clear instructions.

My current quest for pwm is related to SHRDLU, in the sense that giving someone a cooking recipe is related to manually telling them how to cook step by step. The difference is that if you can manage to give someone a recipe, then you don’t need to be there instructing them at every step. In the case of computers, this means that they can perform tasks for you that involve not just a few blocks, but millions of blocks (or millions of web pages).

Yesterday I described the general idea of programming without math to my mom, and she got the idea instantly. She say “oh I see, you don’t need math, just logic.” My mom got it exactly right (as she generally does). Indeed, what is required for pwm – and you see it in the SHRDLU dialog – is that the person doing the programming understand that they are dealing with a computer – a creature built of logic.

To me, the most interesting thing about the success of SHRDLU is not what the computer is doing, but rather what the person is doing. The person is adapting to the literal ways of the computer – using the powerful human mind to understand what is going on, and adjusting accordingly. People are amazing at adapting to new situations, once we know what to expect from those situations.

And that is one reason I think pwm – if it is done right – has a real chance of being accessible, and useful, to a very large number of people.

There is so much more to say on the subject of programming without math, and it will take many more posts to even begin doing justice to the subject. But I don’t want this blog to be hijacked by a single topic. Therefore I will adopt an alternate days strategy – posting about pwm on odd days, and other topics on even days, until the pwm discussion has run its course.

Yesterday evening I was having a fascinating and exciting discussion on a far ranging set of topics, with a friend who is reliably brilliant and full of unique insights. In other words, a perfect evening. Topics ranged freely from early pre-Hayes code talkies to the plays of Alan Ayckbourn to the origins of Indoeuropean languages, while touching upon a vast array of other subjects besides.

All was well – joyous and inquisitive and freely explorative – until the topic turned to politics. Suddenly my friend closed up like a book. The arms went across the chest, the body language stiffened, and the tone became bitter and angry. Why have the bankers been given free handouts? What will become of the people out of work, and why isn’t more being done? Why are we back in Afghanistan? I no longer felt that I was talking to an individual, but rather to a representative of familiar talking points brought into the room from other conversations.

I tried to shift the discussion to something more positive, such as our president’s recent announcement of a $260 million consortium to use games to help childrens’ education (a topic I actually know something about). But my friend would have none of it. It seems that my unwillingness to share in the general community bitterness was coming across as a kind of avoidance. My friend seemed to want to know – why wasn’t I sufficiently angry?

Today, in the sunshine of a new day, after I had described my experience of the evening before as clearly as I could, my friend completely came around to agreeing with my views on this, that the “warlike stance” of political discussions seems to take us quite out of our own actual identities and into some sort of weird reflexive group mind.

Yet I have run into this dynamic before in recent times. Two weeks ago I found myself talking to some Germans who were very angry and bitter about their government. Last week I was in Sao Paulo, and encountered one conversation after another with Brazilians who were completely appalled and disgusted by their government.

Yes, I understand – politics is rotten. Nation states are self-interested and highly combative creatures whose realities are generally far less palatable than their national myths. That hero you thought you elected into office generally turns out to be – of all things – a politician. I really do get it.

But why does this obvious reality need to hijack our personal interactions? Why do we cease to be ourselves and become blind with rage when confronted with these sad truths?

Personally, I am quite happy to accept that my president and my country are far from perfect, and at the same time to throw my efforts into helping that same president to improve kids’ education. It seems to me that this is a much more productive use of my psychic energies than yet another angry conversation about how rotten it all is.

Does this refusal to be sufficiently infused with indignant rage make me a hopeless Polyanna?

There is indeed a highly innovative project to introduce children to programming. It’s called the Scratch project (http://scratch.mit.edu), and it was started at MIT by professor Mitch Resnick and his students six years ago. By now there are several hundred thousand kids around the world who play with it. You can go to the site and download it for yourself, or you can just look at the following snippet of a screenshot to get the idea:

The general idea is that you drag virtual tiles from the left side of the screen, and they snap together to build a computer program. This whole “drag and snap together” idea gets rid of the problem of syntax errors — tiles will only snap together in ways that form a syntactically correct result.

The Scratch system provides a library of building blocks that allow kids to incorporate drawings (which they can move and flip through to create animations), recorded sounds, and detect user events from the mouse and keyboard. This is enough to form a kind of kid-friendly equivalent to authoring in Flash — kids can (and do) create animations, games, interactive stories.

This is a great project. But I would argue that there are two important problems it does not solve:

(1) It’s a single integrated project, not a general way of working. You can only build things in the Scratch environment. You can’t use it in your own web authoring tool, word processor, spreadsheet, music composing program, or anywhere else that’s outside of the Scratch sandbox.

(2) There is no way to fully connect the “kid’s toy” environment of Scratch to the sorts of serious computer progamming that these kids will later encounter in their freshman year of college — a good eight or nine years later. As kids go through middle school and on to high school, their experience with Scratch does not continue to stay relevant to their evolving interests.

(3) It’s still very math-centric. Philosophically, starting kids off with Scratch is a bit like teaching English by looking at electrical engineering technical manuals. Scratch is a version of an E.E. technical manual for ten to twelve year olds — it assumes you will eventually be using programming to do mathematical things.

I think the Scratch project is wonderful. I also think we might be able to do better, in two ways:

(1) Rather than create a single monolithic program, create a flexible way of putting these sorts of tile blocks into any computer program, so that a user can assemble programming tiles anywhere.

(2) Create a “core” language that does not require the use of math. Ideally we’d want to make this language powerful enough so that it could handle many different types of problems.

Tomorrow I will describe such a language, and various ways it can be used.

I agree with all of the commenters on yesterday’s post that different problems require different tools. But God, as they say, is in the details.

To address Douglas’ point: I’m not suggesting that people learn a fundamentally weaker programming language. What I’m suggesting is that we radically rethink the order in which we teach programming. It seems to me that a K-12 computer programming curriculum should culminate in, say, a solid working knowledge of Python. But many of the features of Python need not be introduced in the earlier grades, and even then some of those features could be elective, not mandatory.

The mandatory features are the ones that focus on temporal logic, such as sequencing, conditionality and repeated/looped operations – basically the same operations you use when following a cooking recipe. That’s really the core for any use of computer programming. After that we can bring the ideas of how to do common sub-operations. In programming we call these functions. The equivalent in cooking are the sorts of things you already know how to make from another recipe.

Certainly some applications, such as ray tracing, are going to call for tools of mathematical modeling to be readily available. But other applications might require very different sorts of tools. The point is not to remove the math from the toolkit, but rather to not require learning math as a critical path on the way to learning programming.

Andras expresses a utopian ideal of being able to express things in broad and inexact strokes. Alas, I’m not sure we can deliver on that one. A statement as simple as “John is strong as an ox” would confuse the hell out of any computer program. The statement could be interpreted as saying that the phrase “strong as an ox” is an alternate name for John, or that John is strong whenever he is being an ox. And don’t even get me started on “Time flies like an arrow.” The only reason we can glean the proper sense from such statements is that our human brain performs extraordinary feats of culturally informed analysis — something that is quite beyond computer software in its current state.

But what can we do? Well, for one thing, I think we should start out by reducing “data” to the bare minimum needed for describing an interactive virtual world. As far as the program is concerned, the world consists of objects, and each object either does or doesn’t have each of various properties. We can modify objects over time by adding and taking away their properties.

Tomorrow I’ll go into more detail.

Continuing the thread from yesterday…

There is a reason that programming necessarily started out being very math-centered. It had no choice. Back in the 1950s when FORTRAN (essentially the first high level programming language) burst upon the scene, there was no interactive graphics, virtual reality, or real-time graphics acceleration. There were no high resolution LCD monitors and there certainly was no computer mouse. The very idea of interacting with “objects” in real-time on a computer screen – or any nascent glimmering of such an idea – was still the stuff of science fiction.

In such a world, math became central to programming because math is the great recourse when all else fails. You may not be able to see a world, to touch it or sense it, but you can indeed use mathematics to describe it. In the absence of being able to see, say, a round spherical ball on my computer screen, I can always type in the mathematical description of a sphere, and use that description to help me program my computer to make the ball roll, or bounce, or do anything else.

This was the world faced by the early computer programmers. They were literally working blind. All they had were their mathematical descriptions of things, and so naturally computer programming grew around that central, highly empowering way to describe everything. It’s no wonder that the concepts and methods of mathematical description became deeply embedded into the DNA of programming culture and practice.

Even something as seemingly fundamental to programmers as the “=” assignment operator is a math concept, not a programming concept. It basically says “give this thing a new identity.” In the real world, we rarely do such a thing. Rather, we impart new properties to things that already exist. We paint a wall, or we drive ourselves to work, or we make another person smile. Yet we take it for granted that programmers are supposed to write things like “a = b”, because we are so used to seeing the math embedded in the programming that we forget it’s math, not programming.

Programming without math would be much more likely to have operations like “make John smile”, or “drive to work” than such an abstraction as “a = b”. And that would make it much more interesting to a lot more people.

When people think of computer programming, they generally also think of math and engineering – equations, numeric variables, arrays of numbers. Both programmers and non-programmers share this association. To people who program, this mindset generally means that programming becomes associated with engineering tasks – things like running plumbing lines through a house, or building a bridge, or designing an integrated circuit.

To people who don’t program (that is, most people), this association generally has them running as fast as they can for the nearest exit. The whole conversation taps into the deep well of math-phobia that pervades much of our society.

Most people who talk seriously about “universal programming literacy” are the same folks who have no problem with math and engineering. They often wonder why it’s so difficult to get the rest of the population on board.

But I’ve been wondering recently, maybe programming has been misunderstood, abused even, the product of a dysfunctional childhood. Perhaps the same people who lovingly birthed it and raised it – the mathematicians and engineers – have also doomed it to a marginal existence within society, by declaring it to be a sub-branch of math and engineering.

In its most general form, computer programming has nothing at all to do with math and engineering. It’s really all about explaining a cooking recipe to a computer, and then having the computer go ahead and follow that recipe. The computer is rather stupid, but it’s incredibly fast, so if it manages to understand your recipe, it can finish cooking you up a batch of a million or a billion cookies around the time you’d be taking the first dozen out of the oven.

When you start talking not about cookies but about searching for a particular size LCD TV in your price range, or finding references to your favorite author in a library, or putting together a new musical passage by looking at all the songs that were ever in the top ten charts, or locating the best Indian restaurant in Pittsburgh within walking distance of an ATM, things start to get interesting. None of these tasks, or countless other tasks that might require a computer program, is inherently about math or engineering.

Rather, they are about getting the computer to follow through on making the decisions that you would have made, except the computer can do it millions of times faster than you ever could.

I would argue that programming without math has been sadly neglected – the concept of sequential logic, things like “if this is true, then try that”, or “look through all of these things, and as soon as I have found the one I want, send a message to my friend”. We shouldn’t be trying to teach kids programming as a special case of math and engineering. We should be teaching them languages that let them get computers to do the things they want computers to do for them.

And that will require us, in our teaching, to move away from the “math/engineering” mindset, and toward a new way of thinking that is centered on what is truly essential about computer programming.