As promised, the explanation. Which will require a little detour into the research of my brilliant colleague Yann LeCun, who creates machine learning algorithms. That is, he figures out how to get computers to learn. One of his inspirations comes from the way the human brain itself learns. And that can lead to some pretty interesting questions.
At some level, a computer’s memory consists of bits – and each bit can either be one or zero. There is something analogous that goes on in our brains. Each synapse of a neuron can be thought of as a bit. In response to stimuli from the world, a neuron can either fire a signal across that synapse (one) or not (zero).
It’s not hard to see that it takes a lot more energy for a neuron to make a one than to make a zero. In fact, if all of the neurons in our brain were to fire their synapses at once, the heat generated would fry our brain from the inside, and we’d be dead in less time than it takes to say “fried brains”.
Not surprisingly, the “bits” of our neural synapses tend to spend most of their time at zero, and much less time at one. This tendency saves our brains from frying, but at a cost: it means we need a lot more neurons. I’ll explain.
Given, say, a million different things to distinguish, if you had to have a synapse for each one, then you’d need to use a million synapses. This is the base one representation we talked about yesterday – like representing the number 42 with forty two zeroes. But there are many many millions of things that humans need to distinguish, and if we worked this way, pretty soon our brains would run out of space. This is sometimes known as the “Grandmother problem”: If there had to be a special synapse that fired only when you recognized your Grandmother (as well as one dedicated synapse for each such recognition task), then your head would probably need to be as big as a house. And that would most likely upset your Grandmother.
There are two opposing influences here: Our brains need to have a lot more zeroes than ones, so they won’t fry anytime we think too hard, but they can’t have all zeroes, or you run into the Grandmother problem. What the brain actually does is a compromise – something between base one (Grandma gets her own synapse) and base two (deep fried brain from too many synapses firing all the time). The brain actually works somewhere between base one and base two: mostly zero bits, and then the occasional one bit.
Yann uses something like this property in his work. Yann’s learning algorithms represent aspects of the things to be recognized as long strings of bits. In any given string, almost all the bits are zero, and only a few are one. Like 0000100001000000000100000100. It turns out that this is a good way to make machine learning algorithms efficient: by allowing the occasional one bit, the strings can be kept reasonably short (ie: you can avoid the Grandmother problem). And by making sure most of the bits are zero, you can get away with a lot less computation, because you can concentrate all of the computing work just where the one bits are. Just as your brain avoids getting fried by not firing too many synapses at once, Yann’s algorithms avoid frying the computer’s CPU by keeping down the number of computations.
This is all very interesting, but it’s a little abstract, isn’t it? Are we really talking here about something like base 1.7 or base 1.3? Well, yes we are. But in order to explain properly, I’m going to need to recruit that army of short people in elevators. Which I will do tomorrow.
“In fact, if all of the neurons in our brain were to fire their synapses at once, the heat generated would fry our brain from the inside, and we’d be dead in less time than it takes to say ‘fried brains’.”
I smell a DoD funding opportunity – the neural activator!