Memory Network – Computer Says “YES”
Ever wonder how much information we put in our heads? The answer: a lot. For starters, a typical vocabulary is 50,000-250,000 words. And then there are all the little details that stretch back decades — the house we grew up in, the time we spilled orange juice on our test back in third grade, the solution to a quadratic equation (for some of us).
So where do we put it all? If we had hard drives in our heads, the answer would be easy: we would store memories as 0s and 1s. But we don’t, we have neurons, connected by synapses, and storing memories in such systems is a lot harder than putting 0s and 1s on a hard drive. Nevertheless, about two decades ago John Hopfield showed that memories could be stored by modifying the strength of synapses in a particular way.
Importantly, the number of memories that could be stored using his scheme was proportional to the number of neurons in the network. This solved the storage problem: there are about 50 million neurons in a cubic centimeter of cortex, plenty of room for both a vocabulary and spilled orange juice.
Recently, Roudi and Latham from University College London threw a monkey wrench into this picture. In a study publishing in PLoS Computational Biology on September 7, 2007, they show that for realistic networks of spiking neurons, the number of memories is not proportional to the number of neurons, it’s proportional to the number of connections per neurons — at most about 10,000.
Moreover, they provided evidence that the constant of proportionality is small, not more than a few percent, and they eliminated one of theorists’ favorite tricks — reducing the number of neurons involved in any one memory — for increasing that constant. Thus, if networks use the algorithm proposed by Hopfield, they can store at most about 500 memories, no matter how many neurons they contain.
So we’re not exactly back to square one, but we’re not much farther than square two: we no longer know how the brain holds so many memories. Roudi and Latham speculate that the answer lies in multiple, weakly coupled networks. However, until that, or some other idea, is shown to be correct, we will have to be content with just remembering, without the added knowledge of how we remember.
A previous version of this article appeared as an Early Online Release on June 5, 2007 (doi:10.1371/journal.pcbi.0030141.eor).
PLoS Comput Biol 3(9): e141 doi:10.1371/journal.pcbi.0030141
A Balanced Memory Network
Yasser Roudi*, Peter E. Latham
1 Gatsby Computational Neuroscience Unit, University College London, London, United Kingdom
A fundamental problem in neuroscience is understanding how working memory—the ability to store information at intermediate timescales, like tens of seconds—is implemented in realistic neuronal networks. The most likely candidate mechanism is the attractor network, and a great deal of effort has gone toward investigating it theoretically. Yet, despite almost a quarter century of intense work, attractor networks are not fully understood. In particular, there are still two unanswered questions. First, how is it that attractor networks exhibit irregular firing, as is observed experimentally during working memory tasks? And second, how many memories can be stored under biologically realistic conditions? Here we answer both questions by studying an attractor neural network in which inhibition and excitation balance each other. Using mean-field analysis, we derive a three-variable description of attractor networks. From this description it follows that irregular firing can exist only if the number of neurons involved in a memory is large. The same mean-field analysis also shows that the number of memories that can be stored in a network scales with the number of excitatory connections, a result that has been suggested for simple models but never shown for realistic ones. Both of these predictions are verified using simulations with large networks of spiking neurons.