Why the brain uses “sparse codes”

Gabor Filter showing a receptive field of a hypothetical simple-cell in the visual cortex (Wikipedia)

Gabor Filter showing a receptive field of a hypothetical simple-cell in the visual cortex (Wikipedia)

In 1972, Horace Barlow, great-grandson of Darwin, wrote an article in which he put forth “A neuron doctrine for perceptual psychology?” (The question mark at the end shows that Barlow was a contemplative guy.) With this doctrine, Barlow tried to relate the firing of neurons in sensory pathways with subjectively experienced sensation. One of the principles he introduces is: at progressively higher levels of sensory processing, information is carried by fewer neurons because the system is organized to a near complete a representation with the fewest active neurons. In other terms, the encoding of sensory information gets ‘sparser’ as one moves up into higher levels of sensory processing. Since then, a number of lines of evidence have converged, supporting Barlow’s proposition and the general value of ‘sparse codes.’

On the blog today I’ll be reviewing one such paper, “Sparse coding of sensory inputs” by Bruno Olshausen and David Field. This paper defines ‘sparse coding’ as a computational strategy where brains encode sensory information using a small number of simultaneously active neurons at a given time[1].

This paper puts forth four ideas for why sparse coding theoretically might be a good strategy:

  1. More memories can be stored
  2. Makes use of the statistical structure of natural signals
  3. Represents data in a convenient way for further processing
  4. Save metabolic energy, by decreasing neuronal firing rates

When model neurons are trained to optimize spare representations of natural scenes, the receptive fields that emerge represent the simple-cells of the primary visual cortex.

Learned receptive fields to maximize sparseness of natural scenes (Olshausen and Field, 2004, Current Opinion in Neurobiology)

Learned receptive fields to maximize sparseness of natural scenes (Olshausen and Field, 2004, Current Opinion in Neurobiology)

One interesting feature of sensory representations that this paper points out, is that as visual signals move from the thalamus to the visual cortex, there is a 25:1 expansion (of axonal projects to cortex versus axonal projections to the LGN of the thalamus). They suggest that this 25:1 expansion possible emerges as a compromise between the trade-offs: with high sparseness eventually ending in ‘grandmother cells’ where a single unique neuron represents each element of a sensory[2], and low sparseness incurring developmental and metabolic costs of having to use many neurons to encode each element. Additionally, the more a neuron spikes the more energy is needed to maintain it’s electrochemical gradients through pumps like the sodium-potassium ATPase. From what we know about the metabolic use of the cortex, scientists have estimated that only 1/50th of neurons are active at any given time.

Another interesting point this paper makes, is that some experimental results show neurons with much higher firing rates than would be predicted by metabolic estimates. Because these experiments often involve searching for firing neurons with an electrode, we may be systematically biasing our studies towards a minority of neurons that fires “less sparsely” than the general populations. Solutions to this bias include chronically implanting electrodes where the positioning is set anatomically or using antidromic stimulation to identify neurons as opposed to stimulus elicited firing.

Sparse coding beyond sensory systems

This paper also points out that sparsely firing neurons are observed in motor cortex during movements, and that experimentally driving a single neuron can be enough to initiate whisker movements in rats (Brecht et al., 2004). In the zebra finch song production pathway, HVC neurons fire sparsely at precise points in song, and precise spike-timing in RA may be important as well.

How does one actually measure “sparseness”

Olshausen and Field, 2004, Current Opinion in Neurobiology

Olshausen and Field, 2004, Current Opinion in Neurobiology

Olshausen and Field write that a standard measure of “sparseness” is kurtosis, with a larger value indicating a “sparser” distribution.

Kurtosis equation sparseness

They also describe another method developed by Rolls and Tovee, the activity ratio, which is specialized for one sided distributions (and therefore good at modeling neurons since firing rate cannot drop below zero).

Activity Ratio equation sparseness

Finally, The activity ratio can then be scaled from 0-1, using Vinje and Gallant’s[3] sparse coding scale transformation:



I think there is compelling evidence for brains using of sparse coding in sensory systems and there are good theoretical reason for why brains should use sparse coding. It will be interesting to see if these findings hold up for motor systems as well. If they find evidence of sparse coding in relatively simple and blunt movements like locomotion, I would guess that they will be more present in higher premotor areas encoding complicated and learned movements.

I big issue in machine learning is how to process the data before unleashing machine learning algorithms on it. I think neurophysiology shows that a good strategy might be to duplicate the data and turn it into over complete sparse representations. I assume scientists working with big data are already doing this, but honestly I have no idea.


Barlow HB: Single units and sensation: a neuron doctrine for perceptual psychology? Perception 1972, 1:371-394.

Brecht M, Schneider M, Sakmann B, Margrie TW: Whisker movements evoked by stimulation of single pyramidal cells in rat motor cortex. Nature 2004, 427:704-710

Olshausen BA, Field DJ (2004). Sparse Coding of Sensory Inputs. Current Opinion in Neurobiology, 14: 481-487.

Rolls ET, Tovee MJ: Sparseness of the neuronal representation of stimuli in the primate temporal visual cortex. J Neurophysiol 1995, 73:713-726.

Vinje WE, Gallant JL: Sparse coding and decorrelation in primary visual cortex during natural vision. Science 2000, 287:1273-1276.

[1] I had always thought of sparse coding from the perspective of individual neurons, as a strategy where a given sensory neuron fires only at a very specific stimuli or aspect of a stimuli, so therefore it fires ‘sparsely.’ I realized in reading this, that really this paper’s definition and my mental understanding are two sides of the same coin if the neurons are firing independently. However, even with a population of neurons that only fire very rarely might not be ‘sparsely coding’ by Olshausen and Field’s definition if they are all firing together and then silent together.

[2] How does that neuron spur the rest of the brain into action and what occurs when that neuron dies?

[3] This is the same gallant that did some of the coolest experiments ever decoding neural activity from fMRI activation: https://neuroamer.wordpress.com/2011/10/31/scientists-record-lucid-dreams-with-eeg-and-fmri-simultaneously/


No Responses Yet to “Why the brain uses “sparse codes””

  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: