« Back to Table of Content || Next »

Chapter III. The Neurological Basis of Conscious Experience >> The representation of mental images in the brain

The representation of mental images in the brain

Due to the subjective nature of mental images, the process of mental image representation is difficult to study in experimental animals. It is difficult to control an animal’s mental images when the animal’s eyes are closed. Thus, much of the work has been done not with mental images, but with visual percepts (the perception formed by a visual input). Since there is strong evidence that visual percept and mental image share underlying neural processes, we will first discuss the neural correlates of visual percepts.

Neural correlates of visual percept

Images on the retina are transmitted to the primary visual cortex. There is strong evidence that the brain depicts the visual percepts literally, using space in the cortex to represent space in the outside environment. We say that neurons in the primary visual cortex are organized topographically. For example, when a monkey was trained to stare at a pattern (left panel), the physical pattern of active neurons (right panel, darker pixels correspond to greater neuronal activity) was clearly a geometrical representation of the pattern physically laid-out on the cortex (Tootell, 1982).

Macaque monkeys were trained to stare at a pattern (left panel) while injected with radioactive glucose (¹⁴Clabeled 2-deoxy-D-glucose). The radioactive glucose was absorbed and metabolized by active neurons to a much greater extent than by other neurons. After the experiment, the animals were sacrificed and the cortical radioactivity pattern was analyzed. This method provides high resolution radioactive labeling of active neurons. The physical pattern of active neurons (right panel, darker pixels correspond to greater neuronal activity) is clearly a geometrical representation of the pattern physically laid-out on the cortex. This experiment clearly demonstrates that the visual cortex relies on topographical representation of spatial information (adapted from Tootell, 1982).

Further, if a part of the primary visual cortex (V1) is damaged (because of stoke or tumor), a blind spot develops in the corresponding part of the visual field. When two parts of the visual cortex are damaged, the closer the damaged areas are physically in the cortex, the closer the blind spots are in the visual field.

Finally, transcranial magnetic stimulation of the visual cortex demonstrates a similar result. Stimulation of a small area in the visual cortex elicits a perception of a bright flash of light. When two stimulated sites are located close to each other, the bright flashes of light appear close in space. When two stimulated sites are farther apart in the physical cortex, the flashes of light appear farther apart in space (Kastner, 1998).

It should also be mentioned that the active neurons in the figure above do not just code the geometrical pattern. Neurons in the visual area can be specific to orientation of line segments, color, and binocular disparity (important for depth appreciation). However, we can still conclude that the visual cortex relies on the topographical representation of spatial information. In very simplistic terms, when we focus on a cup, the pattern of activity in the visual cortex represents the cup topographically. The topographic organization of the primary visual cortex should not be surprising. After all, many other cortical areas, including somatosensory and motor cortices (see illustration in Appendix 2), are organized topographically. Even the primary auditory cortex is organized tonotopically: certain cells in the auditory cortex are sensitive to specific sound tones and cells that are sensitive to tones with similar pitch are located in close proximity.

Relationship between visual percept and mental image

As pointed out by Kosslyn and colleagues (Kosslyn, 2006), there is strong evidence that visual percept and mental image share underlying neural processes. The most simplistic way to picture this is to assume that cortical neurons activated during visual perception of an object are re-activated during the recall of the object from memory. Here is a summary of experimental evidence:

1. Klein asked six subjects to visualize bow tie patterns. These patters were oriented either vertically or horizontally. The activation of the visual cortex was monitored by functional magnetic-resonance imaging (fMRI, a device that makes images showing the location of increased blood-flow that is normally associated with increased neuronal activity). The pattern of activation during the visualization task closely matched the pattern of activation during visual perception of similar objects. The patterns of visual cortex activation differed significantly between the horizontal and the vertical imagined stimuli, and “colocalized with the horizontal and vertical visual field meridians, as determined for each individual using a retinotopic localizer task” (Klein, 2004).

2. Mental images can interfere with visual perception and this interference is location specific: the interference is greatest when the mental image is formed directly over the visual percept (Craver-Lemley, 1992).

3. Parallel deficits were reported in mental image formation and in visual perception in patients with brain damage (see Ganis, 2003, for review). First, patients with unilateral damage in the occipital lobe who had lost visual perception in half of the visual field had trouble forming mental images in that part of space (Butter, 1997). Second, patients with damage localized in one hemisphere, who neglect half of the space during visual perception, also neglect the same half of space when forming a mental image. A stroke in the right hemisphere can lead to neglect of the left side of the visual field, causing a patient to behave as if the left side of the sensory space were nonexistent. In extreme cases, a patient with neglect may fail to eat the food on the left half of the plate, even though the patient may complain of being hungry. If a patient with neglect is asked to draw a clock, the drawing may show only the numbers 12 to 6, with the other side of the clock left blank. Patients with neglect may also ignore one side of their body, shaving or applying make-up only to the non-neglected side.

Finally, damage to the visual cortex (due to exposure to toxins such as carbon monoxide) often leads to a patient’s inability to recognize objects (object agnosia) while other visual functions such as acuity, color vision, and brightness discrimination may still be intact. Patients with object agnosia have trouble visually recognizing, copying, or discriminating between different objects. Often, such patients also have difficulties imagining those objects. For example, patients who are impaired at face recognition often have difficulty imagining faces (Shuttleworth, 1982; Levine, 1985).

4. Transcranial magnetic stimulation is a noninvasive method to excite neurons in the brain. Electromagnetic induction from a coil of wire (incased in plastic) triggers some neurons to fire action potentials. When transcranial magnetic stimulation is applied repetitively, it can desensitize (inactivate) a patch of neurons under the coil. Kosslyn and colleagues used transcranial magnetic stimulation to temporarily desensitize small patches of the cortex (Kosslyn, 1999). Following the desensitization of the visual cortex, the performance of subjects in both visual imagery and visual perception tests was degraded. Statistical calculations indicated that not only was the performance degraded, but that it was degraded by the same degree in visual imagery and visual perception. Desensitization of the cortex outside of the visual cortex did not degrade a subject’s performance in either test. The fact that desensitization of the visual cortex degrades performance in mental imagery tasks indicates that the visual cortex plays an active role in performing the task.

5. Cortical activity measured by Positron emission tomography, functional MRI, and Single photon emission computed that the tomography during visual perception overlaps with brain activity measured during mental imagery. The evidence of cortical activity measured during mental imagery is summarized in the book “The Case for Mental Imagery” (Kosslyn, 2006). The book also provides an excellent in depth discussion of the physical properties of mental images. The authors conclude that visual percepts and mental images share underlying neural processes.

Thus, experimental evidence indicates that the same cortical real estate is used for representation of visual percepts and mental images. Why would nature use the same cortical areas for tasks that are seemingly so different?

Why do visual percept and mental image share an underlying neural representation?

One of the major roles performed by visual perception is that of distance measuring: measuring the size of an object, the distance to an object, and the distance between objects. If you see an animal, you want to know how big it is; if you want to hide inside a cave, you want to know if you could fit into the cave opening; if you want to throw a stone at a predator, you need to know how far the predator is; if you are planning to run to a tree for cover, your visual system needs to calculate how far the tree is. The visual cortex, of course, is essential for all these tasks. Humans can also perform these tasks in their mind with their eyes closed by imagining the scene and then mentally measuring the distance.

There is strong experimental evidence that measuring distances in a visual percept and in a mental image share the same mechanism: the time to scan across an imagined object increases linearly with the distance scanned, suggesting a mechanism similar to that of distance measurement of a physical object (see Denis & Kosslyn, 1999, for review).

It makes perfect evolutionary sense that expensive cortical real estate is shared between systems that perform identical tasks (just like scientists from multiple departments tend to share expensive equipment or the way tenants in a condominium share one swimming pool). It would be highly inefficient to have two different cortical areas involved in distance measurement: one for measurement of physical objects and another one for measurement of imaginary objects (when the eyes are closed). In addition, if the systems were separate, it is unclear how the mental imagery system could have been trained to measure distances. After all, the visual perception system is trained to measure distances to objects and distances between objects during the organism’s interaction with physical objects (during child’s development). The brain makes predictions based on the visual percept and these predictions are tested by touching the physical objects. In this process, the neuronal connections are refined to analyze the relationship between object properties (such as object size) and the spatial parameters (such as distance to the object). The rules for measuring the distance are ingrained into the visual cortex and applied automatically. The mental imagery system, on the other hand, does not have physical objects to test its predictions. Thus, shared neuronal real estate allows the mental imagery system to re-use the facilities developed by the visual percept system: just like scientists from one laboratory use methodology developed in another laboratory. This is a simple and elegant solution.

Insights from blind patients

Blindness, that is, the inability to form a visual perception can be caused by (1) visual information not reaching the cortex (for example due to damage to both eyes or to a pathway from the eyes to the cortex) or (2) damage to the visual cortex. In the first case, completely blind patients may still be able to use the visual cortex to form mental images (note that these patients cannot form a visual percept). Sadato and colleagues compared the brain activity in blind and non-blind people using the Braille system for reading (Sadato, 1996). The Braille system was originally developed in response to Napoleon's demand for a code that soldiers could use to communicate silently at night. Each Braille character is made up of six raised dots, arranged in a rectangle containing two columns of three dots each. The brain activity of blind subjects reading text written in Braille characters, measured by positron emission tomography, showed an activation of visual cortical areas. The non-blind subjects performing the same task showed a deactivation of visual cortical areas. The authors concluded that “in blind subjects, cortical areas normally reserved for vision may be activated by other sensory modalities” (Sadato, 1996).

In the second case, damage to the visual cortex may be associated with a specific deficit in mental imagery. In the most profound case of bilateral damage to the visual cortex, patients are cortically blind. These patients cannot form a visual percept and the majority of these patients have difficulty forming visual mental images.

However, there are reports that some of these patients can solve visual-based tasks. How can these patients solve visual-based tasks without the ability to form mental images? Here is an insight provided by Kosslyn and colleagues (Kosslyn, 2006, page 116):

“One possible account for this result rests on the idea that the tasks may not have required object-based imagery. For example, one of us once saw a patient who had a major damage to her occipital lobes. When asked whether the uppercase letter A had any curved lines, she literally drew it in the air, and then responded, “No”. She clearly was using kinesthetic feedback to answer. Her ability to draw need not have been guided by visual imagery; rather, purely spatial or motoric information would have been sufficient to allow her to make this response.

Another account focuses on the fact that topographically organized areas are present not only in occipital and parietal lobes, but at many levels of hierarchy in the pathways within these systems.”

Thus, patients with damage to the visual cortex can sometimes draw upon other cortical areas for the tasks normally solved in the visual cortex. This should not surprise us as redundancy is a hallmark of nature. There are normally several ways to solve any problem: to reach a banana hanging on a tree, one can climb the tree, jump, use a stick to break the banana off the tree, build a pile of dirt under the tree and climb on it to reach the banana, or swing the tree. It is likely that some tasks that are normally solved in the visual cortex can be solved in the motor cortex, or maybe even in the somatosensory cortex. After all, both motor and somatosensory cortices are organized topographically (see Appendix 2). In addition, the brain can often reprogram its hardware, especially when the damage occurs at a young age. In this case the brain area normally reserved for one task can be rewired to perform a very different task.

From the evidence presented above, we can conclude that the visual cortex contributes to both visual perception and mental imagery. It is likely that, in individuals with no brain damage, mental images share neuronal real estate with that used for visual perception of physical objects. In other words, when you close your eyes and generate a mental image of a cup, tens of thousands of neurons located in the primary visual cortex, V2, V4, inferior temporal cortex, and other parts of the visual cortex are activated. It is likely that many of those neurons are the same neurons that respond to a physical cup visible to the eyes.

Now that we have discussed the cortical location of neurons involved in the formation of mental images, we are ready to switch our attention to the pattern of neuronal activity necessary for conscious awareness of those images. There is strong evidence that for a subject to be consciously aware of an object, neurons representing that object must fire synchronously.