« Back to Table of Content || Next »

Chapter III. The Neurological Basis of Conscious Experience >> Neuroscience of perception

Neuroscience of perception

In this section, we will consider the experiments that are most revealing of the workings of the mind. We will then discuss the experimental data in the context of the theory of mental synthesis.

Functional differentiation between the left and right hemispheres

The remarkable functional differentiation between the left and right hemispheres of the brain has been demonstrated in a number of experiments performed on so-called split-brain patients. In these patients, for whom epilepsy could not be controlled with drugs, the two hemispheres are disconnected in order to disrupt the spread of epileptic activity from one hemisphere to the other. The connection between the two hemispheres, called the corpus callosum, is cut using a medical procedure developed over 50 years ago. Strikingly, the disconnection between the two hemispheres does not lead to changes in everyday life, intellect, or temper. Michael Gazzaniga and other scientists have studied the effects of the disconnection for over 35 years. Scientists have been able to develop ways to experiment with one hemisphere at a time (e.g. Gazzaniga, 2000). For example, pictures positioned in the left visual field are only available to the right hemisphere, and pictures shown in the right visual field are only available to the left hemisphere. Therefore, by showing a picture for a short period of time to the left or to the right of the fixation point, scientists ensured that only one hemisphere was allowed to view the picture. When a picture was shown to the left hemisphere, the patients could describe what they saw. But when a picture was shown to the right hemisphere the patients said that they saw nothing. This occurs because linguistic centers are located in the left hemisphere. The right hemisphere, which saw the picture, could not transmit any information to the talking left hemisphere, and the left hemisphere declared that it saw nothing. Did the right hemisphere really see the picture? Yes. When the patients were asked to point to the object they saw among a series of pictures, they did it with ease. This experiment shows us that both hemispheres contain an apparatus for matching a visual percept to memory.

Experiments with split-brain patients are not the only way to study the differences between the left and right hemispheres. In patients undergoing half-brain electroconvulsive therapy, the two hemispheres are disabled one at a time. In this treatment, a strong electric current flows between two electrodes, one on the forehead and one on the posterior side of the head. Tests can be performed on the unaffected hemisphere during the short time interval following the shock that disables the other hemisphere.

The combined experimental evidence from both approaches indicates that the two hemispheres process visual information differently. The right hemisphere’s visual processing is holistic while the left hemisphere is analytical. For example, the right hemisphere has difficulty in recognizing objects that are missing even minor details and is completely unable to recognize objects based only on a minor detail. The right hemisphere is best at recognizing whole, complete objects. On the other hand, the left hemisphere tends to analyze the visual percept into details and base its recognition on the major available detail. In an experiment in which a picture is shown for a very short time, the left hemisphere will not even notice that a small detail is missing. For example, when shown a cow without a tail, the left hemisphere will never report that the tail is missing.

In peoplewith intact brains, of course, both hemispheres process visual information simultaneously: the right hemisphere identifies complete objects and stores neuronal ensembles encoding complete objects in memory (we refer to this type of processing as parallel), while the left hemisphere analyzes the visual percept into the smallest meaningful details, each detail encoded by its own neuronal ensemble. Neuronal ensembles representing each detail are processed sequentially (we refer to this type of processing as serial) and can be recalled from memory separately.

There is some evidence that a functional differentiation between the left and right hemispheres exists in non-human primates. Split-brain monkeys were better able to distinguish monkey faces while using the right hemisphere as opposed to the left hemisphere (Hamilton, 1988). Also, it has been reported that, in monkeys, there exists a specialization of the left hemisphere in auditory processing (Heffner, 1984). These observations are consistent with a conjecture that the last common ancestor of chimpanzees and humans exhibited, to a small degree, a functional differentiation between the left and right hemispheres. During the five million years of evolution on its own, the hominid brain has taken functional differentiation to an extreme. The right hemisphere remained mainly holistic and animal-like in its visual processing, while the left hemisphere developed visual analysis of objects, target recognition based on elements of the object (an ear or a tail visible above the grass), and, eventually, the capacity to synthesize visual elements represented by distinct neuronal ensembles into a new, never-before-seen object.

Neural mechanisms of visual analysis

On a neuronal level, visual analysis is associated with dissecting a visual image into a number of smallest meaningful neuronal ensembles. Each ensemble is then stored in memory for matching to a stimulus at a later time. Recall the short-term memory experiment conducted by Dr. Matsuzawa. In this experiment, five numbers were flashed on the screen for less than a second, after which the numbers were replaced by white squares. Subjects had to remember the position of the numbers and touch the white squares in the order corresponding to that of ascending numbers. When the numbers were displayed for about 700 milliseconds, the chimp Ayumu and the college students were both able to correctly remember about 80% of the numbers. However, when the numbers were displayed for just 210 milliseconds, the chimp scored about 80%, while the best humans scored 40%. Why is it that humans could only remember two numbers out of five, while the chimp was able to remember four numbers out of five?

It is likely that automated visual analysis by the dominant left hemisphere is to blame for the humans’ poor performance. When presented with five numbers on the screen, humans automatically rush to analyze the visual percept and match each number to its representation in memory. Let us hypothesize about what is happening in the brain during this process. The attention rhythm is about 15Hz. In other words, there are 15 frames per second in which visual information can be analyzed, each frame lasting for about 60 milliseconds. In the first frame, the complete visual percept is represented in the visual cortex. In order to match the first (lowest) number to its representation in memory, the brain must separate the neuronal ensemble representing the first number from the rest on the visual percept. It is likely that this is accomplished by shifting all neurons except those coding for the first number out-of-phase with the attention rhythm. If we could measure the neuronal activity of the visual cortex during the second frame at the upswing of the attention rhythms wave, we would detect the action potentials only in neurons representing the first number. This activity allows the human to match the visual pattern to memory and thus remember where the first number is located. Note that the human has already spent 120 milliseconds (two frames) to remember the location of the first number. To identify the second lowest number, the human must re-synchronize the complete visual percept with the attention rhythm. This happens in the third conscious frame. Matching of the second number to memory occurs in the fourth conscious frame. The complete process of matching all five numbers takes ten conscious frames and 600 milliseconds (10 frames x 60 milliseconds). In the experiment conducted by Dr. Matsuzawa, both humans and chimpanzees performed equally and identified 80% of the numbers correctly when the numbers were displayed for 700 milliseconds. However, when the numbers were displayed for only 210 milliseconds, human performance dropped to 40%. This makes sense because in 210 milliseconds, humans only had enough time to identify the two lowest numbers. Identification of the two numbers takes four frames (4 frames x 60 milliseconds=240 milliseconds). Some time during the fourth frame, the numbers were substituted by white squares. When the human mind re-synchronized a complete visual percept with the attention rhythm in the fifth frame, the visual percept consisted of white squares and further number identification was impossible.

Visual analysis being performed by a human subject in the short-term memory experiment conducted by Dr. Matsuzawa. No more than one neuronal ensemble can be attended to by the consciousness in one mental frame. Visual analysis automatically rushes to analyze the five numbers. It takes at least 10 mental frames or 600 ms to analyze five numbers. Mental frame 1 (after 0 ms): the complete visual percept (the screen) is synchronized with the attention rhythm. Mental frame 2 (after 60 ms): the neuronal ensemble representing the first number is separated from the rest on the visual percept and its location is memorized. Mental frame 3 (after 120 ms): the complete visual percept is synchronized with the attention rhythm. Mental frame 4 (after 180 ms): the neuronal ensemble representing the second number is separated from the rest on the visual percept and its location is memorized. Mental frame 5 (after 240 ms): the complete visual percept is synchronized with the attention rhythm. Mental frame 6 (after 300 ms): the neuronal ensemble representing the third number is separated from the rest on the visual percept and its location is memorized. Mental frame 7 (after 360 ms): the complete visual percept is synchronized with the attention rhythm. Mental frame 8 (after 420 ms): the neuronal ensemble representing the fourth number is separated from the rest on the visual percept and its location is memorized. Mental frame 9 (after 480 ms): the complete visual percept is synchronized with the attention rhythm. Mental frame 10 (after 540 ms): the neuronal ensemble representing the fifth number is separated from the rest on the visual percept and its location is memorized.

When the numbers are substituted by squares 210 ms after the start of the test, a human subject can identify no more than two numbers.

The chimpanzee has a different strategy. The chimpanzee does not rush to visual analysis but remembers a complete visual percept; the number identification is performed at a later time. The difference in visual percept processing reflects the difference in the evolutionary path over the last five million years. While chimpanzees stayed in the relative safety of the treetop canopies, hominids foraging in savannas acquired visual analysis for faster predator identification based on few visual clues. It is possible that if the left hemisphere with its visual analysis was suppressed or otherwise not used in the experiment (for example by showing the display to the right hemisphere only), human subjects matching the numbers using only their right hemisphere could improve their performance and achieve a result that is comparable to that of the chimpanzee.

Organization of visual neuronal ensembles in the cortex

It can be assumed that the basic rules for organization of neuronal ensembles are similar for both the left and the right hemispheres. The difference, of course, is in the nature of neuronal ensembles. While the right hemisphere stores information about complete objects, the left hemisphere tends to split an object into the smallest meaningful elements; each element then becomes a neuronal ensemble on its own.

As we discussed above, it is likely that the cells that form neuronal ensembles are located in many parts of the cortex, particularly in multiple visual areas along the ventral stream in the occipital and temporal lobes. The specificity of neurons along the ventral stream increases from the primary visual area to the areas in the inferior temporal lobe: the neurons in the primary visual area (V1) are the least specific; the neurons in the inferior temporal lobe are the most specific - they only fire action potentials when their specific target appears in the visual field. In other words, the neurons of V1 can be part of many neuronal ensembles, while neurons in the inferior temporal lobe are likely to be involved in only one or a few neuronal ensembles. If a neuron in the temporal lobe is electrically stimulated, a complete neuronal ensemble may be activated as a result. In that case, the subject will report a conscious experience of seeing an object. Neurosurgeon Wilder Penfield invented a procedure in which it was possible to treat patients with severe epilepsy by destroying the nerve cells in the brain where the seizures originated. Before operating, Penfield stimulated the brain with electrical probes while the patients were conscious on the operating table, and observed their responses. In this way, he could more accurately target the responsible areas of the brain, reducing the side-effects of the surgery. Penfield reported that stimulation of the temporal lobes could lead to vivid recall of memories (Penfield, 1975; also see Appendix 2 for a detailed description and a discussion of Penfield’s observations). Conversely, when neurons in V1 were electrically stimulated, subjects only reported seeing a flash of light. Since neurons in V1 are part of a great many neuronal ensembles, stimulation of one cell does not evoke activity in any complete neuronal ensemble and the subjects do not experience seeing any objects.

Thus, the neurons in the temporal lobe play an especially important role in activating and maintaining the neuronal ensembles. For example, damage (particularly bilateral damage) to the fusiform gyrus of the inferior temporal lobe may result in prosopagnosia, a complete inability of a subject to identify faces (prosopagnosia comes from the Greek prosopon meaning "face" and agnosia meaning "non-knowledge") . In the absence of bonding neurons in the fusiform gyrus, the neuronal ensembles encoding faces cannot activate into synchronous firing that is necessary for face recognition.

A simplistic model of the visual system along the ventral stream can work like this: during the visual observation of a physical stimulus, the neurons firing in synchrony with the attention rhythm change the strength of the connections between them, forming a neuronal ensemble in the process. You can visualize this ensemble as a pyramid that includes a high number of relatively non-specific neurons in V1 that encode the contours of the object, its color, the spatial depth (by means of binocular disparity), etc; a fewer number of relatively more specific cells in the extrastriate areas such as V2 and V4 that encode simple shapes; and even a fewer number of very specific neurons in the inferior temporal lobe.

A schematic representation of a neuronal ensemble. A neuronal ensemble pyramid includes a high number of relatively non-specific neurons in V1 that encode the contours of the object, its color, the spatial depth, etc. (pyramid’s base); a fewer number of relatively more specific cells in the extrastriate areas such as V2 and V4 that encode simple shapes; and even a fewer number of very specific neurons in the inferior temporal lobe (pyramid’s tip).

The memory of the stimuli is stored in the increased connection strength between all the cells in the ensemble: those in V1, extrastriate areas, and inferior temporal lobe. When one’s eyes are closed, the image of the stimulus can be recalled by activating the neuronal ensemble corresponding to the stimulus. The recall occurs when neurons of the ensemble fire synchronously in-phase with the attention rhythm during at least one conscious frame. Any part of the ensemble can influence the self-organization of the neurons into synchronous firing. However, the cells in the temporal lobe have the highest probability for triggering this event. This model likely explains why Penfield’s patients reported vivid recall of memories upon stimulation of the temporal lobe; the stimulation elicited the synchronization of a complete neuronal ensemble. Meanwhile, stimulation of a few neurons in V1 was not enough to trigger activity of a complete neuronal ensemble. One needs to stimulate a significantly larger proportion of neurons located in V1 to trigger activity of the complete neuronal ensemble.

The model also explains the dramatic loss of specific memories (such as faces) following damage to the temporal lobe; it is more difficult for the ensemble to activate without the organizing role of neurons in the temporal lobe. Finally, the model suggests a mechanism for the conscious control of mental images: rather than directly stimulating all neurons in an ensemble, the frontal lobe can trigger the synchronous firing of the ensemble by stimulating a few cells in the temporal lobe.

Visual perception of ambiguous images

When you first look at the painting by Salvador Dalí reproduced below, what do you see?

Slave Market with the Disappearing Bust of Voltaire  by Salvador Dalí ©Salvador Dali Museum, St. Petersburg, Florida, USA/ The Bridgeman Art Library/Artists Rights Society

Most people immediately perceive the slave market with two nuns in the center of the painting under the arch. However, when you look again, the visual percept rearranges itself and you perceive the bust of Voltaire. The faces of the nuns emerge as Voltaire’s eyes and the arch opening above the nuns emerges as Voltaire’s head. The painting’s title is Slave Market with the Disappearing Bust of Voltaire. When Dalí created this painting, he was interested in the capacity of the mind to perceive different images from the same painting. Other Dalí paintings that result in an ambiguous visual perception include: Old Age, Adolescence, Infancy (The Three Ages), The Hallucinogenic Toreador, and many others.

I first looked at the painting Slave Market with the Disappearing Bust of Voltaire when I was about 15 years old. I spent hours staring at the painting, trying to comprehend what was changing in my brain when the visual percept changed from the nuns to Voltaire and back. Since the image on my retina did not vary, the change in the perception must have been associated with changes in the activity of the cortical neurons. Where are those cortical neurons that command perception located? How does their activity change? What part of the brain controls the change in perception? We can now try to answer these questions.

When one first looks at the painting, one’s visual system rushes to analyze the painting and to identify the smallest meaningful elements (in the process of visual analysis). As a result of visual analysis, the two nuns are perceived under the arch (remember that hominids were trained during their evolution in savannas to look for the smallest meaningful visual detail such as a tail visible above the grass). The left and right nuns are each represented by their own neuronal ensembles. This means that in at least one conscious frame, all the neurons representing the nuns (first the left nun, then the right nun), were shifted in-phase with the attention rhythm. During that frame, a visual memory of the nun was formed: the group of neurons representing the nun (the neuronal ensemble of the nun), that is, the cells that are active and synchronous with the attention rhythm in that particular frame, form an internally connected and coordinated unit. The connections between the neurons in the ensemble are modified in such a way that in the future these neurons can be activated as one unit in a memory recall process. In other words, after the neuronal ensemble of the nun is formed, you can close your eyes and imagine the nun.

When the bust of Voltaire is recognized, the neurons restructure into a different neuronal ensemble: the one representing the bust of Voltaire. All the neurons of the Voltaire ensemble fire in-phase with the attention rhythm. Neurons representing other parts of the painting are shifted out-of-phase with the attention rhythm; these parts are still represented by neuronal activity in V1 and other parts of the cortex, however one is not aware of their activity since they fire out-of-phase with the attention rhythm. Note that some neurons are part of both ensembles. For example, the neurons representing the nun’s white apron just below the waistline also represent Voltaire’s chin. The neurons are shared by both the nun ensemble and the Voltaire ensemble. The shared neurons are expected to stay in-phase with the attention rhythm independent of whether one is aware of the nuns or of Voltaire.

The phenomenon of visual perception of rivalrous stimuli was studied for over twenty years by David A. Leopold, Nikos K. Logothetis, and their colleagues. When presented with rivalrous stimuli, a monkey’s visual perception (just like a human’s) lapses into a sequence of spontaneous alternations. The switching between the two perceptions occurs every few seconds (again, just like in humans). The scientists trained the monkeys to report which stimulus they perceive while using microelectrodes to record the activity of single neurons located in different parts of the monkey’s cortex. The microelectrodes can measure action potentials in neurons without damaging the neurons. The scientists reported that in V1, most cells’ activity depends on the visual stimulus and does not depend on what the monkey’s perception is. Change in the perception affected the firing rate in fewer than 10% of V1 and V2 neurons. In V4, 40% of the neurons changed their activity when the perception changed. Finally, in the inferior temporal lobe, 90% of the neurons changed their activity when the perception changed.

Let us analyze these findings. Clearly, a monkey’s visual perception lapsed into a sequence of spontaneous alternations between the two neuronal ensembles, each representing a rivalrous stimulus. The question is, how are these ensembles formed and what part of the cortex defines which neuronal ensemble is perceived. We start with V1. We know that the activity of neurons in V1 during visual perception is primarily defined by the image on the retina. Since the image on the retina was not altered in the experiment, we should not expect a significant change in the firing of neurons in V1. That, in fact, is exactly what was observed in the experiment. We should expect, however, a shift in the firing phase of V1 neurons when the perception changes: the neurons of the perceived ensemble are shifted to fire in-phase with the attention rhythm. The phase shift was not investigated in the experiment conducted by Leopold and Logothetis. However, other scientists reported a change in the firing phase consistent with the conjecture put forth in this book (Fries, 1997; Srinivasan, 1999; Fries, 2002; Doesburg, 2005).

For the neurons in V1 to fire synchronously, there must be a master that organizes them into one ensemble. It is likely that the master neurons are located in the temporal lobe. As noted above, neurons in the temporal lobe are likely to be essential for bonding and synchronization of the neuronal ensemble. In other words, the stimulus is perceived only when the master neurons in the temporal lobe are active synchronizing neurons representing the stimulus in V1 (as well as V2, V4, and other visual areas) with the attention rhythm. As soon as the master cells in the temporal lobe cease their activity, the ensemble disintegrates and the corresponding stimulus in no longer perceived. This model is consistent with the experimental finding that most neurons in the inferior temporal lobe change their activity when perception changes. In other words, the neurons in the inferior temporal lobe fire only when their subordinate neuronal ensemble is perceived.

Recall that neurons in the temporal lobe were also implemented in memory recall. In Penfield’s procedure, stimulation of neurons in the temporal lobe activated a complete neuronal ensemble, consistent with the central role of the temporal lobe neurons in activating and maintaining of a neuronal ensemble. We have also discussed that patients with damage to the fusiform area of the inferior temporal lobe were unable to identify faces, suggesting that master neurons in the fusiform area are essential for memory matching. Thus the master neurons in the temporal lobe are essential for all of the following: visual perception of a physical object, forming a mental image of an object from memory, and matching the physical object to memory. We can conclude that (1) in all of these processes, the same neuronal ensemble representing the object (the pyramid) has to be activated and synchronized with the attention rhythm and (2) that the neurons in the temporal lobe play a central role in triggering the self-organizing neuronal ensemble into synchronized activity.

Look again at Dalí’s Slave Market with the Disappearing Bust of Voltaire, and let us discuss what processes are happening in your brain. First, your brain matches visual cues (represented by neurons in V1) to the memory of the nuns (represented by the neurons in the temporal lobe or TL). As soon as the nuns are matched, the neurons in the temporal lobe activate and synchronize the neuronal ensemble representing the nuns with the attention rhythm. At that moment the nuns are perceived. After a few seconds of staring at the painting, the neurons representing the nuns in the temporal lobe cease firing and the neuronal ensemble representing the nuns disintegrates. The mind, freed from perceiving the nuns attempts to match the visual cues to other objects in memory. Thus the visual cues of the bust of Voltaire (represented by neurons in V1) are matched to the memory of Voltaire. At that moment, the neurons in the temporal lobe activate and synchronize the neuronal ensemble representing Voltaire with the attention rhythm and Voltaire is perceived. After a few seconds, the process repeats itself. The process is involuntary: no matter how hard you try to keep the perception of Voltaire, your mind will switch to perceiving the nuns after a few seconds.

Conscious perception flickers between perceiving the nuns and the bust of Voltaire. When the nuns are perceived, the neuronal ensemble representing the nuns (light green pyramid) is activated. When the bust of Voltaire is perceived, the neuronal ensemble representing Voltaire (dark blue pyramid) is activated. Note that most neurons in V1 belong to either one or the other neuronal ensemble, while only a small population of neurons (e.g. representing the white apron) belong to both the Voltaire and the nuns ensembles. The proportion of neurons that belong to both neuronal ensembles decreases as we ascend to more specific areas in V4 and TL.

From an evolutionary perspective, it is clear why ambiguous images are always flickering between the two states. If two interpretations are possible based on the same visual cues, it is beneficial for the organism to consider both perceptions. Suppose that you see a stick at a distance that can be perceived as a stick and as a snake. It benefits one’s survival to consider both possibilities. The species that was limited by the first possible interpretation would never be able to recognize the snake and would eventually be eliminated from the gene pool. Thus, the visual system always attempts to match all available memories to all available visual cues. When it is possible to match two objects to an ambiguous visual cue, the brain will always flicker between the two perceptions so that an individual can decide the best cause of actions based on both interpretations.

For humans, there is a voluntary aspect to the flickering. One can regulate, to a certain degree, when the mind switches between the two interpretations. It is likely that the prefrontal cortex connections to the neurons in the temporal lobe play a role in the voluntary aspect of the flickering.

Analysis of the Necker cube

In Dalí’s Slave Market with the Disappearing Bust of Voltaire, visual cues for the nuns and Voltaire are overlapping but not identical. Some of the neurons in V1 are only part of the visual cues for the nuns while a separate population of neurons is only part of the visual cues for Voltaire. However, this does not need to be the case in order for the drawing to be ambiguous. Consider the Necker cube. The three-dimensional cube can be perceived in two different ways, depending on whether you see the lower-left corner as being on the front face of the cube or on its rear face.

Necker cube. The three-dimensional cube can be perceived in two different ways, depending on whether you see the lower-left corner as being on the front face of the cube or on its rear face.

In the case of the Necker cube, the visual cues for both interpretations are identical. A diagram of the neuronal ensemble formation is shown in the illustration on the opposite page. (1) When one first looks at the Necker cube, the lines are projected from the retina to V1. The population of neurons coding for the lines is indicated by the thick red line. No 3D cube is yet perceived. (2) The mind attempts to fit 3D objects from memory to the available visual cues. Thus the neural ensemble representing the 3D cube is activated (shown in light green) and synchronized with the attention rhythm. At that moment, the 3D cube is perceived. Note that, in addition to V1, the pyramid includes neurons in extrastriate areas (marked V4) and temporal lobe (TL). Also note that additional neurons in V1 were activated to encode perceived depth. (3) After some staring at the drawing, a second possible interpretation is found. The first neuronal ensemble disintegrates and the second neuronal ensemble (shown in dark blue) is synchronized with the attention rhythm. At this time, a second interpretation of the 3D cube is perceived.

A diagram of the Necker cube neuronal ensemble formation.

A comparison of the neuronal ensembles representing the two interpretations of the Necker cube identifies two non-overlapping populations of neurons. These neurons encode the depth information. Population A is only active when the light green ensemble is activated and the first 3D cube is perceived; population B is only active when the dark blue ensemble is activated and the second 3D cube is perceived.

A comparison of the neuronal ensembles representing the two interpretations of the drawing identifies two non-overlapping populations of neurons. These neurons encode the depth information. Population A is only active when the light green ensemble is activated and the first 3D cube is perceived; population B is only active when the dark blue ensemble is activated and the second 3D cube is perceived. Thus, it is likely that the activity of some cortical neurons changes with alteration of the percept. Specifically, the neurons that encode depth information will change their activity when the percept changes. Note that these neurons will not just synchronize/desynchronize with the attention rhythm but will activate/deactivate depending on which 3D cube is perceived. This prediction yields itself to an experiment in which the depth coding neurons can be identified and their activity correlated with perception.

One interpretation at a time

There are a number of other very interesting ambiguous images. Please look at some of them, illustrated here. While looking at the ambiguous image, note how your mind rushes to identify the first possible object based on visual cues. The second interpretation pops up later, sometimes after a significant effort (if you give up, there is a hint written in small grey font).

My Wife and My Mother-In-Law by W. E. Hill. Most people immediately perceive the young lady (the “wife”) Can you recognize the older lady (the mother-in-law”) as well? If you give up, read the hint written in small grey font. The chin of the wife is the nose of the mother-in-law.

Duck-Rabbit illusion by Joseph Jastrow. The beak of the duck forms the ears off the rabbit.

Note that it is impossible to think of both interpretations simultaneously: when a neuronal ensemble representing one interpretation is consciously experienced the other neural ensemble is not. The best you can do is to flicker between the two interpretations. If each neuronal ensemble were represented by a pyramid (as described above), then the flickering occurs between the two pyramids of neurons. Why can’t we see both interpretations in one conscious frame? The reason seems to be simple. In all ambiguous images, some of the neurons are part of both ensembles. They can synchronize with one neuronal ensemble or the other but not with both ensembles simultaneously. This may be a fundamental rule of perception: one neuron can only be part of one neuronal ensemble during one conscious frame. As simple as it sounds, this perception rule implies the presence of a destruction mechanism. When the interpretation changes, the new neuronal ensemble self-organizes, the neurons in the temporal lobe (at the tip of the pyramid) are activated, and the previously perceived neuronal ensemble has to be deactivated. The deactivation must literally shut down the competing neurons in the temporal lobe. Recall that David A. Leopold, Nikos K. Logothetis, and their colleagues reported that 90% of neurons in the temporal lobe changed their rate of firing when the perception changed. This means that nearly all neurons at the tip of the pyramid were actively inhibited. The mechanism of this inhibition is unclear but may involve the prefrontal cortex for the following reasons: (1) the prefrontal cortex executes mental synthesis presumably by activating the neurons in the temporal lobe (at the tip of each pyramid); (2) the duration of each ambiguous image interpretation in humans can be controlled to some extent, presumably by the prefrontal cortex.

Here are illustrations of some impossible objects. The mind attempts to interpret all available visual cues. This process results in perceptual flickering between two neuronal ensembles in the case of the blivet and three in the case of the Penrose triangle.

The blivet.

The Penrose triangle.

Lateralization of mental synthesis

Is mental synthesis normally conducted by one hemisphere or by both? In most people, the left hemisphere executes mental synthesis. In a letter-height comparison experiment (letter classification task), a split-brain subject, JW, was asked to classify letters of the alphabet according to the relative heights of their lower case forms (Farah, 1985). For example, the letter ‘a’ and the letter ‘h’ were shown to the subject and the subject was asked to identify which letter was taller. It is generally believed that people perform this task by generating a mental image of the two letters next to each other and examining the image to compare letter heights.

In other words, we remember the discrete images of ‘a’ and ‘h’, but we do not store in memory the image of ‘a’ next to ‘h’ - the image of ‘a h’. Therefore, in order to compare the heights, we generate a new image of ‘a h’ in the process of mental synthesis. After the image ‘a h’ is synthesized, it can be mentally examined and determined that the letter ‘h’ is taller.

In the control experiment, in which the patient needed to compare two letters shown simultaneously on the screen, the patient had no problem. Both separated hemispheres (tested one at a time) were able to identify the taller letter in nearly 100% of trials. When the letters were introduced one at a time with a short pause in between, the left hemisphere performed at 100% but the right hemisphere performed randomly (it was correct 45% times, that is, worse than 50% - the result that would be expected if the answers were generated randomly). The letter-height comparison experiment clearly indicates that, for the split-brain subject JW, only the left hemisphere was capable of mental synthesis. His right hemisphere was incapable of mental synthesis.

The fact that mental synthesis in patient JW was colocalized in the left hemisphere along with linguistic, analytical, and interpretive functions should come as no surprise. All these functions depend on each other and, therefore, benefit from shorter interconnections. About 95% of right-handed and 80% of left-handed people have the language function localized in the left hemisphere. Most people are right-handed because it is faster for the brain to control the right hand than the left hand. The right hand is controlled by the left hemisphere and the left hemisphere in most people is dominant. The dominant left hemisphere dictates letters during writing and executes mental synthesis. The left hemisphere is considerably superior to the right hemisphere in solving “simple problems requiring making inferences and seeing causal relationships” (Gazzaniga, 1984). The dominant left hemisphere usually first imagines a task in the process of mental synthesis, and then it commands the execution of the task in the physical world. The transmission of signals from the left hemisphere to the right hand only involves neurons located in the left hemisphere. The transmission of signals from the left hemisphere to the left hand involves a transmission of signals to the neurons in the right hemisphere via the corpus callosum. The latter neuronal path, of course, takes a longer time. Therefore right-handedness in most people is the result of left hemispheric dominance.

If the left hemisphere is surgically removed in an adult (for example, as a result of treatment for brain cancer), that person will likely have problems with his or her speech. Further, a complete section (cut) of the corpus callosum (known as a colostomy) in cases where speech cortical areas and manual dominance are in the opposite hemispheres may result in a prolonged loss of spontaneous speech. We have to conclude that in these cases the hemisphere that is involved in mental synthesis cannot direct the opposite hemisphere to generate speech.

Is there something special about the anatomy of the left hemisphere that makes it dominant in most people? The answer, most likely, is no. People who lose their left hemisphere at a young age, have no problem developing their language skills in the right hemisphere. Thousands of operations in which one hemisphere was completely removed have been performed to date. The surgery is often a last resort for treatment of seizures that resist all medication. Following the operation, patients lose control of the hand on the side opposite of the hemisphere that was removed, but remarkably, the surgery has no apparent effect on personality or memory. Further, if the left hemisphere was lost before a critical age, there is usually a complete recovery of language skills. In general, the younger a person is when they undergo the operation, the less disability they have in talking.

This observation inevitably leads to a conclusion that the right hemisphere has all the anatomy necessary for language development and mental synthesis. It is likely that colocalization of dominance and linguistic areas in one hemisphere is the result of training. During child development, the dominant hemisphere continuously executes mental synthesis that is complemented by a parallel flow of words. If cortical areas executing mental synthesis were in one hemisphere and language areas were in the other hemisphere, communication via the corpus callosum would add more than 30 milliseconds to every transaction. The result may be that the flow of thoughts and the flow of words would not be so “parallel” in that case. Further, in adults, such cross-dominance would result in an increased delay between generating a plan in the process of mental synthesis and explaining the plan in a speech to fellow humans.

The fact that children, whose left hemisphere was surgically removed, can develop into fully functional adults means that the right hemisphere is capable of becoming the dominant hemisphere. In other words, both hemispheres have the potential to become dominant. However, in the process of child development only one hemisphere takes the leading role. Usually the left hemisphere becomes the dominant hemisphere. Nature does not like two leaders in one brain.

In normal subjects, the prefrontal cortex activation is stronger in the left hemisphere during verbal tasks, while its activation in the right hemisphere is stronger in non-verbal tasks. The specialization, though, is far from absolute. The hemispheres normally complement each other; the activation of both hemispheres (as revealed by imaging techniques) becomes stronger with increased task complexity. There is also strong evidence of prefrontal cortex adaptability after damage: several days following a stroke affecting the left prefrontal cortex, a verbal task that usually activates this region can instead produce activation in the right prefrontal cortex. Such adaptability is easily comprehensible if the full distribution of relevant cells extends to both hemispheres, but with a particular focus on the left.

Hallucinations

Mental synthesis involves voluntary synthesis of new mental images. It should be noted that spontaneous, uncontrolled generation of new images is possible as well. This phenomenon is referred to as hallucination. In a classical example of a schizophrenic hallucination, the patient may be convinced that a real small homunculus is sitting on the table giving orders to the patient. It is likely that animals can also hallucinate. Several substances including marijuana, nutmeg, and peyote are known to produce delusional behavior in cats: hunting for nonexistent objects or running away from non-existent danger.

Hallucinations can be explained by a spontaneous formation of neuronal ensembles. Moreover, several neuronal ensembles can fire in synch with the attention rhythm to create never-before-seen images. I will never forget the hallucination I experienced when I was 19, following the ingestion of a large dose of codeine pills (an opiate used to suppress cough). I was serving in the army at the time and needed to stop a cough that I could not suppress by any other means. I ended up eating several times the recommended dose of codeine until the cough was finally under control. In a few hours I experienced the most dramatic hallucination of my life: a large number of small colorful cars were driving over the roots of a huge tree. Under the influence of codeine, my mind activated neuronal ensembles representing the cars and the tree. In other words, neurons encoding cars (thousands of neurons encoding shape, color, and texture) were firing action potentials in sync with neurons encoding the tree (thousands of neurons encoding shape, color, and texture) and in synch with the attention rhythm. Note that during the hallucination, the synchronization occurred spontaneously and involuntarily. No matter how hard I tried, I was not able to stop the hallucination.

Recently, a team headed by Stuart C. Sealfon and Jay A. Gingrich demonstrated that in mice, hallucinogens such as LSD, psilocybin, and mescaline, act on a serotonin receptor on the pyramidal cells. It is possible that mental images caused by hallucinogenic compounds come from the direct activation of the pyramidal cells via serotonin receptors, which, in turn, trigger the self-organization of neuronal ensembles.