How multilingualism alters multisensory perception

We know that different people can perceive sounds and images like the blue/gold dress in divisive ways depending on how their brain processes them. But how can bilingualism alter the way that light and sound information are delivered to the brain?

Sayuri Hayakawa & Viorica Marian

First, the gold/blue dress “melted the internet.” Then, Yanny/Laurel “divided America.” Illusions like these ignite interest and controversy because they violate the intuition that our perceptions are direct, unfiltered versions of the external environment. Color, after all, shouldn’t be a matter of opinion. Which is why it blows our minds when someone insists that the gold dress we see is blue, or that what we clearly hear as “Yanny” sounds to them like “Laurel.”

The reality is that sensory perception can be nudged, distorted, and radically transformed by anything from the languages we speak to the surrounding visual context.

The reality is that sensory perception can be nudged, distorted, and radically transformed by anything from the languages we speak to the surrounding visual context – something that is dramatically illustrated by the strength of perceptual illusions. Like a virus, a well-designed illusion contains just the right combination of features to essentially hi-jack the “machinery” of the perceptual system, including learned heuristics and innate functions, and make it do what it always does – interpret sensory input based on (usually) reliable cues. Unlike a virus, perceptual illusions are generally harmless (other than inciting the occasional internet war), and can even provide a glimpse into the inner-workings of our minds. They can also shed light on how our personal experiences, such as experiences with different languages, shape how we perceive the world around us.

A neutral density filter held in front of a landscape
Credit: Robert Emperley [CC BY-SA 2.0]
To illustrate, consider the audiovisual illusion called the McGurk Effect. When we hear a speech sound, such as “ba,” but see an incongruent lip movement, such as “ga,” the brain tries to reconcile this discrepancy, often resulting in the perception of a completely different sound, such as “da.” Our research shows that bilinguals are more likely to experience the McGurk Effect than monolinguals, suggesting that multilingual experience alters multisensory integration. One potential explanation is that bilinguals may need to (at least initially) rely more on visual information to make sense of complex, multilingual speech. Indeed, there is evidence that babies growing up in multilingual households attend to the mouths of speakers to a greater degree than infants in monolingual households. As described in our recent review paper published in Behavioral and Brain Functions, these early differences in how bilinguals and monolinguals attend to speech-relevant inputs continue to shape how, and even how well, individuals process sensory information throughout their lives.

Utilizing neuroimaging tools such as EEG and fMRI, researchers have demonstrated that bilingual children and adults show visible differences in processing of auditory stimuli compared to their monolingual peers. These effects of bilingualism have been observed in neural regions devoted to higher-level cognitive functions, such as those responsible for controlling attention, as well as in the evolutionarily ancient auditory brainstem, which can encode sounds so faithfully that it is possible to play back recognizable speech and music using the electrical signals generated by the brain.

Bilingualism can therefore alter bottom-up processes that deliver light and sound waves up to the brain, as well as top-down processes that guide how we interpret and organize this information. It is possible that bilingualism generally increases audiovisual integration due to stronger encoding of sensory stimuli (think, a more sensitive mic or camera). However, because bilingualism also impacts top-down functions, the influence of language experience on sensory perception is likely to be more nuanced. Specifically, bilinguals may integrate sensory information differently due to more precise control over whether inputs should be combined. Indeed, researchers have found that auditory and visual stimuli need to be timed more perfectly for bilinguals to experience the double-flash illusion, whereby hearing two tones transforms a single flash of light into what appears to be two flashes. In other words, bilinguals are better able to determine when sights and sounds likely originated from different sources (based on features such as timing), and subsequently suppress inputs that would distort their perception.

"Hello" written in several languages on a blackboard

Over time, language experience can even alter the physical structure of the brain. For instance, it has been shown that bilingual experience can lead to greater brain matter density and volume in regions associated with sensory processing, such as the primary auditory cortex, as well as executive function, such as the prefrontal cortex. The behavioral correlates of these physical changes can be significant, as greater gray matter in a region of the primary auditory cortex predicts better speech perception. Increased gray matter in the prefrontal cortex is additionally associated with enhanced cognitive control Attentional Network Task→ → ← → →).

Given the importance of both sensory and executive functions for language learning, one of the consequences of bilingualism is a greater capacity to continue learning new languages, thereby perpetuating the cycle of neural reconfiguration that can begin with exposure to multilingual speech.

The film maker, Federico Fellini, famously said that “a different language is a different vision of life”. Through a combination of behavioral and neurological investigations, it has now been confirmed that learning a different language can literally change what you see.

Sayuri Hayakawa & Viorica Marian

Sayuri Hayakawa & Viorica Marian

Sayuri Hayakawa is a Post-Doctoral Researcher in the Department of Communication Sciences and Disorders at Northwestern University.

Viorica Marian is the Ralph and Jean Sundin Endowed Professor of Communication Sciences and Disorders and Professor of Psychology at Northwestern University. Follow her on Twitter.
Sayuri Hayakawa & Viorica Marian

Latest posts by Sayuri Hayakawa & Viorica Marian (see all)

View the latest posts on the On Biology homepage

Comments