Depiction of the vOICe encoding scheme. A camera mounted on glasses records video that is converted to sound by a computer and transmitted to headphones in real time. Credit: Shimojo Lab/Caltech
Caltech researchers have now discovered that intrinsic neural connections- crossmodal mappings -can be used by assistive devices to help the blind detect their environment without requiring intense concentration or hundreds of hours of training. This new multisensory perspective on such aids (called sensory substitution devices) could make tasks that were previously attention-consuming much easier, allowing nonsighted people to acquire a new sensory functionality similar to vision.
“.. 99% of our daily life depends on multisensory- also called multimodal processing.” As an example, he says, if you are talking on the phone with someone you know very well, and they are crying, you will not just hear the sound but will visualize their face in tears. “This is an example of the way sensory causality is not unidirectional–vision can influence sound, and sound can influence vision.”
Shimojo and Noelle Stiles have exploited these crossmodal mappings to stimulate the visual cortex with auditory signals that encode information about the environment. They explain that crossmodal mappings are ubiquitous; everyone already has them. Mappings include the intuitive matching of high pitch to elevated locations in space or the matching of noisy sounds with bright lights. Multimodal processing, like these mappings, may be the key to making sensory substitution devices more automatic.
The researchers conducted trials with both sighted and blind people using a sensory substitution device, called a vOICe device, that translates images into sound. It is made up of a small computer connected to a camera that is attached to darkened glasses, allowing it to “see” what a human eye would. A computer algorithm scans each camera image from left to right, and for every column of pixels, generates an associated sound with a frequency and volume that depends upon the vertical location and brightness of the pixels. A large number of bright pixels at the top of a column would translate into a loud, high-frequency sound, whereas a large number of lower dark pixels would be a quieter, lower-pitched sound. A blind person wearing this camera on a pair of glasses could then associate different sounds with features of their environment.
The intuitively identified textures used in the experiments exploited the crossmodal mappings already within the vOICe encoding algorithm. “When we reverse the crossmodal mappings in the vOICe auditory-to-visual translation, the naive performance significantly decreased, showing that the mappings are important to the intuitive interpretation of the sound,” explains Stiles.
“We found that using this device to look at textures–patterns of light and dark–illustrated ‘intuitive’ neural connections between textures and sounds, implying that there is some preexisting crossmodality,” says Shimojo.
“Auditory regions are activated upon hearing sound, as are the visual regions, which we think will process the sound for its spatial qualities and elements. The visual part of the brain, when processing images, maps objects to spatial location, fitting them together like a puzzle piece,” Stiles says. To learn more about how the crossmodal processing happens in the brain, the group is currently using functional magnetic resonance imaging (fMRI) data to analyze the crossmodal neural network.
“Our research has shown that the visual cortex can be activated by sound, indicating that we don’t really need our eyes to see. It’s very profound–we’re trying to give blind people a visual experience through other senses.” http://www.caltech.edu/news/seeing-sound-48579
Recent Comments