One Brain Implant Decoded Words and Gestures at the Same Time

When paralysis takes away speech, communication technology can sometimes return the words. But words are not the whole conversation.

A nod can turn a statement into agreement. A shrug can replace a sentence. A wave can say hello before anyone begins to speak. Strip those movements away and communication may remain functional, but it becomes flatter, a transcript of a person rather than their full presence.

Researchers at the University of California, San Francisco, have now demonstrated a brain-computer interface that tries to restore both layers in digital form. In a proof-of-concept study published in Nature Neuroscience, implants decoded selected attempted phrases and upper-body gestures in parallel. The phrases appeared as text while personalized full-body avatars performed the predicted movements.

The especially revealing part was what happened when the researchers tried to combine the two capabilities. Speech and gesture could not be treated as entirely independent decoding problems.

One implant, two channels

The study used electrocorticography, usually shortened to ECoG. Unlike arrays whose electrodes penetrate brain tissue, an ECoG array rests on the cortex's surface and records the combined electrical activity of nearby neural populations.

Each participant had a thin array containing 253 recording contacts placed over a broad section of the left sensorimotor cortex. That coverage mattered. Speech-related activity tended to appear in more ventral regions, closer to areas associated with the mouth and vocal tract, while gesture-related activity was generally farther up, near representations of the hand and arm.

The boundary was not clean. Some electrodes responded during both kinds of attempted movement.

Three people with severe paralysis participated in the broader study, allowing the team to examine neural activity associated with speech, the face and multiple parts of the body. The simultaneous communication experiments focused on two of them.

One, identified as Bravo-1r, had impaired speech and movement following a brainstem stroke. He silently attempted five phrases and minimally attempted four gestures. The other, Bravo-6, had amyotrophic lateral sclerosis, or ALS. He overtly attempted ten phrases and imagined performing ten gestures.

The system did not translate unrestricted thoughts. During most testing, a screen told the participant which phrase, gesture or combination to attempt. Separate participant-specific decoders then classified the neural activity into one of the small number of options they had learned, plus a resting state.

That distinction matters. This was a controlled test of intended motor actions, not a device listening to an uncensored inner monologue.

Why two separate decoders were not enough

An obvious design would be to train a speech decoder on speech, train a gesture decoder on gestures and then run them side by side. The researchers tested the underlying idea. Models trained only on isolated behavior did not transfer reliably when a participant attempted both actions together.

The neural pattern for a simultaneous phrase and gesture was not simply the isolated speech pattern added to the isolated gesture pattern. The activity overlapped, and performing both tasks changed the signals enough to confuse models that had encountered only one behavior at a time.

The solution was to train the models on that interaction.

The team used a mixture of speech-only, gesture-only and simultaneous trials. It also showed each decoder examples in which only the other communication channel was active. That helped the speech decoder learn when there was no speech to report and the gesture decoder learn when the avatar should remain still.

Training across contexts improved performance when the behaviors occurred alone and when they occurred together. The model analysis also indicated that concurrent training changed how heavily the decoders relied on different electrodes, favoring signals that better separated the two tasks.

This may be the study's most useful lesson for future neural prostheses. A multifunctional interface cannot necessarily be assembled by independently perfecting several single-purpose systems. If the behaviors interact in the brain, the training data must capture those interactions.

What the system could actually do

Bravo-6 completed the largest real-time test. He worked with ten phrases and ten gestures, producing 100 possible pairings. Testing the full grid prevented an important shortcut. If “hello” always accompanied a wave, for example, a gesture decoder might learn to predict the wave from speech-related activity without identifying the intended gesture itself.

During real-time simultaneous trials, the gesture decoder was correct 66 percent of the time and the speech decoder 70 percent of the time. Each model's nominal chance level was about 9 percent because it selected among ten targets and a resting state. Predictions appeared as text or moved the avatar after several-second, cue-aligned recording windows.

Those results are well above chance, but they are not yet smooth, open-ended conversation. The system selected discrete phrases rather than composing arbitrary sentences. Its gesture vocabulary was tiny compared with natural body language. The implant was wired through a connector mounted to the skull and depended on external computers.

The researchers also staged cued conversations in which the participants responded to an experimenter using speech, gestures or both. The demonstrations included a fully virtual exchange between two avatars, but they involved too few trials to establish how the system would perform during ordinary conversation.

A more encouraging result came from offline tests of unfamiliar combinations. When training one decoder, the researchers withheld concurrent examples involving a particular target from the other channel. Accuracy did not fall significantly when the models later encountered those withheld pairings.

That suggests a future system might not need to rehearse every phrase with every gesture, a requirement that would quickly become impractical as both vocabularies expanded. It would still need experience with concurrent behavior, but perhaps not every conceivable combination.

Restoring more than a message

Brain-computer interfaces have already produced text, synthetic voices and animated facial expressions for people with paralysis. The UCSF group previously demonstrated a system that decoded speech into text, audio and movements of a facial avatar.

The new study extends that idea to upper-body gestures. A gesture can stand alone, reinforce a phrase or contradict it. Even a limited avatar can convey something that a line of text cannot.

“Conversation is about much more than the words being spoken,” study leader Edward Chang said in an NIH announcement.

The experiment remains small, the vocabulary restricted and the equipment experimental. Two people using a system in cued trials cannot establish how it will perform across the many ways paralysis affects the brain and body. The researchers say larger vocabularies, lower-latency continuous decoding, more participants and other recording technologies still need to be explored. The NIH reports that the team also plans to test a fully implantable wireless version.

The paper discloses that Chang and coauthor Jessie Liu are inventors on relevant patent applications and that Chang is a cofounder of Echo Neurotechnologies.

For now, the achievement is narrower but still striking. The implant did not merely decode a few intended phrases. It showed that a neural prosthesis could begin to represent some of the movement around those phrases, the part of communication that conveys not only what a person means, but something of how they mean it.