Brain-computer interface turns mental writing into on-screen text


Researchers have, for the first time, decoded the neural signals associated with writing letters and then displayed typed versions of those letters in real time. They hope their invention will one day help paralyzed people communicate.

Scientists are exploring a number of ways people with disabilities communicate with their thoughts. The newest and fastest comes back to a vintage medium: handwriting.

For the first time, researchers decipher brain activity associated with trying to write letters by hand. Working with a paralyzed participant who has sensors implanted in his brain, the team used an algorithm to identify the letters as he attempted to write them down. Then the system would display the text on a screen – in real time.

The innovation could, with further development, allow people with paralysis to type quickly without using their hands, says study co-author Krishna Shenoy, a researcher at the Howard Hughes Medical Institute at Stanford University who has Co-supervised the work with Jaimie Henderson, a Stanford neurosurgeon.

While attempting to write by hand, the study participant typed 90 characters per minute – more than double the previous record for typing with such a “brain-computer interface,” Shenoy and colleagues report in the newspaper. Nature on May 12, 2021.

This technology and others like it have the potential to help people with all kinds of disabilities, says Jose Carmena, a neural engineer at the University of California, Berkeley, who was not involved in the study. Although the results are preliminary, he says, “this is a big step forward in the field.

Brain-computer interfaces turn thought into action, says Carmena. “This paper is a perfect example: the interface decodes thought from writing and produces action.

Communication fueled by thought

When injury or illness deprives a person of the ability to move, the neural activity in the brain to walk, have a cup of coffee, or speak a phrase remains. Researchers can tap into this activity to help people with paralysis or amputation regain their lost abilities.

The need varies depending on the nature of the disability. Some people who have lost the use of their hands can still use a computer with voice recognition and other software. For those who have difficulty speaking, scientists have developed other ways to help people communicate.

Two tiny arrays of implanted electrodes relayed information from the area of ​​the brain that controls the hands and arms to an algorithm, which translated them into letters that appeared on a screen. Credit: F. Willett et al./Nature 2021 / Erika Woodrum

In recent years, Shenoy’s team has decoded neural activity associated with speech in the hope of reproducing it. They also devised a way for participants with implanted sensors to use their thoughts associated with attempted arm movements to move a cursor on a screen. By pointing and clicking letters in this manner, people type around 40 characters per minute, the previous speed record for typing with a brain-computer interface (BCI).

No one, however, had looked at the handwriting. Frank Willett, an HHMI research specialist and neuroscientist in Shenoy’s group, wondered if it would be possible to harness the evoked brain signals by putting a pen on paper. “We want to find new ways to allow people to communicate faster,” he says. He was also motivated by the opportunity to try something different.

The team worked with a participant enrolled in a clinical trial called BrainGate2, which tests the safety of BCIs that transmit information directly from a participant’s brain to a computer. (The trial director is Leigh Hochberg, a neurologist and neuroscientist at Massachusetts General Hospital, Brown University, and Providence VA Medical Center.) Henderson implanted two tiny sensors in the part of the brain that controls the hand and arm. , thus allowing the person, for example, to move a robotic arm or a cursor on a screen while attempting to move their own paralyzed arm.

The participant, who was 65 at the time of the research, suffered a spinal cord injury that left him paralyzed from neck to toe. Using signals picked up by sensors in individual neurons when humans imagined writing, a machine learning algorithm recognized the patterns their brain produced with each letter. With this system, the man could copy sentences and answer questions at a pace similar to that of a person his age typing on a smartphone.

This so-called “brain-to-text” BCI is so fast because each letter sparks a very distinctive pattern of activity, which allows the algorithm to tell them apart relatively easily, Willett explains.

A new system

The Shenoy team plans to use an attempted handwriting for text entry as part of a more comprehensive system that also includes point-and-click navigation, much like that used on current smartphones, and even an attempt to decode speech. “Having those two or three modes and switching between them is something we do naturally,” he says.

Tracing the interpretation of letters
As the participant imagined writing a letter or symbol, sensors implanted in his brain detected patterns of electrical activity, which an algorithm interpreted to trace the path of his imaginary pen. Credit: F. Willett et al./Nature 2021

Next, Shenoy says, the team plan to work with a participant who cannot speak, such as someone with amyotrophic lateral sclerosis, a degenerative neurological disorder that causes loss of movement and speech.

The new system could potentially help people with paralysis caused by a number of conditions, Henderson adds. These include a stroke of the brainstem, which afflicted Jean-Dominique Bauby, the author of the book The Diving Bell and the Butterfly. “He was able to write this moving and beautiful book by carefully selecting the characters, one at a time, using eye movement,” says Henderson. “Imagine what he could have done with Frank’s handwriting interface! “



Francis R. Willett, Donald T. Avansino, Leigh R. Hochberg, Jaimie M. Henderson, and Krishna V. Shenoy. “High-performance brain-to-text communication via handwriting. ” Nature. Published online May 12, 2021. doi: 10.1038 / s41586-021-03506-2

CAUTION: Investigation device. Restricted by Federal law to experimental use.

About Nell Love

Check Also

Tales of 5 Orlando Roller Coasters

Central Florida is on a roll with its coasters. Epcot’s new Guardians of the Galaxy: …

Leave a Reply

Your email address will not be published.