Abbreviations
Introduction
Brain–computer interface (BCI) systems provide a means for those who are paralyzed to interact with the world around them, usually by detecting brain activity associated with motor intent and translating it into control commands for output devices. One classic example is an intracortical BCI that decodes a user's motor intent into computer cursor movements [1]. More complex BCI systems allow for bidirectional flow of information, e.g., providing hapticsensory feedback to help shape motor control of robotic limbs [2]. Such BCI systems have potential to return sensorimotor function to those with spinal cord injury (SCI) by rerouting neural signals, bypassing damaged tissue, and reconnecting the brain with peripheral sensory receptors, nerves, muscles, or their robotic surrogates. Individuals with mid-cervical SCI who lack the ability to grasp objects are particularly interested in the potential of BCIs for home use to restore hand function [3,4], facilitate independent self-care, and improve quality of life [5,6]. When patient-stakeholders were asked about the readiness of this technology for home use, most deemed the technology “ready for translation” with the caveats that systems need to be optimized for portability and safety, ease of use without a technician, and to provide naturalistic and functional object manipulation (Solzbacher et al., Basic-Translational-Clinical Roundtable on Neuroprosthetic Devices, Society for Neuroscience Annual Meeting, San Diego CA, November 2018; Roderick et al. NIH SCI 2020: Launching a Decade for Disruption in Spinal Cord Injury Research, Bethesda MD, February 2019; Day 1 NIH VideoCasting; URL: https://videocast.nih.gov/summary.asp?live=30194&bhcp=1) [7].
BCI—neuroprosthetic components for upper limb sensorimotor restoration
The ultimate goal of BCI-controlled upper limb neuroprosthetics is to approximate normal grasp force, speed, and dexterity, making the systems useful for performing self-care activities, such as eating and dressing, without the assistance of others. To reconnect body and brain, BCI systems utilize three main components: neural interfaces that record from but may also stimulate the brain, signal processing, and decoding algorithms and effectors that evoke movement and may transduce sensory input (Figure 1). Neural interfaces and effector devices can range in invasiveness from wearable, noninvasive components to devices surgically implanted for long-term use. For example, electrical brain potentials associated with upper limb movement intent may be captured noninvasively by scalp electroencephalography (EEG) [8, 9∗, 10∗, 11∗], through semiinvasive electrocorticographic (ECoG) arrays placed under the skull or dura [12, 13, 14, 15, 16∗, 17∗], or intracortically implanted microelectrode arrays (MEAs) [1,18, 19, 20, 21, 22∗∗]. Similarly, voluntary movement of a paralyzed limb may be effected through a powered exoskeleton [8,22,23] via neuromuscular electrical stimulation (ESTIM) through cutaneous electrodes [9,12,21,24, 25, 26, 27∗, 28∗∗, 29∗, 30∗∗], by stimulation of an implanted array of intramuscular electrodes [22,31, 32, 33], or replaced entirely by use of a robotic prosthetic [19,34, 35, 36, 37∗∗]. The choice of which set of components to use will involve tradeoffs between clinical risk, device cost, and device functionality; as such, potential use cases need to be matched to individual user characteristics (e.g., residual function, intended use, and body habitus) and performance expectations.
Figure 1. Components of BCI systems: neural interfaces that record from and may stimulate the brain (left), signal processing and decoding algorithms (middle), and effectors that evoke movement and may transduce sensory input(right). Top right: MEA-BCI–controlled robotic limb (UPMC); Middle right: MEA-BCI–controlled hybrid mobile arm support with implanted, upper limb ESTIMsystem (BrainGate2); Bottom right: MEA-BCI controlled noninvasive, forearm ESTIM array (Ohio State/NeuroLife®) (Image credit: M. Bockbrader, N. Austin). BCI, brain–computer interface; MEA, microelectrode array; ESTIM, electrical stimulation; UPMC, University of Pittsburgh Medical Center; EEG, electroencephalography; EOG, electrooculography.Scope of work
The scope of the present review is to identify and compare state of the art BCI systems and components that have demonstrated the ability to capture upper limb (and especially grasp) motor intent, translate this neural activity into effector commands, and return arm and hand function to those with tetraparesis. The purpose of this work is to identify for researchers and stakeholders the relative performance expected or attained by each approach to evaluate whether BCI systems deliver naturalistic and functional grasp speed, force, and dexterity and also to identify areas in which more work is needed to facilitate comparison. Where possible, the results from using standardized tasks to evaluate upper limb motor function are provided to illustrate objective comparisons of performance.
Neural interfaces for upper limb sensorimotor control
Limited clinical BCI trials have explored the use of lower risk neural interfaces such as EEG and ECoG for individuals with paralysis for offline [9,11,12] and online [8,11,31,33,38] control of hand functions. Because both EEG and ECoG reflect the summed activity of populations of neurons and have lower spatial and temporal signal resolution compared with MEAs, their information transfer rates are slower [39,40], which restricts processing speed, accuracy, and degrees of freedom (DoF) of decoding (e.g., compare BCIs in Table 1[8,11,28,31,33,38,41,42]). These qualities mean that even though arm and hand features can be detected using EEG [8,10,11] and ECoG [12,16,17,41], BCIs using these less invasive and lower risk neural interfaces may have a harder time delivering naturalistic speed and coordination of motor control. For example, acceptable “time to initialize” movement for fluent motor control with EEG is <3 s [43], which is similar to response time reported for online ECoG BCI performance for a person with SCI [41], whereas response time for MEA-based BCIs can be as low as 0.6–1 s [42,44].
Table 1. Comparison of online processing speed, accuracy, and degrees of freedom (DoF) of decoding by persons with spinal cord injury (SCI) using brain–computer interface (BCI) systems.
| Study author & year (Number of SCI participants) | Neural interface: features | Decoder | Response latency | Success rate | DoF |
|---|---|---|---|---|---|
| Lauer 1999 [31] (n = 1) | EMG/EEG: SMR at F3 | ARSE (25–28 Hz) | 2 s | 93.5% of trials | 1 (grasp) |
| Müller-Putz 2005 [33] (n = 1) | EEG: ERD/ERS (12–14 Hz & 18–22 Hz) at Cz and C4 | LDA | >1.5 s | 68–73% of trials | 1 (switch grasp phase) |
| Osuagwu 2016 [38] (n = 7) | EEG: Time domain parameters (7–30 Hz) at CP3–CF3, CPz–CFz and CP4–CF4 | LDA | 1.5–2 s | Not reported | 1 (hand movement) |
| Soekadar 2016 [8] (n = 6) | EEG/EOG: SMR-ERD at C3 | ARSE (8–13 Hz) | >1.5 s | Not reported | 1 (grasp); horizontal EOG controlled hand open |
| Ofner 2019 [11] (n = 1) | EEG: MRCP at Cz | LDA | 2.2–2.5 s | 26.6–36.9% of trials | 2 (palmar grasp, hand open) |
| Wang 2013 [41] (n = 1) | ECoG: High gamma | ARSE (40–200 Hz) | 2.9 s | 80% of trials | 3 (thumb, elbow, wrist) |
| Colachis IV 2018 [28] (n = 1) | Motor cortex MEA: MWP (234–3750 Hz) | SVM | <1 s | 96–99% across all trials | 7 (6 GRT object-specific grasps + hand open) |
| Skomrock [42] (n = 1) | Motor cortex MEA: MWP (234–1875 Hz) | SVM, DNN | 0.6–1 s | 76% across trials | 6 (palmar grasp, hand open, D2 extension, D2 flexion, wrist flexion, wrist extension) |
ARSE, autoregressive spectral estimation; D2, second digit (index); DNN, deep neural network; DoF, degrees of freedom; ECoG, electrocorticography; EEG, electroencephalography; EOG, electrooculography; ERD/ERS, event-related desynchronization/synchronization; GRT, grasp and release test; LDA, linear discriminant analysis; MEA, microelectrode array; MRCPs, movement-related cortical potentials; MWP, mean db4 wavelet power; SMR-ERD, sensorimotor rhythm event-related desynchronization of μ-rhythm; SVM, support vector machine.
Given the inherent system lag and lower DoF control available to EEG and ECoG, less invasive BCIs may struggle to meet performance expectations of users or be unsuited for tasks like meal preparation that involve sharp or hot objects which could cause injury if mishandled. Moreover, human factors limit the proportion of the population who are able to use EEG-based BCI effectively [45,46]; these factors may be exacerbated in individuals with paralysis [47], negatively impacting EEG-based BCI performance for those who may benefit most from improved sensorimotor control. Additional practical considerations may present barriers to widespread utilization of EEG-based BCIs—specifically the time, inconvenience, and dependence on others to set up scalp EEG electrodes. While work is in progress to address barriers to EEG-BCI translation [9], these less invasive, lower clinical risk interfaces may currently be better accepted as therapeutic aids than assistive devices. Indeed, emerging evidence supports the benefit of EEG-BCI augmented therapies for individuals with acute or incomplete SCI to drive nervous system reorganization and neurorecovery through use-dependent plasticity [38].
Other less invasive approaches, such as ECoG arrays [41] or minimally invasive endovascular stent electrodes [48], continue to be developed and explored for future BCI use. One advantage of these approaches is that they avoid the brain trauma of direct cortical implantation, as well as the concomitant risk for posttraumatic gliosis, neuroinflammation, and subsequent signal degradation. However, chronic implant studies in humans with SCI have yet to be performed to establish that ECoG or endovascular options are capable of delivering sufficiently reliable signals to support accurate, long-term decoding of motor intent for neuroprosthetic control.
Clinical trials with intracortical MEA-based BCIs for upper limb motor control
In contrast, open label, investigational, and clinical device trials using more invasive, higher risk, intracortical MEA-based BCIs have demonstrated initial safety, recording longevity beyond 5 years, and up to 10 DoF control by humans with SCI (see Table 2) [1,2,19,21,22,26, 27∗, 28∗∗, 29∗, 30∗∗,34,35,37,42,44,49∗, 50∗, 51∗, 52∗]. Recent work has shown that individuals with tetraparesis are able to use intracortical MEA-based BCIs to not only control communication devices [53], computer cursors [18,54,55], driving simulators [52], and robotic arms [2,18,19,34,35,37,49,50], but they are also effective for restoring functional grasp to paralyzed limbs [21,22,28∗∗, 29∗, 30∗∗,44], which impacts independence for self-care [30]. In each of these trials, users of intracortical MEA-based BCI systems have been able to intuitively evoke desired upper limb movements by thinking directly about the intended movement [19,21,22,26, 27∗, 28∗∗, 29∗, 30∗∗,34,35,37]. Users of the systems have also demonstrated ability to manipulate objects with sufficient skill to independently perform activities of daily living (ADL) such as eating, brushing their teeth, or holding a telephone [21,22,29,30]. Trials are limited to a small number of participants, so generalizability of the results requires investigation. Nevertheless, MEA-based BCI systems are increasingly being requested for use as assistive devices to replace lost function by those with chronic or complete SCI because of their potential to reduce disability and enhance quality of life for those with paralysis (F Solzbacher et al., Basic-Translational-Clinical Roundtable on Neuroprosthetic Devices, Society for Neuroscience Annual Meeting, San Diego CA, November 2018; JL Contreras-Vidal et al., NIH SCI 2020: Launching a Decade for Disruption in Spinal Cord Injury Research, Bethesda MD, February 2019; Day 2 NIH VideoCasting; URL: https://videocast.nih.gov/Summary.asp?Live=30198&bhcp=1) [7].
Table 2. Sensorimotor microelectrode array––brain––computer interface (MEA-BCI) systems with clinical trial results from participants with spinal cord injury (SCI).
| BrainGate/BrainGate2 NCT00912041NCT03482310 |
Ohio State/Battelle NeuroLife® clinical trial NCT01997125 |
UPMC/Pittsburgh VA NCT01364480NCT01894802 |
|
|---|---|---|---|
| Highlights |
|
|
|
| User requirements | ESTIM requires preserved innervation of muscles | ESTIM requires preserved innervation of muscles; Reaching relies on intact shoulder strength | No limb preservation required |
| Participants & characteristics |
(1-MN) C4 ASIA A SCI [1] (2-T8) C4 ASIA A SCI [22] |
(1) C5 ASIA A SCI [21,26, 27∗, 28∗∗, 29∗, 30∗∗,42,44,51,52] |
(1) Tetraparesis from spinocerebellar degeneration [19,34,37,49] (2) C5 ASIA B SCI [2,35, 36, 37∗∗,49,50] |
| Neural interface |
96 channel NeuroPort array: (1) 1 array in motor cortex [1] (2) 2 arrays in motor cortex [22] |
96-channel NeuroPort array: (1) 1 array in motor cortex [21,26, 27∗, 28∗∗, 29∗, 30∗∗,42,44,51,52] |
96-channel NeuroPort array: (1) 2 arrays in motor cortex [19,34,35,37,49] 88-channel cortical recording and stimulating (CRS) arrays and 32-channel arrays: (2) 2 CRS arrays in motor cortex, 2 32-channel arrays in somatosensory cortex [2,35,37,49,50] |
| Neural Features |
Threshold crossing: (1) 50 ms bins [1] (2) 20 ms bins +average power (250–3000 Hz) [22] |
Mean wavelet power (MWP) in multi-unit activity range: 1) 100 ms bins (db4 scales 3–6: 234–3750 Hz) [21,26,28∗∗, 29∗, 30∗∗,51,52] or (db4 scales 4–6: 234–1875 Hz) [27,42,44] Compared offline to: MWP at other frequency (db4 scales 7–11: 0–234 Hz) [51] and (db4 scales 1–2: >3750Hz) [51] Threshold crossing [28,51] Multi-unit activity (300–6000 Hz) [51] Local field potential (0–100 Hz) [51] |
Threshold crossing: (1) 30 ms bins [19,34,35,37] (2) 20 ms bins [2,35,37,50] |
| Decoding algorithms | Proportional control methods: Linear decoder (Kalman filter) for reaching [1,22] |
Proportional control methods: Beta regression [27], Non-linear support vector regression [27] Classification methods: Nonlinear support vector machine [21,26,28∗∗, 29∗, 30∗∗,42,44,51,52], Deep Neural Network (fixed, supervised updating, unsupervised updating, transfer learning) [42,44], Linear discriminant analysis (LDA) [44], Naïve Bayes [44] |
Proportional control methods: Linear decoder [2,19,34,35,37] |
| Arm function effector device |
(1–MN) Liberating Technologies prosthetic hand or Lynx 5 Series robotic arm [1] (2-T8) Powered Focal Meditech exoskeleton (shoulder-elbow-wrist) & 36 implanted ESTIM electrodes (hand, elbow, shoulder) [22] ESTIM parameters: current-controlled, biphasic, pulses (12.5 Hz pulse rate, pulse amplitude 20 mA, 0–200 μs pulse width) |
(1) NeuroLife ESTIM cuffs or sleeve (130–150 electrodes) ESTIM parameters: current-controlled, monophasic, rectangular pulses (50 Hz pulse rate, 500 μs pulse width, pulse amplitude 0–20 mA, 100 ms pulse duration) ESTIM pattern interleaving for combinatorial motion [29] |
(1 & 2) Modular Prosthetic Limb robotic arm [2,19,34,36,37] (1 & 2) WAM robotic arm [35] |
| Sensory feedback | Visual only |
Visual Residual somatosensory: Tactile & below threshold sensations decoded from motor cortex and used as grasp feedbackb |
Visual Intracortical microstimulation elicits tactile sensations [36] Intracortical feedback improves grasp [2]c |
| Motor function (postimplant days, PID) |
Translation and grasp: (1—MN) Reach or grasp (PID ∼30–300 [1]) (2—T8) Reach (PID 311–707); Grasp (<PID 717 [22]) |
Rotation, wrist (WF, WE, WU, WR), grasp (HO, spherical grip, cylindrical grip, palmar power grasp, lateral grasp, pinch grasp), and individual finger movements (D1E, D1F, D2E, D2F, D3F): (1) Proof-of-concept (PID 30–350 [21]); Rhythmic vs. ballistic (PID 108–798 [26]); Proportional wrist (PID 314–1151 [27]); Decoder comparison (PID 279–1144 [44] & 771–1371 [42]); Grip with rotation (PID 1196–1283 [29]); Feature stability (PID 30–1220 [51]); Standardized tasks: GRT (PID 702–1043 [28]); ARAT, GRASSP, BBT, GRT, CUE (PID 137–1478 [30] & 137–1854a); GRT + haptic feedback (PID 1130–1854b) |
Translation, rotation, and grasp (hand shaping: pinch, scoop, finger abduction, thumb opposition): (1) ARAT objects (PID 32–95 [19] & 119–280 [34]); Object interaction (PID 795–850 [37]); Semiautonomous grasp/computer vision (PID <987 [35]) (2) Somatosensory stimulation (PID 7–140 [36]); Semi-autonomous grasp/computer vision (PID 50–100 [35]); Object interaction (PID 661–673 [37]); Somatosensory feedback control (PID 717–738c) |
| DoF |
(1—MN/prosthetic hand) 1 (HO/HC) (1—MN/robotic limb) 1grasp and transfer [1] 2—T8/exoskeleton + ESTIM: 1–3 Reach/grasp [22] |
(1) 1: Wrist (WF) [27], grasp [30] 2: Hand (HO, HC) [30,51] 3: Grasp (HC, palmar grasp, pinch grasp) integrated shoulder/elbow [21,30] 4: Wrist (WF, WE, WU, WR) [21], Wrist/hand (D2E, D2F, WE, WF) [44,51], GRT object grips (can, fork, peg, HO) [44], Rotation & grasp (HO, Palmar grasp, Supinate/grasp, Pronate/grasp) [29]; Wrist/hand (HO, HC, Pronate, Supinate) [52] 6: Wrist/hand (D1E, D1F, D3F, WF, WE, HO) [21], Wrist/hand (D1E, D1F, D1O, WF, WE, WO) [26], Wrist/hand (D2E, D2F, WE, WF, HO, HC) [42,44] 7: (6 GRT object-specific grasps + HO) [28,30] |
(1 & 2) 5:Reach + grasp MPL [37] 7: Reach (3) + Rotate (3) + Grasp (1) MPL [19] or WAM [35] 10: Reach (3) + Rotate (3) + Grasp (4) MPL [2,34] |
| Performance (improvement from baseline paralysis) on validated measures |
GRASSP: N/R GRT: N/R BBT: N/R ARAT: N/R CUE: N/R QIF-SF: N/R |
(GRASSP: 1) Functional neurologic level C7/T1 [21,30] (improved from C5/6) (GRT: 1) enabled grip/transfer of objects [28] (faster transfer rates for peg, weight, fork, can and tape but not block [30]) (BBT: 1) 9 blocks/min [30] (from 12 blocks/min) (ARAT: 1) Total score 30/57 [30] (from 18/57); Modified ARAT: 24/27 [30], grip/transfer speed met norms for bar, cylinders, and blocks CUE-unilateral arm/hand: (1) 49 or 82% of normal [30] (increased from 27 or 45% of normal) (QIF: 1) expected ADL change to 13 (from 4) [30] |
GRASSP: N/R GRT: N/R (BBT: 1-MPL) Modified BBT: 0.1–1 block/min [34] (ARAT: 1-MPL) Modified ARAT: 15–17/27 [19], 12–17/27 [34] (2-MPL + Tactile) Modified ARAT: median increased from 18/27 to 21/27, achieving “normal” speed of grasp for 15 vs. 4 trialsc (1-WAM and 2-WAM) Modified ARAT: score N/R [35] CUE: N/R QIF-SF: N/R |
| ADLs and other object manipulation | (4-exoskeleton + ESTIM) self-paced reach to pick up coffee and drink, self-feed; reach/grasp speed 15–25 s/attempt [22] |
(1) Bottle grasp, bottle pour, stir stick grasp, stir stick transfer [21] GRT objects: transfer speeds from ∼4 to 20 s/object [28,42,44]; Telephone [29] Toothbrush, fork, cup, book, soda can, board game [30] CARLA virtual driving simulator [52] |
(1-MPL) Cone stacking [19]; 5 cm cylinder 0.35–1.09 transfer/min [37] (2-WAM) 7.5 cm cube grasp & lift [35] (2-MPL) 5 cm cylinder 4.65–5.28 transfer/min [37]; speed improved with tactile feedbackc |
ARAT, Action Research Arm Test; ASIA, American Spinal Injury Association; BBT, Box and Block Test; BCI, brain–computer interface; CRS, cortical recording and stimulating; CUE, capabilities of upper extremity; D1E, first digit (thumb) extension; D1F, first digit (thumb) flexion; D2E, second digit (index) extension; D2F, second digit flexion; D3F, third digit(middle) flexion; db4, db4 wavelet; DNN, deep neural network; DoF, degrees of freedom; ESTIM, electrical stimulation; GRASSP, Graded Redefined Assessment of Strength, Sensation, and Prehension; GRT, Grasp and Release Test; HC, hand close; HO, hand open; LDA, linear discriminant analysis; MCP, metacarpal phalangeal; MEA, microelectrode array; MPL, Modular Prosthetic Limb; N/R, not reported; PID, postimplant day; QIF, Quadriplegia Index of Function—Short Form; SCI, spinal cord injury; SVM, support vector machine; WE, wrist extension; WF, wrist flexion; wNSP, portable neural signal processor; WR, wrist radial deviation; WU, wrist ulnar deviation.
- a
-
Bockbrader et al. abstract in American Congress of Rehabilitation Medicine Abstracts, Chicago, IL, 2019.
- b
-
Ganzer P et al., bioRxiv https://doi.org/10.1101/604108.
- c
-
Flesher S et al., bioRxiv https://doi.org/10.1101/653428.
- d
-
Weiss et al. Asilomar, CA: BCI Society; 2018:131–132.