^{1}

^{1}

^{2}

^{3}

^{4}

^{5}

^{6}

^{1}

^{2}

^{3}

^{4}

^{5}

^{6}

Edited by: Cuntai Guan, Institute for Infocomm Research, Singapore

Reviewed by: Alireza Mousavi, Brunel University, UK; Jose L. ‘Pepe’ Contreras-Vidal, University of Maryland, USA

*Correspondence: Gerwin Schalk, Wadsworth Center, New York State Department of Health, C650 Empire State Plaza, Albany, NY, USA. e-mail:

This article was submitted to Frontiers in Neuroprosthetics, a specialty of Frontiers in Neuroscience.

This is an open-access article subject to a non-exclusive license between the authors and Frontiers Media SA, which permits use, distribution and reproduction in other forums, provided the original authors and source are credited and other Frontiers conditions are complied with.

Brain–computer interfaces (BCIs) use brain signals to convey a user’s intent. Some BCI approaches begin by decoding kinematic parameters of movements from brain signals, and then proceed to using these signals, in absence of movements, to allow a user to control an output. Recent results have shown that electrocorticographic (ECoG) recordings from the surface of the brain in humans can give information about kinematic parameters (e.g., hand velocity or finger flexion). The decoding approaches in these studies usually employed classical classification/regression algorithms that derive a linear mapping between brain signals and outputs. However, they typically only incorporate little prior information about the target movement parameter. In this paper, we incorporate prior knowledge using a Bayesian decoding method, and use it to decode finger flexion from ECoG signals. Specifically, we exploit the constraints that govern finger flexion and incorporate these constraints in the construction, structure, and the probabilistic functions of the prior model of a switched non-parametric dynamic system (SNDS). Given a measurement model resulting from a traditional linear regression method, we decoded finger flexion using posterior estimation that combined the prior and measurement models. Our results show that the application of the Bayesian decoding model, which incorporates prior knowledge, improves decoding performance compared to the application of a linear regression model, which does not incorporate prior knowledge. Thus, the results presented in this paper may ultimately lead to neurally controlled hand prostheses with full fine-grained finger articulation.

Brain–computer interfaces (BCIs) allow people to control devices directly using brain signals (Wolpaw,

Substantial efforts in signal processing and machine learning have been devoted to decoding algorithms. Many of these efforts focused on classifying discrete brain states. The linear and non-linear classification algorithms used in these efforts are reviewed in (Muller et al.,

The main question we sought to answer with this study is whether appropriate integration of prior knowledge improves the fidelity of decoding of individual finger movements. To do this, we used a switching non-parametric dynamic system (SNDS) to build a model that integrates information from prior knowledge and from the output of a simple regression model that was established between ECoG signals and finger flexion. This method is an extension to the switched linear dynamic system (SLDS) method. SLDS has been successfully applied in a variety of domains (Azzouzi and Nabney,

We attribute this improvement to the following technical advances. First, and most importantly, we introduce a prior model based on SNDS, which takes advantage of prior information about finger flexion patterns. For example, movements of fingers generally switch between extension, flexion, and rest, and there are some constraints that govern the transition between these movement states. Second, to effectively model the duration of movement patterns, our model solves the “Markov assumption” problem more efficiently by modeling the dependence of state transition on the continuous state variable. Third, because estimation of continuous transition is crucial to accurate prediction, we applied kernel density estimation to model the continuous state transition. Finally, we developed effective learning and inference methods for the SNDS model.

In this study, we used a dataset that was collected for a previous study (Kubánek et al.,

The ECoG signals from the electrode grid were recorded using the general-purpose BCI2000 system (Schalk et al.,

Feature extraction was identical to that in (Kubánek et al., _{q}

We defined a movement period as the time between 1000 ms prior to movement onset and 1000 ms after movement offset. Movement onset was defined as the time when the finger’s flexion value exceeded an empirically defined threshold (one-fifth of the largest finger flexion value). Conversely, movement offset was defined as the time when the finger’s flexion value fell below that threshold and no movement onset was detected within the next 1200 ms (Kubánek et al.,

The output of the pace regression algorithm was combined with prior knowledge using a switching non-linear dynamic system (SNDS). The SNDS is a Bayesian decoding model that infers the posterior distribution of finger flexion by combining a prior model and a measurement model (as shown in Figure

Section

In this section, we will develop a set of constraints that guide the movement of the fingers. These constraints commonly exist but are generally ignored by most decoding algorithms. Conventional decoding algorithms (such as pace regression) may make predictions that may be outside of these constraints. For example, a conventional decoding algorithms may produce a prediction of a finger that flexes past physical constraints, or may result in predictions in which a finger immediately proceeds from full extension to full flexion.

Figure

The movement of fingers can be categorized into three states: extension (state _{1}), flexion (state _{2}), and rest (rest state _{3}).

For each state, there are particular predominant movement patterns. In the extension state _{1}, the finger keeps moving away from the rest position. In the flexion state _{2}, the finger moves back to the rest position. In the rest state _{3}, there are only very small movements.

For either state _{1} or state _{2}, the movement speed is relatively low toward full flexion or full extension, but faster in between. For the rest state, the speed stays close to zero.

The natural flexion or extension of fingers are limited to certain ranges due to the physical constraints of our hand.

The transition between different states is not random. Figure

Figure

In summary, the observations described above provide constraints that govern finger flexion patterns. Using the methods described below, we will build a computational model that incorporates these constraints and that can systematically learn the movement patterns from data.

In this section, we show how the constraints summarized above are incorporated into the construction of the SNDS model. The SNDS is an extension of SLDS (Pavlovic et al., _{1}), flexion state (_{2}), and rest state (_{3}). The middle layer (continuous state variable) represents the real finger position. We discuss these two layers in detail below.

_{t}, Y_{t}, Z_{t}

In the standard SLDS, the probability of duration

where _{ii}

This limitation of the state duration model has been addressed for hidden Markov models (HMMs) in the area of speech recognition by introducing explicit state duration distributions (Ferguson, _{t}_{t − 1} but also from _{t − 1}:

where _{t−1}|_{t−1}) is a normalization term with no relation to _{t}_{t}_{t−1}) is the state transition, which is same with that in HMM and standard SLDS. _{t−1}|_{t−1},_{t}_{t−1} given state transition from _{t−1} to _{t}_{t−1}|_{t−1},_{t}

Figure

_{t−1} given _{t−1} = _{t}_{t−1} given _{t−1} = _{t}

In SLDSs, the

where _{t−1} and _{t}

where _{i}_{−} _{1},_{i}

Figure _{t−1} and _{t}_{t}_{t−1}, i.e., fingers are moving up. Also the farther the kernel locations are from the diagonal, the larger the value of _{t}_{t−1}, which implies greater moving speed at time _{t}_{t−1} could be a measurement of the speed). In the extension state, the moving speed around average flexion is statistically greater than that around the two extremes (full flexion and extension). Similar arguments can be applied to the flexion state in Figure _{t}_{t−1}, i.e., fingers are not moving. The capability of being able to model the non-linear dependence of speed on position under each state is critical to make a precise prediction of the flexion trace.

_{t}_{t}

Parameters ^{(s)}, ^{(s)}, and

where

All variables of the SNDS model are incorporated during learning. Finger flexion states are estimated from the behavioral flexion traces (e.g., Figure

All parameters _{t}_{t−1}) and _{t−1}|_{t−1},_{t}_{t}_{t−1}) can be simply obtained by counting. However, here we need to enforce the constraints described in Section 3.1(v). The elements in the conditional probability table of _{t}_{t−1}) corresponding to the impossible state transitions are set to zero. _{t−1}|_{t−1},_{t}^{(s)}, ^{(s)}, and

Given the time course of ECoG signals, our goal is to infer the time course of finger flexion. This is a typical filtering problem, that is, recursively estimating the posterior distribution of _{t}_{t}_{1:t}:

where _{t−1},_{t−1}, _{1:t}) is the filtering result of the former step. However, we note that not all the continuous variables in our model follow a Gaussian distribution, because kernel density estimation was used to model the dynamics of the continuous state variable. Hence, it is infeasible to update the posterior distribution _{t}_{t}_{1:t}) analytically in each step. To cope with this issue, we adopted a numerical sampling method based on particle filtering (Isard and Blake,

Initialization

For _{0}) and _{0}|_{0}).

Importance sampling

For

For

Normalize the importance weights:

Resampling

For

Once the N samples and normalized importance weights have been constructed, the finger flexion at time step

where

The main question we set out to answer in this study was to determine whether appropriate incorporation of prior knowledge can improve the performance of the decoding of finger flexion. We began our analyses by using SNDS in combination with a linear method (i.e., pace regression) as the underlying decoding algorithm. We previously used pace regression alone (i.e., without SNDS) to decode finger flexion using the same dataset (Kubánek et al.,

To give a qualitative impression of the improvement of the SNDS algorithm combined with pace regression compared to pace regression alone, we first provide an example of the results achieved with each of these two approaches on the decoding of index finger flexion of subject A. These results are shown in Figure

These results demonstrate that the state of finger flexion [which cannot be directly inferred using a method that does not incorporate a state machine (such as pace regression)] can be accurately inferred using SNDS.

In addition to the qualitative comparison provided above, Table

Subject | Alg. | Thumb | Index finger | Middle finger | Ring finger | Little finger | Avg. |
---|---|---|---|---|---|---|---|

A | Pace (MSE) | 0.58 ± 0.05 | 0.64 ± 0.02 | 0.77 ± 0.03 | 0.86 ± 0.06 | 0.81 ± 0.04 | 0.73 |

SNDS (MSE) | 0.35 ± 0.07 | 0.44 ± 0.05 | 0.63 ± 0.07 | 0.73 ± 0.09 | 0.59 ± 0.06 | 0.54 | |

Pace (CC) | 0.70 ± 0.03 | 0.68 ± 0.01 | 0.61 ± 0.02 | 0.56 ± 0.03 | 0.59 ± 0.02 | 0.63 | |

SNDS (CC) | 0.82 ± 0.04 | 0.76 ± 0.03 | 0.64 ± 0.04 | 0.58 ± 0.04 | 0.65 ± 0.03 | 0.69 | |

B | Pace (MSE) | 0.65 ± 0.10 | 0.63 ± 0.18 | 0.68 ± 0.16 | 0.52 ± 0.08 | 0.60 ± 0.15 | 0.62 |

SNDS (MSE) | 0.46 ± 0.12 | 0.44 ± 0.20 | 0.49 ± 0.14 | 0.39 ± 0.10 | 0.40 ± 0.17 | 0.43 | |

Pace (CC) | 0.68 ± 0.04 | 0.68 ± 0.09 | 0.65 ± 0.09 | 0.74 ± 0.04 | 0.71 ± 0.08 | 0.69 | |

SNDS (CC) | 0.76 ± 0.05 | 0.75 ± 0.10 | 0.70 ± 0.06 | 0.77 ± 0.05 | 0.76 ± 0.07 | 0.75 | |

C | Pace (MSE) | 0.83 ± 0.15 | 0.78 ± 0.03 | 0.87 ± 0.08 | 0.89 ± 0.08 | 0.97 ± 0.12 | 0.87 |

SNDS (MSE) | 0.53 ± 0.20 | 0.46 ± 0.07 | 0.54 ± 0.05 | 0.61 ± 0.10 | 0.73 ± 0.14 | 0.56 | |

Pace (CC) | 0.58 ± 0.07 | 0.61 ± 0.01 | 0.56 ± 0.06 | 0.55 ± 0.04 | 0.51 ± 0.07 | 0.56 | |

SNDS (CC) | 0.70 ± 0.09 | 0.74 ± 0.03 | 0.69 ± 0.04 | 0.61 ± 0.06 | 0.57 ± 0.09 | 0.66 | |

D | Pace (MSE) | 1.29 ± 0.10 | 1.07 ± 0.12 | 0.99 ± 0.05 | 1.09 ± 0.08 | 1.27 ± 0.12 | 1.14 |

SNDS (MSE) | 1.15 ± 0.11 | 0.94 ± 0.13 | 0.87 ± 0.03 | 0.96 ± 0.05 | 1.0 ± 0.10 | 0.98 | |

Pace (CC) | 0.35 ± 0.06 | 0.46 ± 0.07 | 0.50 ± 0.04 | 0.45 ± 0.04 | 0.38 ± 0.06 | 0.42 | |

SNDS (CC) | 0.38 ± 0.07 | 0.51 ± 0.07 | 0.55 ± 0.02 | 0.50 ± 0.03 | 0.46 ± 0.05 | 0.48 | |

E | Pace (MSE) | 1.03 ± 0.05 | 0.96 ± 0.15 | 0.80 ± 0.15 | 0.94 ± 0.07 | 0.90 ± 0.13 | 0.93 |

SNDS (MSE) | 0.84 ± 0.10 | 0.75 ± 0.18 | 0.63 ± 0.13 | 0.73 ± 0.07 | 0.68 ± 0.19 | 0.71 | |

Pace (CC) | 0.49 ± 0.03 | 0.52 ± 0.08 | 0.60 ± 0.08 | 0.54 ± 0.05 | 0.55 ± 0.07 | 0.54 | |

SNDS (CC) | 0.57 ± 0.06 | 0.61 ± 0.09 | 0.65 ± 0.08 | 0.61 ± 0.05 | 0.65 ± 0.10 | 0.62 |

The previous section demonstrated how we incorporated prior knowledge into a computational model to improve decoding of finger flexion. We were interested to what extent each aspect of the prior knowledge contributed to the improvement of the results. Thus, we incrementally incorporated each of these aspects into the model and determined the resulting effect.

The different models are shown in Figure _{t−1} to _{t}

Figure

Because SNDS is a general probabilistic framework to incorporate prior knowledge into the decoding process, we also studied, as an example for the use of a different decoding algorithm, the combination of SNDS and a non-linear method (i.e., a Gaussian process Rasmussen,

The practical application of Gaussian processes is affected by its computational complexity, which is cubic to the data size. Sparse Gaussian processes (SPGP; Snelson and Ghahramani,

Algorithm | Sub. A | Sub. B | Sub. C | Sub. D | Sub. E |
---|---|---|---|---|---|

Pace | 0.73 | 0.62 | 0.87 | 1.14 | 0.93 |

SNDS (pace) | 0.54 | 0.43 | 0.56 | 0.98 | 0.71 |

SPGP | 0.60 | 0.57 | 0.73 | 1.07 | 0.90 |

SNDS (SPGP) | 0.46 | 0.39 | 0.53 | 0.96 | 0.75 |

This paper demonstrates that prior knowledge can be successfully captured to build switched non-parametric dynamic systems to decode finger flexion from ECoG signals. We also showed that the resulting computational models improve the decoding of finger flexion compared to when prior knowledge was not incorporated. This improvement is possible by dividing the flexion activity into several moving states (_{t}_{t}

Generally, this improvements in decoding performance likely results from the different types of constraints on the possible flexion predictions that are realized by the computational model. In other words, the model may not able to produce all possible finger flexion patterns, although it is important to point out that the constraints that we put on finger flexions are those of natural finger flexions. Yet, it is still unclear to what extent these constraints and the same methodology used here may generalize to those of other natural movements, such as simultaneous movements of multiple fingers or hand gestures during natural reaches. In particular, our method improves results in part because it infers discrete behavioral states, but there may be many such states during natural movements, in particular when multiple degrees of freedom (e.g., movements of different fingers, wrist, hand, etc.) are considered. However, individual degrees of freedom of natural hand/finger movements are usually not independent, but are coordinated to form particular movement patterns such as those during reach-and-grasp. Thus, the number of possible states will usually be dramatically less than the number of all possible states. In this case, using techniques to reduce the dimensionality (such as principal component analysis, PCA) of the behavioral space should also limit the number of states. In other words, it would be straightforward to change the definition of states from movements of individual fingers to estimation of different grasp patterns. The general structure of the model would be the same while the parameterization and physical meanings of the variables would be somewhat different. At the same time, as the amount of prior knowledge decreases, e.g., movements along different degrees of freedom occur unpredictably and independently, the benefit of using prior knowledge will likely decrease.

There are some directions in which this work could be further improved. First, to reduce the computational complexity caused by kernel density estimation, non-linear transition functions can be used to model the continuous state transitions. Second, more efficient inference methods could be developed to replace standard particle sampling. Finally, the methods presented in this paper could be extended to allow for simultaneous decoding of all five fingers instead of one at a time.

In conclusion, the results presented in this paper demonstrate that, with appropriate mathematical decoding algorithms, ECoG signals can give information about finger movements that in their specificity and fidelity goes substantially beyond what has previously been demonstrated using any other method in any species. With further improvements to current ECoG sensor technology, in particular to the density and form factor of current implants, and extension of current methods to real-time capability, it may ultimately be possible to develop neurally controlled hand prostheses with full fine-grained finger articulation. This eventual prospect is exciting, because even simpler capabilities may offer distinct advantages, e.g., the restoration of select grasp patterns in stroke patients. More generally, the possibility that ECoG may support practical, robust, and chronic brain–computer interfaces was recently further substantiated: the study by Chao et al. (

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

This work was supported in part by grants from the US Army Research Office [W911NF-07-1-0415 (Gerwin Schalk) and W911NF-08-1-0216 (Gerwin Schalk)] and the NIH [EB006356 (Gerwin Schalk) and EB000856 (Gerwin Schalk)].