^{1}

^{2}

^{3}

^{1}

^{4}

^{*}

^{1}

^{2}

^{3}

^{4}

Edited by: Giovanni Mirabella, Sapienza University of Rome, Italy

Reviewed by: Giancarlo Ferrigno, Politecnico di Milano, Italy; Giuseppe D'Avenio, Istituto Superiore di Sanità, Italy

*Correspondence: Alex Ossadtchi

This article was submitted to Neural Technology, a section of the journal Frontiers in Neuroscience

This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

In recent years, several assistive devices have been proposed to reconstruct arm and hand movements from electromyographic (EMG) activity. Although simple to implement and potentially useful to augment many functions, such myoelectric devices still need improvement before they become practical. Here we considered the problem of reconstruction of handwriting from multichannel EMG activity. Previously, linear regression methods (e.g., the Wiener filter) have been utilized for this purpose with some success. To improve reconstruction accuracy, we implemented the Kalman filter, which allows to fuse two information sources: the physical characteristics of handwriting and the activity of the leading hand muscles, registered by the EMG. Applying the Kalman filter, we were able to convert eight channels of EMG activity recorded from the forearm and the hand muscles into smooth reconstructions of handwritten traces. The filter operates in a causal manner and acts as a true predictor utilizing the EMGs from the past only, which makes the approach suitable for real-time operations. Our algorithm is appropriate for clinical neuroprosthetic applications and computer peripherals. Moreover, it is applicable to a broader class of tasks where predictive myoelectric control is needed.

Handwriting is a unique development of human culture. A skill learned during the early childhood, it remains among the primary means of communication and self-expression throughout the course of life. From the physiological point of view, handwriting is a complex interplay between the nervous system and the numerous muscles of the upper extremity. Despite several attempts to study this intricate activity theoretically (Plamondon and Maarse,

The relationship between the muscle force and the pen trajectory is complicated by the motor redundancy phenomenon (Bernstein,

Additional obstacle in studying the physiology of handwriting is the difficulty of measuring the muscle force directly. Surface and intramuscular Electromyography (EMG) are the common methods to register neuromuscular activity during a motor task. Surface EMG is a non-invasive method which implies placing the electrodes on the skin above the muscles of interest. Being easy and safe to implement, this technique, however, does not yield sufficiently accurate biomechanical measurements, due to the complex relationship between the EMG and muscle force, complicated anatomy of muscles and the inability to record from all the muscles involved, especially from the deep muscles. Intramuscular EMG (iEMG) is invasive and uses needle electrodes inserted into the muscle tissue to yield more spatially specific measurements with less leakage and disturbance. The invasive nature of iEMG limits its utility.

Both EMG registration methods posit several substantial difficulties, primarily related to signal quality and associated issues of noise filtering and source extraction from the observed data. Nonetheless, it was shown that even the surface EMG carries valuable information about the neuromuscular interactions and can therefore be used effectively in modeling and interpreting movements (Reaz et al.,

Despite the evident difficulties of measuring and interpreting neuromuscular activity with the currently available techniques, understanding such complex motor tasks as handwriting is important for both theoretical and practical reasons. Once we learn how to model the relationship between EMG patterns and pen movements during handwriting, we can introduce this knowledge to many rapidly expanding fields and practices, including biomedical engineering, robotics and biofeedback therapy. For instance, we can substantially improve the existing treatment and rehabilitation techniques for patients with a loss or an injury of an upper limb (Xiao and Menon,

However, such appealing advances and practices are still in their infancy. To date, the existing research on decoding of handwriting from electromyography is small and restricted to laboratory conditions. Several papers addressed the question of written character classification based on surface EMG, which involved implementation of machine-learning techniques to distinguish between muscle activation patterns for different written characters, such as digits, alphabet letters or simple geometric shapes. Linderman et al. (

The other studies considered a rather complex task of on-line decoding of the pen traces, based on the incoming EMG signals from the measurement electrodes. Among the most successful methods known to the authors, is the Wiener Filter (Linderman et al.,

In this paper, we consider the same multisubject data-set as in Linderman et al. (

In its classical formulation the Kalman Filter (KF) (Kalman,

In our application, the first information source is the dynamical model that captures the physical properties of the arm-wrist-pen device and is formalized as a multivariate autoregressive (MVAR) process, whose parameters are estimated from the data. The noisy vector of EMG measurements is the second source of information, whose relation to the pen coordinate is modeled via multivariate linear regression equation with coefficients determined from the training data-set.

For simplicity, the derivations provided in this section are based on the assumption of multivariate normality of the fused sources (Faragher,

As demonstrated by the statistical tests described in the Appendix (Section Testing the Assumptions of the Model), we could not reject the hypothesis of independence for the majority of trials. Based on this and, for the simplicity reasons, we base our developments in this paper on the assumption of independence of the two fused sources. In case the independence assumption is violated, the performance gained by taking into account the dynamics of the reconstructed process could have been more sizable should we use a slightly modified form of equations (Shimkin,

As the first information source, we assume that, at each time moment

where

_{t} is a [6

Based on Equation (1), we can derive the following expressions connecting the mean and the covariance matrix of the state vector at the two consecutive time moments.

_{1t} = _{1(t−1)} is a 6K-dimensional mean state vector at time

Detailed derivations of the model parameters can be found in the Appendix (Section Testing the Assumptions of the Model).

Usually, in the KF framework, the measurement equation appears in the

where

_{t} is a [8

_{t} is a [8_{t} is assumed to be independent from the process noise _{t}.

The 6

Since we do not model _{t} as a stochastic process, the covariance matrix of _{t} reduces to covariance matrix of the measurement noise, so that

As outlined in the previous two subsections, we have two independent sources of information about the state vector. The first endogenous source bases its predictions on the dynamical characteristics of the pen coordinates during the handwriting and yields _{1}(_{t}|_{t−1}) as the state vector distribution (red distribution in Figure _{2}(_{t}|_{t}) as the state vector distribution (blue distribution in Figure

The joint conditional estimate of the state vector is distributed as _{fused}(_{t}|_{t−1}, _{t}) (green distribution in Figure _{t}|_{t−1}, _{t}) reduces to a simple multiplication of the two probability density functions, i.e.,

The product of two multivariate normal distributions is also a multivariate normal (Rencher,

Specifically, _{fused} is the covariance matrix of the fused distribution can be computed as

and the 6

It is instructive to reformulate the expressions for the mean and the covariance matrix of the new distribution and to separate the influence of the two distributions being fused.

Using the matrix inversion lemma (Henderson and Searle,

we can rewrite Equations (5) and (6) as

and

For detailed derivation of the parameters see the Appendix.

The term _{t} in Equation (7), commonly known as the Kalman Gain, plays a crucial role of the dynamic scaling factor reflecting the distribution of trust in each of the two information sources. It depends on the relative amount of uncertainty present in the estimates by each of the information sources alone, and varies over time.

Since covariance matrices are positive definite, Equation (8) shows that by fusing the two distributions we reduce the variation associated with the state estimate, proportionally to the Kalman gain. At the same time, the fused mean (Equation 9) becomes the weighted average of the endogenously predicted and the measurements-based mean estimates.

Based on the above equations we are now ready to formulate the algorithm for calculating the Kalman Filter estimate. At each time moment the computation can be split into three consecutive steps.

Endogenous state prediction and error covariance update:

Kalman Gain Calculation:

Measurement Update:

In order to relate our approach to the classical KF paradigm in the equations above, we assigned the variables used in the previous subsection to the standard symbols, commonly employed in the KF literature. In the above algorithm, the first step is to use the State Transition Equation only and to calculate the so-called _{t|t−1} (Equations 10 and 11).

Then, we calculate the Kalman Gain based on the

Finally, the _{t|t} (Equation 15), which shows that, in the final estimate, the

Six healthy participants were instructed to write symbols from 0 to 9, repeating each symbol approximately 50 times. At the same time, muscle activity was recorded with eight bipolar-surface EMG electrodes, placed on each participants leading hand muscles:

Before applying the algorithm, we preprocessed EMG signals to extract the envelope via the standard rectification procedure. For each channel separately, we first calculated the absolute value of the EMG signals and then low-pass filtered the result with a second-order Butterworth Filter with the cut-off frequency of _{c}. We optimized the value of the cut-off frequency based on the training subset of the recorded data to obtain the best reconstruction performance. In the final results reported here _{c} = 2 Hz. Additionally, we have applied square-root transformation to each signal's envelope, obtained via the described rectification procedure.

Half of the trials of each symbol was randomly assigned to training the parameters of the model, while the remaining half was used for testing the performance (Figure

^{2}) is used as a measure of efficiency of reconstruction.

We used two basic experimental designs to calibrate our pen tracking algorithm.

Within-Group Design

A single set of parameters (

Between-Group Design

A separate set of parameters (^{n}, ^{n}, ^{n} and ^{n},

Note that for Within-Group Design, only one set of matrices was estimated by pooling all the training samples together, while in Between-Group Design the four matrices were estimated separately for each of the ten symbols. Then, the out-of-training sample measurements were used to reconstruct handwriting via the recursive process outlined in Section 2.1.5.

The testing procedure was the same within each experimental design (see Figure ^{2} between the actual coordinate and its fused estimate (Figure _{t}) explained by the reconstructed ones (Explained Sum of Squares _{e}), i.e.,

The accuracy was computed within each trial, and then averaged across trials for each symbol. Confidence intervals were computed to account for the standard errors associated with the variation across the participants.

We compared the accuracy of our model to the accuracy of the Wiener Filter (WF) estimate, which was originally tested on the same data-set by Linderman et al. (

In the framework of handwriting recognition from electromyography reported in Linderman et al. (_{1}, _{2}]. The unknown weights, mapping rectified EMG signals into the pen-tip coordinate vector, were estimated using the Ordinary Least Squares Method. It is important to stress that, in contrast to the approach reported in Linderman et al. (_{1},

The dynamical model (Equation 1) and the measurement model (Equation 2) contain model order parameters ^{2} as a metric of the goodness of reconstruction achieved (Figure

To search for the combination of ^{2})∕^{2}) ratio. The expected value ^{2}) and the standard deviation ^{2}) were computed over the trials in the data-set used for cross-validation. Therefore, high values of ^{2}.

^{2} for the mean reconstruction accuracy in the two coordinates (

Based on these maps, we set

We first trained one set of parameters for all symbols and used it to reconstruct pen traces from the EMG data in the testing set. Figure

Table

^{2} |
|||
---|---|---|---|

“0” | 0.65 ± 0.15 | 0.57 ± 0.17 | 0.61 ± 0.15 |

“1” | 0.49 ± 0.08 | 0.85 ± 0.08 | 0.67 ± 0.05 |

“2” | 0.71 ± 0.14 | 0.75 ± 0.12 | 0.73 ± 0.13 |

“3” | 0.59 ± 0.20 | 0.79 ± 0.09 | 0.69 ± 0.13 |

“4” | 0.71 ± 0.15 | 0.63 ± 0.17 | 0.67 ± 0.13 |

“5” | 0.67 ± 0.15 | 0.72 ± 0.05 | 0.70 ± 0.08 |

“6” | 0.68 ± 0.18 | 0.71 ± 0.11 | 0.70 ± 0.12 |

“7” | 0.59 ± 0.28 | 0.69 ± 0.22 | 0.63 ± 0.25 |

“8” | 0.61 ± 0.15 | 0.74 ± 0.12 | 0.77 ± 0.11 |

“9” | 0.62 ± 0.22 | 0.71 ± 0.15 | 0.67 ± 0.17 |

All | 0.63 ± 0.17 | 0.73 ± 0.14 | 0.68 ± 0.13 |

Statistically speaking, we managed to achieve the average accuracy of 63 ± 17% and 73 ± 14% with 95% confidence, for the two reconstructed coordinates, as estimated for the six participants of the experiment. Note that the average accuracy in both coordinates is higher than that found by Linderman et al. (

In the Between-group design, we attempted to learn the parameters of the Kalman Filter for each symbol separately, and then reconstruct the traces of the same symbol. Figure

The average reconstruction in the two coordinates between subjects was 78 ± 13% and 88 ± 7%, respectively. Table

^{2} |
|||
---|---|---|---|

“0” | 0.84 ± 0.06 | 0.86 ± 0.05 | 0.85 ± 0.05 |

“1” | 0.63 ± 0.15 | 0.97 ± 0.01 | 0.80 ± 0.08 |

“2” | 0.82 ± 0.11 | 0.92 ± 0.06 | 0.87 ± 0.09 |

“3” | 0.75 ± 0.12 | 0.93 ± 0.06 | 0.84 ± 0.08 |

“4” | 0.81 ± 0.11 | 0.81 ± 0.03 | 0.81 ± 0.05 |

“5” | 0.80 ± 0.10 | 0.86 ± 0.06 | 0.83 ± 0.07 |

“6” | 0.83 ± 0.05 | 0.88 ± 0.05 | 0.85 ± 0.05 |

“7” | 0.76 ± 0.20 | 0.88 ± 0.08 | 0.82 ± 0.14 |

“8” | 0.76 ± 0.12 | 0.86 ± 0.07 | 0.81 ± 0.09 |

“9” | 0.81 ± 0.09 | 0.84 ± 0.08 | 0.83 ± 0.08 |

All | 0.78 ± 0.13 | 0.88 ± 0.07 | 0.83 ± 0.08 |

In the previous subsection we have shown that, as expected, the Kalman Filter improves the reconstruction performance, as compared to the previously proposed method (Linderman et al.,

To consider the increase in accuracy, specifically associated with the dynamical model, we reconstructed pen traces of several symbols by Within-group design (one set of parameters for all symbols), and then repeated the procedure on the same samples using the Wiener Filter, fixing all other external parameters, including those related to the training-testing split of the data and data preprocessing techniques.

On average, introduction of the KF framework leads to a significant increase in accuracy for both reconstructed coordinates across the six participants (Figure

^{**}significantly greater than zero at 1% significance level.

^{2}) and the Wiener Filter Accuracy (WF ^{2}) between subjects, measured for each written character independently^{*}significantly greater than zero at 5% significance level, ^{**}significantly greater than zero at 1% significance level.

The ergonomics of the reconstructed handwriting traces plays an important role. The use of the dynamical model to enforce natural smoothness of handwriting yielded improved ergonomics of the recovered traces. Figure

In this work we applied the Kalman Filter approach to reconstruction of handwritten pen traces on the basis of EMG measurements. We have demonstrated that it is possible to obtain accurate and ergonomic reconstruction of pen traces and still remain in the linear framework to utilize all the benefits associated with it. Our results show a significant improvement over the figures, previously reported by Linderman et al. (

Although the accuracy of reconstruction does not generally go above 90%, in terms of the coefficient of determination, the method still offers a reliable and ergonomic reconstruction for all symbols. Approximately 25 trials of each symbol were enough to learn the parameters of the filter and yield comparable results in testing samples.

The method works well for both specific (Between-Group design) and general (Within-Group design) models, which reveals its potential for a wide range of applications. The Between-group design, although quite limited at first sight, is not entirely unrealistic, due to the fact that handwritten figures are very well discriminated between each other on the basis of EMG signals (Linderman et al.,

The Within-group design, on the other hand, is more applicable to on-line handwriting reconstruction, when no prior information about the class of the symbol being written is available. In this framework the causality of our approach offers additional benefits and makes the EMG-controlled handwriting feel natural to the user. Also, the natural smoothness of the traces, recovered with the use of the KF, provides for an improved feedback, which is crucial in the real-life on-line scenario.

While our method has shown improvement over the previously proposed technique, it still requires thorough consideration before it can be reliably applied in neuroprosthetic devices and rehabilitation practices. One of the main difficulties in applying the Linear Kalman filter is the high variability of results across subjects. The confidence intervals in Tables

Such heterogeneous performance across individuals apparently stems from a combination of behavioral and physiological factors, which we could not control in this study. Participants vary in the style and neatness of handwriting, including the way they hold and press the pen, and the strategies they apply to write the same symbol. Anatomical differences, such as the individual muscle length, muscle size and attachment to the bones and the differences in the amount of subcutaneous fat might be significant factors influencing the patterns of the recorded neuromuscular activity (EMGs). Additional source of variability may come from the variation in electrode placement sites.

The problem of EMG variability has received significant attention in the recent experimental literature (Linssen et al.,

Additionally, the results can be further improved by individually tuning the latent parameters, such as the model orders, filter cut-off frequency and sensor locations. In this work, however, we intentionally used a single set of latent variable values in order to emulate the out-of-box performance of such a system. Methods for non-supervised on-line adaptation and individual tuning of the latent variables need to be developed to address these issues and make the fine-tuning seamless to the user.

The linear framework of the filter offers significant benefits, such as stability and good generalization ability. However, the non-linear nature of the relation between the recorded EMG signals and the actuator trajectory prompts to explore the use of non-linear models in this application. The benefits brought in by the non-linearity, however, have to be leveraged against the additional complexity and potential instability associated with the use of such models.

In this paper we investigated the relationship between handwriting and neuromuscular activity measured by electromyography. We built and optimized the Kalman filter in order to reconstruct the pen coordinates based on the dynamical characteristics of handwriting and the corresponding EMG measurements.

We showed that the Kalman filter significantly outperforms previously proposed method (Linderman et al.,

The main attraction of the proposed method is its ability to smooth the noise and, as a result, provide a comprehensible and realistic reconstruction. Further progress in this field would potentially create intelligent rehabilitation techniques for patients with hand injuries, as well as become useful in human-computer interfaces, associated with handwriting.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The authors are grateful for the fruitful comments of Dr. Igor Furtat related to the algorithmic portion of the work. The experimental portion (Section 3) was funded by the government subsidy for improving competitiveness of National Research University Higher School of Economics among the leading world centers for research and education. The data fusion algorithm for reconstruction of pen traces (Section 2) was adapted to the use with the EMG data solely under support of RNF (grant 14-29-00142) in IPME RAS.

Equation (A3) shows the distribution of the estimate of the State Transition Model given by Equation (1).

The mean of the distribution is

The covariance matrix is derived the following way:

Since the state estimate noise _{t} is uncorrelated with the estimate,

and the covariance matrix reduces to:

Under the normality assumption the probability density function (pdf) of the state space vector _{t} can be written as

Since the additive noise term _{t} in Equation (2) is assumed to be Gaussian, while each of the observed EMG signals are not modeled as stochastic processes, the state vector follows the Gaussian distribution

By multiplying pdf in Equation (A3) by pdf in Equation (A4) we get a scaled fused Distribution of the state estimate:

with parameters

and a normalization constant

Let us reformulate the expressions for the mean and Covariance matrix of the new distribution in such a way that they separate the influence of the two fused distributions.

Using the Matrix Inversion lemma (the Woodbury matrix identity) for two matrices of the same dimension (_{1t} and _{2t}) and setting

We ran the Royston test for multivariate normality on the residuals of each training sample and concluded that for most samples the assumption of normality is not violated (at 1% significance level). Table

State transition model | 92 | 79 | 92 | 96 | 83 | 73 | 86 |

Measurement model | 93 | 85 | 92 | 94 | 80 | 77 | 87 |

Joint model | 89 | 72 | 89 | 93 | 76 | 67 | 81 |

The fusion of the two information sources described in this paper is heavily based on the assumption that the two merged sources are independent.

Then, we checked to what extent the two information sources can be considered independent. Taking into account the results of the test for multivariate normality, we checked for the absence of correlation between the error terms of the two merged models (Equations 1 and 2). We used the residuals of the two models to calculate the sample cross-correlation matrix and assessed the significance of its elements for each trial.

Given the sample size and the multiplicity of the tested hypothesis (36), we could not reject the null-hypothesis of no-correlation at 5% group level significance for the majority of trials (74%). Detailed distribution of the test performance across the participants is given in Table

Independent trials | 83 | 72 | 76 | 80 | 67 | 68 | 74 |