^{1}

^{2}

^{1}

^{1}

^{2}

^{1}

^{3}

^{1}

^{2}

^{1}

^{2}

^{3}

Edited by: Cecilia Laschi, Sant’Anna School of Advanced Studies, Italy

Reviewed by: Dongming Gan, Khalifa University, United Arab Emirates; Sunil L. Kukreja, National University of Singapore, Singapore

Specialty section: This article was submitted to Bionics and Biomimetics, a section of the journal Frontiers in Robotics and AI

This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

The rich variety of human upper limb movements requires an extraordinary coordination of different joints according to specific spatio-temporal patterns. However, unvealing these motor schemes is a challenging task. Principal components have been often used for analogous purposes, but such an approach relies on hypothesis of temporal uncorrelation of upper limb poses in time. To overcome these limitations, in this work, we leverage on functional principal component analysis (fPCA). We carried out experiments with 7 subjects performing a set of most significant human actions, selected considering state-of-the-art grasp taxonomies and human kinematic workspace. fPCA results show that human upper limb trajectories can be reconstructed by a linear combination of few principal time-dependent functions, with a first component alone explaining around 60/70% of the observed behaviors. This allows to infer that in daily living activities humans reduce the complexity of movement by modulating their motions through a reduced set of few principal patterns. Finally, we discuss how this approach could be profitably applied in robotics and bioengineering, opening fascinating perspectives to advance the state of the art of artificial systems, as it was the case of hand synergies.

Human hands represent an extraordinary tool to explore and interact with the external environment. Not surprisingly, a lot of studies have been devoted to model how the nervous system can cope with the complexity of hand sensory-motor architecture (Mason et al.,

For these reasons, in addition to many works devoted to analyze hand behavior, it is also possible to find studies modeling human upper limb motor workspace, either from a kinematic point of view, or from a muscular or neural point of view. In Heidari et al. (

Typical approaches based on principal component analysis are not suitable in this case because of the underlying hypothesis of temporal uncorrelation of upper limb poses in time. For this reason, to achieve this goal, we propose to use for the fist time functional principal component analysis (fPCA) to study upper limb motions. fPCA is a statistical method for investigating dominant modes of variation of functional data in time and has been widely used in one-dimensional or multi-dimensional time series analysis in chemistry, weather phenomena, and medicine (Aguilera et al.,

The choice to use fPCA as main data analysis tool is motivated by the fact that it allows to include some important features of the signal, such as shape and time dependence, which cannot be taken into account by other simpler data dimensionality reduction techniques (e.g., principal component analysis). To achieve this goal, we propose an experimental setup for studying upper limb movements, based on a Motion Capture (MoCap) system (^{®}). Using this tool, we carried out a series of experiments with human considering a comprehensive dataset of daily living activities (ADLs) and grasping/manipulation actions. These actions were selected relying on the study of grasping taxonomies (Cutkosky,

In order to develop a comprehensive study of human upper limb movements, one of the key features for the generation of a valid dataset is the definition of a set of meaningful actions (Santello et al.,

Protocol actions.

# | #Cutkosky | Class | Description |
---|---|---|---|

1 | Intransitive | Ok gesture (lifting hand from the table) | |

2 | Intransitive | Thumb down (lifting hand from the table) | |

3 | Intransitive | Exultation (extending the arm up in the air and keeping it in with closed fist) | |

4 | Intransitive | Hitchhiking (extending the arm along the frontal plane, laterally, parallel to the floor, with extended elbow, closed fist, extended thumb) | |

5 | Intransitive | Block out sun from own face (with open hand, touching the face with the palm and covering the eyes) | |

6 | Intransitive | Greet (with open hand, moving wrist) (three times) | |

7 | Intransitive | Military salute (with lifted elbow) | |

8 | Intransitive | Stop gesture (extending the arm along the sagittal plane, parallel to the floor, with extended elbow, open palm) | |

9 | Intransitive | Pointing (with index finger) of something straight ahead (with outstretched arm) | |

10 | Intransitive | Silence gesture (bringing the index finger, with the remainder of the hand closed, on the lips) | |

11 | 2 | Transitive | Reach and grasp a small suitcase (placed along own frontal plane) from the handle, lift it and place it on the floor (close to own chair, along own sagittal plane) |

12 | 3 | Transitive | Reach and grasp a glass, drink for 3 s (stop signal by the examiner) and place it in the initial position |

13 | 4 | Transitive | Reach and grasp a phone receiver (placed along own sagittal plane), carry it to own ear for 3 s (stop signal by the examiner) and place it in the initial position |

14 | 6 | Transitive | Reach and grasp a book (placed overhead on a shelf), put in on the table and open it (from right side to left side) |

15 | 8 | Transitive | Reach and grasp a small cup from the handle (2 fingers + thumb), drink for 3 s (stop signal by the examiner) and place it in the initial position |

16 | 11 | Transitive | Reach and grasp an apple, mimic biting, and put it in the initial position |

17 | 12 13 | Transitive | Reach and grasp a hat (placed on the right side of the table) from its top and place it on own head |

18 | 12 | Transitive | Reach and grasp a cup from its top, lift it and put it on the left side of the table |

19 | 15 | Transitive | Receive a tray from someone (straight ahead, with open hand) and put it in the middle of the table |

20 | 16 | Transitive | Reach and grasp a key in a lock (vertical axis), extract it from the lock and put it on the left side of the table |

21 | 1 | Tool mediated | Reach and grasp a bottle, pour water into a glass, and put the bottle in the initial position |

22 | 2 3 4 | Tool mediated | Reach and grasp a tennis racket (placed along own frontal plane), and play a forehand (the subject is still seated) |

23 | 5 | Tool mediated | Reach and grasp a toothbrush, brush teeth (horizontal axis, one time on left side one time on right side), and put the toothbrush inside a cylindrical holder (placed on the right side of the table) |

24 | 6 | Tool mediated | Reach and grasp a laptop and open the laptop (without changing its position) (4 fingers + thumb) |

25 | 7 8 9 | Tool mediated | Reach and grasp a pen (placed on the right side of the table) and draw a vertical line on the table (from the top to the bottom) |

26 | 7 | Tool mediated | Reach and grasp a pencil (placed along own frontal plane) (3 fingers + thumb) and put it inside a squared pencil holder (placed on the left side of the table) |

27 | 9 | Tool mediated | Reach and grasp a tea bag in a cup (1 finger + thumb), remove it from the cup, and place it on the table on the right side of the table |

28 | 10 | Tool mediated | Reach and grasp a doorknob (disk shape), turn it clockwise, and counterclockwise and open the door |

29 | 13 | Tool mediated | Reach and grasp a tennis ball (with fingertips) and place it in a basket placed on the floor (close to own chair) |

30 | 14 | Tool mediated | Reach and grasp a cap (2 fingers + thumb) of a bottle (held by left hand), unscrew it, and place it overhead on a shelf |

We focused on kinematic recordings, which were achieved using a commercial system for 3D motion tracking with active markers (^{®}). Ten stereo-cameras working at 480 Hz tracked 3D position of markers, which were fastened to supports rigidly attached to upper limb links. In this manner, 20 markers were accommodated on the upper limb so that the distance between elements of each support was fixed. Supports were suitably designed for these experiments and printed in ABS (see Figure

In these figures, we show the experimental setup. In

Seven adult right-handed subjects (5 males and 2 females, aged between 20 and 30) performed the experiment. Each task was repeated three times in order to increase the robustness of collected data. The experimenter gave the starting signal to subjects. In the instructions, the experimenter emphasized that the whole movement should be performed in a natural fashion. The object order was randomized for every subject. Each subject performed the whole experiment in a single day. No subject knew the purpose of the study, and had no history of neuromuscular disorders. Each participant signed an informed consent to participate in the experiment, and the experimental protocol was approved by the Institutional Review Board of University of Pisa, in accordance with the declaration of Helsinki. The complete experimental setup is reported in Figures

An accurate description of human upper limb is challenging due to the high complexity of the kinematic structure, e.g., for axis location and direction, which are usually time varying. In order to explore the system complexity, the interested reader can refer to Maurel and Thalmann (_{1}, …, _{7}: _{1} is associated with the shoulder abduction–adduction; _{2} is associated with the shoulder flexion–extension; _{3} is associated with the shoulder external–internal rotation; _{4} is associated with the elbow flexion–extension; _{5} is associated with the elbow pronation–supination; _{6} is associated with the wrist abduction–adduction; _{7} is associated with the wrist flexion–extension. In Figure

System parametrization. In _{1}, …, _{7} refers to joints of the model. _{ref}_{ref}_{S}_{S}_{E}_{E}_{W}_{W}_{H}_{H}_{ref}_{S}_{S}_{E}_{E}_{W}_{W}_{H}

In order to describe the forward kinematics of the arm, 5 different reference systems was defined: _{ref}_{ref}_{S}_{S}_{E}_{E}_{W}_{W}_{H}_{H}_{ref}_{S}_{S}_{E}_{E}_{W}_{W}_{H}_{1}, …, θ_{k}_{j}^{T}

Links movements were tracked by fastening optical active markers to upper limb links. Markers positioning is inspired by Biryukova et al. (

Markers positioning. In the left figure, we report the arm markers position; in the central figure, we report the forearm markers position; and in the right figure, we report the hand marker position.

The model is completely parameterized using 14 parameters (different for each subject) collected in a vector _{G}_{G}_{G}_{1}, …, _{7}]^{T}

As previously mentioned, the parameters of the kinematic model must be adapted for the specific subject that performs the experiments. The optimal parameters were obtained by solving a constrained least-squares minimization problem:

The residual function _{k}_{k}_{k}_{G}_{k} _{k}_{G}_{k}_{k}_{G}_{x}_{p}_{k}_{G}_{k}_{k}_{G}_{k}_{k}_{k} in a sample task. Taking inspiration from Gabiccini et al. (_{k}_{k}_{k}_{k}_{k}_{k}_{k}

Mean squared error obtained in the estimation procedure in a sample. Initial error value is 57.1 mm, related to the filter initialization.

Given the state at time frame _{k}

The performance of the estimation tool for time frame k can be evaluated by calculating the mean squared error (MSE) _{k}_{markers}_{k}_{k}

The goal of this work is the study of functional motor synergies of upper limb. This is accomplished using functional PCA, a statistical method that allows to study the differences in shapes between functions. In order to avoid the inclusion in this analysis of undesired features due to misalignments in time or in velocity of the samples, we performed the following pre-processing techniques: segmentation, to divide the repetition of each task, time warping, to synchronize in time all the elements of the dataset.

For each task, the three repetitions have been segmented using the following procedure (see Figure

select the data elements of the third DoF (_{3});

find the first two peaks _{1}, _{2} of the signal;

evaluate the mean slope _{1}, _{2} of the signal in a section close to each peak;

calculate the segmentation point as

repeat points 2–4 using the second and the third peaks and obtain _{2}.

Segmentation procedure. In the left figure, we report a sample of joint evolution in time. In the central figure, we show two peaks of the signal _{1}, _{2} and the mean slope of the signal _{1}, _{2} evaluated in two ranges close to each peak. The segmentation point _{1} is evaluated as

_{3} data (i.e., shoulder flexion–extension) was used for segmentation because it almost always contains three distinct peaks. If the peaks were not detectable, another DoF with detectable peaks was used instead. Note that the segmentation is performed using the same couple of segmentation points for all the 7 DoFs.

Considering different subjects and tasks, differences between shapes are evident (see Figure

Segmentation and time warping.

The synchronization between two signals allows to increase the affinity by conforming starting-time and speed of the action. This can be achieved by finding the optimal time-shift and time-stretch of one signal w.r.t. the other one. This problem is known in literature as _{1} and _{2}, the affinity between the two signals is increased by the solution of the following least-squares minimization problem:
_{2} and _{2}. The dataset elements were time-warped w.r.t. a reference time series, selected in the set as the element whose length is the mean value w.r.t. the length of all dataset elements. For each element,

Scheme of data analysis.

To explain fPCA, it is useful to start from classic principal component analysis (PCA). Principal component analysis (PCA) is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. This transformation is defined in such a way that the first principal component (PC) has the largest possible variance (that is, it accounts for the largest part of the variability in the data). The other components explain an amount of variance in decreasing order, with the constraint that each principal component is orthogonal to the previous ones. Hence, the resulting vectors represent an orthogonal basis set. Principal components are calculated as eigenvectors of the covariance matrix of data. The variance explained by each PC is calculated as normalization of the corresponding eigenvalue. Given the first eigenvector ξ_{1}, the principal component score _{1}|| = 1; the second eigenvector ξ_{2} maximizes _{2}|| = 1 and

Functional PCA can be described as a functional extension of PCA. The first functional principal component ξ_{1}(_{2}(_{2}|| = 1 and

Consider a dataset of functions _{i}

Remove the mean calculated in step 1 from each data element by

Define a basis function. The basis must contain a number of functions large enough to consider all possible modes of variations of data. Usually basis elements are exponential functions, splines, Fourier basis (Ramsay and Silverman,

Given the basis functions _{1}, …, _{N}

Then each function is described by a vector of coefficients Θ = (θ_{1}, …, θ_{N}

PCA is now performed on these vectors. This leads to define the PCs, which are vectors of coefficients;

Each PC is, then, transformed into the corresponding function principal components (fPCs) using basis elements as

Each fPC explains a certain percentage of variance. The variance explained by an fPC is quantified normalizing (w.r.t. the sum of the eigenvalues) the corresponding eigenvalue of the covariation matrix.

We used fPCA on this dataset after the post-processing phase reported in previous sections. 15 fifth order spline basis elements were used, taking inspiration for the polynomial description in Flash and Hogan (_{k}^{th} knot. The fPCs can be used to reconstruct the data sample by adding M fPCs weighted by coefficients _{i}

This analysis allows to infer that the first fPC by itself account for 60–70% of the variation w.r.t. the mean function, as reported in Figure _{rec}

Explained variance for different DoFs and for each fPC.

In

In

Figure _{RMS}_{RMS}_{RMS}_{RMS}_{RMS}

In this work, we have shown that the complexity of upper limb movements in activities of daily living can be described using a reduced number of functional principal components. To achieve this goal, we developed an experimental setup, which is based on kinematic recordings but also allows to include additional sensing modalities. Kinematic data are based on a 7 DoFs model and are quantified through a calibration-identification procedure. Collected data were used to characterize upper limb movements through functional analysis. The findings of this work can be used to pave the path toward a more accurate characterization of human upper limb principal modes, opening fascinating scenarios in rehabilitation, e.g., for automatic recognition of physiological and pathological movements (e.g., stroke affected subjects) through machine learning.

At the same time, the here reported results and future investigations could also offer a valuable inspiration for the design and control of robotic manipulators. First, recognizing that few principal modes describe most of kinematic variability could provide insights for a more effective planning and control of robotic manipulators. For the planning phase, using input trajectories as combinations of the main functional components, which explain most of the kinematic variability, could represent a successful initial guess to control the movement of the robot—eventually combined with a feedback correction. This combination of feedforward and feedback components could be successfully employed also with soft robotic manipulators, i.e., robots designed to embody safe and natural behaviors relying on compliant physical structures purposefully used to achieve desirable and sometimes variable impedance characteristics. In these cases, standard methods of robotic control can effectively fight against or even completely cancel the physical dynamics of the system, replacing them with a desired model—which defeats the purpose of introducing physical compliance. To overcome this limitation in Della Santina et al. (

This study was carried out in accordance with the recommendations of Regione Toscana, D.G.R. no. 158 23/02/2004, “Direttive regionali in materia di autorizzazione e procedure di valutazione degli studi osservazionali,” with written informed consent from all subjects in accordance with the Declaration of Helsinki, and in observation of the “Guideline for good clinical practice E6(R1), International Council for Harmonization of Technical Requirements for Pharmaceuticals for Human Use (ICH).” The protocol was approved by the local Ethical Committee, i.e., “Comitato Etico di Area Vasta Nord-Ovest (CEAVNO).”

GA, CS, MB, and AB designed the study. GA, FF, and MB designed the protocol. GA, EB, and FF designed and developed the experimental setup. GA and FF performed the experiments. GA, CS, and MB performed data analysis. All authors contributed to writing the manuscript.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.