^{1}

^{*}

^{1}

^{2}

^{1}

^{3}

^{1}

^{2}

^{3}

Edited by: Pedro Antonio Valdes-Sosa, Clinical Hospital of Chengdu Brain Science Institute, China

Reviewed by: Daniel Cozzolino, Central Queensland University, Australia; Anirban Dutta, University at Buffalo, United States

This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

Neuroscience research shows a growing interest in the application of Near-Infrared Spectroscopy (NIRS) in analysis and decoding of the brain activity of human subjects. Given the correlation that is observed between the Blood Oxygen Dependent Level (BOLD) responses that are exhibited by the time series data of functional Magnetic Resonance Imaging (fMRI) and the hemoglobin oxy/deoxy-genation that is captured by NIRS, linear models play a central role in these applications. This, in turn, results in adaptation of the feature extraction strategies that are well-suited for discretization of data that exhibit a high degree of linearity, namely, slope and the mean as well as their combination, to summarize the informational contents of the NIRS time series. In this article, we demonstrate that these features are inefficient in capturing the variational information of NIRS data, limiting the reliability and the adequacy of the conclusion on their results. Alternatively, we propose the linear estimate of differential entropy of these time series as a natural representation of such information. We provide evidence for our claim through comparative analysis of the application of these features on NIRS data pertinent to several working memory tasks as well as naturalistic conversational stimuli.

Recent years witness a growing interest in Near-Infrared Spectroscopy (Ferrari and Quaresima,

Utilization of NIRS for monitoring the brain activity becomes more attractive, considering the non-invasive operational setup of NIRS-related devices. These devices are easy to use with portable, light-weighted headsets that are comparatively more immune to body movement (Dieler et al.,

An important characteristic that is attributable to NIRS and fMRI time series is the underlying linear property of the hemodynamic responses that are measured by these devices (Dale and Buckner,

These results, in turn, help explain the choice of feature extraction methodologies that are well-suited for discretization of data with a high degree of linearity (i.e., linear changes in hemodynamics in response to a given stimulus), namely, the slope (Herff et al., ^{1}

On the other hand, research suggests a direct correspondence between variational behavior of the brain activity and its information content (Miller, ^{2}

These findings are in line with the concept of entropy in information theory (Cover and Thomas,

In particular, linear estimate of differential entropy (Xiong et al.,

In this article, we address these shortcomings through a systematic investigation of the mathematical foundation of the degree of correspondence between DE and these time series. In line with this perspective, we argue that averaging-based feature spaces are inefficient in representing the information content of the NIRS time series of the brain activity of human subjects. In doing so, we adapt the viewpoint of Miller (

Our contributions are twofold. Firstly, we present the mathematical bases for the shortcomings of the averaging-based feature spaces. Secondly, we prove that efficiency of the linear estimate of differential entropy of the time series of brain activity is due to its functional correspondence with the underlying spiking rate of neural activity. This is in line with the results in the literature, implying the correspondence between brain regional activation and the increase of the blood flow (Gusnard and Raichle,

Our results suggest the potential that utilization of the linear estimate of the differential entropy of NIRS time series data can provide to the solution concept of the analysis of the brain activity of human subjects in response to working memory tasks whose variational intensities are nontrivial.

As we presented in section 1, the central role of linear models in NIRS-based applications result in adaptation of the feature extraction strategies that are well-suited for discretization of data with a high degree of linearity, namely, the slope, the mean, and their combination to summarize the informational contents of the NIRS time series. In this section, we demonstrate the shortcomings of such feature spaces. Subsequently, we propose the use of linear estimate of DE of these time series as a natural choice for extracting the information content of these series.

It is apparent that slope is optimum if the data of a given time series is collinear and monotonic in its order. This is a substantial limiting factor since most physical system, including the brain, have their limits defined by their power limitation which corresponds to a limit on the variance of their outcome (Stone,

Proposition 2.1.

where _{i} ∈ _{i} ∈

Furthermore, for a monotonically increasing independent variable (e.g., time and/or indices of data point in time series), we have:

Given the Equations (2) and (3), we have:

where

■

A customary practice in NIRS analysis is the transformation of data to ensure the reduction of the effect of the overall brain activity that is unrelated to the events of interests. These include subtraction/division of data with resting data (Fazli et al., _{X} and σ_{X}, and scaling within [0…1] interval (Hong et al., _{min} and _{max} indicating the minimum and maximum values in

Proposition 2.2.

where _{X}, σ_{X}, _{min}, and _{max} are mean, standard deviation, minimum, and maximum values of

The mutual information between

This implies that:

Furthermore,

Using Equations (8) and (9), we have:

and _{X} < 1, σ_{X} ≠ 0 and −1 ≤ _{max} − _{min} < 1, _{max} − _{min} ≠ 0. However, this requirement is unwarranted since _{max} − _{min} ∈ ℝ and σ_{X} ∈ ℝ.

■

Although averaging provides an effective tool for the analysis of certain aspects of the naturalistic stimuli (e.g., neural coupling between the speaker and the listener, Stephens et al.,

Proposition 2.3.

_{i} ⊆ _{i}|| =

where _{X} are the sample mean of _{i} and the mean of the time series

due to the law of large numbers. Therefore, these values reveal the general trend of their respective segments, thereby tending to the overall trend i.e., the expected value of

■

A common practice in the literature pertinent to analysis of the effect of the stimuli on the hemodynamic and/or the neural activity of the human subjects is the application of the sliding window on the given time series that is associated with these signals. In such a setting, the two consecutive segments that are extracted from the time series

On the other hand, Eden and Kramer (

where,

with _{X} is:

■

We conducted two series of experiments, referred to as

WME: It consisted of four different working memory tasks, namely, Listening Span Test (LST) (Osaka et al.,

LST: There were two subtasks, namely, L1 and L2, consisting of two and three sentences, respectively. These sentences were readout to the participants sequentially. Participants were instructed to judge the validity of each sentence once its reading was over [e.g., Sun sets in the west. (yes/no?)]. Once reading of sentences of a given subtask were complete, participants were required to recall the first word of each of the sentences. This resulted in two and three words recall in case of L1 and L2, respectively.

B: It included a one-back (B1) and a two-back (B2) WM tasks. We used a recorded call-out of numerical sequences (0 through 9) in which participants were required to respond to sequential (i.e., B1) and every-other (i.e., B2) repetition of these digits via clicking the arrow keys on the computer keyboard.

S: It contained two subtasks, involving two-color (i.e., S1) and three-color (i.e., S2) streams. Both of these subtasks consisted of a sequence of twenty words (i.e., name of a color such as “red,” or “green”) that were randomly matched/mismatched with their corresponding colors (e.g., word “red” was shown with its matching color, red, or a mismatching color such as blue). We used the color/word “red, blue“ in S1 and “red, blue, green” in S2.

M: It comprised of two subtasks, requiring the mental addition of a two-digit number with a single-digit number (M1) and two two-digit numbers (M2), respectively. There were four addition operations in each of these subtasks, resulting in eight arithmetic operations in total. In case of M2, half of these operations resulted in carryover.

Every subject participated in all of these four WM tasks. We acquired a 1-min-long resting data of the participants (with their eyes closed) prior to start of each subtask which was followed by its corresponding task. Furthermore, we randomized the ordering of these WM tasks while keeping the order of their corresponding subtasks intact for all participants. We used the PsychoPy (Peirce,

CTE: This paradigm comprised of 3-min-long conversation sessions in which we discussed four different topics (two easy and two difficult) with the participants (in Japanese). We communicated with our participants through minimalist anthropomorphic android, the Telenoid, to eliminate the potential effect of human characteristics such as gender and age. This resulted in four separate sessions, per participant. In every session, we began with acquiring a 1-min-long resting data, followed by its corresponding 3-min-long experimental session. We kept the content of conversations intact in all sessions. Every subject participated in all of these settings. However, we randomized the order of the easy/difficult among our participants. We provided our participants with a 1-min-long resting break (while staying at their seat with their eyes closed) prior to the commencement of each of these session. We maintained an approximately 1.2 m distance between the seat of the participant and the Telenoid. A male person conversed with our participants in all four conversational sessions.

WME: Thirteen young adults (nine females and four males,

CTE: Our participants included twenty two individuals (fourteen females and eight males,

All participants were right-handed [confirmed using FLANDERS (Nicholls et al.,

We used Near-Infrared Spectroscopy (Ferrari and Quaresima, _{1}, _{3}, _{1}, and _{3}, as shown in Figure _{1} and _{1} have a 1.0 cm and _{3} and _{3} have 3.0 cm source-detector distances, respectively. Findings in the literature on brain region activation during memory and language processing suggest a left-lateralized activation in both genders with higher specificity in females (Haut and Barch, _{3} of this device in the present study. It is worth noting that the source-detector distance of 3.0 cm is adequate for proper data acquisition of cortical brain activity using NIRS-based devices (Ferrari and Quaresima,

The NIRS device

First, we normalized the data corresponding to the selected NIRS channel via subtracting the mean of the 1 min resting period as a baseline from its data. Next, we applied a one-degree polynomial butter worth filter on this normalized data with 0.01 Hz and 0.6 Hz for low and high bandpass values, respectively. This was followed by its linear detrending.

After data preprocessing step, we applied Wilcoxon signed-rank test on calculated features of time series data of our participants in both, WME and CTE experimental paradigms, to investigate the utility of different feature spaces in capturing potential differences in NIRS time series of brain activity. It is worth noting that we chose this non-parametric test to avoid any assumption on the underlying distribution of the data, as it is the case for two-sample

In case of WME, we first computed the mean, combined mean & slope, moving average, and DE features from the entire time series of corresponding subtasks, per participant. Figure

Feature extraction from _{i} ∈ _{1}) implies calculating mean of the 1st 5-s-long segment while adapting mean feature extraction strategy with

In case of CTE, we segmented each time series data into non-overlapping 5-s-long segments. For each segment, we extracted a feature using mean, mean & slope, moving average, and DE. Figure

For cluster analysis in

_{j} ∈ _{i} refers to _{i} represents the feature vector of this

In this section, We validate our mathematical proofs on effectiveness of the linear estimate of differential entropy (DE) of NIRS time series in contrast with the conventional averaging-based feature extraction strategies through statistical analysis (Wilcoxon test) of the brain activity of human subjects. Our analyses pertain to two different experiments, namely, Working Memory Tasks Experiment (WME) and Conversational Tasks Experiment (CTE). Results in each of these sections are structured as follows.

WME: section 3.1.1 includes the analyses of Listening Span Test (LST), N-Back, Stroop, and Mental Arithmetic (MA) Working Memory (WM) tasks where each of these tasks comprised of two subtasks of different cognitive loads. In this subsection, we use features that are calculated based on actual value of NIRS time series data of the participants. On the other hand, we present results of this test on normalized NIRS time series data of the participants in section 3.1.2.

CTE: section 3.2 provide evidence on ability of linear estimate of DE in resolving the shortcoming of the averaging-based features in case of naturalistic stimuli (Spiers and Maguire,

Wilcoxon test on application of mean as adapted feature space implied significant difference between two subtasks of Listening Span Test (LST) Working Memory (WM) tasks, i.e., L1 and L2 [_{(24)} = −4.31], as well as subtasks S1 and S2 in Stroop WM [_{(24)} = −6.65]. However, it indicated non-significant with respect to Mental Arithmetic M1 and M2 [_{(24)} = 1.20] as as well as N-Back B1 and B2 [_{(24)} = 0.72] WM tasks.

Similarly, combination of mean & slope indicated significant difference between L1 and L2 [_{(24)} = −4.31], as well as S1 and S2 [_{(24)} = −6.65]. However, it implied non-significant with respect to M1 and M2 (_{(24)} = 0.72].

Although, application of moving average showed significant difference between L1 and L2 [_{(24)} = −4.31] as well as S1 and S2 [_{(24)} = −6.65], it implied non-significant with regards to M1 and M2 [_{(24)} = 1.17] as well as B1 and B2 [_{(24)} = −0.23].

On the other hand, DE indicated significant differences between L1 and L2 [_{(24)} = −4.31], S1 and S2 [_{(24)} = −4.72], as well as M1 and M2 [_{(24)} = −2.55] while suggesting non-significant difference between B1 and B2 [_{(24)} = −0.47]. Table

Working Memory Experiment (WME): Mean (M), Standard Deviation (SD), and Standard Error (SE) of Moving Average, DE, Mean, and combined Mean & Slope feature spaces with respect to N-Back (B1 and B2), Listening Span Test (L1 and L2), Stroop (S1 and S2), and Mental Arithmetic (M1 and M2) WM tasks.

B1 | 763.20 | 2.54 | 0.48 | 10.39 | 0.20 | 0.04 | 374.54 | 0.13 | 0.02 | 374.54 | 0.13 | 0.02 |

B2 | 762.81 | 3.71 | 0.70 | 10.39 | 0.17 | 0.03 | 376.85 | 48.57 | 0.09 | 374.52 | 0.09 | 0.02 |

M1 | 87.83 | 26.95 | 5.01 | 5.86 | 1.12 | 0.21 | 42.60 | 13.27 | 2.46 | 42.62 | 13.26 | 2.46 |

M2 | 80.44 | 29.06 | 5.40 | 6.68 | 1.03 | 0.19 | 39.10 | 14.25 | 2.65 | 39.12 | 14.25 | 2.64 |

S1 | 106.03 | 14.70 | 2.68 | 6.22 | 0.77 | 0.14 | 50.90 | 7.23 | 1.32 | 50.91 | 7.23 | 1.32 |

S2 | 212.98 | 27.80 | 5.08 | 7.51 | 1.24 | 0.23 | 105.82 | 14.12 | 2.58 | 105.82 | 14.12 | 2.58 |

L1 | 286.48 | 33.48 | 9.29 | 7.61 | 1.95 | 0.54 | 142.85 | 16.61 | 4.61 | 142.85 | 16.61 | 4.61 |

L2 | 771.07 | 0.69 | 27.93 | 10.47 | 0.31 | 0.09 | 376.85 | 48.57 | 13.47 | 376.85 | 48.57 | 13.47 |

Figure _{(24)} = −2.41, SD = 0.26] and Stroop, i.e., S1 and S2 [_{(24)} = −1.99, SD = 0.22] where mean [LST: _{(24)} = 0.05, SD = 0.27, Stroop: _{(24)} = −1.42, SD = 0.24], mean & slope [LST: _{(24)} = −1.43, SD = 0.27, Stroop: _{(24)} = −1.43, SD = 0.24], as well as moving average [LST: _{(24)} = 0.0, SD = 0.28, Stroop: _{(24)} = −1.65, SD = 0.23] were unable to determine such differences. Most interesting is the result of utilization of these features in analysis of mental arithmetic (MA) where DE indicated an apparent significant between M1 and M2 (_{(24)} = −2.02, SD = 0.28] while other features implied a tendency [Mean: _{(24)} = 1.43, SD = 0.28, Mean & Slope: _{(24)} = 1.43, SD = 0.28, Moving Average: _{(24)} = 1.48, SD = 0.28].

DE, mean, combined mean & slope, and moving average of the feature vectors of NIRS time series of our participants, scaled to fit within [0.…1] interval using

Wilcoxon test implied non-significant between NIRS time series of the participant with respect to the topic of conversation, i.e., easy and hard, using mean [_{(746)} = 0.76], moving average [_{(746)} = 0.83], as well as combined mean & slope [_{(746)} = 0.69]. However, it indicated significant based on application of DE on these time series [_{(746)} = 4.00]. Table

Conversational Tasks Experiment (CTE): Mean (M), Standard Deviation (SD), and Standard Error (SE) of Moving Average, DE, Mean, and combined Mean & Slope feature spaces with respect to easy and hard conversational topics during CTE.

Easy | −20.31 | 179.05 | 9.46 | 1.73 | 0.84 | 0.04 | −14.87 | 63.25 | 5.16 | −14.83 | 61.20 | 7.16 |

Hard | −15.82 | 149.23 | 7.56 | 2.73 | 1.99 | 0.07 | −8.62 | 76.61 | 9.26 | −8.90 | 78.67 | 10.26 |

In this article, we argued that averaging-based feature extraction strategies that are inspired by high degree of linearity in NIRS time series of brain activity of human subjects results in suboptimal solution in capturing the variational information of these signals, thereby limiting the reliability of an adequate conclusion on their outcomes. Alternatively, we proposed the linear estimate of differential entropy of these time series as a natural representation of such information. We provided evidence for our claim through theoretical and empirical comparative analyses of the application of these features on NIRS data pertinent to a number of WM tasks with varying level of cognitive loads. Concretely, we demonstrated the utility of DE in contrast with mean, combination of mean & slope, and moving average feature spaces in analysis as well as differentiation of subtasks of a several WM tasks into their corresponding classes. These WM tasks included Listening Span Tests (L1 and L2), Stroop (S1 and S2), N-Back (B1 and B2), and Mental Arithmetic (M1 and M2). We further showed the confounding effect of these averaging-based feature extraction strategies via analysis of a naturalistic conversational tasks with differing level of difficulty in their respective topics, thereby providing evidence on inability of these features in representing the significance in responses of our participants to varying contextual complexity of conversational topics. Subsequently, we presented the sensitivity of DE in extracting this information. Moreover, we illustrated the substantial similarities between distribution of data based on mean and combination of mean & slope, thereby indicating the negligible contribution of the slope in representation of the information content of the brain activity of human subjects in response to WM as well as naturalistic conversational tasks with varying degree of difficulty in their topics.

Although we found similar indication of non-significant difference between N-back subtasks through application of DE as well as averaging-based feature spaces of the NIRS time series of the brain activity of the participants, we suggest that such a similarity is due to the significant resembling dynamics of these subtasks, and consequently, their equivalence in imposed cognitive loads on human subjects. This claim is due to comprehensive results in study and analysis of N-Back WM task (Owen et al.,

Fano factor (Fano, ^{2} ≤ 1 in

Another source of evidence on significance of the variational information of brain activity is due to the results of the analyses of physiological systems from perspective of their dynamical complexity. Research suggests that increase in complexity is an inherent attribute of healthy physiological systems (Lipsitz and Goldberger,

Research on working memory (WM) (Baddeley,

An important implication of analytical studies of pattern of brain activity of human subjects during cognitive tasks is their integration in real-life applications (Mitchell et al.,

Clusters generated through application of K-mean clustering algorithm (Liao,

Apart from the capability of DE in capturing the variational information that is implicit in the responses of the brain activity of human subjects to mental tasks with varying cognitive loads, DE presents a reliable tool for quantitative measurement of the amount of information in these activities. Concretely, we found that the difference in the amount of information (measured in bit i.e., base 2 logarithm) in B1 (Mean = 5.10, SD = 1.08) was significantly above one standard deviation from L1 (Mean = 3.77,

Proposal of the linear estimate of differential entropy as a feature extraction strategy for NIRS time series implies two assumptions. Firstly, it assumes a linearity of the underlying dynamics of the time series data under investigation (Kaiser and Schreiber,

In this article, we studied the shortcoming of averaging-based feature extraction strategies in capturing the information content of brain activity of human subjects, as represented by NIRS time series, during WM and conversational tasks. Furthermore, we demonstrated the efficiency of linear estimate of differential entropy (DE) in quantification of information content of such time series, thereby presenting its correspondence with the underlying spiking neural activity.

We validated our mathematical analyses through application of these features in analysis of a number of working memory (WM) tasks. Our results suggested that DE shows higher sensitivity to brain activity of human subjects in comparison with mean, slope, combination of mean & slope, as well as moving average feature spaces. In addition, we showed that DE has higher sensitivity with regards to information gain through comparative analysis of its results in contrast with averaging-based feature spaces after the application of normalization and scaling. It is worth noting that such steps as baseline correction, normalization, and scaling are of crucial importance since they help refine data, thereby reducing detrimental effect of undesirable variation such as effect of outliers and biasing.

We validated the sensitivity of DE in capturing the variational information of time series of brain activity of human subjects to more naturalistic scenarios through comparison of its results with respect to averaging-based feature spaces on data pertinent to conversational time series. Whereas mean, slope, combination of mean & slope, as well as moving average feature extraction strategies implied non-significant in brain activity of human subjects in response to topics of conversation (i.e., easy topic such as daily activities vs. difficult such as conversation on a controversial topic), DE indicated significant difference in time series of brain activity in response to these conversational topics.

Although our results suggested the utility of DE in analysis of brain activity of human subjects pertinent to WM tasks with varying degree of cognitive loads, they do not imply its utility as a universal NIRS feature. For instance, it is necessary to examine its utility on NIRS time series of other mental states such as relaxation, resting, and vigilance, to name a few. Therefore, our results primarily represent the first step toward realization of the potential of this feature extraction strategy. Accordingly, future research that is devised with larger sample sizes along with more rigorous experimental settings and quantitative validation measures is necessary to derive an informed conclusion on its performance.

SK carried out the proofs and analyses. HS conceived the experiments and supervised their progress. RY conducted the conversational experiment. As the head of Hiroshi Ishiguro Laboratories (HIL), HI oversees the entire activity of all research teams and themes, ensuring the soundness of all proposals, quality of results, and their validity. SK and HS contributed equally in preparation of this manuscript.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

This study was carried out in accordance with the recommendations of the ethical committee of Advanced Telecommunications Research Institute International (ATR) with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the ATR ethical committee (approval code:16-601-1).

In our analyses, we found similar indication of non-significant difference between N-back subtasks through application of DE as well as mean of the NIRS time series of the brain activity of the participants. We further suggested that such a similarity is due to the significant resembling dynamics of these subtasks, and consequently the equivalence in their imposed cognitive loads on human subjects, as opposed to insensitivity of DE. In addition, we provided support for our claim from the comprehensive results on study and analysis of this N-Back WM task (Owen et al.,

Referring to Figure _{(38)} = 4.21, SD = 0.86], mental arithmetic [_{(38)} = 2.19, SD = 0.87] and Stroop [_{(38)} = −2.37, SD = 1.0], it was non-significant in case N-back WM tasks [_{(38)} = 1.41, SD = 0.18].

Grand averages of the mutli-scale entropy (MSE) values of the NIRS time series data of our participants. We use the pattern length

^{1}Let _{i} and _{i} be the mean and slope of the _{i}, _{i}].

^{2}Fano factor or coefficient of variation is a measure of dispersion of a probability distribution of a random process, _{X} representing the variance and mean of