^{1}

^{2}

^{1}

^{2}

This article was submitted to Systems Interactions and Organ Networks, a section of the journal Frontiers in Network Physiology

This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

An information theoretic reduction of auto-regressive modeling called the Reduced Auto-Regressive (RAR) modeling is applied to several multivariate time series as a method to detect the relationships among the components in the time series. The results are compared with the results of the transfer entropy, one of the common techniques for detecting causal relationships. These common techniques are pairwise by definition and could be inappropriate in detecting the relationships in highly complicated dynamical systems. When the relationships between the dynamics of the components are linear and the time scales in the fluctuations of each component are in the same order of magnitude, the results of the RAR model and the transfer entropy are consistent. When the time series contain components that have large differences in the amplitude and the time scales of fluctuation, however, the transfer entropy fails to detect the correct relationships between the components, while the results of the RAR modeling are still correct. For a highly complicated dynamics such as human brain activity observed by electroencephalography measurements, the results of the transfer entropy are drastically different from those of the RAR modeling.

To understand the dynamical properties of any complicated systems including those in physiology, we have to analyze a set of signals generated by the system under consideration, varying in time and interrelated with each other, which is referred to as multivariate time series. Though it is surely important to understand the time dependence of each component of the time series separately, it is also crucial to detect the directed relationships among the components, in which the structure and functionality of the system are partially embodied. In many cases including those in physiology, however, the system is so complicated that we have no theoretical argument to identify the relationships from the first principle and we have to detect them only from observed data.

There are several common techniques for such detection. Among them, the Granger causality is probably the most classical and well-known [

An important feature of these techniques is that they are pairwise measures. In other words, these measures are calculated by taking all pairwise combinations out of a set of the components contained in the time series. It is not obvious, however, whether the relationships among components more than three can always be broken into pairwise relationships. For instance, let us consider the case in which two pairs of components, (A, B) and (A, C), are directly related within each pair. Despite that there is no direct relationship between B and C, the pairwise measure would detect a non-zero value of indirect relationship

When the number of components are large

In this article, we investigate multivariate time series of a moderate number of components up to 10 and show that pairwise measures such as transfer entropy might fail in detecting relationships among components even for time series of this relatively small number of components. As a technique that enables us to extract relationships from an entire set of components without pairwise break-up and threshold, we take the Reduced Auto-Regressive (RAR) modeling firstly proposed by

This article is organized as follows. In

We consider a set of multivariate time series,

The time series modeling for multivariate time series, _{
i
}, for each component and an appropriate value of the maximum lag, _{
i,0} is the constant term in the modeling of the _{
i
}(_{
j
}(_{
i,j,k
}), at previous time with lag _{
i,j,k
} and parameter _{
i,j,k
}. The subscripts of the lags and the parameters,

Here we take another model, which is an information theoretic reduction of linear models and referred to as the Reduced Auto-Regressive (RAR) model [

To be concrete, let us assume that we have a set of observed values of four-component multivariate time series,

Here, we represent the value of the model for the _{
i
}(_{
j
}(_{
i,j,k
}) included in the model are selected from a “pool of terms”, which is called a “dictionary”. For example, if we take the maximum lag as _{
i0}, _{
i,j,k
} corresponding to the extracted terms by minimizing a suitably chosen information criterion.

Information criteria have a general form,

The mean square prediction error is the average of the squared norm of the prediction error vector, _{
i
}(_{
i,0} and _{
i,j,l
}, are chosen to minimize the sum of squares of the prediction errors, ^{
T
} ⋅_{
i
} (_{
i
}, see

To extract the optimal subset to minimize

A typical result of RAR modeling takes the form_{0}(_{0}(_{1}(_{3}(_{0} is affected by _{1} and _{3} apart from _{0} itself. It should also be emphasized that there are strong information theoretic arguments to support that the RAR model can detect any periodicities built into given time series [

Transfer entropy is an information theoretic measure for quantifying the information flow between two univariate time series, which we denote here as _{
i
}) is the probability for _{
i
}. The extra information gain of the state of _{
i
} by obtaining the state of _{
i
} under the condition of _{
i
}, which is

Noticing that

Here we slightly extend the definition by Schreiber to include the time difference

The transfer entropy is non-negative and becomes zero when

In this section, we apply the RAR modeling technique to two artificial systems, both of which are represented by linear combinations of the terms of three components with various distinctive lags to investigate the directional relationship among the components and compare the results to the ones obtained from the calculated values of transfer entropy. The difference between these two time series is the time scales of fluctuations of each component. While the time scales of fluctuations of all components in the first system (System 1) are similar, the time scales in the second system (System 2) differ from each other.

The time series of System 1 are generated by the following linear equations:_{1} are generated by other components, _{0} and _{2} and not related to the previous values of _{1} itself. In

Three-component time series data generated by

We generate 10000 data points for each component of System 1 after sufficient number of iterations to erase initial value dependence to build the RAR model. In the modeling, we set the maximum time delay

The notation for the values of the parameters such as 0.41(2) represents that the mean value of the parameter of _{0}(

The values of transfer entropy of each component from other components for time delay (lag) up to 20. We plot the values for

Let us examine the results of for _{0}, the large values of transfer entropy come from component _{1} at lag 4 and component _{3} at lag 7. Compared to the generator of _{0} defined by _{1}(_{3}(_{0}. For component _{1}, peaks appear at lag 2 for component _{0} and at lag 9 for component _{2}, which are also consistent with the terms _{0}(_{2}(_{1}, _{2}, the large value of transfer entropy at lag 2 for component _{0} is consistent with the term _{0}(_{0}, which does not have any corresponding term in _{1} are almost zero, which is reasonable, since component _{2} is independent of _{1}. For the results of

The time series of System 2 are generated by the following linear equations:_{0} fluctuates slowly over about 50 iterations, component _{1} fluctuates rapidly in almost every iteration, and component _{2} fluctuates intermediately in time scale between those of _{0} and _{1}. It should also be noticed that component _{1}, which has the smallest amplitude and is independent of other components, affects components _{0} and _{2}. In this regard, System 2 has more complicated characteristics than System 1, even though the dynamics is represented by linear equations.

Three-component time series data generated by _{0} fluctuates slowly, component _{1} fluctuates rapidly, and component _{2} fluctuates intermediately.

As in the case of System 1, we generate 10,000 data points for each component of System 2 to build the RAR model, then we divide these 10,000 data points into 10 intervals each of which contains 1,000 data points and compare the corresponding results of RAR modeling. We set the maximum lag as

As in the case of System 1, all terms and parameters are correctly recovered within reasonable statistical errors for System 2 in spite of the differences in the amplitude and the time scale of fluctuation for each components.

_{0} shows no distinctive peaks, which is remarkably different from those of components _{1} and _{2}. Moreover, the values from component _{2} are always larger than those of component _{1}, though the generator of _{0} defined by _{2}. This deceptive result might be caused by the fact that the amplitudes of components _{0} and _{2} are in the same order. For _{1}, the values are very small around 0.0075 and the large values come from _{2} at lags 2 and 3, though there are no such terms in the generator of _{1}, _{1} is independent of other components, though for decisive conclusion for the independence we need to estimate the effect of dynamical and/or observational noise using a method like surrogate generation based approach. For component _{2}, the large values of transfer entropy come from _{1} at lags 3 and 4 that might corresponds to the term _{1}(

The values of transfer entropy of each component from other components for various values of time delay (lag) up to 20. We plot the values for

In this section, we apply the RAR modeling to electroencephalography (EEG) data and compare the results to the values of transfer entropy. The EEG signals used here were recorded from a healthy human adult during resting state with eyes closed in an electrically shielded room and have been analyzed by other methods in

The placement of 10 electrodes in International 10–20 System for electroencephalography measurements. The top (bottom) is the front (back) direction of the head. The digits over the circles representing electrodes are the component numbers used in the RAR modeling.

The 10 channel electroencephalography signals analyzed here are plotted in

The plots of the 10 channel electroencephalography signals analyzed in the present section. All plotted data are normalized and dimensionless.

From this model, component Fz is influenced by component Cz at lag 1 and 2 and component F3 at lag 5, 13, and 20. In

For the transfer entropy, we use the same data points as those used in RAR model building and calculate the information gain from the correlation in the signals between all pairs of component Fz and each of other channels up to the maximum lag 30. According to the analysis described in

The values of transfer entropy of component Fz from other components with respect to the lags up to 30. All values are in the same order of magnitude and do not show distinct peaks.

Plot of the components and the maximum values of transfer entropy of component Fz for each lag up to 30 sorted in the descending order of the values of transfer entropy. The red bars are the top five values of transfer entropy. For component Fz, all top five values come from component C3.

We summarize these results for component Fz in

Pictorial summary of the results of the RAR model and the transfer entropy for component Fz. The target component Fz is represented by the red circle. The circles from which the arrows emanate, which are Cz and F3, represent the components contained in the RAR model with the width of the arrows being proportional to the number of appearance of the component in the RAR model. For the case of component Fz, component Cz appears two times and component F3 appears three times. The orange circles represent the components that give the top 5 values of transfer entropy for each lag, which is only C3.

As for the other nine components, we show only the summarized results in

Summarized results for other components. Red circles represent the target components against which the RAR models are built. The arrows are the directed relationships indicated by the corresponding RAR models. The orange circles are the components that gives large values of transfer entropy to the target nodes. See the caption of

For two artificial linear systems described in

For the application to EEG data in

In summary, we have applied the RAR modeling technique to several multivariate time series as a method to detect the relationships among the components in the time series and compared the results with those of a pairwise measure, transfer entropy in this article. When the relationships between the dynamics of the components are linear and the time scales in the fluctuation of each component are in the same order of magnitude, the results of the RAR model and the transfer entropy are consistent. When the time series contain components that have large differences in the amplitude and the time scales of fluctuation, however, the transfer entropy fails to capture the correct relationships between the components, while the results of the RAR modeling are still correct. For a highly complicated dynamics such as human brain activity observed by electroencephalography measurements, the results of the transfer entropy are drastically different from those of the RAR modeling.

The original contributions presented in the study are included in the article/Supplementary Material; further inquiries can be directed to the corresponding author.

TT conceived the idea of numerical calculations presented in this article and carried out the computation. TT and TN verified and discussed the results. TT took the lead in writing the manuscript with the critical feedback from TN.

TT and TN would like to acknowledge Paul E. Rapp of Uniformed Services University for providing us with the EEG data used in

TT is employed by Toyota Motor Corporation.

The remaining author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

In this Appendix, we show the behaviors of simulated EEG signals generated by the RAR models and their power spectral densities in the frequency domain. Here the RAR models are constructed from the first 769 observations of each EEG channels to compare the simulated signals to the last 256 observations.

Comparative plots of the observed EEG signals and the signals generated by the RAR models, which are constructed from the first 769 observations of each channel. The observed EEG signals are the last 281 (= 25 + 256) signals of each channel and the RAR signals are generated by the corresponding RAR models with 25 observed signals prior to the last 256 signals as initial values. Simulated signals also contain Gaussian random numbers with mean 0 and standard deviations determined from the fitting errors of each channels in RAR modeling as dynamic noise.

Plots of the power spectral densities of simulated signals from the RAR models. Plotted values are the averages of the power spectral densities over 100 independent runs of simulation. Significant contributions come from frequencies up to about 20 Hz.