^{1}

^{2}

^{3}

^{4}

^{5}

^{*}

^{1}

^{2}

^{3}

^{4}

^{5}

Edited by: Konstantinos Michmizos, Rutgers, The State University of New Jersey, United States

Reviewed by: Jun Ma, Lanzhou University of Technology, China; Dimitrios Mylonas, Harvard Medical School, United States

This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

Systems of coupled dynamical units (e.g., oscillators or neurons) are known to exhibit complex, emergent behaviors that may be simplified through coarse-graining: a process in which one discovers coarse variables and derives equations for their evolution. Such coarse-graining procedures often require extensive experience and/or a deep understanding of the system dynamics. In this paper we present a systematic, data-driven approach to discovering “bespoke” coarse variables based on manifold learning algorithms. We illustrate this methodology with the classic Kuramoto phase oscillator model, and demonstrate how our manifold learning technique can successfully identify a coarse variable that is one-to-one with the established Kuramoto order parameter. We then introduce an extension of our coarse-graining methodology which enables us to learn evolution equations for the discovered coarse variables via an artificial neural network architecture templated on numerical time integrators (initial value solvers). This approach allows us to learn accurate approximations of time derivatives of state variables from sparse flow data, and hence discover useful approximate differential equation descriptions of their dynamic behavior. We demonstrate this capability by learning ODEs that agree with the known analytical expression for the Kuramoto order parameter dynamics at the continuum limit. We then show how this approach can also be used to learn the dynamics of coarse variables discovered through our manifold learning methodology. In both of these examples, we compare the results of our neural network based method to typical finite differences complemented with geometric harmonics. Finally, we present a series of computational examples illustrating how a variation of our manifold learning methodology can be used to discover sets of “effective” parameters, reduced parameter combinations, for multi-parameter models with complex coupling. We conclude with a discussion of possible extensions of this approach, including the possibility of obtaining data-driven effective partial differential equations for coarse-grained neuronal network behavior, as illustrated by the synchronization dynamics of Hodgkin–Huxley type neurons with a Chung-Lu network. Thus, we build an integrated suite of tools for obtaining data-driven coarse variables, data-driven effective parameters, and data-driven coarse-grained equations from detailed observations of networks of oscillators.

We study coupled systems comprised of many individual units that are able to interact to produce new, often complex, emergent types of dynamical behavior. The units themselves (motivated by the modeling of large, complex neuronal networks) may be simple phase oscillators, or may be much more sophisticated, with heterogeneities and parameter dependence contributing to the emerging behavior complexity. Each unit is typically described by a system of ordinary differential equations (ODEs) (1), where

Examples of such systems from across the scientific domains range from coupled reactor networks (Mankin and Hudson,

Here, we present and illustrate an alternative, automated, data-driven approach to finding emergent coarse variables and modeling their dynamical behavior. As part of our methodology we make use of manifold learning techniques, specifically diffusion maps (DMAPs) (Coifman and Lafon,

We begin our presentation with a brief outline of our key manifold learning tool, the diffusion maps technique, followed by an abridged introduction to the associated geometric harmonics function extension method (Coifman and Lafon,

Motivated by the intended applications of our methodology to neurological systems, we choose to illustrate our techniques with one of the simplest neurobiologically salient models available, the classic Kuramoto coupled phase oscillator model (Kuramoto,

We make use of the classic Kuramoto coupled phase oscillator model and its variations throughout this paper as prototypical examples to showcase our methodology. In our concluding discussion and future work section we also mention the possibility of discovering effective emergent

Introduced by Coifman and Lafon (^{n} based on data, with

At the heart of the Laplace operator construction lies the kernel, _{ij} is the kernel evaluated on data points

where ϵ is a tunable kernel bandwidth parameter and ∥·∥_{2} is the Euclidean norm. By selecting different kernel bandwidth parameters it is possible to examine features of the data geometry at different length scales. Several heuristics are available to select the value of the kernel bandwidth parameter (Dsilva et al., _{i} as a starting point for our ϵ tuning.

After defining a kernel, the Laplace operator can be approximated as follows. First, one forms the diagonal matrix

A challenge of using the eigenfunctions of a Laplace operator to parameterize the manifold on which the data lie is the existence of “repeated” coordinates, that is, coordinates which parameterize an already discovered direction along the manifold, referred to as

Before the local linear regression method can be applied, the eigenfunctions of the Laplace operator must be sorted by their corresponding eigenvalues, all of which are real, from greatest to smallest. The main idea behind the local linear regression method is that the repeated eigenfunctions, the harmonics, can be represented as functions of the fundamental eigenfunctions, which correspond to unique directions. The local linear regression method checks for this functional relationship by computing a local linear fit (3) of a given eigenfunction

The coefficients of the fit, α_{k}(

The weighting kernel

where ϵ_{reg} is a tunable scale for the kernel of the regression. Choosing ϵ_{reg} to be one third of the median of the pairwise distances between the eigenfunctions Φ_{k−1}(

The quality of the fit is assessed by the normalized leave-one-out cross-validation error (6) or the residual _{k} of the fit. Values of the residual near zero indicate an accurate approximation of ϕ_{k} from Φ_{k−1}, indicative of a harmonic eigenfunction, while values near one correspond to a poor fit, and are therefore suggestive of a significant, informative eigenfunction corresponding to a new direction in the data. Thus, by computing residual values for a collection of the computed eigenfunctions, it is possible to identify the fundamentals and the harmonics in a systematic way,

Throughout this paper we employ the DMAPs algorithm to identify and parameterize manifolds as well as the local linear regression method to identify the significant, fundamental (non-harmonic) eigenfunctions. In all cases, we use α = 1 to remove the influence of the sampling density of the data, and the Gaussian kernel with the Euclidean distance defined in (2) as our similarity measure. For the local linear regression, we utilize a Gaussian kernel (5) and select ϵ_{reg} as one third of the median of the pairwise distances between the Φ_{k−1} as our regression kernel scale, in accordance with Dsilva et al. (

Consider a set

The first step of computing the geometric harmonics extension is to define a kernel

Consider ^{m × n} with entries _{ij} = _{i}, _{j}), where _{i} is an unknown point, _{j} is a sampled point with known value of _{j}), and

where ψ^{(l)} is the ^{n × d}, the first

Any observable of the data (e.g., the phase or the amplitude or any state variable for each particular oscillator in our network) is a function on the low-dimensional manifold, and can therefore be extended out-of-sample through geometric harmonics. This provides us a systematic approach for translating back-and-forth between physical observations and coarse-grained “effective” or “latent space” descriptions of the emergent dynamics.

Here we illustrate how manifold learning techniques, in this case diffusion maps, can be used to identify coarse variables for coupled oscillator systems directly from time series data. We consider the simple Kuramoto model with sinusoidal coupling as a test case, to demonstrate how we can learn a variable that is equivalent to the typical Kuramoto order parameter,

The Kuramoto model is a classical example of limit cycle oscillators that can exhibit synchronizing behavior. The basic version of the model consists of a number of heterogeneous oscillators (_{i}, selected from a distribution _{i} − θ_{j}).

The strength and presence of coupling among the oscillators is expressed by the coupling matrix _{ij} (

Originally, Kuramoto considered a mean-field coupling approximation (Strogatz, _{ij} =

The degree of phase synchronization of the oscillators can be expressed in terms of the complex-valued order parameter (a coarse variable) introduced by Kuramoto as follows (Kuramoto,

where 0 ≤

Time series of

In general, if the coupling is chosen such that the system exhibits complete synchronization (characterized by the lack of “rogue” or unbound oscillators), then a steady state exists in a rotating reference frame. Typically, the rotating frame transformation is taken to be θ_{i}(_{i}(

Throughout the remainder of this paper we employ the simple all-to-all coupled Kuramoto model and its variations, some with more complicated couplings, to illustrate our methodology. We emphasize the synchronizing behavior of the Kuramoto model as it facilitates coarse-graining, and repeatedly make use of this property to simplify our calculations throughout the rest of the paper.

For this section we consider the simple Kuramoto model with all-to-all mean-field coupling (10). Our goal is to: (a) first identify a coarse variable from time series of phase data; and then (b) demonstrate that this discovered variable is equivalent to the established Kuramoto order parameter. Throughout, we consider a rotating frame in which only the magnitude of the order parameter

We begin our analysis by simulating 8,000 Kuramoto oscillators (

In order to reduce the finite sample noise in ^{−4}, to generate our frequencies in a systematic and symmetric manner. We select ϵ to place a cap on the maximum absolute value of the frequencies in order to facilitate numerical simulation. For these parameter values, _{∞} ≈ 0.71. Thus, in order to sample the entire range of potential ^{−7} and 10^{−4}, respectively. After discarding the initial transients, we transform the phase data into a rotating frame and then sample it at discrete, equidistant time steps to form our time series data (

Plot of the magnitude

Before applying the DMAPs algorithm to these time series of phase data to identify a coarse variable, we need to select a suitable kernel. It is crucial to select a kernel that will compare the relevant features of the data in order to produce a meaningful measure of the similarity between data points. In this case, it is important to choose a kernel that measures the degree of clustering or equivalently the phase synchronization of the oscillators

Since the synchronization of the oscillators is related to how the oscillator phases group together, it is a natural choice to consider the phase density as a meaningful observable. The oscillator phase density captures phase clustering while being permutation independent, and should thus provide a meaningful similarity measure between system snapshots. Therefore, we first pre-process the phase data by computing the density of the oscillators over the interval [−π, π] before defining the kernel. We approximate this density with a binning process (histogram) that uses 200 equally spaced bins over the interval [−π, π]. As part of the binning process we are careful to reduce the phases of the oscillators mod 2π to ensure that all of them are captured in the [−π, π] interval (

Normalized oscillator densities of the Kuramoto oscillator phases sampled at equal time intervals from the two simulation runs. The densities are colored by whether they originated from the simulation run that began above/below the steady state (blue/orange)

Using time series of density data instead of individual phase data simplifies the comparison of different snapshots, and thus facilitates the selection of the kernel in the DMAPs algorithm. For this we select the standard Gaussian kernel with the Euclidean metric for the comparison of our oscillator density vectors (^{200}) with diffusion map tuning parameters of α = 1 selecting the Laplace-Beltrami operator, and ϵ ≈ 1.54 as the kernel bandwidth parameter. This results in a single diffusion map coordinate (eigenfunction) that is deemed significant by the local linear regression method, see

_{1} suffices to parameterize the behavior. _{1} and the magnitude of the typical Kuramoto order parameter

Comparing this diffusion map coordinate ϕ_{1} to the Kuramoto order parameter

While this process is both systematic and automated, we must point out that the key step in this process, the choice of the relevant features of the data, still requires a modicum of insight. Different choices of the relevant features of the data, either through pre-processing and/or selection of the kernel, will produce different coarse variables, that may well be “reconcilable,” i.e., one-to-one with each other and with

Once descriptive coarse variables have been identified, it is desirable to find descriptions of their behavior (evolution laws, typically in the form of ordinary or partial differential equations, ODEs or PDEs). However, finding analytical expressions for these descriptions (“laws”) is often extremely challenging, if not practically infeasible, and may rely on

where

In the following sections, we present an alternative, data-driven approach to learning the behavior of coarse variables directly from time series of observational data. As part of this approach we make use of a recurrent neural network architecture “templated” on numerical time integration schemes, which allows us to learn the time derivatives of state variables from flow data in a general and systematic way. We illustrate this approach through an example in which we learn the aforementioned ODEs (14) that govern the behavior of the Kuramoto order parameter in the continuum limit from data. We then compare the result of our neural network based approach to standard finite differences complemented with geometric harmonics. Throughout this example we only observe the magnitude of the order parameter

Numerical integration algorithms for ODEs rely on knowledge of the time derivative in order to approximate a future state. If an analytical formula for the time derivative is not available or unknown, it can be approximated from time series of observations through, say, the use of finite differences, with known associated accuracy problems, especially when the data is scarce. Here, we discuss an alternative, neural network based approach to learning time derivatives (the “right hand sides” of ODEs) from discrete time observations.

Artificial neural networks have gained prominence for their expressiveness and generality, and especially for their ability to model non-linear behavior. These qualities have led to their widespread adoption and use in areas as diverse as image (Simonyan and Zisserman,

This feedfoward approach is a method for approximating the functional form of the right-hand-side of systems modeled through autonomous ODEs (Rico-Martinez et al.,

In addition to its application to learning the right-hand-sides of systems of ODEs, this type of neural network architecture can also be extended to learn the right hand sides of PDEs discretized as systems of ODEs through a method of lines approach (Gonzalez-Garcia et al.,

A schematic of the feedforward recurrent neural network architecture we construct templated on a fourth order, fixed-step Runge–Kutta algorithm is illustrated in _{1}, _{2}, _{3}, _{4}). That is, there is a single copy of the neural “sub”-network

The neural network approach for learning unknown ODE right-hand-sides templated on a fourth order, fixed-step Runge–Kutta numerical integration algorithm. The neural “sub”-network

In the following section, we use this Runge–Kutta scheme to learn the ODE governing the behavior of the magnitude of the Kuramoto order parameter

As discussed in the previous section, we need to generate pairs of flow data of the coarse variable, (_{i} drawn from a Cauchy distribution with γ = 0.5.

One of the difficulties of generating flow data for the order parameter is adequately sampling the entire range of

After generating the starting points of the flows

This entire process results in two collections of training data, each consisting of 2,000 pairs of the form (

Our neural network architecture is based on a standard fixed step-size, fourth order Runge–Kutta integration algorithm complemented with a feedforward neural network with 3 hidden layers of 24 neurons each, ^{−3} and a mean squared error loss on the final flow point

We train one neural network for each time horizon data set, Δ

The training and validation loss of the neural network training for:

As an added point of comparison between our neural network right-hand-sides and the analytical expression, we integrate flows for a variety of initial conditions with both of the neural network approximated right-hand-sides and the analytical (theoretical, infinite oscillator limit) right-hand-side. As

Plots of trajectories generated from the learned, neural network, evolution law right-hand-side (blue, purple) vs. the analytically known evolution law (orange). The learned trajectories exhibit nearly identical behavior to the analytical trajectories and, importantly, converge to the correct steady state.

One of the greatest utilities of this approach is its generality, which enables it to be applied to cases in which analytical approaches have not yet been devised, or may not be practical. Throughout the previous example we assumed knowledge of the coarse variable, the Kuramoto order parameter, and learned its governing ODEs. However, it is important to realize that this same technique can also be applied to our “discovered” coarse variables, that we identified earlier in this paper through manifold learning techniques. In this way, this methodology allows one to

Following the same methodology outlined in the “Order Parameter Identification” section, we combine the phase data corresponding to the two (_{1}.

The coarse variable identified by our manifold learning procedure applied to data generated in this section. Only the single coarse coordinate ϕ_{1} was deemed significant by the local linear regression method. Even in the presence of noisy data, there is nearly a one-to-one map between the discovered coordinate ϕ_{1} and the analytical order parameter

By keeping track of the ϕ_{1} values corresponding to the _{1}(_{1}(_{1}. Using an identical procedure to the one used to learn the right-hand-side of the _{1} in an architecture templated on a fixed step-size, fourth order Runge–Kutta algorithm. As before, we initialize this neural network integration procedure with a uniform Glorot procedure for the kernels, and zeros for the biases. We define the loss to be the mean squared error on the final point of the training flow and then train the network with full batch training with the Adam optimizer with a learning rate of 10^{−3}. _{1} time derivative for Δ

The learned evolution law (ODE right-hand-side) of our discovered order parameter ϕ_{1} for each reporting time horizon. The steady state of this system in these coordinates is highlighted in red. A forward Euler approximation (black) and its geometric harmonic interpolation is included to provide a point of comparison and to highlight the scatter in the data.

Our recurrent, integrator-based neural network architecture, appears to successfully learn the evolution equation for ϕ_{1} over the domain. However, the fit does not appear to be as accurate as the one found for _{1} is one-to-one, but does not preserve the shape of the sampling density due to its non-linearity, leading to a bias in the sampling of the ϕ_{1} domain. However, despite this bias, the neural network procedure manages to learn a time derivative near the visual average of its observed Euler approximation.

To summarize, we have shown how our order parameter identification method can discover a coarse variable that shows good agreement (i.e., is one-to-one) with a known analytical variable, even in the case of non-ideal, noisy data. Furthermore, we have demonstrated that even with biased sampling, our recurrent, integrator templated neural network architecture is capable of successfully learning the right-hand-side of the evolution of our discovered coarse variable, effectively smoothing its Euler finite difference estimates, and certainly closely approximating the correct steady state.

Up to this point we focused on coarse-graining (reducing the dimension) the system state variables required to formulate an effective dynamic coupled oscillator model. Often, however, complex systems whose dynamics depend explicitly on several parameters can also admit

Throughout this section we investigate variations of the Kuramoto model and show how DMAPs can be used to discover effective parameters that accurately describe the phases of Kuramoto oscillators when synchronized (at steady state in a rotating frame). The general idea is to use an

Since these effective coordinates describe the variability of the steady state variables (which depend on the detailed system parameters), the eigenfunctions themselves provide a new, data-driven set of effective parameters for the model. If fewer eigenfunctions are required to describe the steady state space than there are original parameters, then these eigenfunctions furnish an

We begin by considering a simple variation of the Kuramoto model in which we include the coupling constant _{i} as an oscillator heterogeneity in addition to the frequencies ω_{i}. The governing equations for this model are

Similar to the typical Kuramoto model, the phase synchronization of the oscillators can be described with the usual complex-valued order parameter

As with the typical Kuramoto model, this variation admits a steady state in a rotating frame if there is complete synchronization.

Our goal for this model is to show that even though there are two model heterogeneities, (ω_{i}, _{i}), the steady state phases can be described by a single “effective” parameter. That is, we will demonstrate that there is a new parameter that depends on both of the original two parameters, such that the steady state oscillator phases only depend on this single combination of the original parameters.

In order to find this effective parameter, we apply the DMAPs algorithm to the steady state phase data with an _{i, ∞} = θ_{i}(_{∞}) (where _{∞} is a time after which the oscillators have reached a steady state), and ignore the oscillator heterogeneities (ω_{i}, _{i}). We use the eigenfunctions provided by the DMAPs algorithm to define a change of variables for the parameter space (ω, _{1} is required to describe the phase data. Thus, this change of variables is a many-to-one map and provides a reduction from the two original parameters (ω, _{1}.

For our simulations we consider 1,500 oscillators (

Next, we apply the DMAPs algorithm to the steady state phases with a Gaussian kernel with the Euclidean distance and parameters of α = 1 for the Laplace-Beltrami operator and ϵ = 0.5 for the kernel bandwidth parameter. The kernel is given by

where ∥·∥_{2} is the Euclidean norm in the complex plane^{1}

We use the local linear regression method to verify that only a single diffusion map eigenfunction ϕ_{1} is required to represent this phase data, resulting in a single, data-driven effective parameter. A coloring of the original, two-dimensional parameter space (ω,

_{1}. _{∞}.

Due to the simplicity of this model it is also possible to find an effective parameter analytically. Multiplying the order parameter by

which under steady state conditions yields an effective parameter

As a validation of our data-driven approach, we compare our data-driven parameter ϕ_{1} to the analytical one (24). In _{1}, (b) the analytical parameter, and (c) the steady state phases. As the figure illustrates, the colorings are similar suggesting that our data-driven parameter is indeed an equivalent effective parameter for this model.

This is confirmed by the plot in _{1} in (ω,

We now consider a modification to the Kuramoto model in which we additionally incorporate a “firing term” with coefficient α_{i} to the coupling strength _{i}, and frequency ω_{i} of each oscillator. This model was originally introduced to model excitable behavior among coupled oscillators. If |ω_{i}/α_{i}| < 1 and _{i} = 0, each oscillator exhibits two steady states, one stable and one unstable. A small perturbation of the stationary, stable solution that exceeds the unstable steady state induces a firing of the oscillator, which appears as a large deviation in phase before a return to the stable state (Tessone et al.,

Similar to the typical Kuramoto model, one can express the degree of phase synchronization among the oscillators with the Kuramoto order parameter,

Transforming the model equations in the same way as those of the Kuramoto model with heterogeneous coupling coefficients (18) studied earlier results in

which under steady state conditions yields

Now considering a rotating reference frame, in which ψ is constant, we set ψ = 0 for convenience yielding

By the above manipulations, it is now clear that the steady state phases θ_{i,∞} are a function of _{i}/α_{i} and ω_{i}/α_{i}, meaning that this system can be analytically described by two combinations of the original parameters. We now employ our data-driven approach to discover the effective parameter(s).

We begin by simulating this model using Scipy's vode integrator with _{i} uniformly randomly sampled in [−2, 2], _{i} uniformly randomly sampled in [10, 100], and ω_{i} uniformly randomly sampled in [−π, π]. We select these parameter values to ensure complete synchronization of the oscillators and hence a steady state in a rotating frame. As with the Kuramoto model with heterogeneous coupling coefficients, we transform the phases into a rotating frame and apply the DMAPs algorithm with an output-only informed kernel to the complex transformed steady state phases. We select diffusion map parameters of α = 1, and ϵ = 0.9 and find that a single diffusion map coordinate ϕ_{1} is identified by the local linear regression method, giving rise to a single effective parameter.

This parameter is a non-linear combination of all three original parameters (ω, _{1}, which is itself a combination of the three original system parameters (ω, _{1} can be subsequently explored, if desired, through standard regression techniques.

The level surfaces of the significant eigenfunction ϕ_{1} of the output-only informed diffusion map kernel in the 3D parameter space of the Kuramoto model with firing. These level surfaces were found with the marching cubes algorithm (Lorensen and Cline,

Here, we showcase our data-driven parameter discovery process for systems with more complicated couplings. Instead of the all-to-all coupled model considered by Kuramoto (10), we consider a general Kuramoto model (9) with Chung-Lu type coupling between oscillators (Chung and Lu,

where _{i} is a sequence of weights defined by

for network parameters _{ij} defined by the network parameters through the sequence of weights _{i}.

One of the special properties of the Kuramoto system with a Chung-Lu coupling is that when a steady state exists in a rotating reference frame, the steady state phases of the oscillators θ_{i} lie on an invariant manifold (Bertalan et al., _{i}, and a network property called the degree κ_{i}, as depicted in

_{i, rotating}. There is a one-to-one mapping between the two, indicating that the steady state phases depend on the single diffusion map coordinate uniquely.

Throughout the remainder of this section we consider a collection of 4,000 oscillators (_{i} over the range [0, 1] that are coupled together in a Chung-Lu network with a coupling constant of

After our parameter selection, we integrate the oscillators in time with SciPy's Runge–Kutta integrator (

Now we show that, although there are two heterogeneity parameters, (ω_{i}, κ_{i}), the long term behavior of the Kuramoto oscillators with a Chung-Lu network is intrinsically one dimensional, and can be described by a single effective parameter, which itself is a, possibly non-linear, combination of the original system parameters.

In order to find this effective parameter, we begin by applying the DMAPs algorithm to the complex transformed steady state phase data with an output-only informed kernel. As mentioned before, this means that our observations consist solely of the steady state phases of the oscillators θ_{i,∞}, and ignore the oscillator heterogeneities (ω_{i}, κ_{i}). For our diffusion maps we use α = 1 for the Laplace-Beltrami operator, a kernel bandwidth parameter of ϵ ≈ 2.3*10^{−2}, and the Gaussian kernel with the Euclidean distance

where ∥·∥_{2} is the Euclidean norm in the complex plane of 4,000 oscillator phases, i.e.,

Next, we use the local linear regression method to determine that there is a single significant eigenfunction, ϕ_{1}. _{1}, as claimed.

The relationship between the original parameters and the DMAPs parameter is illustrated by _{1}. In this figure one can observe that there are multiple combinations of the original parameters that correspond to the same value of ϕ_{1} and hence the same steady state phase. The key observation is that the level sets of ϕ_{1} provide the mapping between the original parameters and the new effective parameter. These level sets are depicted in _{1} we can express the steady state phases in terms of a single combination of the original parameters. If necessary/useful, we can try to express this new effective parameter as a function of the original system parameters through standard regression techniques, or even possibly through neural networks (

The application of the diffusion map algorithm to the Chung-Lu coupled Kuramoto model with an output-only-informed kernel yields a single significant diffusion map coordinate, ϕ_{1}. _{1} in the original parameter space (ω, κ). These level sets were found by means of the marching squares algorithm (Maple, _{1}.

To summarize our approach, in each of three examples above we used DMAPs combined with the local linear regression method to determine the intrinsic dimensionality of the output space. The significant eigenfunctions that we obtained from this process provided new coordinates for the output space and can be considered as the “effective” parameters of the system. If there are fewer significant DMAPs eigenfunctions than original parameters, then this change of variables also provides a reduction in total necessary parameters.

Throughout this paper we have presented a data-driven methodology for discovering coarse variables, learning their dynamic evolution laws, and identifying sets of effective parameters. In each case, we used either an example or a series of examples to demonstrate the efficacy of our techniques compared to the established analytical technique and, in each case, the results of our data-driven approach were in close agreement with the established methodology.

Nevertheless, it is important to consider the interpretability (the “X” in XAI, explainable artificial intelligence) of the data-driven coarse variables discovered in our work. Even when performing model reduction with linear data-driven techniques, like Principal Component Analysis, it is difficult to ascribe a physical meaning to linear combinations of meaningful system variables (what does a linear combination of, say, a firing rate and an ion concentration “mean”?). The conundrum is resolved by looking for physically meaningful quantities that, on the data, are one-to-one with the discovered data-driven descriptors, and are therefore equally good at parametrizing the observations. One hypothesizes a set of meaningful descriptors and then checks that the Jacobian of the transformation from data-driven to meaningful descriptors never becomes singular

With that said, we believe that our techniques offer an approach that is both general and systematic, and we intend to apply it to a variety of coupled oscillator systems. One such problem that we are currently investigating is the possible existence, and data-driven identification, of partial differential equation (PDE) descriptions of coupled oscillator systems. Such an alternative coarse-grained description would confer the typical benefits associated with model reduction, such as accelerated simulation and analysis; it will also present unique challenges: for instance, the selection of appropriate boundary conditions for such a data-driven PDE model of coupled oscillators.

As we remarked in our discussion of the integrator-based neural network architecture, there is an extension of the ODE neural network approach that allows one to learn PDEs discretized by the method of lines approach (Gonzalez-Garcia et al.,

Utilizing the properties of the continuous form of the Kuramoto model, one can express the time- and phase- dependent oscillator density

As we pointed out earlier, Ott and Antonsen discovered an invariant attracting manifold for the simple Kuramoto model in the continuum limit with Cauchy distributed frequencies (Ott and Antonsen,

Away from this attracting invariant manifold, the full oscillator density ρ(θ, ω,

We would like to use our neural network based approach to learn equivalent PDEs directly from oscillator density data. As an example of what this would look like for

Furthermore, one can numerically obtain the partial derivative of _{θ} produces a loop, as illustrated in

where

_{1} and ϕ_{2}, colored by _{1}, ϕ_{2}, and time. The black dots are the actual oscillators which were used to produce the surfaces in ϕ_{1} and ϕ_{2} by means of a polynomial chaos expansion.

Moving away from simple phase oscillators, we consider a model of Hodgkin–Huxley neural oscillators studied in (Choi et al.,

for _{syn} defined as

with adjacency matrix _{ij}. The functions τ(_{∞}(_{Na}, _{Na}, _{l}, and _{l} are model parameters.

It has been shown that with a Chung-Lu network (30, 31) these oscillators are drawn to an attractive limit cycle along which their states can be described by two parameters: their applied current _{app} and their degree, κ. These two heterogeneities can also be described by two diffusion map coordinates, ϕ_{1} and ϕ_{2} (Kemeth et al.,

The datasets generated for this study are available on request to the corresponding author.

IK and CL planned the research. TT, MK, and TB performed the computations, and coordinated their design with IK and CL. All authors contributed to writing, editing and proofreading the manuscript.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The authors are pleased to acknowledge fruitful discussions with Drs. F. Dietrich and J. Bello-Rivas, as well as the use of a diffusion maps/geometric harmonics computational package authored by the latter.

^{1}It is necessary to map the oscillator phases to the complex plane to avoid the unfortunate occurrence of the phases lying across the branch cut of the multi-valued argument function. Taking the Euclidean norm of the difference between oscillator phases in this situation would result in the DMAPs algorithm incorrectly identifying two different clusters of oscillators split across the branch cut instead of a single group. By first mapping the phases to the complex plane and then computing the distances there, we avoid this possibility and correctly identify a single group of oscillators.