Learning Entropy: On Shannon vs. Machine-Learning-Based Information in Time Series

The paper discusses the Learning-based information (\({\varvec{L}}\)) and Learning Entropy (\({\varvec{L}}{\varvec{E}}\)) in contrast to classical Shannon probabilistic Information (\({\varvec{I}}\)) and probabilistic entropy (\({\varvec{H}}\)). It is shown that \({\varvec{L}}\) corresponds to the recently introduced Approximate Individual Sample-point Learning Entropy (\({\varvec{A}}{\varvec{I}}{\varvec{S}}{\varvec{L}}{\varvec{E}}\)). For data series, then, the LE should be defined as the mean value of L that is finally in proper accordance with Shannon's concept of entropy \({\varvec{H}}\). The distinction of \({\varvec{L}}\) against \({\varvec{I}}\) is explained by the real-time anomaly detection of individual time series data points (states). First, the principal distinction of the information concept of \({\varvec{I}}\boldsymbol{ }{\varvec{v}}{\varvec{s}}.\boldsymbol{ }{\varvec{L}}\) is demonstrated in respect to data governing law that \({\varvec{L}}\) considers explicitly (while \({\varvec{I}}\) does not). Second, it is shown that \({\varvec{L}}\) has the potential to be applied on much shorter datasets than \({\varvec{I}}\) because of the learning system being pre-trained and being able to generalize from a smaller dataset. Then, floating window trajectories of the covariance matrix norm, the trajectory of approximate variance fractal dimension, and especially the windowed Shannon Entropy trajectory are compared to \({\varvec{L}}{\varvec{E}}\) on multichannel EEG featuring epileptic seizure. The results on real time series show that \({\varvec{L}}\), i.e., \({\varvec{A}}{\varvec{I}}{\varvec{S}}{\varvec{L}}{\varvec{E}}\), can be a useful counterpart to Shannon entropy allowing us also for more detailed search of anomaly onsets (change points).

\(AISLE\):

Approximate Individual Sample Learning Entropy

\(\alpha \):

Vector of re-sampling setups for estimation of \(VFD\)

\(\beta \):

Vector of detection sensitivities for \(LE\)

\(H\):

Shannon Entropy

\(k\):

Discrete index of time

\(LE\):

Learning Entropy (\(AISLE\))

\(n\):

Number of channels

\({n}_{{w}_{i}}\):

Length of vector \({{\varvec{w}}}_{i}\)

\(r\):

Order of \(LE\)

\(\sigma \left(.\right)\):

Standard deviation

\(VFD\):

Variance Fractal Dimension

\({\mathbf{w}}_{i}\):

Vector of all adaptive parameters (neural weights) of \({i}^{th}\) channel

\(x\), x, X:

Scalar, vector, matrix (multidim. array)

\({y}_{i}\left(k\right)\):

Measured data sample of \({i}^{th}\) channel at time \(k\)

\({\tilde{y }}_{i}\left(k\right)\):

Filter output (neural predictor output) of \({i}^{th}\) channel at time \(k\)

The research reported in this paper has been funded by the European Interreg Austria-Czech Republic project “PredMAIn (ATCZ279)”.

Anonymous real EEG dataset courtesy of Department of Neurology, Faculty of Medicine in Hradec Kralove, Charles University, Czech Republic.

Authors and Affiliations

1. Department of Computer Science, Faculty of Science, University of South Bohemia in České Budějovice, České Budějovice, Czech Republic

   Ivo Bukovsky & Ondrej Budik

Corresponding author

Correspondence to Ivo Bukovsky .

Editors and Affiliations

1. Johannes Kepler University of Linz, Linz, Oberösterreich, Austria

   Gabriele Kotsis

2. Technical University of Vienna, Vienna, Austria

   A Min Tjoa

3. Johannes Kepler University of Linz, Linz, Austria

   Ismail Khalil

4. Software Competence Center Hagenberg, Hagenberg, Austria

   Bernhard Moser

5. (WU) Vienna University of Economics and Business, Vienna, Austria

   Alfred Taudes

6. Johannes Kepler University of Linz, Linz, Austria

   Atif Mashkoor

7. Johannes Kepler University of Linz, Linz, Austria

   Johannes Sametinger

8. Software Competence Center Hagenberg, Hagenberg, Austria

   Jorge Martinez-Gil

9. Software Competence Center Hagenberg, Hagenberg, Austria

   Florian Sobieczky

10. Software Competence Center Hagenberg, Hagenberg, Austria

    Lukas Fischer

11. Software Competence Center Hagenberg, Hagenberg, Austria

    Rudolf Ramler

12. Pak-Austria Fachhochschule - Institute of Applied Sciences and Technology (PAF-IAST), Haripur, Pakistan

    Maqbool Khan

13. Software Competence Center Hagenberg, Hagenberg, Austria

    Gerald Czech

Appendix

(See Figs. 5, 6 and 7)

The comparison of AISLE with the other proposed methods on artificial multichannel data (15) with noisy intervals at k = 400…500 and k = 700…800; covDEMT = \(\left\| {{\mathbf{covX}}^{{\Delta }} } \right\|\) does not detect anything (due to various frequencies and phases of sinusoidal channels), VFDT and H gradually detects the changes in data, and AISLE instantly detects the increased learning effort when dynamical behavior is presented at k > 400 and k > 700.

Full size image

The data Y are EEG channels with known epileptic seizure starting at about t = 44 s, its detection with the discussed methods is in further figures.

Full size image

The evaluation of multichannel EEG data (Fig. 2), where covDEMT = does not indicate the seizure onset at t = 44s, VFDT gradually increases, and both H and AISLE indicate the seizure onset rather instantly. The detail of how early H vs. AISLE detect the seizure is shown in Fig. 4.

Full size image

(Detail of Fig. 3) Both methods, i.e., the DEM-based Shannon Entropy trajectory H as well as the Approximate Individual Sample Learning Entropy (AISLE) were found practically feasible for early seizure indication while the AISLE has potential of earlier detection (bottom axis) because it is in principle more sensitive to individual samples of data.

Full size image

Reprints and permissions

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

Policies and ethics