Abstract
Model interpretation methods play a critical role in enhancing the applicability of time series neural networks in high-risk domains. However, existing model interpretation methods, primarily designed for static data like images, do not yield satisfactory results while dealing with time series data. Although some studies have explored the time dimension and evaluated feature importance at each time point through feature removal, they neglect the potential correlations among multiple features that impact the model’s predictive outcomes. To address this limitation, we introduce the concept of Shapley value into the process of time series model interpretation and propose the TFS (Temporal Feature Sampling) algorithm. This algorithm calculates the importance scores of feature subsets, which include the removed features, during the model interpretation process. Additionally, it models the distribution of features by sampling within the time series data. We conducted comparative experiments between TFS and several baseline methods on two synthetic datasets and one real-world dataset, and the experimental results confirmed the efficiency and performance of our algorithm.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Yuan, Y., Zhou, X., Pan, S., Zhu, Q., Song, Z., Guo, L.: A relation-specific attention network for joint entity and relation extraction. In: IJCAI. vol. 2020, pp. 4054–4060 (2020)
Ho, N.H., Yang, H.J., Kim, S.H., Lee, G.: Multimodal approach of speech emotion recognition using multi-level multi-head fusion attention-based recurrent neural network. IEEE Access 8, 61672–61686 (2020)
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., Elhadad, N.: Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1721–1730 (2015)
Zhang, Z., Xie, Y., Xing, F., McGough, M., Yang, L.: MDNet: a semantically and visually interpretable medical image diagnosis network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6428–6436 (2017)
Wang, D., Quek, C., Ng, G.S.: Bank failure prediction using an accurate and interpretable neural fuzzy inference system. AI Commun. 29(4), 477–495 (2016)
Kim, J., Rohrbach, A., Darrell, T., Canny, J., Akata, Z.: Textual explanations for self-driving vehicles. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 577–593. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01216-8_35
Linardatos, P., Papastefanopoulos, V., Kotsiantis, S.: Explainable AI: a review of machine learning interpretability methods. Entropy 23(1), 18 (2020)
Ismail, A.A., Gunady, M., Corrada Bravo, H., Feizi, S.: Benchmarking deep learning interpretability in time series predictions. Adv. Neural. Inf. Process. Syst. 33, 6441–6452 (2020)
Serrano, S., Smith, N.A.: Is attention interpretable? arXiv preprint arXiv:1906.03731 (2019)
Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_53
Tonekaboni, S., Joshi, S., Campbell, K., Duvenaud, D.K., Goldenberg, A.: What went wrong and when? Instance-wise feature importance for time-series black-box models. Adv. Neural. Inf. Process. Syst. 33, 799–809 (2020)
Petsiuk, V., Das, A., Saenko, K.: Rise: randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.07421 (2018)
Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: International Conference on Machine Learning, pp. 3319–3328. PMLR (2017)
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., Müller, K.R.: Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recogn. 65, 211–222 (2017)
Shrikumar, A., Greenside, P., Kundaje, A.: Learning important features through propagating activation differences. In: International Conference on Machine Learning, pp. 3145–3153. PMLR (2017)
Erion, G., Janizek, J.D., Sturmfels, P., Lundberg, S.M., Lee, S.I.: Learning explainable models using attribution priors (2019)
Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. In: Advances In Neural Information Processing Systems, vol. 30 (2017)
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE 10(7), e0130140 (2015)
Kindermans, P.J., et al.: Learning how to explain neural networks: Patternnet and pattern attribution. arXiv preprint arXiv:1705.05598 (2017)
Zhang, J., Bargal, S.A., Lin, Z., Brandt, J., Shen, X., Sclaroff, S.: Top-down neural attention by excitation backprop. Int. J. Comput. Vision 126(10), 1084–1102 (2018)
Ismail, A.A., Gunady, M., Pessoa, L., Corrada Bravo, H., Feizi, S.: Input-cell attention reduces vanishing saliency of recurrent neural networks. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Bento, J., Saleiro, P., Cruz, A.F., Figueiredo, M.A., Bizarro, P.: Timeshap: explaining recurrent models through sequence perturbations. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 2565–2573 (2021)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Choi, E., Bahadori, M.T., Sun, J., Kulas, J., Schuetz, A., Stewart, W.: Retain: an interpretable predictive model for healthcare using reverse time attention mechanism. In: Advances In Neural Information Processing Systems, vol. 29 (2016)
Song, H., Rajan, D., Thiagarajan, J., Spanias, A.: Attend and diagnose: clinical time series analysis using attention models. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)
Guo, T., Lin, T., Lu, Y.: An interpretable LSTM neural network for autoregressive exogenous model. arXiv preprint arXiv:1804.05251 (2018)
Suresh, H., Hunt, N., Johnson, A., Celi, L.A., Szolovits, P., Ghassemi, M.: Clinical intervention prediction and understanding using deep networks. arXiv preprint arXiv:1705.08498 (2017)
Crabbé, J., Van Der Schaar, M.: Explaining time series predictions with dynamic masks. In: International Conference on Machine Learning, pp. 2166–2177. PMLR (2021)
Covert, I.C., Lundberg, S., Lee, S.I.: Explaining by removing: a unified framework for model explanation. J. Mach. Learn. Res. 22(1), 9477–9566 (2021)
Shapley, L.S., et al.: A value for n-person games (1953)
Štrumbelj, E., Kononenko, I.: Explaining prediction models and individual predictions with feature contributions. Knowl. Inf. Syst. 41, 647–665 (2014)
Schlegel, U., Arnout, H., El-Assady, M., Oelke, D., Keim, D.A.: Towards a rigorous evaluation of XAI methods on time series. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 4197–4201. IEEE (2019)
Johnson, A.E., et al.: MIMIC-III, a freely accessible critical care database. Sci. Data 3(1), 1–9 (2016)
Kokhlikyan, N., et al.: Captum: a unified and generic model interpretability library for PyTorch. arXiv preprint arXiv:2009.07896 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Liu, Z., Li, X., Cui, Y. (2024). Time Series Model Interpretation via Temporal Feature Sampling. In: Song, X., Feng, R., Chen, Y., Li, J., Min, G. (eds) Web and Big Data. APWeb-WAIM 2023. Lecture Notes in Computer Science, vol 14331. Springer, Singapore. https://doi.org/10.1007/978-981-97-2303-4_23
Download citation
DOI: https://doi.org/10.1007/978-981-97-2303-4_23
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-2302-7
Online ISBN: 978-981-97-2303-4
eBook Packages: Computer ScienceComputer Science (R0)