research-article

Human-in-the-loop Extraction of Interpretable Concepts in Deep Learning Models

Authors: Zhenge Zhao, Panpan Xu, Carlos Scheidegger, Liu RenAuthors Info & Claims

IEEE Transactions on Visualization and Computer Graphics, Volume 28, Issue 1

Pages 780 - 790

https://doi.org/10.1109/TVCG.2021.3114837

Published: 01 January 2022 Publication History

Abstract

The interpretation of deep neural networks (DNNs) has become a key topic as more and more people apply them to solve various problems and making critical decisions. Concept-based explanations have recently become a popular approach for post-hoc interpretation of DNNs. However, identifying human-understandable visual concepts that affect model decisions is a challenging task that is not easily addressed with automatic approaches. We present a novel human-in-the-Ioop approach to generate user-defined concepts for model interpretation and diagnostics. Central to our proposal is the use of active learning, where human knowledge and feedback are combined to train a concept extractor with very little human labeling effort. We integrate this process into an interactive system, ConceptExtract. Through two case studies, we show how our approach helps analyze model behavior and extract human-friendly concepts for different machine learning tasks and datasets and how to use these concepts to understand the predictions, compare model performance and make suggestions for model refinement. Quantitative experiments show that our active learning approach can accurately extract meaningful visual concepts. More importantly, by identifying visual concepts that negatively affect model performance, we develop the corresponding data augmentation strategy that consistently improves model performance.

References

[1]

J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, and B. Kim. Sanity checks for saliency maps. In Proceedings of the 32Nd International Conference on Neural Information Processing Systems, NIPS’18, pp. 9525–9536. Curran Associates Inc., USA, 2018.

[2]

J. Ba and R. Caruana. Do deep nets really need to be deep? In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, eds., Advances in Neural Information Processing Systems, vol. 27. Curran Associates, Inc., 2014.

[3]

S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and W. Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):1–46, 07 2015.

[4]

G. Bansal, B. Nushi, E. Kamar, D. S. Weld, W. S. Lasecki, and E. Horvitz. Updates in human-ai teams: Understanding and addressing the perfor-mance/compatibility tradeoff. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):2429–2437, Jul.2019.

Digital Library

[5]

M. Bojarski, A. Choromanska, K. Choromanski, B. Firner, L. J. Ackel, U. Muller, P. Yeres, and K. Zieba. Visualbackprop: Efficient visualization of cnns for autonomous driving. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 4701–4708, 2018.

Digital Library

[6]

M. Bostock, V. Ogievetsky, and J. Heer. D3 data-driven documents. IEEE Transactions on Visualization and Computer Graphics, 17(12):2301–2309, Dec.2011.

Digital Library

[7]

C. J. Cai, E. Reif, N. Hegde, J. D. Hipp, B. Kim, D. Smilkov, M. Wat-tenberg, F. B. Viégas, G. S. Corrado, M. C. Stumpe, and M. Terry. Human-centered tools for coping with imperfect algorithms during medical decision-making. CoRR, abs/1902.02960, 2019.

[8]

C. Chen, O. Li, A. Barnett, J. Su, and C. Rudin. This looks like that: deep learning for interpretable image recognition. ArXiv, abs/1806.10574, 2018.

[9]

L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:, 2017.

[10]

E. Choi, M. T. Bahadori, J. Sun, J. A. Kulas, A. Schuetz, and W. F. Stewart. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. In NIPS, 2016.

[11]

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255, 2009.

[12]

R. Fong and A. Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. CoRR, abs/1704.03296, 2017.

[13]

A. Ghorbani, J. Wexler, J. Y. Zou, and B. Kim. Towards automatic concept-based explanations. In Advances in Neural Information Processing Systems, pp. 9273–9282, 2019.

[14]

M. Grinberg. Flask web development: developing web applications with python. “O'Reilly Media, Inc.”, 2018.

[15]

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016.

[16]

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016.

[17]

M. Hind, D. Wei, M. Campbell, N. C. F. Codella, A. Dhurandhar, A. Mo-jsilović, K. Natesan Ramamurthy, and K. R. Varshney. Ted: Teaching ai to explain its decisions. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES’19, p. 123–129. Association for Computing Machinery, New York, NY, USA, 2019.

Digital Library

[18]

D. Ho, E. Liang, I. Stoica, P. Abbeel, and X. Chen. Population based augmentation: Efficient learning of augmentation policy schedules. CoRR, abs/1905.05393, 2019.

[19]

F. Hohman, M. Kahng, R. Pienta, and D. H. Chau. Visual analytics in deep learning: An interrogative survey for the next frontiers. IEEE Transactions on Visualization and Computer Graphics, 25(8):2674–2693, 2019.

Digital Library

[20]

F. Hohman, H. Park, C. Robinson, and D. H. Polo Chau. Summit: Scaling deep learning interpretability by visualizing activation and attribution summarizations. IEEE Transactions on Visualization and Computer Graphics, 26(1):1096–1106, 2020.

Digital Library

[21]

K. Holstein, J. Wortman Vaughan, H. Daumé, M. Dudik, and H. Wallach. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?, p. 1–16. New York, NY, USA: Association for Computing Machinery, 2019.

[22]

G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261–2269, 2017.

[23]

B. Kim, M. Wattenberg, J. Gilmer, C. J. Cai, J. Wexler, F. B. Viégas, and R. Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In ICML, 2017.

[24]

B. M. Lake, R. Salakhutdinov, J. Gross, and J. B. Tenenbaum. One shot learning of simple visual concepts. Cognitive Science, 33, 2011.

[25]

T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2999–3007, 2017.

[26]

Y. Liu, E. Jun, Q. Li, and J. Heer. Latent space cartography: Visual analysis of vector space embeddings. Computer Graphics Forum, 38:67–78, 062019.

[27]

J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431–3440, June2015.

[28]

S. Lundberg and S. Lee. A unified approach to interpreting model predictions. CoRR, abs/1705.07874, 2017.

[29]

Y. Ming, P. Xu, H. Qu, and L. Ren. Interpretable and steerable sequence learning via prototypes. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD'19, p. 903–913. Association for Computing Machinery, New York, NY, USA, 2019.

Digital Library

[30]

D. P. Papadopoulos, J. R. R. Uijlings, F. Keller, and V. Ferrari. Extreme clicking for efficient object annotation, 2017.

[31]

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems32, pp. 8024–8035. Curran Associates, Inc., 2019.

[32]

V. Petsiuk, A. Das, and K. Saenko. RISE: randomized input sampling for explanation of black-box models. CoRR, abs/1806.07421, 2018.

[33]

P. Pinggera, S. Ramos, S. Gehrig, U. Franke, C. Rother, and R. Mester. Lost and found: detecting small road hazards for self-driving vehicles. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1099–1106, 2016.

Digital Library

[34]

M. T. Ribeiro, S. Singh, and C. Guestrin. “why should i trust you?”: Explaining the predictions of any classifier. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD'16, pp. 1135–1144. ACM, New York, NY, USA, 2016.

Digital Library

[35]

C. Seifert, A. Aamir, A. Balagopalan, D. Jain, A. Sharma, S. Grottel, and S. Gumhold. Visualizations of Deep Neural Networks in Computer Vision: A Survey, pp. 123–144. Studies in Big Data. Germany: Springer, 2017.

[36]

O. Sener and S. Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018.

[37]

B. Settles. Active learning literature survey. University of Wisconsin, Madison, 52, 072010.

[38]

B. Settles and M. Craven. An analysis of active learning strategies for sequence labeling tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP'08, p. 1070–1079. Association for Computational Linguistics, USA, 2008.

Digital Library

[39]

K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, abs/1312.6034, 2013.

[40]

K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv, 092014.

[41]

D. Slack, S. Hilgard, E. Jia, S. Singh, and H. Lakkaraju. How can we fool lime and shap? adversarial attacks on post hoc explanation methods, 2019.

[42]

T. Spinner, J. Körner, J. Görtler, and O. Deussen. Towards an interpretable latent space: an intuitive comparison of autoencoders with variational autoencoders. In Proceedings of the Workshop on Visualization for AI Explainability 2018 (VISxAI), 2018.

[43]

J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. A. Riedmiller. Striving for simplicity: The all convolutional net. CoRR, abs/1412.6806, 2014.

[44]

C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9, 2015.

[45]

G. Torlai and R. G. Melko. Latent space purification via neural density operators. Phys. Rev. Lett., 120:240503, Jun2018.

[46]

L. van der Maaten and G. Hinton. Visualizing high-dimensional data using t-sne. Journal of Machine Learning Research, 9:2579–2605, 2008.

[47]

A. Vedaldi and S. Soatto. Quick shift and kernel methods for mode seeking. In Proceedings of the European Conference on Computer Vision, vol. 5305, pp. 705–718, 102008.

[48]

K. Wang, D. Zhang, Y. Li, R. Zhang, and L. Lin. Cost-effective active learning for deep image classification. IEEE Transactions on Circuits and Systems for Video Technology, 27:1–1, 012016.

Digital Library

[49]

J. Wu, C. Zhang, T. Xue, W. T. Freeman, and J. B. Tenenbaum. Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, pp. 82–90. Curran Associates Inc., USA, 2016.

[50]

D. Yoo and I. Kweon. Learning loss for active learning. 2019/EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 93–102, 2019.

[51]

M. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In Computer Vision, ECCV 2014-13th European Conference, Proceedings, vol. 8689 LNCS of Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), pp. 818–833. Springer Verlag, 2014.

[52]

Q. Zhang, Y. N. Wu, and S. Zhu. Interpretable convolutional neural networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8827–8836, June2018.

[53]

Q. Zhang, Y. Yang, H. Ma, and Y. N. Wu. Interpreting cnns via decision trees. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6254–6263, 2019.

[54]

R. Zhang, P. Isola, A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings - 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 586–595. IEEE Computer Society, Dec.2018.

[55]

H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6230–6239, 2017.

Cited By

Zhang XPiazentin Ono JHe WGou LSachan MMa KRen L(2024)Slicing, Chatting, and Refining: A Concept-Based Approach for Machine Learning Model Validation with ConceptSlicerProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645163(274-287)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645163
Quaye JParrish AInel ORastogi CKirk HKahng MVan Liemt EBartolo MTsang JWhite JClement NMosquera RCiro JJanapa Reddi VAroyo L(2024)Adversarial Nibbler: An Open Red-Teaming Method for Identifying Diverse Harms in Text-to-Image GenerationProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658913(388-406)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3658913
Li YWang JAboagye PYeh CZheng YWang LZhang WMa K(2024)Visual Analytics for Efficient Image Exploration and User-Guided Image CaptioningIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.338851430:6(2875-2887)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3388514
Show More Cited By

Recommendations

Learning with Hierarchical-Deep Models

We introduce HD (or “Hierarchical-Deep”) models, a new compositional learning architecture that integrates deep learning models with structured hierarchical Bayesian (HB) models. Specifically, we show how we can learn a hierarchical Dirichlet process (...
Aspect Sentiment Triplet Extraction Using Reinforcement Learning
CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge Management

Aspect Sentiment Triplet Extraction (ASTE) is the task of extracting triplets of aspect terms, their associated sentiments, and the opinion terms that provide evidence for the expressed sentiments. Previous approaches to ASTE usually simultaneously ...
Applying interpretable deep learning models to identify chronic cough patients using EHR data
Highlights
- Evaluated language-based data representation with different learning models to predict chronic cough patients using EHR data.
- The best approach can gain 0.952 and 0.930 sensitivity and specificity, respectively.
- Identify chronic ...
Abstract Background and Objective
Chronic cough (CC) affects approximately 10% of adults. Many disease states are associated with chronic cough, such as asthma, upper airway cough syndrome, bronchitis, and gastroesophageal reflux disease. The lack of an ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Visualization and Computer Graphics

IEEE Transactions on Visualization and Computer Graphics Volume 28, Issue 1

Jan. 2022

1190 pages

ISSN:1077-2626

Issue’s Table of Contents

1077-2626 © 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Educational Activities Department

United States

Publication History

Published: 01 January 2022

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang XPiazentin Ono JHe WGou LSachan MMa KRen L(2024)Slicing, Chatting, and Refining: A Concept-Based Approach for Machine Learning Model Validation with ConceptSlicerProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645163(274-287)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645163
Quaye JParrish AInel ORastogi CKirk HKahng MVan Liemt EBartolo MTsang JWhite JClement NMosquera RCiro JJanapa Reddi VAroyo L(2024)Adversarial Nibbler: An Open Red-Teaming Method for Identifying Diverse Harms in Text-to-Image GenerationProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658913(388-406)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3658913
Li YWang JAboagye PYeh CZheng YWang LZhang WMa K(2024)Visual Analytics for Efficient Image Exploration and User-Guided Image CaptioningIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.338851430:6(2875-2887)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3388514
Prasad Vvan Sloun RVilanova APezzotti N(2024)ProactiV: Studying Deep Learning Model Behavior Under Input TransformationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.330172230:8(5651-5665)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3301722
Prasad Vvan Sloun RElzen SVilanova APezzotti N(2024)The Transform-and-Perform Framework: Explainable Deep Learning Beyond ClassificationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.321924830:2(1502-1515)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1109/TVCG.2022.3219248
Mosqueira-Rey EHernández-Pereira EBobes-Bascarán JAlonso-Ríos DPérez-Sánchez AFernández-Leal ÁMoret-Bonillo VVidal-Ínsua YVázquez-Rivera F(2024)Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approachNeural Computing and Applications10.1007/s00521-023-09197-236:5(2597-2616)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1007/s00521-023-09197-2
Kerdvibulvech CLi Q(2024)Empowering Zero-Shot Object Detection: A Human-in-the-Loop Strategy for Unveiling Unseen Realms in Visual DataDigital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management10.1007/978-3-031-61066-0_14(235-244)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-61066-0_14
Ruangrotsakun AOh DNguyen TLee KSer MHiew ANgo RShureih ZKhanna RKahng M(2023)VIVA: Visual Exploration and Analysis of Videos with Interactive AnnotationCompanion Proceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581754.3584160(162-165)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581754.3584160
Ahn YLin YXu PDai Z(2023)ESCAPE: Countering Systematic Errors from Machine’s Blind Spots via Interactive Visual AnalysisProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581373(1-16)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581373
Wang QL'Yi SGehlenborg N(2023)DRAVA: Aligning Human Concepts with Machine Learning Latent Dimensions for the Visual Exploration of Small MultiplesProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581127(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581127

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents