Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization

We propose a technique for producing ‘visual explanations’ for decisions from a large class of Convolutional Neural Network (CNN)-based models, making them more transparent and explainable. Our approach—Gradient-weighted Class Activation Mapping (Grad-CAM), uses the gradients of any target concept (say ‘dog’ in a classification network or a sequence of words in captioning network) flowing into the final convolutional layer to produce a coarse localization map highlighting the important regions in the image for predicting the concept. Unlike previous approaches, Grad-CAM is applicable to a wide variety of CNN model-families: (1) CNNs with fully-connected layers (e.g.VGG), (2) CNNs used for structured outputs (e.g.captioning), (3) CNNs used in tasks with multi-modal inputs (e.g.visual question answering) or reinforcement learning, all without architectural changes or re-training. We combine Grad-CAM with existing fine-grained visualizations to create a high-resolution class-discriminative visualization, Guided Grad-CAM, and apply it to image classification, image captioning, and visual question answering (VQA) models, including ResNet-based architectures. In the context of image classification models, our visualizations (a) lend insights into failure modes of these models (showing that seemingly unreasonable predictions have reasonable explanations), (b) outperform previous methods on the ILSVRC-15 weakly-supervised localization task, (c) are robust to adversarial perturbations, (d) are more faithful to the underlying model, and (e) help achieve model generalization by identifying dataset bias. For image captioning and VQA, our visualizations show that even non-attention based models learn to localize discriminative regions of input image. We devise a way to identify important neurons through Grad-CAM and combine it with neuron names (Bau et al. in Computer vision and pattern recognition, 2017) to provide textual explanations for model decisions. Finally, we design and conduct human studies to measure if Grad-CAM explanations help users establish appropriate trust in predictions from deep networks and show that Grad-CAM helps untrained users successfully discern a ‘stronger’ deep network from a ‘weaker’ one even when both make identical predictions. Our code is available at https://github.com/ramprs/grad-cam/, along with a demo on CloudCV (Agrawal et al., in: Mobile cloud visual media computing, pp 265–290. Springer, 2015) (http://gradcam.cloudcv.org) and a video at http://youtu.be/COjUB9Izk6E.

1. Empirically we found global-average-pooling to work better than global-max-pooling as can be found in the “Appendix”.

2. We find that Grad-CAM maps become progressively worse as we move to earlier convolutional layers as they have smaller receptive fields and only focus on less semantic local features.

3. We use GoogLeNet finetuned on COCO, as provided by Zhang et al. (2016).

4. c-MWP (Zhang et al. 2016) highlights arbitrary regions for predicted but non-existent categories, unlike Grad-CAM maps which typically do not.

5. The green and red boxes are drawn manually to highlight correct and incorrect focus of the model.

6. Area of overlap between ground truth concept annotation and neuron activation over area of their union. More details of this metric can be found in Bau et al. (2017).

7. https://github.com/karpathy/neuraltalk2.

8. https://github.com/karpathy/neuraltalk2.

This work was funded in part by NSF CAREER awards to DB and DP, DARPA XAI Grant to DB and DP, ONR YIP awards to DP and DB, ONR Grant N00014-14-1-0679 to DB, a Sloan Fellowship to DP, ARO YIP awards to DB and DP, an Allen Distinguished Investigator award to DP from the Paul G. Allen Family Foundation, ICTAS Junior Faculty awards to DB and DP, Google Faculty Research Awards to DP and DB, Amazon Academic Research Awards to DP and DB, AWS in Education Research Grant to DB, and NVIDIA GPU donations to DB. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government, or any sponsor. Funding was provided by Virginia Polytechnic Institute and State University.

Authors and Affiliations

1. Georgia Institute of Technology, Atlanta, GA, USA

   Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh & Dhruv Batra

2. Facebook AI Research, Menlo Park, CA, USA

   Devi Parikh & Dhruv Batra

Corresponding author

Correspondence to Ramprasaath R. Selvaraju.

Communicated by Antonio Torralba.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

A Appendix Overview

In the “Appendix”, we provide:

1. I

   Ablation studies evaluating our design choices

2. II

   More qualitative examples for image classification, captioning and VQA

3. III

   More details of Pointing Game evaluation technique

4. IV

   Qualitative comparison to existing visualization techniques

5. V

   More qualitative examples of textual explanations

B Ablation Studies

We perform several ablation studies to explore and validate our design choices for computing Grad-CAM visualizations. This includes visualizing different layers in the network, understanding importance of ReLU in (2), analyzing different types of gradients (for ReLU backward pass), and different gradient pooling strategies.

1.1 1. Grad-CAM for Different Layers

We show Grad-CAM visualizations for the “tiger-cat” class at different convolutional layers in AlexNet and VGG-16. As expected, the results from Fig. 13 show that localization becomes progressively worse as we move to earlier convolutional layers. This is because later convolutional layers better capture high-level semantic information while retaining spatial information than earlier layers, that have smaller receptive fields and only focus on local features (Fig. 14).

Grad-CAM localizations for “tiger cat” category for different rectified convolutional layer feature maps for AlexNet

Full size image

1.2 2. Design Choices

We evaluate different design choices via top-1 localization errors on the ILSVRC-15 val set (Deng et al. 2009). See Table 3.

1.2.1 2.1. Importance of ReLU in (3)

Removing ReLU [(3)] increases error by 15.3%. Negative values in Grad-CAM indicate confusion between multiple occurring classes.

1.2.2 2.2. Global Average Pooling Versus Global Max Pooling

Instead of Global Average Pooling (GAP) the incoming gradients to the convolutional layer, we tried Global Max Pooling (GMP). We observe that using GMP lowers the localization ability of Grad-CAM. An example can be found in Fig. 15 below. This may be due to the fact that max is statistically less robust to noise compared to the averaged gradient.

Grad-CAM visualizations for “tiger cat” category with Global Average Pooling and Global Max Pooling

Full size image

1.2.3 2.3. Effect of Different ReLU on Grad-CAM

We experiment with Guided-ReLU (Springenberg et al. 2014) and Deconv-ReLU (Zeiler and Fergus 2014) as modifications to the backward pass of ReLU.

Guided-ReLU Springenberg et al. (2014) introduced Guided Backprop, where the backward pass of ReLU is modified to only pass positive gradients to regions of positive activations. Applying this change to the computation of Grad-CAM introduces a drop in the class-discriminative ability as can be seen in Fig. 16, but it marginally improves localization performance as can be seen in Table 3.

Deconv-ReLU In Deconvolution (Zeiler and Fergus 2014), Zeiler and Fergus introduced a modification to the backward pass of ReLU to only pass positive gradients. Applying this modification to the computation of Grad-CAM leads to worse results (Fig. 16). This indicates that negative gradients also carry important information for class-discriminativeness.

Grad-CAM visualizations for “tiger cat” category for different modifications to the ReLU backward pass. The best results are obtained when we use the actual gradients during the computation of Grad-CAM

Full size image

Visualizations for randomly sampled images from the COCO validation dataset. Predicted classes are mentioned at the top of each column

Full size image

C Qualitative Results for Vision and Language Tasks

In this section we provide more qualitative results for Grad-CAM and Guided Grad-CAM applied to the task of image classification, image captioning and VQA.

1.1 1. Image Classification

We use Grad-CAM and Guided Grad-CAM to visualize the regions of the image that provide support for a particular prediction. The results reported in Fig. 17 correspond to the VGG-16 (Simonyan and Zisserman 2015) network trained on ImageNet.

Figure 17 shows randomly sampled examples from COCO (Lin et al. 2014b) validation set. COCO images typically have multiple objects per image and Grad-CAM visualizations show precise localization to support the model’s prediction.

Guided Grad-CAM can even localize tiny objects. For example our approach correctly localizes the predicted class “torch” (Fig. 17a) inspite of its size and odd location in the image. Our method is also class-discriminative—it places attention only on the “toilet seat” even when a popular ImageNet category “dog” exists in the image (Fig. 17e).

We also visualized Grad-CAM, Guided Backpropagation (GB), Deconvolution (DC), GB \(+\) Grad-CAM (Guided Grad-CAM), DC \(+\) Grad-CAM (Deconvolution Grad-CAM) for images from the ILSVRC13 detection val set that have at least 2 unique object categories each. The visualizations for the mentioned class can be found in the following links.

“computer keyboard, keypad” class: http://i.imgur.com/QMhsRzf.jpg

“sunglasses, dark glasses, shades” class: http://i.imgur.com/a1C7DGh.jpg

1.2 2. Image Captioning

Guided Backpropagation, Grad-CAM and Guided Grad-CAM visualizations for the captions produced by the Neuraltalk2 image captioning model

Full size image

Guided Backpropagation, Grad-CAM and Guided Grad-CAM visualizations for the answers from a VQA model. For each image-question pair, we show visualizations for AlexNet, VGG-16 and VGG-19. Notice how the attention changes in row 3, as we change the answer from Yellow to Green (Color figure online)

Full size image

We use the publicly available Neuraltalk2 code and model^{Footnote 8} for our image captioning experiments. The model uses VGG-16 to encode the image. The image representation is passed as input at the first time step to an LSTM that generates a caption for the image. The model is trained end-to-end along with CNN finetuning using the COCO (Lin et al. 2014b) Captioning dataset. We feedforward the image to the image captioning model to obtain a caption. We use Grad-CAM to get a coarse localization and combine it with Guided Backpropagation to get a high-resolution visualization that highlights regions in the image that provide support for the generated caption (Fig. 18).

1.3 3. Visual Question Answering (VQA)

We use Grad-CAM and Guided Grad-CAM to explain why a publicly available VQA model (Lu et al. 2015) answered what it answered.

The VQA model by Lu et al. uses a standard CNN followed by a fully connected layer to transform the image to 1024-dim to match the LSTM embeddings of the question. Then the transformed image and LSTM embeddings are pointwise multiplied to get a combined representation of the image and question and a multi-layer perceptron is trained on top to predict one among 1000 answers. We show visualizations for the VQA model trained with 3 different CNNs—AlexNet (Krizhevsky et al. 2012), VGG-16 and VGG-19 (Simonyan and Zisserman 2015). Even though the CNNs were not finetuned for the task of VQA, it is interesting to see how our approach can serve as a tool to understand these networks better by providing a localized high-resolution visualization of the regions the model is looking at. Note that these networks were trained with no explicit attention mechanism enforced.

Notice in the first row of Fig. 19, for the question, “Is the person riding the waves?”, the VQA model with AlexNet and VGG-16 answered “No”, as they concentrated on the person mainly, and not the waves. On the other hand, VGG-19 correctly answered “Yes”, and it looked at the regions around the man in order to answer the question. In the second row, for the question, “What is the person hitting?”, the VQA model trained with AlexNet answered “Tennis ball” just based on context without looking at the ball. Such a model might be risky when employed in real-life scenarios. It is difficult to determine the trustworthiness of a model just based on the predicted answer. Our visualizations provide an accurate way to explain the model’s predictions and help in determining which model to trust, without making any architectural changes or sacrificing accuracy. Notice in the last row of Fig. 19, for the question, “Is this a whole orange?”, the model looks for regions around the orange to answer “No”.

D More Details of Pointing Game

In Zhang et al. (2016), the pointing game was setup to evaluate the discriminativeness of different attention maps for localizing ground-truth categories. In a sense, this evaluates the precision of a visualization, i.e. how often does the attention map intersect the segmentation map of the ground-truth category. This does not evaluate how often the visualization technique produces maps which do not correspond to the category of interest.

Hence we propose a modification to the pointing game to evaluate visualizations of the top-5 predicted category. In this case the visualizations are given an additional option to reject any of the top-5 predictions from the CNN classifiers. For each of the two visualizations, Grad-CAM and c-MWP, we choose a threshold on the max value of the visualization, that can be used to determine if the category being visualized exists in the image.

We compute the maps for the top-5 categories, and based on the maximum value in the map, we try to classify if the map is of the GT label or a category that is absent in the image. As mentioned in Sect. 4.2 of the main paper, we find that our approach Grad-CAM outperforms c-MWP by a significant margin (70.58% vs. 60.30% on VGG-16).

E Qualitative Comparison to Excitation Backprop (c-MWP) and CAM

In this section we provide more qualitative results comparing Grad-CAM with CAM (Zhou et al. 2016) and c-MWP (Zhang et al. 2016) on Pascal (Everingham et al. 2009).

Visualizations for ground-truth categories (shown below each image) for images sampled from the PASCAL (Everingham et al. 2009) validation set

Full size image

We compare Grad-CAM, CAM and c-MWP visualizations from ImageNet trained VGG-16 models finetuned on PASCAL VOC 2012 dataset. While Grad-CAM and c-MWP visualizations can be directly obtained from existing models, CAM requires an architectural change, and requires re-training, which leads to loss in accuracy. Also, unlike Grad-CAM, c-MWP and CAM can only be applied for image classification networks. Visualizations for the ground-truth categories can be found in Fig. 20. Qualitative examples comparing Grad-CAM with existing approaches can be found in Selvaraju et al. (2016).

More Qualitative examples showing visual explanations and textual explanations for VGG-16 trained on Places365 dataset (Zhou et al. 2017). For textual explanations we provide the most important neurons for the predicted class along with their names. Important neurons can be either be persuasive (positively important) or inhibitive (negatively important). The first 3 rows show positive examples, and the last 2 rows show failure cases

Full size image

F Visual and Textual Explanations for Places Dataset

Figure 21 shows more examples of visual and textual explanations (Sect. 7) for the image classification model (VGG-16) trained on Places365 dataset (Zhou et al. 2017).

Reprints and permissions