Probabilistic Neural Symbolic Models for Interpretable Visual Question Answering

Ramakrishna Vedantam, Karan Desai, Stefan Lee, Marcus Rohrbach, Dhruv Batra, Devi Parikh
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:6428-6437, 2019.

Abstract

We propose a new class of probabilistic neural-symbolic models, that have symbolic functional programs as a latent, stochastic variable. Instantiated in the context of visual question answering, our probabilistic formulation offers two key conceptual advantages over prior neural-symbolic models for VQA. Firstly, the programs generated by our model are more understandable while requiring less number of teaching examples. Secondly, we show that one can pose counterfactual scenarios to the model, to probe its beliefs on the programs that could lead to a specified answer given an image. Our results on the CLEVR and SHAPES datasets verify our hypotheses, showing that the model gets better program (and answer) prediction accuracy even in the low data regime, and allows one to probe the coherence and consistency of reasoning performed.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-vedantam19a, title = {Probabilistic Neural Symbolic Models for Interpretable Visual Question Answering}, author = {Vedantam, Ramakrishna and Desai, Karan and Lee, Stefan and Rohrbach, Marcus and Batra, Dhruv and Parikh, Devi}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {6428--6437}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/vedantam19a/vedantam19a.pdf}, url = {https://proceedings.mlr.press/v97/vedantam19a.html}, abstract = {We propose a new class of probabilistic neural-symbolic models, that have symbolic functional programs as a latent, stochastic variable. Instantiated in the context of visual question answering, our probabilistic formulation offers two key conceptual advantages over prior neural-symbolic models for VQA. Firstly, the programs generated by our model are more understandable while requiring less number of teaching examples. Secondly, we show that one can pose counterfactual scenarios to the model, to probe its beliefs on the programs that could lead to a specified answer given an image. Our results on the CLEVR and SHAPES datasets verify our hypotheses, showing that the model gets better program (and answer) prediction accuracy even in the low data regime, and allows one to probe the coherence and consistency of reasoning performed.} }
Endnote
%0 Conference Paper %T Probabilistic Neural Symbolic Models for Interpretable Visual Question Answering %A Ramakrishna Vedantam %A Karan Desai %A Stefan Lee %A Marcus Rohrbach %A Dhruv Batra %A Devi Parikh %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-vedantam19a %I PMLR %P 6428--6437 %U https://proceedings.mlr.press/v97/vedantam19a.html %V 97 %X We propose a new class of probabilistic neural-symbolic models, that have symbolic functional programs as a latent, stochastic variable. Instantiated in the context of visual question answering, our probabilistic formulation offers two key conceptual advantages over prior neural-symbolic models for VQA. Firstly, the programs generated by our model are more understandable while requiring less number of teaching examples. Secondly, we show that one can pose counterfactual scenarios to the model, to probe its beliefs on the programs that could lead to a specified answer given an image. Our results on the CLEVR and SHAPES datasets verify our hypotheses, showing that the model gets better program (and answer) prediction accuracy even in the low data regime, and allows one to probe the coherence and consistency of reasoning performed.
APA
Vedantam, R., Desai, K., Lee, S., Rohrbach, M., Batra, D. & Parikh, D.. (2019). Probabilistic Neural Symbolic Models for Interpretable Visual Question Answering. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:6428-6437 Available from https://proceedings.mlr.press/v97/vedantam19a.html.

Related Material