Abstract
In Explanatory Interactive Machine Learning (XIML), counterexamples refine machine learning models by augmenting human feedback. Traditionally created through random sampling or data augmentation, the emergence of Large Language Models (LLMs) now allows an infinite amount of new training instances to be queried through simple natural language prompts. However, validation of LLM results becomes crucial as they may produce potentially inaccurate or “hallucinated” content, which has led to an increased incorporation of logical reasoning with LLMs in recent literature. We present LlmXiml, a framework that integrates logically constrained LLMs into XIML. Our results indicate that LLM-generated counterexamples improve the model performance and logical reasoning increases the counterexamples’ correctness.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Chung, J.J.Y., Kamar, E., Amershi, S.: Increasing diversity while maintaining accuracy: text data generation with large language models and human interventions. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, vol. 1, pp. 575–593 (2023). https://aclanthology.org/2023.acl-long.34.pdf
Hammond, K.J., Leake, D.B.: Large language models need symbolic AI. In: d’Avila Garcez, A.S., Besold, T.R., Gori, M., Jiménez-Ruiz, E. (eds.) Proceedings of the 17th International Workshop on Neural-Symbolic Learning and Reasoning, La Certosa di Pontignano, Siena, Italy, 3–5 July 2023. CEUR Workshop Proceedings, vol. 3432, pp. 204–209. CEUR-WS.org (2023). https://ceur-ws.org/Vol-3432/paper17.pdf
Heidrich, L., Slany, E., Scheele, S., Schmid, U.: FairCaipi: a combination of explanatory interactive and fair machine learning for human and machine bias reduction. Mach. Learn. Knowl. Extr. 5, 1519–1538 (2023). https://doi.org/10.3390/make5040076
Kimmig, A., Demoen, B., Raedt, L.D., Costa, V.S., Rocha, R.: On the implementation of the probabilistic logic programming language ProbLog. Theory Pract. Log. Program. 11(2–3), 235–262 (2011). https://doi.org/10.1017/S1471068410000566
Nye, M.I., Tessler, M.H., Tenenbaum, J.B., Lake, B.M.: Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. In: Ranzato, M., Beygelzimer, A., Dauphin, Y.N., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, 6–14 December 2021, Virtual, pp. 25192–25204 (2021). https://proceedings.neurips.cc/paper/2021/hash/d3e2e8f631bd9336ed25b8162aef8782-Abstract.html
Raedt, L.D., Dries, A., Thon, I., den Broeck, G.V., Verbeke, M.: Inducing probabilistic relational rules from probabilistic examples. In: Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, 25–31 July 2015, pp. 1835–1843. AAAI Press (2015). http://ijcai.org/Abstract/15/261
Ribeiro, M.T., Lundberg, S.M.: Adaptive testing and debugging of NLP models. In: Muresan, S., Nakov, P., Villavicencio, A. (eds.) Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, 22–27 May 2022, pp. 3253–3267. Association for Computational Linguistics (2022). https://doi.org/10.18653/V1/2022.ACL-LONG.230
Shivaswamy, P., Joachims, T.: Coactive learning. J. Artif. Intell. Res. 53, 1–40 (2015). https://doi.org/10.1613/jair.4539
Slany, E., Ott, Y., Scheele, S., Paulus, J., Schmid, U.: CAIPI in practice: towards explainable interactive medical image classification. In: Maglogiannis, I., Iliadis, L., Macintyre, J., Cortez, P. (eds.) AIAI 2022, vol. 652, pp. 389–400. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-08341-9_31
Slany, E., Scheele, S., Schmid, U.: Bayesian CAIPI: a probabilistic approach to explanatory and interactive machine learning. In: Nowaczyk, S., et al. (eds.) ECAI 2023, vol. 1947, pp. 285–301. Springer, Cham (2024). https://doi.org/10.1007/978-3-031-50396-2_16
Teso, S., Kersting, K.: Explanatory interactive machine learning. In: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES 2019, Honolulu, HI, USA, 27–28 January 2019, pp. 239–245. ACM (2019). https://doi.org/10.1145/3306618.3314293
Yang, Z., Ishay, A., Lee, J.: Coupling large language models with logic programming for robust and general reasoning from text. In: Findings of the Association for Computational Linguistics: ACL 2023, pp. 5186–5219 (2023). https://aclanthology.org/2023.findings-acl.321.pdf
Zecevic, M., Willig, M., Dhami, D.S., Kersting, K.: Causal Parrots: Large Language Models May Talk Causality But are Not Causal (2023). https://doi.org/10.48550/arXiv.2308.13067
Acknowledgments
Funded by BMBF Germany, Project hKI-Chemie (# 01IS21023A).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors have no competing interests to declare.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Slany, E., Scheele, S., Schmid, U. (2024). Explanatory Interactive Machine Learning with Counterexamples from Constrained Large Language Models. In: Hotho, A., Rudolph, S. (eds) KI 2024: Advances in Artificial Intelligence. KI 2024. Lecture Notes in Computer Science(), vol 14992 . Springer, Cham. https://doi.org/10.1007/978-3-031-70893-0_26
Download citation
DOI: https://doi.org/10.1007/978-3-031-70893-0_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-70892-3
Online ISBN: 978-3-031-70893-0
eBook Packages: Computer ScienceComputer Science (R0)