Nothing Special   »   [go: up one dir, main page]

Skip to main content

Increasing the Accuracy of LLM Question-Answering Systems with Ontologies

  • Conference paper
  • First Online:
The Semantic Web – ISWC 2024 (ISWC 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15233))

Included in the following conference series:

  • 84 Accesses

Abstract

There is increasing evidence that question-answering (QA) systems with Large Language Models (LLMs), which employ a knowledge graph representation of an enterprise SQL database (Text-to-SPARQL), achieve higher accuracy compared to systems that answer questions directly on SQL databases (Text-to-SQL). The objective of this research is to further improve the accuracy of these LLM Question Answering systems. Our approach, Ontology-based Query Check (OBQC), is to check the LLM generated SPARQL query against the semantics specified by the ontology. A query will be flagged as incorrect and prevented from execution if it does not align with the ontological semantics. The study also explores the LLM’s capability in repairing a SPARQL query given an explanation of the error (LLM Repair). Our methods are evaluated using the chat with the data benchmark. The primary finding is our method further increases the accuracy overall by 21.59% thus pushing the overall accuracy level to 65.63%. These results provide further evidence that investing knowledge graphs, namely the ontology, provides higher accuracy for LLM powered question answering systems. Our method is a component of the data.world AI Context Engine which is being widely used by customers in Generative AI production use cases that enable business users to chat with SQL databases.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 74.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/datadotworld/cwd-benchmark-data.

  2. 2.

    using RDFLib nomenclature.

  3. 3.

    https://github.com/datadotworld/cwd-benchmark-data.

  4. 4.

    https://data.world/ai/.

  5. 5.

    https://data.world/ai/.

References

  1. Green, B.F., Wolf, A.K., Chomsky, C., Laughery, K.: Baseball: an automatic question-answerer. In: Papers Presented at the May 9-11, 1961, Western Joint IRE-AIEE-ACM Computer Conference (New York, NY, USA, 1961), IRE-AIEE-ACM ’61 (Western), Association for Computing Machinery, pp. 219–224 (1962)

    Google Scholar 

  2. Green, C.C., Raphael, B.: The use of theorem-proving techniques in question-answering systems. In: Proceedings of the 1968 23rd ACM National Conference (New York, NY, USA, 1968), ACM ’68, Association for Computing Machinery, pp. 169–181 (1968)

    Google Scholar 

  3. Hendrix, G.G., Sacerdoti, E.D., Sagalowicz, D., Slocum, J.: Developing a natural language interface to complex data. ACM Trans. Database Syst. 3(2), 105–147 (1978)

    Google Scholar 

  4. Woods, W.A.: Transition network grammars for natural language analysis. Commun. ACM 13(10), 591–606 (1970)

    Google Scholar 

  5. Dahl, D.A, et al.:Expanding the scope of the ATIS task: the ATIS-3 corpus. In: Proceedings of the Workshop on Human Language Technology, pp. 43–48 (1994)

    Google Scholar 

  6. Yu, T., et al.: Spider: a large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3911–3921 (2018)

    Google Scholar 

  7. Zelle, J.M., Mooney, R.J.: Learning to parse database queries using inductive logic programming. In: Proceedings of the Thirteenth National Conference on Artificial Intelligence 2, pp. 1050–1055 (1996)

    Google Scholar 

  8. Tang, L.R., Mooney, R.J.: Automated construction of database interfaces: intergrating statistical and relational learning for semantic parsing. In: 2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pp. 133–141 (2000)

    Google Scholar 

  9. Sequeda, J., Allemang, D., Jacob, B.: A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model’s Accuracy for Question Answering on Enterprise SQL Databases. CoRR arxiv preprint arxiv: abs/2311.07509 (2023)

  10. Allemang, D., Sequeda, J.: Increasing the LLM Accuracy for Question Answering: Ontologies to the Rescue! CoRR arxiv preprint arxiv:abs/2405.11706 (2024)

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Dean Allemang .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Allemang, D., Sequeda, J. (2025). Increasing the Accuracy of LLM Question-Answering Systems with Ontologies. In: Demartini, G., et al. The Semantic Web – ISWC 2024. ISWC 2024. Lecture Notes in Computer Science, vol 15233. Springer, Cham. https://doi.org/10.1007/978-3-031-77847-6_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-77847-6_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-77846-9

  • Online ISBN: 978-3-031-77847-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics