research-article

Measuring expert performance at manually classifying domain entities under upper ontology classes

Authors:

Robert Stevens,

Nicolas MatentzogluAuthors Info & Claims

Volume 57, Issue C

https://doi.org/10.1016/j.websem.2018.08.004

Published: 01 August 2019 Publication History

Abstract

Background:

Classifying entities in domain ontologies under upper ontology classes is a recommended task in ontology engineering to facilitate semantic interoperability and modelling consistency. Integrating upper ontologies this way is difficult and, despite emerging automated methods, remains a largely manual task.

Problem:

Little is known about how well experts perform at upper ontology integration. To develop methodological and tool support, we first need to understand how well experts do this task. We designed a study to measure the performance of human experts at manually classifying classes in a general knowledge domain ontology with entities in the Basic Formal Ontology (BFO), an upper ontology used widely in the biomedical domain.

Method:

We recruited 8 BFO experts and asked them to classify 46 commonly known entities from the domain of travel with BFO entities. The tasks were delivered as part of a web survey.

Results:

We find that, even for a well understood general knowledge domain such as travel, the results of the manual classification tasks are highly inconsistent: the mean agreement of the participants with the classification decisions of an expert panel was only 51%, and the inter-rater agreement using Fleiss’ Kappa was merely moderate (0.52). We further follow up on the conjecture that the degree of classification consistency is correlated with the frequency the respective BFO classes are used in practice and find that this is only true to a moderate degree (0.52, Pearson).

Conclusions:

We conclude that manually classifying domain entities under upper ontology classes is indeed very difficult to do correctly. Given the importance of the task and the high degree of inconsistent classifications we encountered, we further conclude that it is necessary to improve the methodological framework surrounding the manual integration of domain and upper ontologies.

References

[1]

Noy N.F., Semantic integration: a survey of ontology-based approaches, SIGMOD Rec. 33 (4) (2004) 65–70.

Digital Library

[2]

Schmidt D., dos Santos C.T., Vieira R., Analysing top-level and domain ontology alignments from matching systems, in: OM@ ISWC, 2016, pp. 13–24.

[3]

Frické M., Review of building ontologies with basic formal ontology Robert Arp, Barry Smith and Andrew D. Spear. Cambridge: The MIT Press, 2015, 248 pp. $30.00 (Paperback) (ISBN: 9780262527811), JASIST 68 (3) (2017) 801–804.

[4]

Mascardi V., Cord V., Rosso P., A Comparison of Upper Ontologies, 2006.

[5]

A. Gangemi, N. Guarino, C. Masolo, A. Oltramari, L. Schneider, Sweetening Ontologies with DOLCE, in: Knowledge Engineering and Knowledge Management. Ontologies and the Semantic Web, 13th International Conference, EKAW 2002, Siguenza, Spain, October 1-4, 2002, Proceedings, 2002, pp. 166–181 https://doi.org/10.1007/3-540-45810-7_18.

[6]

Herre H., General formal ontology (GFO): A foundational ontology for conceptual modelling, in: Theory and Applications of Ontology: Computer Applications, Springer Netherlands, Dordrecht, 2010, pp. 297–345.

[7]

Niles I., Pease A., Towards a standard upper ontology, in: FOIS, 2001, pp. 2–9.

[8]

Smith B., Ceusters W., Ontological realism: A methodology for coordinated evolution of scientific ontologies, Appl. Ontol. 5 (3–4) (2010) 139–188.

[9]

Mascardi V., Locoro A., Rosso P., Automatic ontology matching via upper ontologies: a systematic evaluation, IEEE Trans. Knowl. Data Eng. 22 (5) (2010) 609–623.

[10]

Smith B., Ashburner M., Rosse C., Bard J., Bug W., Ceusters W., Goldberg L.J., Eilbeck K., Ireland A., Mungall C.J., OBI Consortium, Leontis N., Rocca-Serra P., Ruttenberg A., Sansone S.A., Scheuermann R.H., Shah N., Whetzel P.L., Lewis S., The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration, Nature Biotechnol. 25 (11) (2007) 1251–1255.

[11]

C.M. Keet, The use of foundational ontologies in ontology development: an empirical assessment, in: The Semantic Web: Research and Applications - 8th Extended Semantic Web Conference, ESWC 2011, Heraklion, Crete, Greece, May 29-June 2, 2011, Proceedings, Part I, 2011, pp. 321–335, https://doi.org/10.1007/978-3-642-21034-1_22.

[12]

S. Borgo, P. Leitão, The role of foundational ontologies in manufacturing domain applications, in: On the Move to Meaningful Internet Systems 2004: CoopIS, DOA, and ODBASE, OTM Confederated International Conferences, Agia Napa, Cyprus, October 25-29, 2004, Proceedings, Part I, 2004, pp. 670–688, https://doi.org/10.1007/978-3-540-30468-5_43.

[13]

V. Mascardi, V. Cordì, P. Rosso, A comparison of upper ontologies, in: WOA 2007: Dagli Oggetti agli Agenti. 8th AI*IA/TABOO Joint Workshop “From Objects to Agents”: Agents and Industry: Technological Applications of Software Agents, 24–25 September 2007, Genova, Italy, 2007, pp. 55–64. http://woa07.disi.unige.it/papers/mascardi.pdf.

[14]

Guarino N., Welty C.A., An overview of ontoclean, in: Handbook on Ontologies, Springer Publishing Company, 2009, pp. 201–220.

[15]

Alsubait T., Parsia B., Sattler U., Ontology-based multiple choice question generation, KI 30 (2) (2016) 183–188.

[16]

Leo J., Capturing temporal aspects of bio-health ontologies, University of Manchester, 2016.

[17]

Matentzoglu N., Stevens R., Malone J., Lord P., Measuring Expert Performance at Manually Classifying Domain Entities under Upper Ontology Classes: Data and Scripts, figshare, 2017, (Figshare). Output generated by the script can be seen here: https://doi.org/10.6084/m9.figshare.5706016.v1.

[18]

Mungall C.J., McMurry J.A., Khler S., Balhoff J.P., Borromeo C., Brush M., Carbon S., Conlin T., Dunn N., Engelstad M., Foster E., Gourdine J.P., Jacobsen J.O.B., Keith D., Laraway B., Lewis S.E., NguyenXuan J., Shefchek K., Vasilevsky N., Yuan Z., Washington N., Hochheiser H., Groza T., Smedley D., Robinson P.N., Haendel M.A., The monarch Initiative: an integrative data and analytic platform connecting phenotypes to genotypes across species, Nucleic Acids Res. 45 (D1) (2017) D712–D722. https://academic.oup.com/nar/article/45/D1/D712/2605791.

[19]

Cooke N.J., Knowledge elicitation, in: Handbook of Applied Cognition, 1999, pp. 479–510.

[20]

Welbank M., An overview of knowledge acquisition methods, Interact. Comput. 2 (1) (1990) 83–91.

Cited By

Zhang SDong YZhang YPayne TZhang JDastani MSichman JAlechina NDignum V(2024)Large Language Model Assissted Multi-Agent Dialogue for Ontology AlignmentProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems10.5555/3635637.3663238(2594-2596)Online publication date: 6-May-2024
https://dl.acm.org/doi/10.5555/3635637.3663238

Index Terms

Measuring expert performance at manually classifying domain entities under upper ontology classes
1. Theory of computation

Index terms have been assigned to the content through auto-classification.

Recommendations

WebProtégé: A Cloud-Based Ontology Editor
WWW '19: Companion Proceedings of The 2019 World Wide Web Conference

We present WebProtégé, a tool to develop ontologies represented in the Web Ontology Language (OWL). WebProtégé is a cloud-based application that allows users to collaboratively edit OWL ontologies, and it is available for use at https://...
Chowlk: from UML-Based Ontology Conceptualizations to OWL
The Semantic Web
Abstract
Ontology conceptualization is an ontology development task that consists in generating a preliminary model based on the requirements that the ontology should represent. This activity is often carried out by generating the models as diagrams in a ...
Measuring ontology information by rules based transformation

Ontologies have currently attracted much attention of researchers and engineers in many fields such as knowledge management, etc. It is attractive for ontology engineers to select and reuse the existing ontologies by measuring and evaluating them ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Web Semantics: Science, Services and Agents on the World Wide Web

Web Semantics: Science, Services and Agents on the World Wide Web Volume 57, Issue C

Aug 2019

109 pages

ISSN:1570-8268

Issue’s Table of Contents

The Authors.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 01 August 2019

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang SDong YZhang YPayne TZhang JDastani MSichman JAlechina NDignum V(2024)Large Language Model Assissted Multi-Agent Dialogue for Ontology AlignmentProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems10.5555/3635637.3663238(2594-2596)Online publication date: 6-May-2024
https://dl.acm.org/doi/10.5555/3635637.3663238

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents