2 Collaborative OE methodologies

As already pointed out, we require that an OE methodology supports the active involvement of different roles during the ontology lifecycle, providing tools for their active participation in all stages of ontology lifecycle. The roles involved are the following:

1. (a) Domain experts: These are the persons who have the knowledge/expertise of the domain and/or data sources. These usually are practitioners, not acquainted with ontology languages, specifications, etc.

2. (b) Ontology engineers: These are the persons who have the knowledge/expertise to make ontological specifications and coordinate an OE task.

3. (c) Knowledge workers: These are the persons who exploit the ontology in ‘operational’ conditions, so as to solve problems, or perform data-driven analysis tasks. These may be domain experts but, in some cases, this is not necessary.

In addition to the roles, we require that an OE methodology supports the engineering of live, evolved and reusable ontologies. Towards analyzing and presenting this requirement, we characterize an ontology according to its status throughout its lifetime in three corresponding dimensions:

1. (a) Liveness: An ontology is live at a particular time instant if it is active at that instant. It may be under development, under evolution, maintenance or update, evaluation or in-use.

2. (b) Evolution: An evolved ontology is an ontology that has been evolved over time, that is, there are available more than one version, recorded during its life cycle.

3. (c) Reusability: A reused ontology is an ontology that has been used to specify other ontologies or used in applications/projects beyond its origin.

According to the above dimensions of the ontology lifetime, an evolved ontology may be dead, and a live ontology may have been not evolved yet. A reused ontology may be alive or dead, evolved or not. An ontology may have also use other ontologies for its specification but still not been reused by others. Authors propose those three dimensions (liveness, evolution, reusability) for their analysis, based on the requirements for OE as introduced and discussed in the Introduction section. Authors do not claim that these are the only dimensions possible for such an analysis, however, to be able to support the aim of the paper that is, to examine the impact of collaborative OEMs for the evolution of living and reused ontologies, these are the necessary requirements.

In the following, we provide a succinct overview of the most recent collaborative OEMs satisfying our basic requirements for the continuous evolution of living and reused ontologies. The descriptions focus also on characteristics related to tool-supported engineering of ontologies towards live, evolved and/or reused ontologies. Table 1 presents those methodologies in relation to how they meet those requirements and characteristics (labeled in Table 1 and cited in the OEM descriptions below, paragraphs OEM1 to OEM9), as discussed in the Introduction section.

Table 1 Collaborative OEMs and supported features

At this point it must be stated that a collaborative OEM generally comprises three main phases: (a) ontology specification, (b) ontology development and (c) ontology exploitation and evaluation phase. Different organizations of OE tasks into phases exist (e.g. as recently included in Keet’s OE textbook (Keet, Reference Keet2018: 92)); however, all include the development of the ontology as one of the OE phases, and the implementation task as an integrated task of this phase (the term ‘implementation’ is used interchangeably with ‘development’ in some cases). Also, the distinction between the isolated, single, stand-alone, ontology development vs. collaborative development of ontologies and ontology networks is highlighted, even in very recent related work (e.g., Keet (Reference Keet2018: 93)). Based on this discussion, we do not present approaches that emphasize only the development phase (and related task) of ontologies, as in data-driven (e.g. (D’Amato et al., Reference D’Amato, Fanizzi and Esposito2010)) or test-driven (e.g. (Maria Keet & Ławrynowicz, Reference Maria Keet and Ławrynowicz2016)) ontology development approaches. We conjecture that such development approaches should be viewed as integrated ontology development tasks in the development phase of a collaborative OEM. Thus, although those approaches are used for the development of ontologies, they should be examined as part (integrated tasks) of an OE methodological approach.

1. OEM1. The Human-Centered Collaborative Ontology Engineering Methodology (HCOME) (Kotis & Vouros, Reference Kotis and Vouros2006; Kotis & Papasalouros, Reference Kotis and Papasalouros2010) supports the decisive involvement of knowledge workers in an ontology life cycle. It supports a human-centred approach to the collaborative engineering of ontologies, where the active participation of knowledge workers (Table 1(a)) in the ontology life cycle, in close collaboration to domain experts and ontology engineers (Table 1(b)), is emphasized. Ontologies are managed according to the capacities of knowledge workers are developed individually as well as collaboratively and are put in the context of workers’ experiences and their working settings as an integrated part of a ‘knowing’ process. Leveraging the role of knowledge workers by their active participation in the ontology life cycle, this human-centred approach requires the use of tools that provide greater opportunities for managing and interacting with conceptualizations in a direct and continuous mode in threads of discussion, resolving various issues. An iterative approach to the execution of tasks (Table 1(c)) at all three phases (specification, conceptualization, exploitation/evaluation) is supported, emphasizing on tasks related to the argumentation/discussion of conceptualizations, detailed versioning of evolving specifications, and ontology management such as import/export, comparison and merging of ontological definitions. Leveraging the role of knowledge workers, HCOME (although not intended to that when proposed) emphasizes on the exploitation of data in operational contexts. In conjunction to that, a data-driven (bottom-up) conceptualization approach is also proposed (Table 1(d)) in the updated version of the HCOME methodology (Kotis & Papasalouros, Reference Kotis and Papasalouros2010), supported by learning seed ontologies (knowledge in that case is extracted from query logs). In terms of tooling support, the methodology was supported (until 2010) by a stand-alone OE integrated environment (namely, HCONE) to support management and versioning tasks in the personal space of knowledge workers (Table 1(i, g, h)), and a semantic MediaWiki-based shared environment (namely, Shared-HCONE) (Kotis et al., Reference Kotis, Papasalouros, Vouros, Pappas and Zoumpatianos2011) for supporting the shared space collaborative and evaluation tasks (Table 1(i, e, g)) and the argumentation-based discussion model (Table 1(f)). Currently, this gap of support is covered by related tools such as Protégé (for developing of local ontology versions) and Web-Protégé (for a collaborative editing experience) as well as by using e-mail lists and Google technology (groups and documents).

2. OEM2. Similarly to HCOME, the DIstributed, Loosely-controlled and evolving Engineering of oNTologies (DILIGENT) (Vrandečić et al., Reference Vrandečić, Pinto, Tempich and Sure2005; Pinto et al., Reference Pinto, Tempich, Staab, Staab and Studer2009) methodology focuses on the evolutionary life cycle of ontologies (Table 1(c)) and on a user-centric ontology development. The ontology development process begins with the involvement of different stakeholder groups (Table 1(a)), that is, domain experts, knowledge workers and ontology engineers, as they collaboratively build (Table 1(b)) the first version of an ontology (seed ontology). This version is the result of fast agreement (between all participants) on the high-level terms (Table 1(f)). Furthermore, knowledge workers begin to work with the ontology and locally adapt it—introducing subclasses—to their specific requirements. A control board is introduced, collecting requests for changes to the shared (core) ontology (Table 1(e)). The set of different local ontologies is analyzed by the control board, to find similarities (Table 1(h)). Based on this analysis, a new version of the shared ontology is introduced (Table 1(g)). The control board revises the shared ontology frequently. The ontology engineers are responsible for maintaining the ontology based on the decisions of the board. Knowledge workers are able to locally update the local ontologies by reusing new terms instead of using their previously defined local terms. As a result, ontology reuse (Table 1(h)) is maximized among all knowledge workers. As it is obvious, the approach emphasizes the distributed and collaborative construction of ontologies. As in HCOME, leveraging the role of knowledge workers supports the exploitation of data in operational contexts (Table 1(d)). In terms of tooling support, a standard Wiki is used to allow a traceable discussion between all participants (Table 1(i)). The agreed ontology is placed to the Wiki in order to visualize the ontology and ease the discussion upon it. Evaluation and validation processes are also supported. A data-driven (bottom-up) conceptualization approach (Table 1(d)), supported via appropriate tools (Table 1(i)) for ontology learning, is described at the conceptual level. Ontology management is partially supported.

3. OEM3. Melting Point (MP) (Garcia et al., Reference Garcia, Neill, Garcia, Lord, Corcho and Gibson2010) is a collaborative and human-centered methodology that has been engineered to support decentralised communities of practice for which the designers of technology and the users may be the same group of stakeholders. MP was developed by taking best practices from earlier methodologies. As in HCOME and DILIGENT methodologies, MP emphasizes the collaborative development (Table 1(b)) of ontologies by communities of practice (Table 1(a)), utilising the collective intelligence of the application domain. In addition, MP is not prescriptive about specific techniques or methods, since it recommends that ontology developers should consider the use of those that best suit their particular situation (Table 1(i)). Managerial activities happen throughout the whole ontology life cycle (Table 1(h)). The iterative interaction (Table 1(c)) amongst stakeholders ensures (a) the quality of the ontology (Table 1(e)) and (b) that a set of predefined control activities take place. This methodology has been shaped from experiences gathered by its authors as best practice within a real OE setting of bio-ontology development, reusing tasks from other methodologies. The recommendation of specific tools to support each task and phase of the methodology is missing.

4. OEM4. Ontology-Grounded Methods and Applications (DOGMA) (de Moor et al., Reference de Moor, De Leenheer, Meersman, Schärfe, Hitzler and Øhrstrøm2006) and its extension DOGMA-MESS (Leenheer & Debruyne, Reference Leenheer and Debruyne2008) follows a data-driven (bottom-up) approach to OE (Table 1 (d)). It is a collaborative OEM (Table 1(b) that supports the modeling of shared ontologies in stakeholders’ own terminology and context (Table 1(a). To accomplish that, four modes of knowledge conversion are introduced: socialization, externalization, combination and internalization. Technically, at the center of the approach is an Ontology Server, which is embedded in a central ontology evolution support system (Table 1(c)). There are three types of participants (Table 1(a): the core domain expert, the domain expert and the ontology engineer. The ontology evolution process is driven by social knowledge conversion modes. This process is iteratively performed (Table 1(c) until an optimum trade-off between differences and commonalities of organizational and common perspectives is reached. The development task is driven by data (Table 1(d)). Ontology management is partially supported (Table 1(h) via the import and identification of conflicts of ontological definitions. In terms of tooling support, the methodology was supported by DOGMA Studio (Leenheer & Debruyne, Reference Leenheer and Debruyne2008) in all its phases and tasks (Table 1(i, g)).

5. OEM5. A facet-based OEM for the development of large-scale geospatial ontologies in the space domain (FMCLGO) (Giunchiglia et al., Reference Giunchiglia, Dutta, Maltese and Farazi2012) has been proposed, with a minimal set of ontology development guiding principles inspired by the faceted approach, originally proposed for the library science. A facet is a hierarchy of homogeneous terms describing an aspect of the domain, where each term in the hierarchy denotes a different concept. The approach focuses on the domain and facet aspects, refining and extending existing resources including GeoNamesFootnote 1, WordNetFootnote 2 and the Italian part of Multi-WordNetFootnote 3. The ontology development process is manual in terms of identifying and categorizing all the terms denoting classes into facets, relations and attribute names. On the other hand, the process is automatic in terms of populating the ontology with entities and corresponding attribute values. The terms collected and disambiguated during the identification of the terminology phase were used as building blocks for the construction of the facets that eventually constitute the Space ontology (analysis and synthesis phases). This methodology follows a top-down approach to OE (Table 1(d)). The involvement of different roles in a collaborative processing at all steps (Table 1(b)), emphasizing the participation of domain experts and knowledge workers (Table 1(a)), is missing (apart from typical interviews of domain experts). Iterative processing (Table 1(c)) is limited at the stage of mapping resources to the ontology (by repeating all previous tasks), and this only if disagreements have been identified between engineers. Argumentation, exploitation and evaluation of the ontology are not part of the methodological tasks (Table 1(e, f)), and typical management tasks such as versioning, importing, comparing and merging ontological definitions are not discussed (Table 1(g, h)). In terms of tooling support, there is lack of information regarding specific tools proposed or used for each phase and task of the methodology (Table 1(i)).

6. OEM6. A recent methodology for lightweight and rapid OE is UPON Lite (De Nicola & Missikoff, Reference De Nicola and Missikoff2016). It has been designed for domain experts, knowledge workers and even casual users in the relevant business domain. The approach aims to reduce the role of ontology engineers, placing the responsibility for ontology development to the community of end-users (knowledge workers) (Table 1(a)). This is achieved through a participative social approach (Table 1(b, f)), supported by easy-to-use methods and tools (Table 1(i)). UPON Lite is designed for rapid ontology prototyping and is based on three main aspects: (a) a user-centered approach that accentuates the role of domain experts, (b) a social approach that accentuates the collective intelligence of domain experts to progressively (Table 1(c)) perform all tasks and (c) an ontology building process that is organized in well-defined steps. The process of ontology building and management is carried out on a social media platform. Provided examples are based on Google DocsFootnote 4 suite, in particular with shared Google Sheets for OE, in conjunction with Google Forms and Google+ for other functions (such as debating and voting). Argumentation and versioning support are proposed (Table 1(f, g)) and supported via inherited functionality of social media technology. In addition to these tools, MappingMaster text mining tool and Protégé OE integrated environment were in use. This methodology follows a data-driven (bottom-up) approach to OE. It involves different roles in a collaborative processing (Table 1(b)) at all steps, emphasizing the participation of knowledge workers, following an agile process (Table 1(c). Exploitation and evaluation of the ontology are included in the methodological tasks (Table 1(e)), but typical management tasks such as importing, comparing and merging ontological definitions are not discussed (Table 1(h)).

7. OEM7. A scenario-based OEM, NeOn (Pérez et al., Reference Pérez, Baonza and Villazón2008), emphasizes the development of ontology networks as well as the reuse of existing ontological and non-ontological resources to the development of an ontology. One of the key principles of this OEM is the involvement of developers and ontology practitioners at the same time (Table 1(a)). NeOn considers the existence of multiple ontologies in ontology networks, the collaborative development of ontologies (Table 1(b)) and the reuse and re-engineering of knowledge (Table 1(g, h)). NeOn definition has been based on the analysis of a set of nine ontology development scenarios. A data-driven conceptualization approach (Table 1(d)) is not reported. Also, argumentation/discussion (Table 1(f)) for supporting consensus/agreement reach is not explicitly reported, but the methodology was supported by the NeOn toolkit (Table 1(i)), which provided support for discussing issues (based on DILIGENT OEM) via a plugin that was connected to Cicero (Dellschaft et al., Reference Dellschaft, Engelbrecht, Barreto, Rutenbeck and Staab2008). Cicero was a dialogue management platform for keeping track of ontological discussions between ontology developers and users.

8. OEM8. Grounding Ontologies with Social Processes and Natural Language (GOSPL) (Debruyne et al., Reference Debruyne, Tran and Meersman2013) is a collaborative (Table 1(b)) ontology evolution (Table 1(c)) approach that has been based on the notion of hybrid OE and DOGMA-MESS (Leenheer & Debruyne, Reference Leenheer and Debruyne2008). A hybrid ontology is an ontology where concepts are described both in a formal and an informal manner, where context identifiers refer to communities supporting the social processes leading to agreements (Debruyne et al., Reference Debruyne, Tran and Meersman2013) (Table 1(f)). In hybrid OE, communities of practice are first-class citizens, meaning that the interactions within an evolving community result in series of ontology evolution actors (Table 1(a, c)). All the agreements reached in the involved communities are the result of social interactions (Table 1(f)). Natural language is central and vital to this approach. GOSPL supports involved stakeholders to interpret and model shared ontologies in their own terminology and context and provide feedback of the results to the community. Social processes are explicitly defined. GOSPL is discussion-oriented, that is, the ontology evolves only if the community is reaching an agreement (through a voting system) A data-driven conceptualization approach (Table 1(d)) is not reported. Ontology management is partially supported via the importing of ontologies and the identification of conflicts of ontological definitions (Table 1(h)). In terms of tooling support (Table 1(i)), there is support of a discussion-oriented tool to handle hybrid ontology description and glossary, discussions’ links, the management of communities, the commitments of applications to the ontology, the OWL implementation of the hybrid ontology and a voting system, among other provided functionalities.

9. OEM9. Very recently, SAMOD (Peroni, Reference Peroni2017) OEM has been proposed for simplified and agile development of ontologies. Such an approach specifies the steps of an iterative processing (Table 1(c)) focusing on the development of documented ontologies starting from typical examples of domain descriptions. This work has been partially inspired by test-driven development processes in software engineering. Although domain experts and ontology engineers are present, knowledge workers are not placed in the loop (Table 1(a)). Collaboration appears in between domain experts and ontology engineers at certain initial steps of the development process (Table 1(b)). The approach is not data driven, but it is driven by motivating scenarios and informal competency questions (Table 1(d)). Example data populating the ontology test the validity of the evolving models iteratively (Table 1(e)). Versioning support exists at the level of the models as a whole (Table 1(g)), but no detailed tracking of ontological definitions is supported explicitly. Typical management tasks such as importing, comparing and merging ontological definitions are not discussed (Table 1(h)), but no arguments/discussions mechanism is reported as a mean for reaching consensus (Table 1(f)). In terms of tooling support, the methodology was partially supported by tools such as transformation of diagrams into OWL and automatic generation of ontology documentation (Table 1(i)).

3 Selected Ontologies

The ontologies presented in this paper were selected from a large set of well-known, widely-used, representative and recently published ontologies. As already pointed out, this list is not by any means exhaustive: It rather presents representative examples of ontologies (domain, generic, upper, application oriented and data driven) at different stages of their life cycles (from mature to very recent and from those with high reusability to those of no-use beyond the original intended one), claiming the use of an OEM, maybe by means of appropriate tools.

Ontologies were selected from (a) ontology repositories (Table 2) and (b) articles from the latest (last 3 yr) International Semantic Web (ISWC) and Extended Semantic Web (ESWC) conferences. In addition, ontologies known very well to us, since they that have been built in our laboratory and have been added to the list (SO13, SO24): These have been published in conference articles (ESWC’17, SEMANTICS’17) and in a journal article (International Journal of Distributed Systems and Technology), representing knowledge for key research domains (semantic trajectories and IoT respectively). Also, prominent (in terms of acceptance, reuse and community engagement) ontologies that have been reused in OE tasks during the last 10 years of our OE experience have been added to the list (SO14, SO15, SO16, SO17, SO19, SO21, SO22, SO30 and SO31). Furthermore, ontologies that are prominent and are presented in articles discussing the followed OE methodologies have been added (SO8, SO11, SO14, SO15, SO16, SO17, SO18, SO19, SO20, SO21, SO22, SO23, SO25, SO26, SO27, SO28 and SO29). Finally, there is a set of ontologies that were selected due to its intersection with other sets, that is, ontologies that are prominent, well-known and widely used (SO8, SO11, SO14, SO15, SO16, SO17, SO19, SO21 and SO22). The description of the ontologies is ordered according to their publication type and year (conferences first—from recent to older events, journals and Web resources afterwards).

Table 2 Ontology repositories that were used for the ontology selection process

In Table 3, the status of the SOs is presented (a detailed descriptive presentation is provided in the Appendix). More specifically, the table presents: (a) the aim and scope of the ontologies, (b) their life status (alive or dead), (c) their status of evolution, that is, whether they have been evolved so far (or until death) and (d) their status of reuse, that is, how and in what extent they have been reused (or planned to be reused) in other OEM efforts. Also, they are linked to their related published resources (Web and papers in journals or conferences) as well as to the related OEM adopted for their development and/or evolution. Finally, additional information has been gathered to be considered in the analysis, such as the reuse of other ontologies/vocabularies and the use of tools to support a collaborative engineering approach.

Table 3 Selected ontologies, their relation to OEMs, and their status

To validate the details of each SO, knowledge engineers or/and authors of related papers have been contacted by e-mail (sharing the short descriptions). From the thirty contacted ontology development teams/ontologists, eighteen have been explicitly (by providing their feedback by e-mail) verified them, updating the collected information accordingly.

4 OEMs impact Analysis, Findings and Recommendations

4.1 OEMs impact to SOs

In this section, we present the impact of OEMs to the development and evolution of SOs: Specifically, we examine the impact of OEMs to the ontologies’ life status, status of evolution and status of reuse. The analysis involves thirty-two ontologies (single or groups of ontologies as in the case of SPAR and Upon Lite) presented in Section 3, following a pragmatic approach (i.e., based on practical rather than theoretical considerations, since we consider OEMs impact of ontologies in practice, during the engineering of real (pragmatic) ontologies, in real engineering settings).

Starting from our conjecture that, ideally, an OE methodology must support all involved stakeholders during the ontology life cycle, we focus on OE methodologies that support the active and decisive participation of domain experts, knowledge workers and ontology engineers, in all phases of OE, by means of appropriate tooling support. This places some bias towards collaborative ontology specification, development, exploitation and evaluation of ontologies. However, for the purposes of our analysis, we need to comparatively examine the impact of non-collaborative OEMs and other custom OEMs (OEMs that are not classified in the other two types), as these are reported by the ontology engineers in the respected ontology articles.

Therefore, following the pragmatic approach described, we report per ontology examined the following features: (a) collaborative tools’ support, (b) life status, (c) evolution status, (d) reused-in status and (e) reuses status. In addition to these, for each ontology, the analysis involves its classification in one of the three OEM types (collaborative, non-collaborative and custom). Furthermore, we examine the effect of combined features for each ontology, with the aim to identify which of the classified ontologies in a specific OEM type is alive, has been evolved and has been reused or satisfies all these properties.

Subsequently, we report on the number of alive, evolved and reused ontologies, as well as the number of ontologies that are alive, evolved and reused at the same time, out of the number of ontologies that have followed a specific OEM type. Figures 3 and 4 in the following section show how the different OEM types for the SOs satisfy the impact criteria. In Figure 3, the OEMs are comparatively depicted over the three criteria and their combination. In Figure 4, the distribution is shown for all the above criteria taking into account the OEMs that satisfy the criteria, thus providing a view starting from the impact criteria towards the OEMs. As an example of each figure, the following is extracted from the analysis:

• Figure 3 example: Number of live SOs from all SOs followed a Collaborative OEM (4/10 = 0.4).

• Figure 4 example: Number of live SOs following a Collaborative OEM from all the live SOs per OEM type (4/20 = 0.2).

4.1.1 Analysis

Based on the analysis conducted, five main criteria (OEM type, liveness, evolution, reusability, collaborative tool support) and one secondary (reuses other ontologies), and their distribution, have been derived to report the following results:

1. (a) 10 ontologies among those examined followed a collaborative OEM, 4 ontologies followed a non-collaborative OEM and 18 ontologies followed a custom OEM. More than the half of them (19) have been supported by collaborative tools, and most of them (14) have been used in the context of custom OEM (Figure 1).

2. (b) Regardless the OEM followed, from all the SO, 20 ontologies are still alive, 27 ontologies have been evolved, 22 ontologies have been reused and 20 ontologies have reused others in their specifications. Evolution of ontologies (either ended or ongoing) is the feature with the highest percentage (84%) (Figure 2). Out of the alive ontologies, the largest proportion are the ones followed a Non-Collaborative and Custom OEM. Almost all the SOs followed a Custom OEM are live or reused or evolved, while 76% of them exhibit all the above qualities. The latter is very important as a combined metric since it separates the SOs followed a Custom OEM from the rest by a very large margin. This is especially obvious in Figure 3 where the distribution of impact is shown per criterion for the total number of SOs under each criterion instead of the total number of SOs in the study. The SOs followed a Custom OEM are overwhelmingly impactful across the criteria.

3. (c) Impact of collaborative OEM: from all the SOs, 4 ontologies have followed a collaborative OEM and are alive (13%), 5 ontologies have followed a collaborative OEM and have been evolved (16%), 5 ontologies have followed a collaborative OEM and have been reused (16%) and only 2 ontologies have followed a collaborative OEM and are alive and have been evolved and have been reused at the same time (7%). Evolution of ontologies (either ended or ongoing) is the feature with the highest percentage (16%), with all features to range above 6% (Figures 2 and 3).

4. (d) Impact of non-collaborative OEM: from all the SOs, 3 ontologies have followed a non-collaborative OEM and are alive (9%), 4 ontologies have followed a non-collaborative OEM and have been evolved (13%), 1 ontology have followed a non-collaborative OEM and has been reused (3%) and none of the ontologies have followed a non-collaborative OEM and is alive and have been evolved and reused at the same time (0%). Evolution of ontologies (either ended or ongoing) is the feature with the highest percentage (13%), with all features to range at 0% (Figures 2 and 3).

5. (e) Impact of custom OEM: from all the SOs, 13 ontologies have followed a custom OEM and are alive (41%), 17 ontologies have followed a custom OEM and have been evolved (53%), 16 ontologies have followed a custom OEM and have been reused (50%) and 13 ontologies have followed a custom OEM and are alive and have been evolved and reused at the same time (41%). Evolution of ontologies (either ended or ongoing) is the feature with the highest percentage (53%), with all features to range above 41% (Figures 2 and 3).

Figure 1 Collaborate tool support per type of OEM.

Figure 2 OEM types and impact criteria: live, evolved, reused SOs per OEM.

Figure 3 OEM types and impact criteria: live, evolved, reused SOs per OEM per criterion.

By a first reading of the data, it may be surprising that a custom OEM has the highest impact on the status of SOs. However, this may be justified by the fact that most (14 out of 19) of the ontologies in this type have been engineered with the support of collaborative tools, as discussed in paragraph (a) above. However, the need for support tools may have preceded the choice of the developers, who required OEMs that could be easily customized to their needs. What is evident (depicted in Figure 6) is that, across the timeline, almost all of the high impact SOs are custom engineered with tool support. That may well be the cause for their impact, that is, the fact that satisfied broader requirements of the community that made them sustainable and contributed to their longevity. Furthermore, the number of SOs that satisfy all features (alive, evolved and reused) at the same time is observed for those ontologies that followed a custom OEM. This is also justified by the fact that the several different options of tool support (e.g. Wiki, GitHub, Blog, mailing lists, OE tools) were provided to the communities of interest (stakeholders, including domain experts, ontology engineers and knowledge workers).

Another interesting dimension of the analysis concerns correlations between the features of different SOs. The most interesting result of those concerns the number of the evolved ontologies, as well as the number of the ontologies that are alive, evolved and reused at the same time, among the ontologies that have followed a specific OEM type:

• Evolution: For the SOs developed using a collaborative OEM (10 ontologies), only the 50% of them have been evolved, whereas for the SOs followed a non-collaborative OEM (4 ontologies) or a custom OEM (18 ontologies), this percentage is above 94%, that is, all the SOs but one was evolved.

• Liveness+Evolution+Reuse: For the SOs developed using a collaborative OEM (10 ontologies), only 20% of them are alive, have been evolved and reused at the same time, whereas for the SOs developed using a non-collaborative OEM (4 ontologies), this percentage is zero. However, for the SOs followed a custom OEM (18 ontologies), this percentage is 72% (Figure 3).

Overall, from this survey and analysis conducted, it is found that the custom OEMs have the highest impact on the engineered SOs.

Following an Open Science approach, the data are available on GitHub at https://github.com/KotisK/OEM-survey.

4.2 OEM Trends, Findings and Recommendations

From the presented analysis, the following trends have been identified (looking at the SOs of the last three years):

1. • The trend is to engineer ontologies following a custom community-driven OEM supported by widely available and known collaborative tools (Figures 4 and 5). This eases the participation of stakeholders who are very well acquainted with these tools.

2. • The trend is to make the engineered ontologies open (with appropriate licensing schemes) to enable their reuse, evolution and refinement, as much as possible: Sharing is one of the most crucial aspects when devising an ontology, as it is well known. However, this, in conjunction to the use of appropriate tools, provides opportunities for the OE tasks to support evolution and reuse.

3. • The trend is to use custom, tool-supported OEM supporting collaborative evaluation and validation of evolved versions, integrating feedback from several iterations of evolution and commenting on the published and exploited versions of the ontologies. Thus, agility and multiple iterations are crucial aspects of OEMs, while again, tools for managing the evolution (e.g. versioning, keeping track of changes, justifications etc.) are important.

Figure 4 Time distribution of SOs and the OEM followed.

Figure 5 Time distribution of collaborative tool support development, for all SOs.

These trends are evidenced by the following main findings:

Finding 1: The stakeholder needs for collaboration and technology support seem to be the most probable cause for their decision to device and use custom-engineered OEMs and support them by the use of collaborative tools towards achieving their goals. That results in more focused or expert-driven SOs that are desirable for adoption and reuse by the community. Thus, we conjecture that the methodology by itself may not be sufficient for an ontology engineer to perform a set of collaborative tasks. Collaborative tools developed by engineers may be a justified investment to them as well as the key deciding factors for future stakeholders to reuse and evolve. This is evidenced by (a) the high percentages of alive, evolved and/or reused ontologies among those engineered with either collaborative or custom OEMs and (b) the fact that the collaborative and custom OEMs associated with ontologies reported the use of collaborative tools to support all stakeholders at a very high percentage (Figure 6).

Figure 6 Time distribution of SOs’ full impact (live+ evolved+ reused) and collaborative tool support (support is default).

Finding 2: It is the aim and scope of the ontology that most often determines its reuse status. For instance, if an ontology is developed only for the purpose of enabling the representation and management of specific knowledge within a specific project or application, it is less possible for this ontology to be reused outside the project/application and its related community. This fact may also have a negative impact in the life and evolution of this ontology, without however to be able to predict a certain ‘death’: Other related projects may reuse and further evolve the ontology in their own new/related contexts. The aim and scope of the ten SOs that have not been reused are as follows: Five of them have been devised with a limited project-related aim/scope (SO2, SO13, SO26, SO28, and SO29), while six address a domain-specific aim/scope (SO2, SO5, SO10, SO20, SO26, and SO28). One ontology has an application-specific aim/scope (SO6), while only three have a generic aim/scope (SO12, SO13, SO29).

Finding 3: Although custom OEMs are not rendered to tool-supported tasks for stakeholders to follow and implement, they do demonstrate the use of freely available tools that ease their effort in different ways, from tracking issues related to the refinement of proposed versions by the community (e.g., GitHub) down to the collaborative specification of conceptualizations (e.g., Google spreadsheets, e-mails). The conducted analysis shows that there are several options available in terms of tooling for the support of collaboration in the development of shared and commonly agreed ontologies. This finding is important, given that the availability of free and open source collaborative OE tools (e.g. Web Protégé) and ontology libraries (for reuse of ontologies) nowadays is a ‘silver bullet’ for the engineering of ontologies.

Based on the trends and findings resulted from the analysis made, the following recommendations for the engineering of ontologies can be concluded:

Recommendation 1: Ontologists seem to require custom methods to achieve their immediate OE goals and requirements. Following stakeholders also seem to prefer the custom-engineered OEMs and use them in their OE tasks. With that in mind, the recommendation would be that engineers should develop or reuse an OEM keeping in mind that may be used by others and, therefore, utilize or develop methods and technologies that are widely accepted. Towards this, they need to assure the active, continuous, decisive participation of all interested parties. To do this, ontologists need to investigate the impact of community-driven requirements for the ontologies to methodological features and to choose appropriate, widely-known and used (collaborative) tools, for example, OnToology (Alobaid et al., Reference Alobaid, Garijo, Poveda-Villalón, Santana-Perez, Fernández-Izquierdo and Corcho2019) and WebProtégé (Tudorache et al., Reference Tudorache, Vendetti and Noy2009), for facilitating contributions in a selective, familiar and user-friendly way. This should be done throughout the lifetime of the ontology, to be kept alive, evolved and be reused at the maximum level. This will minimize the risk of developing a single-contributor ontology, with no community consensus, and safeguard the active participation of interested stakeholders.

Recommendation 2: As a general recommendation deduced from the works presented in the papers that describe the SOs, the OEM followed (collaborative/non-collaborative or custom-engineered) should provide early versions of the ontology to be evaluated and validated by knowledge workers, in a continuous, iterative approach. This is very important in the Big Data era, among other things, as multi-source data may necessitate fast adaptation to data sources, and validation of ontological specifications for the needs of data analysis tasks. Early feedback and early involvement of stakeholders are other important aspects of the approach to be followed.

Recommendation 3: The ontology developed should be made available, open and ‘linked’ to other ontologies associated to the interests of a wide (as much as possible) community of domain experts and knowledge workers in a respective field. This means that the ontology should reuse classes/properties and specifications from other ontologies, be shared to the community using appropriate licence schemes and with the appropriate tooling support, as needed, and should assure its reusability either within the ‘strict’ scope concerning its development, but also to a wider (as much as possible) context. The latest can be supported, if appropriate generic constructs and specifications are provided, generalizing upon the terms and specifications used within its ‘strict’ scope. Modularity plays a crucial role here (Grau et al., Reference Grau, Horrocks, Kazakov and Sattler2008; Santipantakis & Vouros, Reference Santipantakis and Vouros2014), and special emphasis should be put to designing ontologies as modular parts of other ontologies, and/or allow splitting ontologies to modular parts that can be reused independently from the rest of the ontological specifications.

Recommendation 4: To support the ontology engineer to follow the previous recommendations, the extension of a modern tool-supported OEM should be recommended in terms of (a) supporting the specification of modular ontologies, w.r.t. formal modularity properties, and supporting mapping between ontologies, at the conceptual and at the instance level and (b) facilitating the agile specification, development, evolution and maintenance of modular ontologies in shared collaborative workspaces with well-known and widely used tools by knowledge workers and domain experts.

5 Related Work

According to our recent search and records, there are few related recent surveys on OE approaches; however, these surveys do not analyze the impact of the OEMs to the status of ontologies. This section presents a chronological overview of related works and their differences/limitations over the analysis presented in this paper.

In Iqbal et al. (Reference Iqbal, Murad, Mustapha and Sharef2013), a survey on OEMs is presented. Authors compare existing methodologies and identify gaps, trends and needs that a modern effort to define a new OE methodology should consider. Based on nine criteria (C1–C9 columns of data analysis, see Excel file on GitHub) identified by the authors, the review concludes that the examined methodologies are not mature enough in terms of the level of provided details for each phase and task of the supported ontological life-cycle. However, the review does not present an analysis of the impact of those methodologies to the already engineered ontologies. More importantly, the briefly presented recommendations are limited to the generic comment that new methods and techniques should be explored for the identification of ontology concepts.

In Stadlhofer et al. (Reference Stadlhofer, Salhofer and Durlacher2013), a survey on OEMs in the domain of public administration is provided. The criteria selected for this analysis are related to general aspects of the ontology development (e.g., strategies for building ontologies) as well as project management-related aspects (e.g., recommended process model or the consideration of collaborative ontology development). The study concludes that none of the examined methodologies were fully mature to serve as an OE methodology for electronic service provisioning in public administration. However, there is not any specific description of a proposed OE methodology, and there is lack of analysis based on the correlation of OE methodologies to engineered ontologies or OE tools.

In Simperl & Luczak-Rösch (Reference Simperl and Luczak-Rösch2014), a survey on the collaborative OEMs is presented, along with a survey on OE environments. Both are compared in terms of specific criteria, and the concluded remarks of this analysis are presented. The presented conclusions are fostering the fact that ontologies should be developed and maintained in a community-driven way, with the support of integrated OE environments, providing explicit and full support to collaboration and user participation. Although this related work follows a different path to the one presented in our paper (since there is no any analysis of the product of the OE approaches that is, the ontologies), the outcome is in line to the trends and concluded recommendations presented in this paper. More importantly, the survey identifies the need to collect and examine detailed case studies and experience reports that might be helpful for defining a research roadmap to collaborative OE for the forthcoming years. Authors emphasize the shift from the development of ontologies from scratch to large-scale reuse of a large number of ontologies, also shifting their focus from a tool-assisted engineering process to a data-driven engineering approach. Finally, authors suggest that the collaborative and consensus-reaching processing, and associated tool support, should be adapted to reflect these changes. The findings, trends and recommendations presented in this paper provide supporting evidence to this survey and they also extend their analysis to non-collaborative and custom OEMs, going beyond the correlation of OEMs to OE tools, by also analyzing the actual impact to ontologies, following a pragmatic approach.

In Yadav et al. (Reference Yadav, Singh Narula, Duhan and Jain2016), a survey on all aspects of OE is briefly presented, that is, ontologies, OE methodologies and OE tools. However, the survey is limited to just a brief presentation of the topic, without going deeper into the analysis, comparison or critical discussion of their correlation.

The analysis of OE communities, among others, helps understanding how ontologies are being built, whether the community follows specific OE processes and if the proposed OE methodologies work. The latest related work presented in Kanza et al. (Reference Kanza, Stolz, Hepp and Simperl2018) contributes to this field of research by analyzing the activities and interactions of the Schema.org community. Although the analysis is deep and extensive, the actual overall impact of the methodological approach in the development and evolution of Schema.org ontology is not discussed. Also, the work is limited only on Schema.org, without comparing the analysis results with the results of other related OE efforts.

6 Discussion

In Corcho et al. (Reference Corcho, Fernández-López and Gómez-Pérez2003), authors research the ‘meeting point’ of methodologies, tools and languages for OE, based on the widely accepted requirement that a methodology must be supported by appropriate tools in all of its phases and tasks, and do so in an integrated way (e.g. IDE such as Protégé). Concludingly, authors emphasize the need for an ontology development workbench. In the more recent related work of Simperl & Luczak-Rösch (Reference Simperl and Luczak-Rösch2014), it is also emphasized that ontologies should be developed and maintained in a community-driven manner, with the help of fully fledged environments providing dedicated support for collaboration and user participation. This review paper discussed several of the methodologies, methods and techniques, and present popular development environments, which can be utilized to carry out, or facilitate specific activities within the methodologies.

Based on the already identified need of OE tools to support OEMs, but in contrast to related work that identifies such a need, the detailed description and analysis of OE tools (or integrated environments) are out of the scope of this paper since the aim is different, that is, to analyze the impact of the OEMs to the status of ontologies. Having said that, the need of OE tools is accentuated in the paper, supported by the review and analysis conducted, focusing on the impact of not only the OEMs followed (or not) for the engineering of the SOs but also on the tools used (or not). The presented analysis identifies the impact of custom-engineered approaches with collaborative tools support (Figures 3 and 4) in the findings. For instance, that the number of SOs that satisfy all features (alive, evolved, and reused) at the same time is observed for those ontologies that followed a custom OEM, with several different options of tool-supported technology (e.g. Wiki, Git, Blog, mailing lists, OE tools) provided to the communities of interest (stakeholders, including domain experts, ontology engineers and knowledge workers).

The argument that tools are not necessarily associated with methodologies but have an important impact in OE; besides, methodologies have been seen in several related works. Recently, the development of a Git-based OE environment called OnToology (Alobaid et al., Reference Alobaid, Garijo, Poveda-Villalón, Santana-Perez, Fernández-Izquierdo and Corcho2019), emphasized the need for automating OE support activities (i.e., documentation, evaluation, releasing and versioning) in a collaborative- and community-oriented manner. While compared to other collaborative ontology development environments such as WebProtégé (Tudorache et al., Reference Tudorache, Vendetti and Noy2009), Moki (Bosca et al., Reference Bosca, Casu, Dragoni and Rexha2014), Neologism (Basca et al., Reference Basca, Corlosquet, Cyganiak, Fernández and Schandl2008), Vocol (Halilaj et al., Reference Halilaj, Petersen, Grangel-González, Lange, Auer, Coskun and Lohmann2016) and Vocbench (Stellato et al., Reference Stellato, Rajbhandari, Turbati, Fiorelli, Caracciolo, Lorenzetti and Pazienza2015), the evaluation was based on specific OE support activities that are however only a subset of the collaborative OEM toolset that is required to support all tasks and phases of the ontology life cycle (as presented in our paper). For instance, the actual development/editing of the ontology (improvise, learn, specify, compare and merge conceptualizations) is not supported by OnToology, but it is supported (partially) by WebProtégé and its offline stand-alone version of Protege. Other tools such as the Neologism (not maintained anymore) support the editing and publishing, but not the evaluation and versioning of ontologies. Another Git-based tool is VoCol, providing support for project management, quality assurance, documentation, visualization and publishing of ontologies. Finally, Moki is based on MediaWiki technology, similar to the discontinued Shared-HCONE (Kotis et al., Reference Kotis, Papasalouros, Vouros, Pappas and Zoumpatianos2011), providing either a light-weight view or a full source-code view of the ontology, but it does not provide any documentation/argumentation functionalities.

7 Conclusions

There are several OE methodologies, each specifying roles, artifacts, stages, processes and tasks related to the systematic engineering of ontologies. In this article, we conjecture that custom engineered OEMs) is a major trend as is the development of collaborative tools. The requirements by the engineers are such that justify the effort for going beyond established methods. Such approaches are not only overwhelming in numbers across the last two decades but also in sustainability and reuse. This also led to continuous evolution and maintenance of ontologies, keeping them ‘live’ to shape knowledge. Starting from this conjecture, we focus on and examine the effectiveness of collaborative methodologies to keep ontologies alive, evolved and increase their potential to be reused. Special emphasis is put to providing support, via appropriate tooling, to a collaborative approach in ontology specification, development, exploitation and evaluation, engaging all stakeholders towards shaping commonly agreed knowledge. Therefore, both the real-world complexity of the requirements and the need for collaboration drive the trends in OEMs.

This paper summarizes a set of collaborative approaches and the features that they support, driven by major requirements that are widely accepted from the OE research community. In addition, but closely related to tools used, a detailed community-driven investigation of the requirements of the ontologies that need to be developed in a project, may result to additional methodological features that need to be considered and incorporated in the final methodological approach to be applied.

In this survey, we have followed a pragmatic approach and examined the impact of OEMs to the status of ‘deployed’ ontologies in terms of their life status, evolution and reuse. The analysis allows for the identification of trends in OE and recommendations for the future ontology engineer and ontologist.

An important conclusion is that, while an OEM must support the active participation of all involved stakeholders during the ontology life cycle, it is more the selection of appropriate OE tools that have a positive impact to the status of the engineered ontologies, supporting collaboration, ontology evolution and reuse. Moreover, the aim and scope of the targeted ontology should be carefully specified at the early stages of the ontology life cycle, emphasizing a generic rather than a specific focus of the conceptualizations to be developed, precluding the limited exploitation of the ontology within a restricted community.

Concluding, non-collaborative OEMs have a negative impact to the status of ontologies, that is, to their liveness, evolution and reusability. Collaborative and custom OEMs, to ensure a positive impact to the status of engineered ontologies, need to be effectively supported by an extended collaboration toolset for the development, maintenance and evolution of modularized ontologies.