Artificial intelligence maturity model: a systematic literature review

Purpose of the literature review

The maturity model is used as a scale for assessing the position on the path of evolution (Becker, Knackstedt & Pöppelbuß, 2009). The position is identified by assessing the competency, capability, and level of sophistication of an organization in a specific domain, based on a set of criteria that is more or less comprehensive (De Bruin et al., 2005). Each level or stage describes criteria and characteristics that need to be satisfied to achieve a specific level of maturity (Bahri & Fauzi, 2018; Becker, Knackstedt & Pöppelbuß, 2009). In addition, the description of the criteria and characteristics in the next level could be used as best-practices guidelines on how to reach a higher level (Fraser, Moultrie & Gregory, 2002).

Alsheibani, Cheung & Messom (2019) mentioned that the academic sources in this topic are scarce, and most of them are produced by analyst organizations, besides the theoretical background is missing. The same conclusion was reached by Ellefsen et al. (2019), who indicated that the AIMMs are established in a limited number of articles from Scopus or Web of Science databases. Various studies related to AIMM were conducted with different objectives. An evaluation is usually performed using a maturity model when organizations need to assess how far they are becoming an “AI-first” organization (Gemmink, 2019). A number of AIMM were developed to specify the degree of preparedness to benefit from AI in terms of improving organizational operations and processes, introducing new AI technologies, linking to the state of the company and its mindset, and estimating the performance of the organization enabled by AI technology (Saari, Kuusisto & Pirttikangas, 2019). Besides, these models must be tested for rigor and thoroughness to ensure their ability to be a well-rounded tool to reflect the current state of AI capability. This acted as a motivation for the following research questions:

RQ1: What are the main objectives in the literature on maturity models in the AI domain?

RQ2: What are the research methods used in the literature on maturity models in the AI domain?

RQ3: What is the scope of maturity models in the AI domain?

RQ4: Which design approach, typology, architecture, purpose of use, and components are used to design maturity models in the AI domain?

Protocol and training

A strategic search strategy was used for analyzing and extracting relevant data. The protocol defined the criteria for inclusion, exclusion, keywords, the source databases, and quality assessment to ensure accuracy and consistency across the reviewers and papers. This step is necessary, regardless of the number of reviewers working on a study.

Searching for literature

During the search process, eight academic digital sources were selected to extract relevant studies from the literature. Expert advice and prior systematic analysis were considered in this study to determine the database sources to be used. These databases cover the latest reliable literature topics and studies across all domains. The seven bibliographic databases considered are Scopus; Web of Science (WoS); Dimensions.ai; SpringerLink; ACM Digital Library; IEEE Xplore; and Pro-Quest. Another source, the academic search engine, Google Scholar has also been used since it is still the most comprehensive source (Martín-Martín et al., 2021). For the search string, the keywords used were “Artificial intelligence”, “AI” and “maturity” to search for relevant papers. During the process, Boolean operators were utilized to gather literature that is as pertinent as possible. To narrow down and retrieve relevant results, “AND” and\or “OR” operators were used for databases and the search engine. In SpringerLink and Pro-Quest, the “NEAR” operator was used, which is equivalent to “AND”.

The same approach is used to find articles where the keywords joined by the operator are within a certain number of words of each other. Since each search engine has its own way to work (e.g., search field and operators), each search string was changed to find the preferred results without reducing the validity of the research by missing relevant references. The changed search string was represented that previously defined in the SLR protocol. For this reason, in ACM Digital Library and IEEE Xplore, the searching process was conducted in abstracts (the most suitable choice) due to the limits of these search engines’ choosing and the topic of research always be mentioned in the abstract. This study used the default number of words describing the allowed separation (10 in SpringerLink, 4 in Pro-Quest). Using this helpful operator in the two databases is due to the popularity of AI as a topic, and there are numerous studies related to it. In addition, “maturity” is a widespread word, and many studies use it as it is generalizable across many domains. Besides that, the two latter databases contain books, theses, and dissertations that consist of mostly a hundred pages, which is costly in terms of time and effort. Therefore, by using “NEAR”, only the relevant papers would be retrieved from the databases. On the other hand, besides “NEAR” and due to its functionality, the “OR” operator is included to ensure that we did not miss any important research that our mapping is seeking to address. In Google Scholar, the number of results was huge, and a high proportion of them was not relevant. Perhaps because it indexes the entire text of every article or recall papers that discussed only the main concepts in the full text (Martín-Martín et al., 2021). Thus, to minimize the results by retrieving only the relevant papers, the search string included “OR” operator. On the contrary, in the rest of the databases, the “OR” operator retrieved irrelevant sheets, which it became necessary to delete because the AI is short for other words like agile. All the academic literature was selected for the publication types, including journal articles, conference papers, book chapters, dissertations, and theses. Therefore, to extract only the relevant materials, the search string was modified with the exact phrase. To define the state-of-the-art papers, only published papers after 2015 until September 2020 and written in English were selected. The databases and the search engine with their queries are listed in Table 2.

Sample description

Three aspects of the main demographic statistical results from the papers are discussed, namely, sources, publication types, and the year of publications, as shown in Figs. 2 and 3. The distribution over databases and Google Scholar discovers that Scopus held most of the publications on this topic. The literature search performed in this review started in 2016, showed that no paper on this topic was published in 2016 and 2017. The findings in this review is summarized in Figs. 2 and 3.

Research objectives and methods

The studies are classified into two categories, namely, analytical and empirical papers. Our analysis reveals that half of the studies approximately performed validation for the maturity models. There are seven conceptual studies wherein no testing or application of the existing models was performed. In terms of model validation, seven out of the 15 models were validated using various empirical research approaches. The validation phase is important to ensure the proposed model’s comprehensiveness, consistency, and problem adequacy (Becker, Knackstedt & Pöppelbuß, 2009). Concerning the validation, only two article used a quantitative survey method, with more than 80 respondents based in Finland. The results show that many organizations have only just started exploiting AI in their organizations. Only three papers out of seven implemented qualitative methods. Jaaksi, Koskinen & Jalava (2018) conducted interviews with 11 experts to validate the model. The highest proportion of validation approaches are mixed methods, such as in Coates & Martin (2019), who used a mixed research method by performing a quantitative survey followed by case studies which analyzed the results quantitatively and qualitatively. Ellefsen et al. (2019) combined survey instruments with multi-case studies that were conducted through interviews. Gentsch (2018) analyzed the content available on the internet of three organizations, with an extremely high level of automation. Regarding the development phase, the majority of the papers used qualitative methods (e.g., action research or case studies), but the dominant method is content analysis. The “Application” column is empty, which means no research has applied any existing AIMM.

Main characteristics and components of AIMMs

This section discusses the main characteristics and components of AIMM. Wendler (2012) indicated that not all the selected research had a clear definition of the term “maturity model” and this is also seen in our chosen studies. Only two studies out of 15 reviews contain different definitions of the term. The variation of the definitions proves that there is no consensus on a unified definition of the term ‘maturity model’. The definitions of the maturity model in our findings studies are (i) an essential tool for describing existing production capabilities (Lee, 2020) and (ii) a measurement tool to evaluate the capabilities of an organization (De Bruin et al., 2005). Our findings were analyzed according to the classification in Table 4.

According to the components, the elements in the maturity models under study have different designations, including “dimensions” (Lamberti et al., 2019), “constructs” (Mikalef, Fjørtoft & Torvatn, 2019), “elements” (Gemmink, 2019), “indicators” (Ellefsen et al., 2019), and “process areas” (Kumar, 2017). These findings reveal that these elements have different naming, and no standard terminology is available. This is in line with Poppelbub & Roglinger (2011), who reached the same conclusion. These elements are considered the heart of any maturity model, because their different capabilities represent the maturity levels. However, not all the selected models illustrated them clearly. Four or five levels are indicated in two models, without information about the dimensions used for evaluation. The authors define the principal components associated with each maturity model. The 15 research studies under review are explained briefly below in alphabetical order of the title. The information in all tables has the same order to simplify reading and comparison.

Study 1: Gentsch (2018) aimed to develop an AIMM for the marketing domain based on the AI business framework. Four levels of maturity, from the non-algorithmic enterprise to the automated enterprise, and finally super intelligence enterprise, the model showed the various stages of development regarding data, algorithms, and AI. Although the author included the “superintelligence enterprise” level as the highest maturity level, he indicated that it is difficult to reach this level at present. The model contained five dimensions: strategy, data, analytics, people/Orga, and decisions. However, there is no information about how these dimensions are evaluated to measure maturity.

Study 2: Kreutzer & Sirrenberg (2020) described an AI maturity map to determine the extent of existing AI and provide a path to start the AI journey in an organization. The analysis included two parts: AI basics and AI fields of application. Each part was evaluated separately by percentage value (a range from 0–20% to 80–100%). The four dimensions of AI basics should be evaluated for all organizations. Although the areas of application identified in this study were customer service, marketing/sales, service delivery, and production, it could decrease the current areas or include others like maintenance, human resources, and related fields based on the organization’s focus. However, dimensions are not described at each level and there is no information about how these dimensions are evaluated to measure maturity.

Study 3: Saari, Kuusisto & Pirttikangas (2019) developed a free web tool to assess AI capability in a company context. The tool was developed by AI Accelerator (FAIA) in Finland, where the AI maturity index was first published in Finnish. This tool can be employed to evaluate the maturity in an organization to improve the understanding of the AI maturity level in different departments of the organization and possible areas of development, which is a valuable gauge before any development projects are launched. The model has six dimensions: strategy and management, products and services, competencies and cooperation, processes, data, and technology. The assessment method is carried out through questions, with five response alternatives to which each dimension has two questions. However, dimensions are not described at each level.

Study 4: Coates & Martin (2019) developed a new free self-assessment instrument reflecting the ethical values by actionable methods that would assess organization maturity to control AI bias efficiently. The instrument was designed to evaluate the project through two stages: Design and Development (during the project development) and Post-Development (after finishing the project’s development). During the first stage, the evaluation can be performed multiple times, with the expectation that companies can increase their maturity after each sprint. The constructs used in the design and development phase are business, people, user, data, algorithm, and compliance, and for the post-development phase are business, data, test, client feedback, and transparency. The maturity gain is a score as the sum of the relevant question’ answers, divided by the maximum possible score, to give a proportion of performance independently. However, there is no evaluation of maturity levels after the scores to understand the level reached.

Study 5: Abele & D’Onofrio (2020) suggested an AI maturity to assess intelligence systems’ intelligence capabilities. It provides a way to assess the level of technology growth in relation to one another or measure performance in a specific context. The model involves a two-dimensional classification where the x-axis represents the ability to deal with uncertainty, while the y-axis reflects the capability to solve problems adaptively. The adaptive method solves problems and shows the ability of a system to be applied in totally different contexts. The model shows that AI systems could be classified into five independent discrete categories. But there is no explanation on how to evaluate these systems.

Study 6: Tu, Lin & Lee (2019) proposed a metadata-driven automation maturity model for a clinical research company. The model intends to help these companies evaluate their processes’ intelligent automation depending on their integration, efficiency, and intelligence. Three indicators were used to assess the maturity, which are: standard adoption (SA), code reusability (CR), and process repeatability (PR). All indicators’ capabilities were described using nine maturity levels.

Study 7: Bangalore Seetharam (2020) presented a digital AI maturity ladder for retail organizations in different industries. The ladder was developed to enable them to know the current state of AI and offer a guideline to increase AI adoption against time, depending on the organization’s ambition. The ladder consists of ten steps with four drops (stages), whereby each step defines the subsequent progression in becoming an AI-driven organization and gaining better value. Drop 2 is considered the start point to embark on AI. There are generic descriptions for drops with descriptors, but the capabilities used to evaluate the drops were not clarified. The author mentions that the evaluation was conducted through interviews.

Study 8: Lichtenthaler (2020) proposed a conceptual framework with three elements used to construct the five levels of maturity to manage AI in a company and helps to recognize the managerial challenges and significant resource limitations of organizations. The three elements are various types of AI, various types of human intelligence, and the meta-intelligence to guarantee that the final architecture is more than the aggregate of the different types of intelligence. This framework was built on the grasp of intelligence architecture based on the concept of Integrated Intelligence and the intelligence-based view of company performance.

Study 9: Jaaksi, Koskinen & Jalava (2018) introduced a model that assesses the current maturity level of AI adoption in organizations to address the requirements of AI transformation. The result indicated that the most important five dimensions were correlated with the AI maturity of the company. Feasible quality and quantity of the data to be used to implement the designed solution, workers who have the expertise and skills to create the designed solutions internally or successfully. The state of the organization with the current implemented solutions and the organization’s mentality to adopt AI solutions. All capabilities of dimensions were indicated in four levels with detailed descriptions.

Study 10: Lee (2020) proposed a maturity capability assessment method of crucial technologies in Industrial AI. The objective is to create a thorough assessment of an enterprise’s level of Industrial AI development to provide a foundation for its decision-maker and motivate to improve and address its deficiencies. By matching the four elements with the intelligent transformation stages, the level of industrial AI maturity would be determined. The assessment is based on four technologies, namely, Data (DT), Analytics (AT), Platform (PT), and Operation (OT) technologies. He explained each technology’s capabilities separately in several maturity stages (all have four stages, except Analytics, which has five stages). However, the author did not explain the procedure to evaluate these capabilities.

Study 11: Yablonsky (2019) used a previous AI automation framework and AI/Big Data (BD)/Advanced Analytics (AA) value framework to produce a multi-dimensional data-driven AI enterprise innovation maturity framework to assist with strategic innovation planning and AI-based automation investment decisions. The framework concentrates on data-driven human-machine relationships and implementing AI at five levels of data-driven automation maturity scope, from activities and tasks to AI processes and platforms. The evaluation method to determine the level does not exist.

Study 12: Burgess (2018) suggested an AI maturity matrix and AI heat map to draw the path for an organization to initiation the AI journey, and offers an opportunity to identify the roles and opportunities of AI palpably and automation across the business which are acceptable and technically and economically feasible. Four criteria in the AI heat map (Strategic objectives, Existing challenges, Benefits, and AI capabilities) were used according to their processes in areas (customer service, organization, operations, and finance). All six maturity levels, from level zero to level five were defined and each level comprises related practices for a predefined set of process areas that enhance the performance of an organization. The evaluation process could be determined through interviews and evidence analysis.

Study 13: Ellefsen et al. (2019) presented an AIMM framework to measure the AI readiness of logistics companies. The authors developed a survey instrument (questionnaire) to understand and evaluate the current situation of existing solutions of Logistics 4.0 and the implementation of AI in logistics companies. The four maturity levels of AI maturity for communication service providers (CSP), produced by Ovum, were used for AI levels evaluation. The results revealed that all the selected companies were still at a novice level.

Study 14: Seger, Miailhe & Mueller (2019) proposed an AI maturity framework for Intergovernmental Organizations (IGO). The framework focuses on executives use, contains five levels of maturity. However, no information about the dimensions or how to apply this framework for evaluation was provided.

Study 15: Alsheibani, Cheung & Messom (2019) introduced a conceptual model to evaluate the AIMM to helps an organization to have a vision to effectively develop and implement AI. the dimensions of the model are: AI functions, data structure, people, and organization, to measure maturity. All capabilities were indicated using five levels with a particular descriptor.

This review proves that maturity models in the AI domain were developed with different components. Table 5 reveals that the AIMM has different designs regarding the number of levels, descriptors, and elements. Various descriptors were identified from the 13 models under study, which differ from one model to another. The elements column discovers different designations and that there is no standard terminology. But the majority used “dimensions”, which is mentioned in seven studies. Only three studies are without element identification.

The main scope of the AIMMs

This section involves answering the third research question: What are is the scope of maturity models in the AI domain? The SLR identified five AIMMs that were developed to be applicable in a generic manner. The other 10 models were created with a particular focus on a specific domain or purpose, as shown in Table 6. Five studies developed models for particular domains, while eight studies developed models for specific purposes. Two of 10 studies strived to evaluate the intelligent automation process. The first one was conducted by Tu, Lin & Lee (2019) to develop a model depending on the environment of clinical research to evaluate intelligent automation. Second, Burgess (2018) developed a model to evaluate automation maturity that focused on AI tools essentially, either at the company level or specific area.

On the other hand, the AI maturity ladder was developed by Bangalore Seetharam (2020) for retail organizations. Ellefsen et al. (2019) developed a framework for evaluating AI in logistic companies. Regarding specific purposes, Coates & Martin (2019) developed a tool to evaluate AI bias governance in an entire projector company, while Yablonsky (2019) developed a framework to evaluate AI innovations as examples. Furthermore, two models were developed to evaluate the intelligence in the systems. Abele & D’Onofrio (2020) introduced a model to assess the intelligence systems based on their intelligence capabilities, while Lee (2020) developed a model to evaluate the technical aspect (AI capabilities) and provide a basic judgment about an enterprise’s intelligent transformation stage.

This review reveals four levels of analysis with a scope: a company, process, project, and system levels. One study involved analysis within the scope of the project level (Coates & Martin, 2019). Six studies (Bangalore Seetharam, 2020; Burgess , 2018; Lee, 2020; Lichtenthaler, 2020; Tu, Lin & Lee, 2019; Yablonsky, 2019) assisted organizations in assessing the single process at the process level, while only one study focused on the system level (Abele & D’Onofrio, 2020). For example, AI management (Lichtenthaler, 2020) assists an organization in achieving an integrated intelligence architecture. At the same time, a data-driven automation framework (Yablonsky, 2019) enables an organization to concentrate on the relationship between data and human-machine interactivity and the implementation of AI at different levels of data-driven automation maturity scopes. The SLR listed 10 maturity models with a company-level scope to evaluate AI maturity of the entire company. The SLR provides evidence that most of the maturity models were developed to be applicable for a particular focus on a specific domain or purpose. Additionally, most maturity models had a scope at the company level, followed by the process level.

Design, typology, and architecture of the AIMMs

This section explores issues related to the models’ designs to answer the last research question. Not all of the studies indicate the design approach of the maturity model. Just four studies followed the bottom-up approach. No studies used a top-down approach. According to the topology, three studies followed Likert-like questionnaires as part of a quantitative approach. At the same time, one study (Burgess , 2018) used a structure like CMM to evaluate automation. Most studies used the qualitative description assessment approach (five studies followed the matrix grid). The dominant form to represent maturity is continuous representation. Only two studies used stage representation.

The final issue discussed in this section involves the benefits and purposes of use. The benefits of using maturity models could be descriptive, prescriptive, or comparative (Becker, Knackstedt & Pöppelbuß, 2009; De Bruin et al., 2005; Mettler, 2011; Poppelbub & Roglinger, 2011). The descriptive approach is used to evaluate the current situation of a domain as it is. The prescriptive approach is used to give a roadmap or vision to improve the AI in an organization. Finally, the comparative approach provides a comparison within an organization among a series of organizations. Poppelbub & Roglinger (2011) identified multiple purposes for using the same model during their analysis. This review also identified multiple purposes of use for most of the models, based on the information mentioned in the studies. For example, Gentsch (2018) explained that the model could be used to evaluate the current status with regards to AI, algorithms, and big data (descriptive), as a guideline to the next maturity level (prescriptive), and compare similar practices between organizations to evaluate maturity in different sectors (comparative). Saari, Kuusisto & Pirttikangas (2019) indicated that the final graph would compare an organization, among organizations, and with a reference group if there are sufficient respondents (comparative). In addition, it is used as a self-assessment tool (descriptive), and its result helps the organization identify the most critical areas for improvement (prescriptive). Alsheibani, Cheung & Messom (2019) indicated that AIMM could be used to assess current AI capability (descriptive), offer managers with insights to adopt AI solutions (prescriptive), and benchmark against other advanced organizations to develop their AI capabilities (comparative) gradually. Most AIMMs have a descriptive characteristic (13 articles), but there is a lower amount that use the prescriptive purpose (only six studies). In contrast, comparative approaches are largely unmet (only three papers). All the previous information is shown in Table 7.