- 1SSDIA, ENSET, Department of Mathematics & Computer Science, Hassan II University of Casablanca, Mohammedia, Morocco
- 2HUman-MAchines INteraction (HUMAIN) Lab, Department of Computer Science, International Hellenic University (IHU), Kavala, Greece
Robots are more and more present in our lives, particularly in the health sector. In therapeutic centers, some therapists are beginning to explore various tools like video games, Internet exchanges, and robot-assisted therapy. These tools will be at the disposal of these professionals as additional resources that can support them to assist their patients intuitively and remotely. The humanoid robot can capture young children’s attention and then attract the attention of researchers. It can be considered as a play partner and can directly interact with children or without a third party’s presence. It can equally perform repetitive tasks that humans cannot achieve in the same way. Moreover, humanoid robots can assist a therapist by allowing him to teleoperated and interact from a distance. In this context, our research focuses on robot-assisted therapy and introduces a humanoid social robot in a pediatric hospital care unit. That will be performed by analyzing many aspects of the child’s behavior, such as verbal interactions, gestures and facial expressions, etc. Consequently, the robot can reproduce consistent experiences and actions for children with communication capacity restrictions. This work is done by applying a novel approach based on deep learning and reinforcement learning algorithms supported by an ontological knowledge base that contains relevant information and knowledge about patients, screening tests, and therapies. In this study, we realized a humanoid robot that will assist a therapist by equipping the robot NAO: 1) to detect whether a child is autistic or not using a convolutional neural network, 2) to recommend a set of therapies based on a selection algorithm using a correspondence matrix between screening test and therapies, and 2) to assist and monitor autistic children by executing tasks that require those therapies.
1 Introduction
About 15% of the world’s population lives with some form of disability, according to the World Health Organization (WHO) (World Health Organization (WHO), 2020). Of these, 2–4% have significant functional difficulties and need assistive technology such as low-vision devices, wheelchairs, and or hearing aids. They are especially critical when a person’s quality of life decreases and their independence is reduced.
In recent years, understanding disability has shifted from a physical or medical perspective to one that considers a person’s physical, social, and ethnic political context (Altman, 2014). Today, disability is understood to be the result of the interaction between a person’s state of health or impairment and the multitude of factors influencing their environment (Field et al., 2007). Significant progress has been made in making the world more accessible to people with disabilities; however, much more work is required to meet their needs especially in psychology; because of the important role in diagnosing ASD, that can be done frequently through careful and proficient examination of medical reports, behavioral history, and interviews with friends and family to obtain more detailed and valuable information about interests, activities, and attitudes, as well as intolerances of the child or adolescent.
Advances in technology can play an important role (Du Toit et al., 2018), as they can enable people with disabilities to receive the health care necessary to lead full and independent lives. Integrated artificial intelligence (AI) researches have allowed us to explore the use of robots in different health care settings, where robots can be deployed as tools for intervention and rehabilitation programs. Robots represent a growing clinical interest in mental health services (Georgescu et al., 2019). With innovations ranging from “virtual psychotherapists” (Riva and Gaggioli, 2008; Pdxscholar and Thompson, 2016) to social robots in the treatment of dementia and autism disorders (Chu et al., 2017; Bagheri et al., 2020), artificially intelligent virtual and robotic agents are becoming increasingly popular. High-level therapeutic interventions (Fiske et al., 2020), which were previously offered exclusively by highly qualified health professionals are now performed by robots. The ethical and social implications of the increasing use of incorporated AI in mental health need to be identified and addressed to enable responsible clinical implementation (Burr et al., 2020).
There has been much interest in using social robotics to improve joint attention in children with autism spectrum disorder (ASD). In a pioneering research study, children diagnosed with ASD showed improvements in their social behaviors when interacting with robots, and this was reflected by an increase in both imitation and shared attention (Cao et al., 2020). A key aspect of their use in therapy is to involve the children in extended therapy sessions and thus keeping them focused on specific physical and social tasks (Wainer et al., 2014). A reason for which children with ASD will typically found engagement with robots stimulating is that they are simple (both in appearance and behavior), predictable, and nonintimidating agents, as humans can be in social complexity (Marchetti et al., 2020). An additional pilot study (Conti et al., 2015) involving three children who had ASD and an intellectual disability explored the efficiency of an NAO robot intervention to enhance children’s body imitation.
This paper proposes a social robot NAO that will help therapists diagnose children with autism spectrum disorders and prescribe a set of therapies according to their particular needs. This robot can deduce if a child suffers from autism by analyzing his picture, using a convolutional neural network which is the most used algorithm in the field of image processing, this algorithm will be applied to a database containing pictures of children with ASD. Then, screening tests are performed to assign an appropriate therapy plan. The robot is autonomously supervised, this means that it is independently working to reach a given therapeutic goal under the supervision of a human therapist who can, if necessary, modify the robot’s actions before they are executed to ensure that only valid therapeutic actions are performed. An ontological knowledge base is used to annotate all the information about the patient, the therapies and, and all the processes. This study applies the convolutional neural network-based algorithm on an ontological knowledge base containing data from different case studies. We start with a review of the literature and follow a discussion of the framework of the methodology we used to equip the humanoid robot to learn skills autonomously and interact with struggling children. These skills can be used to help the robot act as a therapist and prescribe therapy depending on the severity of the disease. Then we moved on to the results, discussions, and the deployment environment to test and validate our algorithms. And finally, we conclude some recommendations for future studies.
2 Related Works
The increasing availability of innovative advances in assistive technology can significantly improve people’s quality of life with disabilities. This section discusses the state of the art in the use of Social Assistant Robots for developmental and clinical purposes, considering all major phases of psychological treatment of ASD. We will present the role of these technologies according to the world classification of functioning. Next, we will present some related work on social robotics and its use for assisting children with autism spectrum disorders.
2.1 Autism Syndrome Disorder
Autism Spectrum Disorder (ASD) is a group of neurodevelopmental disorders that are present at birth and whose first manifestations may appear in early childhood (Ghosn, 2021). They affect the acquisition, assimilation, and or application of skills. They may involve dysfunction in attention, memory, perception, language, and social interaction (Hyman et al., 2020). These symptoms can be more or less present and even evolve. The concept of the spectrum allows us to integrate all the disorders and to define the combination of characteristics that the patient possesses since cases differ from one person to another (Mottron and Bzdok, 2020). The goal is to identify the strengths and weaknesses of each individual and recommend appropriate interventions. ASD is not curable, as it is not a disease, but a neurodevelopmental condition (Bölte, 2014). Nevertheless, there are ways to help the affected child. While difficulties in social interactions and communication characterize autism spectrum disorders, early interventions such as speech therapy, and behavior therapy can improve young people’s development with autism (Landa, 2018; Kasilingam et al., 2019; Kolaski, 2020). These interventions are often expensive, but social assistance robots could help the therapists, making treatment more affordable (McCreadie and Tinker, 2005; Elbeleidy et al., 2021).
2.2 Assistive Technologies
Assistive technologies refer to any article, piece of equipment, software program, or product system used to increase, maintain, or improve people with disabilities (Carver et al., 2016; Federici et al., 2017). it enhances the ability of a person with a disability to participate in daily living activities and perform tasks that would otherwise be difficult or impossible for them to complete (World Health Organisation, 2007; Giesbrecht et al., 2017). The principle of enhanced capacity includes an increased level of independent action, reduced time spent on daily living activities (Portugal et al., 2019), more choice of activities, and greater satisfaction in participating in activities (ISO/TC 173, 2016).
According to the International Classification of Functioning (Pazzaglia and Molinari, 2016), which uses disability as a term covering activity limitations, impairments, and participation restrictions, assistive technology aims to reduce limitations and impairments and promote full participation in the activities of life. In this context, technical assistance devices, or so-called Embodied assistive technology. They include those that improve structure and function (Livingstone et al., 2020), such as mobility devices (e.g., prosthetic legs, cochlear implants, and electronic implants for bladder control) (Azocar et al., 2018; Aiordachioae et al., 2020). And those that improve activity performance (e.g., voice input systems, wheelchairs for riding stairs, and communication panels) (Sasaki et al., 2020; Smith, 2020; Patel and Parmar, 2020). Specialized assistance systems (e.g., vision, hearing, cognition, and communication) (Bubar, 2020; Desideri et al., 2020; Kisanga and Kisanga, 2020; Nganji et al., 2011), as well as hardware and software resources that are software and hardware, that facilitate access to information technology (e.g., computers, mobile devices). Environmental modifications or so-called Assistive environments (e.g., automatic door openers, walk-in entrances, and accessible bathrooms) (Guay et al., 2020; Martinez-Martin et al., 2020; Hu and Hu, 2021), which reduce or eliminate restrictions on participation, are also considered types of assistive technology.
The expression of disability changes with the affected person’s environment; therefore, assistive technology devices are considered part of the environment, reducing the term of disability. They can be used, for example, to build accessibility, to increase communication, to allow access to a computer, to enable environmental control of electronic devices, to modify houses to access them, to facilitate activities personal care and family activities, to improve mobility, to stabilize seats, and to modify workplaces and schools. Nevertheless, this type of technology does not support an independent life when the user has chronic or degenerative motor limitations or cognitive abilities (Hu et al., 2020).
Assistive robotics (AR) made its appearance by integrating robotics with artificial intelligence and machine learning techniques to respond to this problem and with the primary objective of successfully promoting the well-being and autonomy of people with disabilities, taking into account societal needs. Ongoing research has concluded that Robots can assist the majority of tasks and activities of daily living, as is the case with household robots (Qian and Bi, 2015), and rehabilitation robots (Feil-Seifer and Matarić, 2005).
2.3 Social Assistive Robots
One of the main challenges in accepting assistive technology is the way it is perceived. In this sense, the interaction between the robot and the user is a crucial issue. This social interaction led to the development of social assistance robotics (SAR). In (Martins et al., 2020; Narasima Venkatesh, 2020), SAR can be defined as the intersection of AR and socially interactive robotics (SIR), whose main task is the interaction with human individuals. In recent years, social assistance robots have been tested. In 2016, IBM created the MERA (Multi-Purpose Eldercare Robot Assistant) prototype (SoftBank Mobile, 2020), based on the Pepper robot developed by SoftBank Robotics in Japan in 2014 (Arsovski et al., 2019). MERA can analyze videos of a person’s face and measure vital signs like heart and respiratory rates. It can also answer basic health-related questions and determine if an individual has had a fall (Detect Autism from a Facial Image, 2021).
3 Research Methodology
The robot should have task-oriented social behavior to reach the therapeutic goals, it should be supervised by human therapists to ensure safe and ethical behaviors, and it should provide and analyze data (e.g., user performance and performance history, robot operation) recorded in structured formats to different parties. In this section, we first present a therapeutic scenario led by a robot to diagnose a child with ASD. Then, we will model the knowledge database that the robot will use to react in each therapy session. Subsequently, we will present the method of classifying children, selecting therapy, and planning the therapy.
3.1 Modeling of General Knowledge
The representation of Knowledge consists, in a first approach, in the definition and hierarchization of the components of said Knowledge, which itself can be attached to an individual, a group (a community), or to the entire population to pass from raw information, of which the web constitutes, and an inexhaustible source to Knowledge. It is necessary to go through cognitive processing of sanitization (attribution of the meaning) and synthesis. The second approach, oriented towards practice, is schematized by the Semantic Graph: a set of elements of which specific pairs, or multiples, are directly linked by so-called semantic links. The links are oriented and qualified, in particular by a tree structure and a predefined ontological definition. A Profile element, which represents the fundamental element to which a set forming a “Knowledge” is attached, determines the origin (manual, synthetic), as well as the editing levels authorized on the links. Overall, the design (General) consists of documentation work (research, collection, organization of information). In this sense, we have tried to give a general presentation of a situation, where a robot therapist is in direct contact with a child (Figure 1), it can record all the information about him by invoking a conversation, and then store this information in an ontological database explained in Section 3.3. The first step is when the robot will detect if the child is ASD positive by analyzing facial features in his picture using an ASD detection algorithm, this is mentioned in Section 3.4. The used algorithm is trained on the database presented in Section 3.2. The second step consists of doing screening tests to better detect disorders the child suffers from, and then choose the priority list of therapies that can be performed to improve his condition, a detailed description is in Section 3.5. Finally, the third step is the planning of therapies followed by their validation to give feedback to the human therapist who will make the final diagnosis of the child’s condition.
FIGURE 1. Overview of the therapy procedure with robots for children with ASD.
3.2 Database Used: Structures and Metadata
The objective is to provide a primary diagnostic that can help the robot therapist decide whether the child has ASD. For this, we used version 9 of the Kaggle database “Detect Autism from a facial image,” (de Mello and de Souza, 2019) containing 2,536 images divided equally between two classes, either autistic and non-autistic; this is in three folders, learning, test, and validation. The validation images are located in a valid directory. It is also separated into 50 images of autistic children and 50 photos of non-autistic children in the same format as for the training set. The second way the data is provided is in the consolidated directory. This directory also contains the two subdirectories Autistic and Non_Autistic. It represents the consolidation of the train’s files, test, and valid directories into a single set. Users can then partition the consolidated data into their train, test, and validation sets. The pictures vary in size and do not show the child’s face—the children in the age distribution of about 2 years to 14 years. Gender relations are close to their respective populations. Men are diagnosed with autism 3 times more than women. Thus, the distribution of male to female images in the autistic class is close to 3 to 1. In the non-autistic class, the ratio is much closer to 1 to 1.
3.3 Ontological Knowledge Database for Therapy Automation
Psychotherapy helps patients become aware of and change their behavior in the face of immediate emotional conflict and initiate the transformation process through listening, observation, awareness, and interventions (Janssen et al., 2019). And with the rapid development of new types of sensors, memory banks, computers, and electronic control devices, automation has been introduced in many diverse human activity fields (Ogg and Rašticová, 2020). In industrial production, its use brings about a radical change in the existing patterns of working life (Lippi and Da Rin, 2019). It completely changed the scope and speed of mathematical analysis and information research. Attempts are currently underway to apply automation to certain aspects of medical diagnosis and hence to psychotherapy procedures (Cameron et al., 1964; Goldberg et al., 2020; Casas-Bocanegra et al., 2020).
The ontological model of the assistive robot therapist is shown in Figure 2. Ontology tries to combine all clinical concepts to the elements of interaction to provide a model containing both parts; patient and robot; in therapy. The ontology represents all the information contained in the basic knowledge, about the patient information, the screening tests, the set of interventions, and therapies; used in the architecture. We consider all the actions that the robot can perform in each exercise, and therefore in each therapy, and any unexpected situations. These actions can be gestural or sound. The robot control engine is the decision support component. The latter uses the domain and receives the problem with the adapted exercise set up to provide a valid action plan that meets the learning objectives.
FIGURE 2. The ontological model of a robot therapist.
3.4 Classification of Patients Model and Detection of Symptoms
When the robot is in direct contact with the child, it will need information on its profile, some of which can be collected from the interaction protocol. This information is stored in three databases. These are databases of questionnaires, behaviors (images, videos), and physiological signals. The robot will need a machine learning mechanism to decide if the child has ASD.
A recent study about autism has shown that children with autism have a distinct facial structure that differs from that of typically developing controls; We chose to use convolutional neural networks because they are the most efficient models to classify images. This algorithm is different from a neural network because it works on an input volume. It is not necessary to extract features. In this case, the system learns to do feature extraction and uses image convolution and filters to generate invariant features passed to the next layer. Each layer tries to find a useful pattern or information about the data. Figure 3 shows what the architecture of a CNN (Lin et al., 2021) looks like.
FIGURE 3. Feature extraction and classification.
In the symptom detection part, it will be essential for us to choose the appropriate therapy for each child. To do this, we will use a community detection algorithm (McGuire, 1983), which we have developed in our related work. The goal is to classify patients with autism into three communities (Figure 4) that’ll designate the disease’s severity (low, medium, and high).
FIGURE 4. The child classification and symptom discovery model.
3.5 Therapy Selection Model
The selection can be made based on the three resulting communities; therefore, we can predict individual responses to different therapy options using various patient data and recommend which therapy is believed to provide the best result. For this, we will need a clinical database integrating patients suffering from ASD disease and the therapies proposed for each. This phase will also allow us to adjust therapies to suit the patient’s needs (Gutierrez and Sequeda, 2021), as shown in one of our previously published papers. And therefore, to offer therapy to a child. You just have to look for the community to which he belongs to decide which therapy will benefit him. Another method based on screening tests can be performed; based on the routine tests that therapists do to facilitate diagnosis. Another method based on screening tests can be performed (Albawi et al., 2018); based on the routine tests that therapists do; to facilitate diagnosis. This method was designed after extensive research work on screening tests and prescribed interventions for each set of disorders.
3.5.1 Psychometric Tests and Functional Assessment Tools
There are screening and assessment tools that can be used to identify warning signs in children with autism (Gutierrez and Sequeda, 2021). The best known and most widely used is the M-Chat test (Modified Checklist for Autism in Toddlers) (Javed et al., 2018). This questionnaire focuses on the child’s behavior and can be administered as early as 18 months of age. The tests and assessments to establish a diagnosis will come later. Functional assessment tests and tools help establish a diagnosis of Autism Spectrum Disorder. Each test provides an assessment of communication, social and behavioral skills, depending on the age of the candidates, whether they are children, adolescents, or adults. Table 1 represents all psychometric tests by age group. They are used to examine behaviors and skills.
TABLE 1. Psychometric tests for all age ranges.
3.5.2 Interventions Used for ASD
As soon as an autism diagnosis is made, children, and adults must be able to benefit from adapted interventions. These interventions aim to improve the quality of life of people with ASD through actions that promote communication, learning, and socialization while reducing behavioral problems. Table 2 shows us the list of therapies used for ASD.
TABLE 2. Interventions Used for ASD based on troubles type.
3.5.3 An Algorithm of Description the Set of Therapies
After having compared the two preceding tables, we were able to construct a matrix of correspondence between Disorder and therapies, that would allow us to deduce the therapies that must be prescribed for each autistic patient according to his case (Table 3). The columns of the matrix represent screening tests by age group; for children, for adults and adolescents, and tests for all ages. the lines of the matrix represent the existing therapies. If a test is positive, we prescribe the patient the therapies that have the value 1. For example, if the patient is a child, and if the ADI-R test is positive, we will prescribe the therapies: developmental, behavioral, and communication. Thus, we developed a therapy selection algorithm which process is described simply by Algorithme1 below in Figure 5:
TABLE 3. Disorder/therapy correspondence matrix.
FIGURE 5. Therapy selection algorithm.
This algorithm takes as input the child class CC, the age range of patient A and the correspondence matrix M, and returns as output a Sorted Map of the prescribed therapies PTM which is sorted in descending order of peer values. Each pair represents a therapy and its weight which the weight of the therapy is calculated by the number of times a positive screening test STS corresponds to this therapy the priority of therapy is given by its weight which indicates how many screening tests are required to do this specific therapy. we have as output a list of prioritized therapies. the decision will be up to the therapist to choose if the robot will proceed with the first therapy, follow all therapies or cancel a therapy.
3.5.4 Therapy Planning and Scheduling Model
As already mentioned in the previous sections, each therapy consists of a sequence of previously planned exercises (Gbedawo, 2020). However, the output of the high-level planner does not indicate the order of the exercises to be performed (Vivanti, 2020). To control the therapy, the robot uses a low-level scheduling mechanism. Figure 6 represents the robot therapy flow (Billeci et al., 2020).
FIGURE 6. The flow of therapy by the robot.
Indeed, once the robot is running, it goes into a detection position. After the patient has been detected, he will move on to the phase of downloading his profile to generate the therapy plan and the progress thereof in the latter. As mentioned in the previous sections, therapy consists of a set of exercises that constitute a sequence of perceptions to be performed. The robot controller turns on to execute its perceptions one after the other, and check if they are produced correctly by the patient, otherwise, it corrects them until the end of the sequence.
4 Results and Discussion
In this section, we present some of the results obtained by following the methodology described in Section 3. Figure 7 summarizes in some way the path that we have followed.
FIGURE 7. Input, outputs, and processing levels used by the robot therapist’s platform.
4.1 The Construction of the Classification Model Based on the CNN and the TensorFlow Framework.
In this section, we have tried to build the classification part of the children. We used the convolutional neural networks of the TensorFlow Framework for Deep Learning. The test database is “Detect Autism from a facial image” (de Mello and de Souza, 2019) which we described in the data structure section. Figure 8 shows some of the images contained in the database.
FIGURE 8. Part of the images from the database. Detect Autism from a facial image (Guay et al., 2020).
We used 100 epochs to train our model. Figure 9 shows the results of the training. Precision was used to measure the performance of the model during training. And we can see that the model achieves 86% accuracy on the validation set in epoch 100. After that, we plotted the model loss and accuracy data. Figure 10 shows the curves plotted with the designation of the best epoch. We see that our model’s best epoch is number 91, with 88% accuracy and 38% loss. We can see that we need to add epochs to know where the pattern will start to readjust.
FIGURE 9. Part of the training results (from epoch 93 to 100).
FIGURE 10. The loss and precision data curves for 100 epochs.
4.2 The Simulator Used for Model Deployment
In our research, we used the humanoid robot Nao (Figure 11) as a development platform and retrieve medical data from patients. Nao is an interactive, autonomous, and fully programmable humanoid robot manufactured by the company Aldebaran Robotics. It is mainly used for research in universities and laboratories, in education, and also with autistic children and Alzheimer’s patients. In our work, we used Nao in its 6th version which has the following features: Embedded OS: OpenNAO (embedded GNU/Linux based on Gentoo), a Middleware: NAOqi 2.8.6. The NAOqi environment is designed for simple voice interaction and includes a dialog engine, an emotional engine, and an autonomous life. We used also a Programming tool: Choregraphe (graphical programming interface to create and edit movements and edit interactive movements and behaviors) and a Programming language: Drag&Drop, C++, Python, and Java.
FIGURE 11. Nao specifications.
4.3 Integration and Tests
The contextualization of the process from the detection of autism to the selection of therapies has been put in the form of a Choregraphe program which is shown in Figure 12. This behavior is defined in solitary mode and starts only when a person is in the field of view of the robot for at least 3 s and/or it is a preferred moment defined by the patient.
FIGURE 12. Choregraphe program for the contextualization of the process.
The organization of the boxes in the Choregraphe program follows the flowchart shown in Figure 12.
- First of all, the robot must be in “Basic Awareness” mode, which means that it can establish and maintain eye contact. It is then able to react to stimuli such as sounds, movements, face detection, or touch. From this mode, the robot can follow a face, and recognize it.
- The robot checks that the time is appropriate for the examination (e.g., it is not at night; it is not an unavailable time defined by the patient).
- The robot recognizes the face of the person it has in its visual area and checks that it is the patient. If the person is not recognized, it takes a picture of him and adds it to its database. At the same time, he asks questions to complete the patient chart and if the robot has waited 7 s without anyone appearing in its view, the application stops.
- If the patient is recognized, the robot retrieves its photo and classifies it using the CNN model (which we discussed in Section 4.1) that has been trained beforehand and stored in the robot.
- If the result of the classification is “NoAutistic” then the robot informs us that the patient does not have an autism spectrum disorder, otherwise it starts the screening tests to deduce the troubles the patient has.
- For each disorder, the interventions that can be prescribed are recorded in the form of the disorder/therapy correspondence matrix in the corresponding SQL database to define and recommend an “effective treatment”. Once the treatment is prescribed, the robot records it in the patient’s chart.
5 Conclusion
Diagnosing autism is a complicated and expensive process. Fortunately, the correlation between facial features and autism means that a model can be trained to detect it. Using this model, the results can be responsibly communicated by a robot therapist. This article has proposed a general architecture of a robot therapist that can detect if a child has autism spectrum disorder, and offered the majority of modules that this one will need to prescript therapy. The detection of children with ASD was done through the Deep Learning classification algorithm, based on convolutional neural networks, which gave an accuracy of 88% in addition, we have implemented a robot that will be used later to implement our algorithms, and interact with autistic children. In our future work, we want to readjust our model to have more precision. This will be carried out in one of the collaborating laboratories within the framework of the H2020 project of which our laboratory is part. Moreover, real clinical tests on ASD patients will be done by using the matrix of correspondence to select the set of therapies, that will allow us to collect a new database about ASD patient therapies. We will apply fuzzy logic to the correspondence matrix to have more precision. After that, we will apply a deep learning algorithm to the new database to select therapies in a very efficient way.
Data Availability Statement
Publicly available datasets were analyzed in this study. This data can be found here: https://www.kaggle.com/gpiosenka/autistic-children-data-settraintestvalidate/metadata.
Ethics Statement
Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.
Author Contributions
All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.
Funding
This research has been partially supported by the project “CybSPEED: Cyber-Physical Systems for Pedagogical Rehabilitation in Special Education” H2020-MSCA-RISE-2017. This project has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No 777720.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
The handling editor is currently co-organizing a Research Topic with one of the authors (VK), and confirms the absence of any other collaboration.
Publisher’s Note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
Aiordachioae, A., Schipor, O. A., and Vatavu, R. D. (2020). “An Inventory of Voice Input Commands for Users with Visual Impairments and Assistive Smartglasses Applications,” in 2020 15th International Conference on Development and Application Systems, DAS 2020 - Proceedings, 146–150.doi:10.1109/das49615.2020.9108915
Adrien, J. (2008). BECS: Batterie d’évaluation cognitive et socio-émotionnelle: Pratiques psychologiques et recherches cliniques auprès des enfants atteints de TED. Bruxelles, Belgium: de boeck.
Albawi, S., Mohammed, T. A., and Al-Zawi, S. (2018). Understanding of a Convolutional Neural Network. Proc. 2017 Int. Conf. Eng. TechnologyICET 2018, 1–6. doi:10.1109/icengtechnol.2017.8308186
Altman, B. M. (2014). Definitions, Concepts, and Measures of Disability. Ann. Epidemiol. 24 (1), 2–7. doi:10.1016/j.annepidem.2013.05.018
Antonio, D. W.-T. P. C. S.undefined, T. X., and undefined (2012). TEST REVIEWS Wechsler Preschool and Primary Scale of Intelligence. Fourth Edition. Alberta, Canada: Canadian of Shool psychology SAGE Publications.
Arsovski, S., Osipyan, H., David Cheok, A., and Oladele Muniru, I. (2019). Internet of Speech: A Conceptual Model. Dordrecht, Netherlands: Atlantis Press.
ISO/TC (2016). Assistive Products for Persons with Disability-Classification and Terminology COPYRIGHT PROTECTED DOCUMENT ISO/TC 173 ISO 9999:2016.
Azocar, A. F., Mooney, L. M., Hargrove, L. J., and Rouse, E. J. (2018). Design and Characterization of an Open-Source Robotic Leg Prosthesis. Proc. IEEE RAS EMBS Int. Conf. Biomed. Robotics Biomechatronics 2018, 111–118. doi:10.1109/biorob.2018.8488057
Bagheri, E., Esteban, P. G., Cao, H.-L., Beir, A. D., Lefeber, D., and Vanderborght, B. (2020). An Autonomous Cognitive Empathy Model Responsive to Users' Facial Emotion Expressions. ACM Trans. Interact. Intell. Syst. 10, 1–23. doi:10.1145/3341198
Ben Itzchak, E., Lahat, E., Burgin, R., and Zachor, A. D. (2008). Cognitive, Behavior and Intervention Outcome in Young Children with Autism. Res. Dev. Disabilities 29 (5), 447–458. doi:10.1016/j.ridd.2007.08.003
Billeci, L., Caterino, E., Tonacci, A., and Gava, M. L. (2020). Behavioral and Autonomic Responses in Treating Children with High-Functioning Autism Spectrum Disorder: Clinical and Phenomenological Insights from Two Case Reports. Brain Sci. 10 (6), 1–14. doi:10.3390/brainsci10060382
Bölte, S. (2014). Is Autism curable?Developmental Medicine and Child Neurology. Dev. Med. Child. Neurol. 56 (10), 927–931. doi:10.1111/dmcn.12495
Brentani, H., Paula, C. S., Bordini, D., Rolim, D., Sato, F., Portolese, J., et al. (2013). Autism Spectrum Disorders: An Overview on Diagnosis and Treatment. Braz. J. Psychiatry 35 (Suppl. 1), S62–S72. doi:10.1590/1516-4446-2013-S104.–S72
Bubar, A. (2020). Emergency siren Detection Technology and Hearing Impairment: A Systematized Literature Review. in Disability and Rehabilitation: Assistive Technology (Taylor & Francis).
Burr, C., Taddeo, M., and Floridi, L. (2020). The Ethics of Digital Well-Being: A Thematic Review. Sci. Eng. Ethicsspringer 26 (4), 2313–2343. doi:10.1007/s11948-020-00175-8
Cameron, D. E., Levy, L., Ban, T., and Rubenstein, L. (1964). Automation of Psychotherapy. Compr. Psychiatry 5 (1), 1–14. doi:10.1016/s0010-440x(64)80038-9
Cao, H.-L., Simut, R. E., Desmet, N., De Beir, A., Van De Perre, G., Vanderborght, B., et al. (2020). Robot-Assisted Joint Attention: A Comparative Study between Children with Autism Spectrum Disorder and Typically Developing Children in Interaction with NAO. IEEE Access 8, 223325–223334. doi:10.1109/ACCESS.2020.3044483
Cardoso, F. G. C., Formiga, C. K. M. R., Bizinotto, T., Tessler, R. B., and Rosa Neto, F. (2017). Validade Concorrente Da Escala Brunet-Lézine Com a Escala Bayley Para Avaliação Do Desenvolvimento De Bebês Pré-Termo Até Dois Anos. Rev. Paul. Pediatr. 35 (2), 144–150. doi:10.1590/1984-0462/;2017;35;2;00005
Carver, J., Ganus, A., Ivey, J. M., Plummer, T., and Eubank, A. (2016). The Impact of Mobility Assistive Technology Devices on Participation for Individuals with Disabilities. Disabil. Rehabil. Assist. Technol. 11 (6), 468–477. doi:10.3109/17483107.2015.1027295
Casas-Bocanegra, D., Gomez-Vargas, D., Pinto-Bernal, M. J., Maldonado, J., Munera, M., Villa-Moreno, A., et al. (2020). An Open-Source Social Robot Based on Compliant Soft Robotics for Therapy with Children with ASD. Actuators 9 (3), 91. doi:10.3390/act9030091
Christl, M. B. (2017). Examining the Clinical Utility of the Modified Alarm Distress Baby Scale (M-ADBB) for the Detection of Early Signs of Autism Spectrum Disorder. Sydney, Australia: University of Sydney.
Chu, M.-T., Khosla, R., Khaksar, S. M. S., and Nguyen, K. (2017). Service Innovation through Social Robot Engagement to Improve Dementia Care Quality. Assistive Tech. 29 (1), 8–18. doi:10.1080/10400435.2016.1171807
Conti, D., Di Nuovo, S., Buono, S., Trubia, G., and Di Nuovo, A. (2015). "Use of Robotics to Stimulate Imitation in Children with Autism Spectrum Disorder: A Pilot Study in a Clinical Setting," 2015 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 1–6. doi:10.1109/ROMAN.2015.7333589
Schopler, E., Reichler, R. J., DeVellis, R. F., and Dally, K. (1980). Toward objective classification of childhood autism: Childhood Autism Rating Scale (CARS). J. Autism Dev. Disord. 10 (1), 91–103. doi:10.1007/BF02408436
de Mello, F. L., and de Souza, S. A. (2019). Psychotherapy and Artificial Intelligence: A Proposal for Alignment. Front. Psychol. 10, 263. doi:10.3389/fpsyg.2019.00263
Desideri, L., Di Santantonio, A., Varrucciu, N., Bonsi, I., and Di Sarro, R. (2020). Assistive Technology for Cognition to Support Executive Functions in Autism: a Scoping Review. Adv. Neurodev. Disord. 4 (4), 330–343. doi:10.1007/s41252-020-00163-w
Detect Autism from a Facial Image (2021). Detect Autism from a Facial Image Kaggle. [Online]. Available at: https://www.kaggle.com/gpiosenka/autistic-children-data-set-traintestvalidate/metadata
Di Renzo, M. (2020). Developmental Interventions for Children with Developmental Disorders: A Literature Review Assessment of Executive Functions in Preschool-Aged Children with Autism Spectrum Disorders View Project.
DiGuiseppi, C., Hepburn, S., Davis, J. M., Fidler, D. J., Hartway, S., Lee, N. R., et al. (2010). Screening for Autism Spectrum Disorders in Children with Down Syndrome. J. Dev. Behav. Pediatr. 31 (3), 181–191. doi:10.1097/dbp.0b013e3181d5aa6d
Du Toit, R., Keeffe, J., Jackson, J., Bell, D., Minto, H., and Hoare, P. (2018). A Global Public Health Perspective: Facilitating Access to Assistive Technology. Optom. Vis. Scilippincott Williams Wilkins 95 (9), 883–888. doi:10.1097/OPX.0000000000001272
Elbeleidy, S., Shick, A., and Williams, T. (2021). “Teleoperation Interface Usage in Robot-Assisted Childhood ASD Therapy,” in ACM/IEEE International Conference on Human-Robot Interaction, 162–166. doi:10.1145/3434074.3447151
Federici, S., Scherer, M., and Scherer, M. (2017). Assistive Technology Assessment Handbook. Second Edition. New York, US: CRC Press.
Feil-Seifer, D., and Matarić, M. J. (2005). Defining Socially Assistive Robotics. Int. Conf. Rehabil. Robotics 2005, 465–468. doi:10.1109/ICORR.2005.1501143
Ferraro, F. R., Hansen, R., and Deling, L. (2018). Executive Function Index (EFI) Performance in Nonclinical Individuals with High Levels of Autistic Traits. Appl. Neuropsychol. Adult 25 (2), 149–154. doi:10.1080/23279095.2016.1263199
Field, M. J., Jette, A. M., and ofon, I. M. C. D. (2007). The Environmental Context of Disability: The Case of Health Care Facilities. (US).in America: National Academies Press
Fiske, A., Henningsen, P., and Buyx, A. (2020). The Implications of Embodied Artificial Intelligence in Mental Healthcare for Digital Wellbeing. Oxford, UK: Ethics of Digital Well-Being.
Fitzgerald, M. (2017). The Clinical Utility of the ADI-R and ADOS in Diagnosing Autism. Br. J. Psychiatry, 2, 117. doi:10.1192/bjp.211.2.117
Freeman, B. J., Del'Homme, M., Guthrie, D., and Zhang, F. (1999). Vineland Adaptive Behavior Scale Scores as a Function of Age and Initial IQ in 210 Autistic Children. J. Autism Dev. Disord. 29 (5), 379–384. doi:10.1023/a:1023078827457
Frost, K. M., Brian, J., Gengoux, G. W., Hardan, A., Rieth, S. R., Stahmer, A., et al. (2020). Identifying and Measuring the Common Elements of Naturalistic Developmental Behavioral Interventions for Autism Spectrum Disorder: Development of the NDBI-Fi. Autism 24 (8), 2285–2297. doi:10.1177/1362361320944011
Gbedawo, H. (2020). “Autism Spectrum Disorder: A Framework for Integrative Pediatric Medicine,” in Integrative Neurology (Oxford University Press), 378–401. doi:10.1093/med/9780190051617.003.0013
Georgescu, A. L., Koehler, J. C., Weiske, J., Vogeley, K., Koutsouleris, N., and Falter-Wagner, C. (2019). Machine Learning to Study Social Interaction Difficulties in ASD. Front. Robot. AI 6, 132. doi:10.3389/frobt.2019.00132
Ghosn, F. (2021). Early Signs of Autism in Infants Whose Mothers Suffered from a Threatened Preterm Labour: a 30-month Prospective Follow-Up Study. Eur. Child. Adolesc. Psychiatry 1435-165X, 1–13. doi:10.1007/s00787-021-01749-y
Giesbrecht, E. M., Smith, E. M., Ben Mortenson, W., and Miller, W. C. (2017). Needs for Mobility Devices, home Modifications and Personal Assistance Among Canadians with Disabilities. Health Report. 28, 9–15. doi:10.1109/ICORR.2005.1501143
Goldberg, S. B., Flemotomos, N., Martinez, V. R., Tanana, M. J., Kuo, P. B., Pace, B. T., et al. (2020). Machine Learning and Natural Language Processing in Psychotherapy Research: Alliance as Example Use Case. J. Couns. Psychol. 67 (4), 438–448. doi:10.1037/cou0000382
González, J. C., Pulido, J. C., and Fernández, F. (2017). A Three-Layer Planning Architecture for the Autonomous Control of Rehabilitation Therapies Based on Social Robots. Cogn. Syst. Res. 43, 232–249. doi:10.1016/j.cogsys.2016.09.003
Guay, M., Latulippe, K., Auger, C., Giroux, D., Séguin-Tremblay, N., Gauthier, J., et al. (2020). Self-selection of Bathroom-Assistive Technology: Development of an Electronic Decision Support System (Hygiene 2.0). J. Med. Internet Res. 22 (8), e16175. doi:10.2196/16175
Gutierrez, C., and Sequeda, J. F. (2021). Knowledge Graphs. Commun. ACM 64 (3), 96–104. doi:10.1145/3418294
Harkness, A. R., and Lilienfeld, S. O. (1997). Individual Differences Science for Treatment Planning: Personality Traits. Psychol. Assess. 9 (4), 349–360. doi:10.1037/1040-3590.9.4.349
Hu, M., and Hu, M. (2021). “The Nexus between Smart, Sustainable, and Healthy Buildings: Three Case Studies,” in Smart Technologies and Design for Healthy Built Environments (Springer International Publishing), 117–140. doi:10.1007/978-3-030-51292-7_7
Hu, Z., Cong, S., Song, T., Bian, K., and Song, L. (2020). AirScope: Mobile Robots-Assisted Cooperative Indoor Air Quality Sensing by Distributed Deep Reinforcement Learning. IEEE Internet Things J. 7 (9), 9189–9200. doi:10.1109/jiot.2020.3004339
Hyman, S. L., Levy, S. E., and Myers, S. M. (2020). Executive Summary: Identification, Evaluation, and Management of Children with Autism Spectrum Disorder. Pediatrics 145 (1), e20193448. doi:10.1542/peds.2019-3448
Janssen, C. P., Donker, S. F., Brumby, D. P., and Kun, A. L. (2019). History and Future of Human-Automation Interaction. Int. J. Human-Computer Stud. 131, 99–107. doi:10.1016/j.ijhcs.2019.05.006
Javed, M. A., Younis, M. S., Latif, S., Qadir, J., and Baig, A. (2018). Community Detection in Networks: A Multidisciplinary Review. Journal of Network and Computer Applications, 10, 887–111. doi:10.1016/j.jnca.2018.02.011
Kasilingam, N., Waddington, H., and Van Der Meer, L. (2019). Early Intervention for Children with Autism Spectrum Disorder in New Zealand: What Children Get and what Parents Want. Int. J. Disabil. Dev. Educ. 68(4), 521–537. doi:10.1080/1034912x.2019.1696949
Kisanga, S. E., and Kisanga, D. H. (2020). The Role of Assistive Technology Devices in Fostering the Participation and Learning of Students with Visual Impairment in Higher Education Institutions in Tanzania. Disabil. Rehabil. Assist. Technol. 0(0), 1–10. doi:10.1080/17483107.2020.1817989
Kolaski, K. (2020). Are Communication Interventions Effective for Minimally Verbal Children with Autism Spectrum Disorder? A Cochrane Review Summary with Commentary. Dev. Neurorehabil. 23 (8), 557–559. doi:10.1080/17518423.2020.1826143
Landa, R. J. (2018). Efficacy of Early Interventions for Infants and Young Children with, and at Risk for, Autism Spectrum Disorders. Int. Rev. Psychiatry. 30, 25–39. doi:10.1080/09540261.2018.1432574
Lin, J., Zhao, Y., Huang, W., Liu, C., and Pu, H. (2021). Domain Knowledge Graph-Based Research Progress of Knowledge Representation. Neural Comput. Applic 33 (2), 681–690. doi:10.1007/s00521-020-05057-5
Lippi, G., and Da Rin, G. (2019). “Advantages and Limitations of Total Laboratory Automation: a Personal Overview,”. Advantages and Limitations of Total Laboratory Automation: A Personal overviewClinical Chemistry and Laboratory Medicine, 57, 802–811. doi:10.1515/cclm-2018-1323
Livingstone, R. W., Bone, J., and Field, D. A. (2020). Beginning Power Mobility: An Exploration of Factors Associated with Child Use of Early Power Mobility Devices and Parent Device Preference. J. Rehabil. Assist. Technol. Eng. 7 (205566832092604), 2055668320926046. doi:10.1177/2055668320926046
Marchetti, A., Di Dio, C., Manzi, F., and Massaro, D. (2020). Robotics in Clinical and Developmental Psychology. Reference Module Neurosci. Biobehavioral Psychol. B978, 00005–14. doi:10.1016/B978-0-12-818697-8.00005-4
Martinez-Martin, E., Escalona, F., and Cazorla, M. (2020). Socially Assistive Robots for Older Adults and People with Autism: An Overview. Electronics 9 (2), 367. doi:10.3390/electronics9020367
Martins, P. S., Faria, G., and Cerqueira, J. d. J. F. (2020). I2E: A Cognitive Architecture Based on Emotions for Assistive Robotics Applications. Electronics 9 (10), 1590. doi:10.3390/electronics9101590
McCreadie, C., and Tinker, A. (2005). The Acceptability of Assistive Technology to Older People. Ageing Soc. 25 (1), 91–110. doi:10.1017/s0144686x0400248x
McGuire, W. J. (1983). A Contextualist Theory of Knowledge: Its Implications for Innovation and Reform in Psychological Research. Adv. Exp. Soc. Psychol. 16, 1–47. doi:10.1016/s0065-2601(08)60393-7
Mottron, L., and Bzdok, D. (2020). Autism Spectrum Heterogeneity: Fact or Artifact? Mol. Psychiatry. 25, 3178–3185. doi:10.1038/s41380-020-0748-y
Narasima Venkatesh, D. A. (2020). Reimagining the Future of Healthcare Industry through Internet of Medical Things (IoMT), Artificial Intelligence (AI), Machine Learning (ML), Big Data, Mobile Apps and Advanced Sensors. SSRN Electron. J. 9 (1), 3014–3018. doi:10.35940/ijeat.A1412.109119
Nganji, J. T., Brayshaw, M., and Tompsett, B. (2011). Ontology-based E-Learning Personalisation for Disabled Students in Higher Education. Innovation Teach. Learn. Inf. Comp. Sci. 10 (1), 1–11. doi:10.11120/ital.2011.10010001
Novogrodsky, R. (2013). Subject Pronoun Use by Children with Autism Spectrum Disorders (ASD). Clin. Linguistics Phonetics 27 (2), 85–93. doi:10.3109/02699206.2012.742567
Ogg, J., and Rašticová, M. (2020). “Introduction: Key Issues and Policies for Extending Working Life,” in Extended Working Life Policies: International Gender and Health Perspectives (Springer International Publishing), 3–27. doi:10.1007/978-3-030-40985-2_1
O’Hara, N. H., and Szakacs, G. M. (2008). The Recovery of a Child with Autism Spectrum Disorder through Biomedical Interventions. Altern. Ther. Health Med. 14 (6), 42–44. doi:10.1109/ICORR.2005.1501143
Patel, K., and Parmar, B. (2020). Assistive Device Using Computer Vision and Image Processing for Visually Impaired; Review and Current Status,” Disability And Rehabilitation: Assistive Technology. New York, US: Taylor & Francis.
Pazzaglia, M., and Molinari, M. (2016). The Embodiment of Assistive Devices-From Wheelchair to Exoskeleton. Phys. Life Rev. 16, 163–175. doi:10.1016/j.plrev.2015.11.006
Pdxscholar, P., and Thompson, R. B. (2016). Psychology at a Distance: Examining the Efficacy of Psychology at a Distance: Examining the Efficacy of Online Therapy Online Therapy.
Portugal, D., Alvito, P., Christodoulou, E., Samaras, G., and Dias, J. (2019). A Study on the Deployment of a Service Robot in an Elderly Care Center. Int. J. Soc. Robotics 11 (2), 317–341. doi:10.1007/s12369-018-0492-5
Posserud, M.-B., Lundervold, A. J., and Gillberg, C. (2009). Validation of the Autism Spectrum Screening Questionnaire in a Total Population Sample. J. Autism Dev. Disord. 39 (1), 126–134. doi:10.1007/s10803-008-0609-z
Qian, Z., and Bi, Z. (2015). Recent Development of Rehabilitation Robots. Adv. Mech. Eng. 7 (2), 563062. doi:10.1155/2014/563062
Rates of Apparently Abnormal (2021). Rates of Apparently Abnormal WAIS-4 Patterns in the Normal Population. [Online]. Available at: https://insights.ovid.com/archives-clinical-neuropsychology/acln/2011/09/000/rates-apparently-abnormal-wais-patterns-normal/80/00011977
Riva, G., and Gaggioli, A. (2008). Virtual Clinical Therapy. Lecture Notes Comp. Sci. 4650, 90–107. doi:10.1007/978-3-540-89430-8_6
Sasaki, K., Eguchi, Y., and Suzuki, K. (2020). Stair-climbing Wheelchair with Lever Propulsion Control of Rotary Legs. Adv. Robotics 34 (12), 802–813. doi:10.1080/01691864.2020.1757505
Smith, E. M. (2020). Assistive Technology Use and Provision during COVID-19: Results from a Rapid Global Survey. Int. J. Heal. Pol. Manag., Nov. 11 (5), 1–10. doi:10.34172/ijhpm.2020.210
SoftBank Mobile (2020). SoftBank Mobile and Aldebaran Unveil ‘Pepper’ – the World’s First Personal Robot that Reads Emotions. Press Releases | News | About Us | SoftBank.” [Online]. Available at: https://www.softbank.jp/en/corp/group/sbm/news/press/2014/20140605_01/
Topçu, S., Ulukol, B., Öner, Ö., Şimşek Orhon, F., and Başkan, S. (2018). Comparison of Tidos with M-Chat for Screening Autism Spectrum Disorder. Psychiatry Clin. Psychopharmacol. 28 (4), 416–422. doi:10.1080/24750573.2017.1422682
Villa, S., Micheli, E., Villa, L., Pastore, V., Crippa, A., and Molteni, M. (2010). Further Empirical Data on the Psychoeducational Profile-Revised (PEP-R): Reliability and Validation with the vineland Adaptive Behavior Scales. J. Autism Dev. Disord. 40 (3), 334–341. doi:10.1007/s10803-009-0877-2
Vivanti, G. (2020). Autism and Autism Treatment: Evolution of Concepts and Practices from Kanner to Contemporary Approaches. Cham: Springer, 1–24. Zabarazh, Ukraine: doi:10.1007/978-3-030-41160-2_1
Wainer, J., Robins, B., Amirabdollahian, F., and Dautenhahn, K. (2014). Using the Humanoid Robot KASPAR to Autonomously Play Triadic Games and Facilitate Collaborative Play Among Children with Autism. IEEE Trans. Auton. Ment. Dev. 6 (3), 183–199. doi:10.1109/TAMD.2014.2303116
World Health Organisation (2007). Steps to Health: A European Framework to Promote Physical Activity for Health. Copenhagen, Denmark: WHO Regional Office for Europe.
World Health Organization (WHO) (2020). Disability. [Online]. Available at: https://www.who.int/health-topics/disability#tab=tab_1
Keywords: robot-assisted therapy, humanoid social robot, deep learning, reinforcement learning, ontological knowledge database
Citation: Salhi I, Qbadou M, Gouraguine S, Mansouri K, Lytridis C and Kaburlasos V (2022) Towards Robot-Assisted Therapy for Children With Autism—The Ontological Knowledge Models and Reinforcement Learning-Based Algorithms. Front. Robot. AI 9:713964. doi: 10.3389/frobt.2022.713964
Received: 24 May 2021; Accepted: 17 March 2022;
Published: 06 April 2022.
Edited by:
Lyuba Alboul, Sheffield Hallam University, United KingdomReviewed by:
Selene Tognarelli, Sant'Anna School of Advanced Studies, ItalyRoman Zahariev, Bulgarian Academy of Sciences (BAS), Bulgaria
Copyright © 2022 Salhi, Qbadou, Gouraguine, Mansouri, Lytridis and Kaburlasos. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Mohammed Qbadou, qbmedn7@gmail.com