Letrus Writing Skills Program Letrus (Centro de Autoria e Cultura LTDA) Improving students’ writing skills through using artificial intelligence Theme Location Date started The use of AI to innovate education, teaching and learning Brazil 2014 Beneficiaries Target population Digital solution 116,677 Primary and High school students Letrus assists in improving the writing skills of primary and high school students through an adaptive online learning platform 2019 Winner UNESCO King Hamad Bin Isa Al-Khalifa Prize UNESCO – a global leader in education Education is UNESCO’s top priority because it is a basic human right and the foundation for peace and sustainable development. UNESCO is the United Nations’ specialized agency for education, providing global and regional leadership to drive progress, strengthening the resilience and capacity of national systems to serve all learners. UNESCO also leads efforts to respond to contemporary global challenges through transformative learning, with special focus on gender equality and Africa across all actions. The Global Education 2030 Agenda UNESCO, as the United Nations’ specialized agency for education, is entrusted to lead and coordinate the Education 2030 Agenda, which is part of a global movement to eradicate poverty through 17 Sustainable Development Goals by 2030. Education, essential to achieve all of these goals, has its own dedicated Goal 4, which aims to “ensure inclusive and equitable quality education and promote lifelong learning opportunities for all.” The Education 2030 Framework for Action provides guidance for the implementation of this ambitious goal and commitments. Published in 2021 by the United Nations Educational, Scientific and Cultural Organization, 7, place de Fontenoy, 75352 Paris 07 SP, France © UNESCO 2021 This document is available in Open Access under the Attribution-ShareAlike 3.0 IGO (CC-BY-SA 3.0 IGO) license (http://creativecommons.org/licenses/by-sa/3.0/igo/). By using the content of this document, the users accept to be bound by the terms of use of the UNESCO Open Access Repository (http://www. unesco.org/open-access/terms-use-ccbysa-en). The designations employed and the presentation of material throughout this document do not imply the expression of any opinion whatsoever on the part of UNESCO concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. The ideas and opinions expressed in this document are those of the authors; they are not necessarily those of UNESCO and do not commit the Organization. Author: JET Education Services Editors: Fengchun Miao and Iaroslava Kharkova Designed by Huieun Kim Cover and inside photos: © UNESCO/Letrus (Centro de Autoria e Cultura LTDA) Printed by UNESCO Printed in France Letrus Writing Skills Program | Letrus Improving students’ writing skills through using artificial intelligence Summary 'The limits of my language are the limits of my world.’ Ludwig Wittgenstein." _ Luis Junqueira, Letrus, 20191 While Brazil has made great strides towards eliminating illiteracy, functional literacy is still a challenge for the country, with only 2% of students achieving the highest marks in literacy on the 2018 Programme for International Student Assessment (PISA) (OECD, 2019a). The Letrus Writing Skills Program2 is an innovative technology solution that supports the development of student writing in Portuguese through an Artificial Intelligence (AI) platform built at the intersection of linguistics, pedagogy and computer science. Through its AI and comprehensive programme of support, Letrus provides a dynamic learning method and personalized, extensive essay feedback to students and reduces both teacher workload and the typical response time associated with grading essays. Students who submit essays on the Letrus platform receive immediate feedback from the Automated Writing Evaluation (AWE) algorithm, which provides them with formative comments on specific areas of strength and weakness such as adherence to the formal written norms of Portuguese or the length of specific paragraphs. Essays are then evaluated by human graders3 who assign additional comments and final grades in alignment with the criteria used for the National High School Exam (Exame Nacional do Ensino Médio, or ENEM), the largest college admission examination in Brazil. At the same time, teachers access class dashboards to view progress on completion, receive feedback on aggregate and individual scores and connect to Letrus support staff. Teachers receive support with implementation and monitoring, interpreting the results on the dashboard and utilizing results for targeted instruction and/or remediation. More information: https://www.letrus.com.br 1 See: https://www.linkedin.com/company/letrus 2 In this text ‘program’ is used to reference a computer-based application or engagement, while ‘programme’ refers to a general intervention that includes both human and digital support structures. 3 As of 2020, Letrus also offers an AI-only option. See the ‘Results’ and ‘Further developments’ sections for details. 3 Letrus Writing Skills Program | Letrus Improving students’ writing skills through using artificial intelligence Why selected The Letrus Writing Skills Program was selected as a laureate due to its: • Innovative use of AI to automatically detect features of the essay and to classify five different skills that are aligned with the national standards for language learning in Brazil, • hybrid AI-human feedback loop of immediate feedback from the AI and more detailed feedback from human graders, leading to more effective practice opportunities for essay writing, • enabling teachers to follow students’ writing progress and view the writing skills of their students and classes in a structured way, as well as provide the students with personalized feedback, • ability to operate at a large scale, and promising findings demonstrated by initial research that student marks improve as more essays are written on the platform, • aim to improve functional literacy and promote inclusion by helping students, especially from low socio- economic backgrounds, to access higher education institutions (UNESCO, 2019b). Literacy is the foundation of any healthy and just society, and the capacity of expressing oneself in written form is fundamental for the cognitive, human and social development of every student." _ Letrus, 2019 4 Letrus Writing Skills Program | Letrus Improving students’ writing skills through using artificial intelligence Programme PROBLEM While Brazil has made great strides towards universal literacy, functional literacy levels remain below the levels necessary for full economic and civic participation SOLUTION An adaptive learning platform which combines AI and human elements to provide responsive feedback on writing skills and personalized learning experiences for students BENEFICIARIES Teachers and primary and high school students RESOURCES REQUIRED Access to a device with internet access RESULTS • Reach: 957 schools; 1,327 teachers; 116,677 students • Average improvement of 10% after writing five essays on the platform • Proven impact: Experimental evidence shows a significant, positive impact • Proven impact: Letrus could decrease the publicprivate achievement gap for specific skills by 20% CHALLENGES • Internet access / device accessibility • Financing the programme, particularly in low-resource and public schools STRENGTHS • Builds on existing technologies such as Natural Language Processing resources • Rigorous interdisciplinary approach leverages linguistics and computer science • Engagement of academic partnerships • Teachers remain the pedagogical leaders for students. • Applications for remote and hybrid learning models 5 Letrus Writing Skills Program | Letrus Improving students’ writing skills through using artificial intelligence Profile: Implementing agency Letrus (Centro de Autoria e Cultura LTDA) is a private education start-up company launched in December 2014 with headquarters in Sao Paulo, Brazil. Working to achieve universal literacy is Letrus's core mission, and the main goal of its products is to reduce the functional illiteracy rate in Portuguese. To this end, Letrus has developed a programme comprising an online platform, AI tools and a support team to provide semi-automated or fully automated feedback on students’ writing. Letrus helps students from various socio-economic backgrounds to access enhanced writing practice opportunities. The company currently employs 72 staff members and leverages principal investors United States funds Canary and Potencia Ventures, which focuses on education and health impact initiatives. To date, Letrus has partnered with 957 schools in 26 states across Brazil, serving over 116,677 students.4 4 As of December, 2020. 6 Letrus Writing Skills Program | Letrus Improving students’ writing skills through using artificial intelligence Context Brazil is a country with an estimated 210 million people and a population density of 25 people per square kilometer. About 13% of Brazil’s population is rural, with a poverty rate of 4.8%5 (UIS, 2020). By the end of 2010, Brazil had established ICT access for 25 million of its 40 million students, trained over 300,000 teachers in ICT and invested in the development of multimedia resources and a teacher portal for accessing them (Ministry of Education, Brazil, 2010). However, a review of technology use by Souza et al. (2017) identified slow growth of ICT prevalence in schools as well as persistent limitations such as low connectivity, low ratios of devices to students and a need for further teacher training. Brazil serves nearly 40 million students (UIS, 2020), with optional pre-primary education and compulsory education for children between the ages of 6 and 17, both provided by a mix of public and private institutions. The compulsory education system is divided into primary school (ensino fundamental) and high school (ensino médio). Primary school consists of level one in years 1 to 5 and level two in years 6 to 9, and high school consists of a further three years (Presidência da República Casa Civil Subchefia para Assuntos Jurídicos, 1996). Students progress after passing end of year examinations in each grade (Ministry of Education, Brazil, 2009). Education challenges In the final year of high school, students participate in the National High School Exam (Exame Nacional do Ensino Médio, or ENEM), the largest college admission examination in Brazil and the second-largest college admission examination in the world. The ENEM covers language, social sciences, natural sciences and mathematics, and includes multiple choice questions and a composition.6 As of 2016, the Brazilian government invested 6.3% of its gross domestic product (GDP) into education (Ministry of Education, Brazil, 2016), well above the OECD average of 3.2%. According to the 2019 Global Education Monitoring Report (UNESCO, 2019a, pp. 310311), education spending accounted for 16.2% of total government expenditure. However, due to Brazil’s low GDP, the monetary investment per student as well as teacher compensation is lower than the OECD average, both in terms of absolute value and purchasing power (OECD, 2019b). Engagement with ICT Information Communications Technology (ICT) in education policies in Brazil have been active for more than a decade and seek to attain three main objectives: digital literacy training for teachers and students; transforming classrooms into dynamic environments; and helping students build autonomy and authoring skills. National implementation of ICTs in education is outlined in a National Development Plan and focuses on delivering ICT infrastructure, training teachers and developing open educational resources (OERs). 5 Poverty is determined as subsisting on an income of less than USD 1.90 per day. 6 See: https://www.gov.br/inep/pt-br/acesso-a-informacao/perguntasfrequentes (in Portuguese) 7 Letrus Writing Skills Program | Letrus Improving students’ writing skills through using artificial intelligence which measures the scholastic achievement of 15 year olds in mathematics, science and literacy. In 2018, Brazil performed below the OECD average in all three subjects, with the largest difference of more than 100 points in mathematics. The percentage of low performers in all three subjects was 43.2%, one of the highest among participating countries, and only 2% of students attained marks demonstrating high proficiency in literacy. In addition, data submitted regarding education context showed that Brazil had one of the highest student:teacher ratios (OECD, 2019a). Brazil has achieved increasing literacy rates since 1980. As of 2018, 93.2% of the population aged 15 and older was considered literate, though Brazil was still home to roughly 11 million illiterate adults (UIS, 2020), and only 71% of people surveyed were functionally literate.7 In addition, only about 16.5% of adults in Brazil attain a tertiary education (PNAD Educação, 2019), and while young women are 42% more likely to attain tertiary education than young men, they are less likely to be employed. While over 80% of primary and secondary education students are enrolled in public education institutions, over 75% of tertiary students enroll in private universities (OECD, 2019b). At the high school level, Brazil participates in the Programme for International Student Assessment (PISA) examination, an internationally normed assessment 7 Functional literacy here refers to the ability to read and interpret minimal levels of text, e.g. news articles. For more information see Statista Research Department (2020). 8 Letrus Writing Skills Program | Letrus Improving students’ writing skills through using artificial intelligence Digital Solution Following this initial feedback, the essay can be evaluated by human graders who have access to the essay and the results of the AWE algorithm.9 Human graders can assign additional scores, adjust grades and comments and assign the final score to the student within three days. Final scores provide a grade per skill for a maximum of 200 points, aligned to the official grading scale used by the ENEM. The Letrus Writing Skills Program seeks to contribute towards building a fully educated country, a more qualified workforce and citizens with critical thinking able to exercise citizenship through the development of writing skills.8 Letrus uses a combination of Artificial Intelligence (AI) and human feedback to reduce the time between submission and feedback to students, provide extensive and meaningful positive as well as corrective feedback, and reduce teacher workload. The platform enables personalized learning and responsive feedback for both students and teachers and further empowers the learning process through providing the information necessary for teachers to target pedagogy towards addressing student errors. An added benefit is that Letrus provides a practical solution to large class sizes by automating parts of the essay-grading process. As students are writing, information such as progress towards completion, average scores for structure and skills as evaluated by the AWE algorithm are provided to teachers on a personal class dashboard. Teachers can follow the progress of students on the writing task and monitor if they have logged in, started writing or finished the task, and view real-time feedback in aggregate and for individual students. Information on common errors across the class can be taken up for instruction or remediation, and teachers are able to view and adjust comments made to individual students. Students in participating high schools write freely in response to writing prompts on the Letrus platform and are able to keep track of their word count and time remaining. The platform also automatically saves essays periodically. As a student writes, the Letrus Automated Writing Evaluation (AWE) algorithm analyses the text, and once the essay is submitted, the student receives instantaneous feedback on performance related to writing norms and composition structure, together with comments to guide improvement. 8 For more information, see https://www.facebook.com/Letruseducacao (August, 2020). 9 Letrus also developed and tested a fully-automated feedback process. See the ‘Results’ and ‘Further developments’ sections for more details on the AI-only program. Developing the digital solution technology To develop Letrus, a minimum viable product was developed which allowed students to submit essays to the Letrus platform following the ENEM model. These essays were assessed by graders trained and experienced in the ENEM criteria, and formed part of a database of 56,644 ENEM and Letrus platform essays that were analysed to identify patterns in the style, content, grammar and structure of excellent essays. This analysis resulted in eight text indicators: words, connectors, spelling mistakes, colloquialism, paragraphs, sentences, social agents and social intervention elements. The iLetrus Index is the AWE algorithm used to interpret the essay through these text indicators to arrive at a score for the text. An additional resource, iLetrus Levels, automatically attributes a level from 1 to 5 for essays submitted. 9 Improving students’ writing skills through using artificial intelligence Letrus Writing Skills Program | Letrus iLetrus is built using deep neural networks and feature engineering. Deep neural networks are AI programs with multiple layers of processing between the inputs and the outputs (Witten et al., 2017). In this case, the first layer processes individual textual elements (i.e. performs calculations between words and sentences), and a second layer performs calculations based on these outputs to score the composition (i.e. performs calculations between sentences and the essay in full) (Fonseca et al., 2018). Feature engineering is a way of preparing unique data inputs (or features) to maximize the performance of the algorithm (Rençberoğlu, 2019). In Letrus, 681 feature values are mapped for relevance across the five ENEM skills: 1) adherence to formal written norms of Portuguese; 2) conformity to the argumentative text genre and the topic; 3) selection, organization and interpretation of data and arguments in defense of a point of view; 4) usage of argumentative linguistic structures; and 5) elaboration of a proposal of intervention to solve the problem in question. The feature engineering approach is used to continuously evaluate the essay as it is written, and the deep neural network approach is used when the student completes the essay. Comments on the platform respond to specific text indicators (e.g. the use of conjunctions, spelling, length of paragraphs, etc.). Essays also receive a general comment and comments on the five specific ENEM skills. Comments are randomly chosen to respond to specific sets of challenges and levels of achievement, and many will provide a model response. For example, an essay with a poor introduction may generate a comment with an example of a good introduction for the student to refer to. 10 Letrus Writing Skills Program | Letrus Improving students’ writing skills through using artificial intelligence Implementation The Letrus platform is accessed by students and teachers with a programme of support implemented in schools. The Letrus team involves school management and teachers in the partnership and provides support with setting a writing timetable for the school, on-site registration and training in order to orient teachers to the platform and its capabilities. Letrus also has a dedicated support team, including a pedagogical advisor, who assists teachers and students on their learning pathway. In addition to assisting with implementation of the programme with students, the Letrus team supports teachers to interpret the results presented on their personalized dashboard to inform pedagogical practices. As the Letrus programme is designed to empower teachers as the curriculum and instruction leads, teachers are the point of contact between Letrus and the school, and are able to contact the Letrus team via a chat function on the platform or directly to query results, request support and give their own feedback. While students are not encouraged to contact Letrus for pedagogical support, they may contact Letrus via a contact form on the Letrus website for technical support such as logging onto the platform or questions about submitting essays. In the midst of the COVID-19 pandemic, the schools in the Sesi Education Network of Mato Grosso do Sul intensified investments in technology-based education to maintain learning. In preparation for ENEM and other entrance exams in the country, students are improving their writing techniques through the Letrus platform. ‘Sesi chose to invest in Letrus because it is a platform with adherence to all the technology that the Sesi schools offer, including artificial intelligence, and also due to the wide initial receptivity we had with students in a pilot phase. Sesi's schools are always aligned with the most modern and advanced tools available in the country and in the world,’ said Sesi's superintendent, Bergson Amarilla. Through AI, the Letrus program allows students to write essays and obtain immediate feedback on the structure of the text, regardless of the chosen theme. Simultaneously, the teacher receives an individualized analysis of the student's work and can create more effective classes to assist students to improve their writing skills. The platform also minimizes the work of the teacher, who would otherwise spend hours correcting students' texts, and enhances the performance of teachers in the classroom. ‘Letrus software is able to recognize the spelling and grammar of each student, whether the language is colloquial or formal, the argumentative pattern and whether there is a proposal for social intervention, as required by the ENEM test,’ Sesi’s education analyst Gláucia Campos explained, adding, ‘the teacher's role becomes even more strategic and fundamental. They start to have a basis to reorient teaching practice and manage the classroom.’ Source: http://www.fiems.com.br/noticias/com-startup-de-inteligencia-artificial-escolas-do-sesi-de-ms-aprimoram-redacao-dosalunos/31777 (in Portuguese.) 11 Letrus Writing Skills Program | Letrus Improving students’ writing skills through using artificial intelligence Enablers and supports The technologies used in the development of text indicators were a Python Natural Language Toolkit (NLTK) library and NLPNet. NLTK is a platform which supports the development of Python programs that can work with human language data.10 NLPNet contributed to the analysis through automating processes such as part-ofspeech tagging (e.g. labelling a token or word such as ‘quickly’ with metadata that identifies it as an adverb).11 Together these technologies supported the initial analysis used to construct a gold standard corpus12 (in other words, a reliable collection of language data suitable to inform the algorithm training). Letrus staff The Letrus team consists of 72 staff members which includes a technology and pedagogical team. As a multidisciplinary innovation, the technology and pedagogical teams work closely together to develop and refine Letrus products. The technology team is primarily responsible for maintaining, testing and refining the algorithm and other technology tools and developing new products using input from the pedagogical team. The pedagogical team develops content such as comments, supports teachers and trains Letrus graders. iLetrus also makes use of a spell checker to identify words which are outside of the parameters of formal written Portuguese, such as typos, spelling errors and jargon, and provide suggestions for substitutions. The spell checker rules allow developers and facilitators to understand common weak spelling points of students and provide schools, teachers and students with precise feedback which explains mistakes and provides examples to improve understanding. For the development and maintenance of this resource, the main technologies adopted were the Spell Checker algorithm (including handcrafted rules, ElasticSearch13 and Levenshtein Letrus graders Letrus graders access an additional platform where they can see essays available for correction and are paid per correction. The number of graders fluctuates based on the demand of schools, and selection processes are periodically conducted for new graders. An initial analysis of work history prioritizes candidates with a history of working with examinations and grading and/ or teachers. Stage two of the selection process involves a practical essay grading test. Graders are selected not only based on technical grading accuracy and skill, but also on the language and orientation of their feedback, with a preference for those who use developmental and motivational feedback styles. Selected graders attend a training on the text genres, criteria for grading and structuring developmental comments for students. After training, graders are added to a database of individuals who can access the grading platform. distance),14 dictionaries, the Letrus corpus and a general Brazilian-Portuguese Language corpus. In addition, the Letrus platform incorporates plagiarism detection using a resource which identifies the rate of similarity between an essay finished on the platform 10 For more information, see https://www.nltk.org 11 For more information, see https://pypi.org/project/nlpnet Technology 12 A ‘corpus’ refers to a collected body of text within set parameters, stored as an electronic database. For more information, see https://courses.helsinki.fi/sites/ default/files/course-material/4433684/070916part2.pdf A number of existing technologies and technological resources supported the development and support the ongoing delivery of Letrus. 13 For more information, see https://www.elastic.co 14 The Levenshtein distance equation measures the minimum number of changes (such as adding, subtracting or replacing a single letter) required to transform one word into another (for example, the Levenshtein distance between ‘boy’ and ‘toy’ is 1). For more information, see https://dzone.com/ articles/the-levenshtein-algorithm-1. 12 Letrus Writing Skills Program | Letrus Improving students’ writing skills through using artificial intelligence Professor Erica Rodrigues to build an algorithm to map students’ writing processes. Academic partnerships also informed the expertise in linguistics necessary to build the Letrus AI. with other texts in the Letrus plagiarism data bank, which includes the writing prompts and motivation texts presented by the platform to the student as well as texts from the web.15 Partnerships have also contributed to the assessment of the programme, including partnerships with the Abdul Latif Jameel Poverty Action Lab (J-PAL), the Lemann Foundation, Espírito Santo State Education Department, the Center of Public Policies and Education Assessment (CAED), and the Sao Paulo School of Economics (FGV - Partnerships Letrus seeks to build solid partnerships with research centres and organizations that share the programme’s orientation to education development. Academic partnerships have been formed to share knowledge with the Institute of Physics at the University of São Paulo and the Pontifical Catholic University of Rio de Janeiro. The former focuses on a textual evaluation automation project, and the latter leverages the research database of 15 This similarity index is calculated following the n-gram based containment measure, described by Barrón-Cedeño and Rosso (2009). Student's voice ‘The Letrus platform is a great benefit that the school offers us, because in addition to training for writing an essay, we can find and correct our mistakes. We are increasingly prepared for entrance exams and know how to argue about several current issues.’ _Isadora Gotardo, 3rd grade high school student, Sesi Campo Grande School (Escola do Sesi de Campo Grande) ‘Letrus has helped me a lot this year. It provides information on how to improve my writing, and has been helping me to develop texts that are increasingly grounded and cohesive.’ _Diego Soares Ribeiro, 3rd grade high school student, Sesi Capital School (Escola do Sesi de Capital) ‘Using the platform to practice writing has been very good, because, in addition to bringing texts from different themes, it makes notes on what we need to improve our ability. In our writing classes the teacher's partnership with the tool makes us realize how much we are able to write better and better.’ _Maitê Louise Falbo Oliveira e Silva, 8th grade primary school student, Sesi Três Lagoas School (Escola do Sesi de Três Lagoas). Source: http://www.fiems.com.br/noticias/com-startup-de-inteligencia-artificial-escolas-do-sesi-de-ms-aprimoram-redacao-dosalunos/31777 (in Portuguese.) 13 Letrus Writing Skills Program | Letrus Improving students’ writing skills through using artificial intelligence Monitoring and evaluation The platform includes back-end analytics which monitor the engagement of students and teachers, including the number of written essays per number of registered students in each activity and the number of teachers who access the platform monthly. The platform and its AI algorithms are also monitored for quality over four levels: 01 Outputs are validated periodically to ensure they are accurate (e.g. ensuring that spelling errors identified by the AI are actually spelling errors, etc.). This assists with identifying areas in which the AI algorithm needs to be retrained. 02 Comments are collected by the humans interacting with the program, primarily Letrus graders, teachers and students. Feedback is used to identify additional areas which may need adjustments. Letrus’s partnerships with language professionals are also a critical resource at this level. 03 Academic partnerships assist with directed research that validates and improves Letrus technology. This monitoring and evaluation is linked to the development of new functionalities and products. 04 Large-scale partnerships with research organizations and foundations assess impact and contribute to Letrus product development. Evaluations conducted to date have included research to analyze student improvement through the programme and a large-scale randomized control trial. 14 Letrus Writing Skills Program | Letrus Improving students’ writing skills through using artificial intelligence Results incorporating human feedback and the second relying on the AI alone. Each of these treatment groups included 55 schools, with 68 control schools. Data was gathered from the Letrus platform, the 2019 ENEM essay, additional writing samples, and survey questions regarding demographic information and information on school practices and perceptions from teachers and students. Reach At the time of the award, the Letrus Writing Skills Program had been used by over 44,000 students in schools in all 26 states in Brazil, totaling over 200,000 essays made on the platform. Engagement Findings included that both versions of Letrus resulted in significantly improved student essay scores over the control group, and that the inclusion of additional inputs from human graders did not improve the effectiveness of the Letrus program. The research suggests that teachers did not simply delegate tasks to the AI but filled in the gaps in the pure technology solution. As a result of these findings, Letrus dedicated more effort to developing the AI-only option in order to lower costs and increase access. In 2019, a partnership with the State Department of Education of Espírito Santo took the use of Letrus to 54 municipalities, reaching 12,000 students and 400 teachers from 110 schools. In this project, the average engagement of students was between 75% and 80%, and 95% for teachers. Performance A further significant finding was that the use of Letrus could mitigate 9% of the public-private achievement gap in ENEM essay scores, with a reduction of 20% in the skillspecific gap (reported nationally to be 80%). In an analysis of the evolution of students’ grades and performance on specific text indicators conducted in 2019 with public schools in Espírito Santo, students using Letrus demonstrated an average of 10% improvement and were found to write 32% more essays than a control group. Ninety per cent of schools improved their grades. Finally, the research measured control and intervention group performance on writing in the narrative genre, literacy skills and non-literacy skills. Findings indicated that improvements were limited to argumentative essays specifically, without significant spillover effects into other genres or subjects. Ferman et al. (2020) engaged in a stratified randomized control trial to determine the impact on student performance of two versions of Letrus, the first 15 Letrus Writing Skills Program | Letrus Improving students’ writing skills through using artificial intelligence Challenges Letrus is still working to determine both the best model and the best methods to ensure the company is financially sustainable while reaching its largely vulnerable target audience. This is a significant challenge and has resulted in a shifting model as different avenues are explored. Reaching target beneficiaries Affordability The company originally explored a direct-to-student purchase option, in which individual students could pay for access to the platform and support. In this model, private schools were provided access with no direct cost, but would assist Letrus in selling the program to students as a co-curricular support for use at home. The company is considering a sliding scale model, in which cost is linked to the scale of implementation. While the implementation and support of the model requires some funding, Letrus is also committed to decreasing the cost of the program as much as possible. To this end, Letrus is exploring different levels and components of support. As of December 2020, Letrus offers both an AI-only option and one which includes a human grader. In either option, a percentage of text is validated to ensure the AI is still appropriate and accurate. This model has been discontinued in favor of a direct business-to-business model in which schools pay for the service, or a business-to-government model in which the government will procure services for schools. However, challenges remain. Many of the schools which serve vulnerable students are public institutions and do not have the revenue streams to support Letrus. At the government level, there are both buy-in and logistical challenges to overcome: it will require a high degree of political will to achieve the government investment necessary for national implementation of Letrus. Infrastructure Both hardware and an internet connection are necessary to run the Letrus platform and program, and one challenge noted is that access is still an issue Brazil is facing as a country. Municipal governments, which are mainly responsible for access to and funding for primary grades, provide an access point to public schools that may have less bureaucracy and offer an equitable solution at a smaller than national scale. However, fitting into this market requires adjustments to the product. To make all possible accommodations, the program was built to be lite software which can run on a lower connection speed, as well as on devices such as smartphones. Additionally, while the programme model was designed to be flexible and does not require all students to write simultaneously, participating schools were required to have a computer:student ratio of at least 1:20. 16 Letrus Writing Skills Program | Letrus Improving students’ writing skills through using artificial intelligence Further developments Since winning the UNESCO prize in 2019, Letrus has grown to a total of 72 staff members and has expanded its scope to 957 schools, 1,327 teachers and 116,677 students. Over 456,000 essays have been submitted to the portal.16 In addition, Letrus has begun testing a version of its software designed for Grades 6 to 9 in order to reach students earlier in their academic careers, as well as adding additional genres such as descriptive, narrative and news articles. This introduces a new set of challenges, as at the high school level a large quantity of essays and specific criteria were available. The Letrus team now faces the challenge (which is also an opportunity) of helping learners develop along a trajectory of skill in the lower grades. integrate monitoring on the behaviour of writing, such as how often students revise or review their work, and to provide general feedback on the writing process of students as well as their writing performance. In 2020, Letrus also launched an app version of its program for students. Letrus has a five-year vision to become financially sustainable and access wide education networks and particularly support public schools from elementary to higher education. Letrus plans to expand to other Latin American countries and has a broad goal to increase the scale and quality of their services at a price point accessible to citizens of any age. On the technology side, Letrus is working to improve the relevance of platform monitoring. In addition to monitoring general use by teachers, Letrus now monitors how frequently teachers open reports, which will provide more information on how teachers use the platform and how they can be better supported. Letrus also plans to 16 Figures as of December, 2020. Founder’s Story When Letrus co-founder Luis Junqueira was a student at the Institute of Language Studies at the State University of Campinas, he came to realize the impact writing and freedom of expression can have. From his own experience as a teacher, he then launched the First Book Project (Projeto Primeiro Livro) to help children and adults publish their own books. Learning from the challenges they encountered led him to create the Letrus Program (UNESCO, 2020). 17 Letrus Writing Skills Program | Letrus Improving students’ writing skills through using artificial intelligence Impact of the Prize Letrus was awarded the UNESCO-King Hamad Bin IsaAl Khalifa Prize for the Use of ICTs in Education 2019 during the height of the COVID-19 pandemic. During this time, many schools around the world, including in Brazil, closed to contribute to efforts to contain the virus, leading in turn to an anxiety on the part of teachers, parents, governments and international organizations that existing learning divides would widen. Although Letrus received the prize during a time when the world focus was on the COVID-19 pandemic, receiving the award did lead to increased publicity and a deep sense of pride for the organization. As a result of the award, Letrus has appeared in seven national and international articles and has made a television appearance, and the organization has noted an increase in its social media followers. Letrus has also forged closer ties to UNESCO in Brazil and is exploring opportunities to collaborate both in knowledge-sharing and in UNESCO Associated Schools. Given this context, Letrus decided to allocate the funding received from the prize to funding the use of Letrus in public schools for a semester to assist in reaching students during quarantine. The Letrus team first provided a lite version of the software for two months to public schools, but when it was understood quarantines would be prolonged and schools would continue to be closed, Letrus shifted this implementation to a six-month programme which included the remote support of its team with chat support and training to help teachers understand the activities and program. The Letrus team mentions the prize proudly in every presentation to programme participants and partners, and has found the prize to be valuable in enhancing the organization’s reputation, both in terms of the award itself and by placing an international focus on using technology to improve education. 'We wanted to offer support for schools in general, a platform to practice and develop writing skills remotely or in the hybrid education model.' _ Interview with Letrus, September 2020 'The theme ‘Using AI in Classrooms’ caught our attention. What we want is to use AI to fulfill students’ needs and help them progress, and to support educators in their teaching practice. We were very aligned with what the prize wants to promote.' _ Interview with Letrus, September 2020 'The award symbolizes that we are moving in the right direction and that the trail now has more light and possibilities to open new doors in the future.' _ Letrus co-Founder Luis Junqueiras, Press Release Point, 2020 18 Letrus Writing Skills Program | Letrus Improving students’ writing skills through using artificial intelligence References Barrón-Cedeño, A. and Rosso, P. 2009. On automatic plagiarism ——. 2019b. Education at a Glance: Brazil Country Note. Paris, detection based on n-grams comparison. M. Boughanem, Organisation for Economic Co-operation and Development C. Berrut, J. Mothe, and C. Soule-Dupuy (eds). Advances in (OECD). Available at: https://www.oecd.org/education/ Information Retrieval. ECIR 2009. Lecture Notes in Computer education-at-a-glance/EAG2019_CN_BRA.pdf (Accessed 25 Science, Vol. 5478. Berlin, Springer-Verlag Berlin Heidelberg, pp. October 2020.) 696–700. PNAD Educação. 2019. Educação 2019. Rio de Janeiro, Instituto Ferman, B., Lima, L. and Riva, F. 2020. Experimental evidence on Brasileiro de Geografia e Estatistica (IBGE). (In Portuguese.) Artificial Intelligence in the classroom. Munich Personal RePEc Available at: https://biblioteca.ibge.gov.br/visualizacao/livros/ Archive, Paper No. 103934. Available at: https://mpra.ub.uni- liv101736_informativo.pdf (Accessed 25 October 2021.) muenchen.de/103934 (Accessed 25 October 2021.) Presidência da República Casa Civil Subchefia para Assuntos Fonseca, E., Medeiros, I., Kamikawachi, D. and Bokan, A. 2018. Jurídicos. 1996. Lei Nº 9.394, De 20 De Dezembro De 1996. Brasilia, Automatically grading Brazilian student essays. A. Villavicencio Presidência da República. (In Portuguese.) Available at: http:// et al. (eds.) Computational Processing of the Portuguese Language. www.planalto.gov.br/ccivil_03/leis/L9394.htm (Accessed 25 PROPOR 2018, Lecture Notes in Computer Science, Vol. 11122. October 2021.) Canela, Springer, Cham. Press Release Point. 2020. Brazil programme awarded UNESCO Letrus. 2019. Letrus application to the UNESCO King Hamad Prize for using AI to improve writing skills. Chicago, Press Release Bin Isa-Al Khalifa Prize for the Use of ICTs in Education 2019. Point. Available at: https://www.pressreleasepoint.com/brazil- Unpublished (Submitted to UNESCO). programme-awarded-unesco-prize-using-ai-improve-writingskills (Accessed 25 October 2021.) Ministry of Education, Brazil. 2009. Ensino Fundamental de Nove Anos: Passo a passo do processo de implantação. Brasilia, Rençberoğlu, E. 2019. Fundamental techniques of feature Ministério da Educação. (In Portuguese.) Available at: http:// engineering for machine learning. Towards Data Science. portal.mec.gov.br/dmdocuments/passo_a_passo_versao_ Vancouver, Towards Data Science. Available at: https:// atual_16_setembro.pdf (Accessed 25 October 2021.) towardsdatascience.com/feature-engineering-for-machinelearning-3a5e293a5114 (Accessed 25 October 2021.) ——. 2010. Internet em banda larga chega a 24 milhões de estudantes. Brasilia, Ministério da Educação. (In Portuguese.) Souza, E., Garcia, L., Silva, J., Garcia, L. and Letouze, Patrick. 2017. Available at: http://portal.mec.gov.br/ultimas-noticias/210- A Review of the Use of Information Technology in Brazilian 1448895310/14198-internet-em-banda-larga-chega-a-escolas- Schools from 2010 to 2014. International Journal of Information de-minas-e-amazonas (Accessed 27 October 2021.) and Education Technology, Vol. 7, No. 7. Singapore, International Journal of Information and Education Technology, p.p. 284-290. ——. 2016. Indicadores Financeiros Educacionais. Brasilia, Available at: https://www.doi.org/10.18178/ijiet.2017.7.4.882 Ministério da Educação. (In Portuguese.) Available at: https:// (Accessed 25 October 2021.) www.gov.br/inep/pt-br/acesso-a-informacao/dados-abertos/ indicadores-educacionais/indicadores-financeiros-educacionais Statista Research Department. 2020. Brazil: Share of Population (Accessed 25 October 2021.) Considered Functionally Literate 2018. Hamburg, Statista. Available at: https://www.statista.com/statistics/1130373/brazilfunctional-literacy (Accessed 25 October 2021.) OECD. 2019a. 2018 PISA country note: Brazil. Paris, Organisation for Economic Co-operation and Development (OECD). Available at: http://www.oecd.org/pisa/publications/PISA2018_CN_BRA. UIS. 2020. Brazil | UNESCO UIS. Montreal, UNESCO Institute for pdf (Accessed 25 October 2021.) Statistics (UIS). Available at: http://uis.unesco.org/en/country/br (Accessed 3 November 2020.) 19 Letrus Writing Skills Program | Letrus Improving students’ writing skills through using artificial intelligence ——. 2020. UNESCO King Hamad Bin Isa Al-Khalifa Prize for the Migration, displacement and education: Building Bridges, Not Use of ICT in Education: 2019 laureates. Paris, UNESCO. Available Walls. Paris, UNESCO. Available at: https://unesdoc.unesco.org/ at: https://unesdoc.unesco.org/ark:/48223/pf0000373477 ark:/48223/pf0000265866 (Accessed 25 October 2021.) (Accessed 25 October 2021.) ——. 2019b. Recommendation of the International Jury for the Witten, I., Frank, E., Hall, M. and Pal, C. 2017. Deep learning. Data UNESCO King Hamad Bin Isa-Al Khalifa Prize for the Use of ICTs in Mining: Practical Machine Learning Tools and Techniques, 4th edn. Education 2019. Unpublished (UNESCO internal document). Burlington, Morgan Kaufmann Publishers, pp. 417-466. ED/FLI/ICT/2021/10 UNESCO. 2019a. 2019 Global Education Monitoring Report. 20 About the Prize The UNESCO King Hamad Bin Isa Al-Khalifa Prize recognizes individuals, institutions and organizations that use digital technologies to expand educational and lifelong learning opportunities. Funded by the Kingdom of Bahrain and established in 2005, the Prize awards annually US$ 25,000 each to two laureates selected by an international Jury, consisting of five recognized experts. The Prize is managed by the Unit for Technology and Artificial Intelligence in Education within the Future of Learning and Innovation Team, Education Sector, UNESCO. Stay in touch https://on.unesco.org/aboutictprize ictprize@unesco.org @UNESCO @UNESCOICTs #ICTPrize @ICTs in Education