Full download test bank at ebook textbookfull.

com

Addressing Global Challenges and

Quality Education 15th European
Conference on Technology Enhanced
Learning EC TEL 2020 Heidelberg
Germany September 14 18 2020
CLICK LINK TO DOWLOAD

https://textbookfull.com/product/addressing-
global-challenges-and-quality-education-15th-
european-conference-on-technology-enhanced-
learning-ec-tel-2020-heidelberg-germany-
september-14-18-2020-proceedings-carlos-
alario-hoyos/

textbookfull
More products digital (pdf, epub, mobi) instant
download maybe you interests ...

Lifelong Technology-Enhanced Learning: 13th European

Conference on Technology Enhanced Learning, EC-TEL
2018, Leeds, UK, September 3-5, 2018, Proceedings
Viktoria Pammer-Schindler
https://textbookfull.com/product/lifelong-technology-enhanced-
learning-13th-european-conference-on-technology-enhanced-
learning-ec-tel-2018-leeds-uk-september-3-5-2018-proceedings-
viktoria-pammer-schindler/

Adaptive and Adaptable Learning 11th European

Conference on Technology Enhanced Learning EC TEL 2016
Lyon France September 13 16 2016 Proceedings 1st
Edition Katrien Verbert
https://textbookfull.com/product/adaptive-and-adaptable-
learning-11th-european-conference-on-technology-enhanced-
learning-ec-tel-2016-lyon-france-
september-13-16-2016-proceedings-1st-edition-katrien-verbert/

Transforming Learning with Meaningful Technologies 14th

European Conference on Technology Enhanced Learning EC
TEL 2019 Delft The Netherlands September 16 19 2019
Proceedings Maren Scheffel
https://textbookfull.com/product/transforming-learning-with-
meaningful-technologies-14th-european-conference-on-technology-
enhanced-learning-ec-tel-2019-delft-the-netherlands-
september-16-19-2019-proceedings-maren-scheffel/

Software Architecture 14th European Conference ECSA

2020 L Aquila Italy September 14 18 2020 Proceedings
Anton Jansen

https://textbookfull.com/product/software-architecture-14th-
european-conference-ecsa-2020-l-aquila-italy-
september-14-18-2020-proceedings-anton-jansen/
Computer Security ESORICS 2020 25th European Symposium
on Research in Computer Security ESORICS 2020 Guildford
UK September 14 18 2020 Proceedings Part II Liqun Chen

https://textbookfull.com/product/computer-security-
esorics-2020-25th-european-symposium-on-research-in-computer-
security-esorics-2020-guildford-uk-
september-14-18-2020-proceedings-part-ii-liqun-chen/

Software Engineering and Formal Methods: 18th

International Conference, SEFM 2020, Amsterdam, The
Netherlands, September 14–18, 2020, Proceedings Frank
De Boer
https://textbookfull.com/product/software-engineering-and-formal-
methods-18th-international-conference-sefm-2020-amsterdam-the-
netherlands-september-14-18-2020-proceedings-frank-de-boer/

Computer Vision ECCV 2018 15th European Conference

Munich Germany September 8 14 2018 Proceedings Part III
Vittorio Ferrari

https://textbookfull.com/product/computer-vision-eccv-2018-15th-
european-conference-munich-germany-
september-8-14-2018-proceedings-part-iii-vittorio-ferrari/

Computer Vision ECCV 2018 15th European Conference

Munich Germany September 8 14 2018 Proceedings Part IV
Vittorio Ferrari

https://textbookfull.com/product/computer-vision-eccv-2018-15th-
european-conference-munich-germany-
september-8-14-2018-proceedings-part-iv-vittorio-ferrari/

Computer Vision ECCV 2018 15th European Conference

Munich Germany September 8 14 2018 Proceedings Part VII
Vittorio Ferrari

https://textbookfull.com/product/computer-vision-eccv-2018-15th-
european-conference-munich-germany-
september-8-14-2018-proceedings-part-vii-vittorio-ferrari/
 Carlos Alario-Hoyos
María Jesús Rodríguez-Triana
Maren Scheffel
Inmaculada Arnedillo-Sánchez
Sebastian Maximilian Dennerlein (Eds.)

Addressing
LNCS 12315

Global Challenges
and Quality Education
15th European Conference
on Technology Enhanced Learning, EC-TEL 2020
Heidelberg, Germany, September 14–18, 2020
Proceedings
Lecture Notes in Computer Science 12315

Founding Editors
Gerhard Goos
Karlsruhe Institute of Technology, Karlsruhe, Germany
Juris Hartmanis
Cornell University, Ithaca, NY, USA

Editorial Board Members

Elisa Bertino
Purdue University, West Lafayette, IN, USA
Wen Gao
Peking University, Beijing, China
Bernhard Steffen
TU Dortmund University, Dortmund, Germany
Gerhard Woeginger
RWTH Aachen, Aachen, Germany
Moti Yung
Columbia University, New York, NY, USA
More information about this series at http://www.springer.com/series/7409
Carlos Alario-Hoyos María Jesús Rodríguez-Triana
• •

Maren Scheffel Inmaculada Arnedillo-Sánchez

• •

Sebastian Maximilian Dennerlein (Eds.)

Addressing
Global Challenges
and Quality Education
15th European Conference
on Technology Enhanced Learning, EC-TEL 2020
Heidelberg, Germany, September 14–18, 2020
Proceedings

123
Editors
Carlos Alario-Hoyos María Jesús Rodríguez-Triana
Universidad Carlos III de Madrid Tallinn University
Leganés (Madrid), Spain Tallinn, Estonia
Maren Scheffel Inmaculada Arnedillo-Sánchez
Open University Netherlands Trinity College Dublin
Heerlen, The Netherlands Dublin, Ireland
Sebastian Maximilian Dennerlein
Graz University of Technology
and Know-Center
Graz, Austria

ISSN 0302-9743 ISSN 1611-3349 (electronic)

Lecture Notes in Computer Science
ISBN 978-3-030-57716-2 ISBN 978-3-030-57717-9 (eBook)
https://doi.org/10.1007/978-3-030-57717-9

LNCS Sublibrary: SL3 – Information Systems and Applications, incl. Internet/Web, and HCI

© Springer Nature Switzerland AG 2020

Chapters 6, 10, 15 and 48 are licensed under the terms of the Creative Commons Attribution 4.0 International
License (http://creativecommons.org/licenses/by/4.0/). For further details see licence information in the
chapters.
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the
material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now
known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this book are
believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors
give a warranty, expressed or implied, with respect to the material contained herein or for any errors or
omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in
published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
 Preface

Welcome to the proceedings of the 15th European Conference on Technology

Enhanced Learning (EC-TEL 2020), one of the flagship events of the European
Association of Technology Enhanced Learning (EATEL). In addition to the social and
economic crisis, the pandemic has led to a historic inflection point in education. The
impact of COVID-19 worldwide and the closure of schools and universities has led to
numerous changes in how, where and with which tools we learn, and how education
can take place at a distance. One of these changes is that lockdowns triggered the use of
Technology-Enhanced Learning (TEL) at a massive level and throughout the world due
to the urgent need to transform face-to-face into online classes. Such circumstances
pose new challenges and research questions that our community must address.
Looking back, the theme of EC-TEL 2020 (Addressing Global Challenges and
Quality Education) could not have been more timely with COVID-19 affecting edu-
cation and its quality all over the world. Thus, the Sustainable Development Goals of
Quality Education and Reduced Inequalities, as defined in the United Nations Agenda
2030, are more essential than ever. In a global world of great contrasts, we are all
suffering the consequences of this unexpected pandemic, and it is paramount to find
global solutions to provide equitable quality education and promote lifelong learning
opportunities for all. This not only requires improving student performance and success
in general but also supporting under-represented groups and those disadvantaged by
inequality. In this challenging context, the active role of the TEL community may be
crucial to pursue the aforementioned goals.
Another important change this year is that EC-TEL 2020 will be held online for the
first time in its 15-year history. Although EC-TEL 2020 was initially planned to take
place in Heidelberg, Germany, local health authorities advised against organizing large
events. What was to be a major liability has been transformed into a powerful expe-
rience in which to continue learning and evolving as a community thanks to the
researchers who have continued to rely on this conference to submit and present their
contributions. This year, 91 contributions were received despite the difficulties of some
researchers to carry out developments and studies as well as reporting them. All
of these contributions were reviewed by three members of the TEL community, who
also had follow-up discussions to agree on a meta-review. As a result, 24 research
papers (26.4%) were accepted and presented at the conference. This shows the high
competitiveness and quality of this conference year after year. In addition, 20 posters, 5
demos, and 8 impact papers were presented during the conference to fuel the discus-
sions among the researchers. Research papers, posters, and demos can be found in this
volume, while impact papers are published in companion proceedings via CEUR.
EC-TEL 2020 was co-located with another major conference in Germany, the 18th
Fachtagung Bildungstechnologien der GI Fachgrupper Bildungstechnologien (DELFI
2020). This helped create partnerships between the two conferences and allowed
attendees to exchange ideas and benefit from presentations and workshops that were
vi Preface

offered in both conferences. As an example, 14 workshops and 4 keynotes (Linda

Castañeda, Jens Mönig, Samuel Greiff, and Sabine Seufert) were co-located between
EC-TEL 2020 and DELFI 2020.
The last words are words of gratitude. Thanks to the researchers who sent their
contributions to EC-TEL 2020. Thanks to the members of the Programme Committee
who devoted their time to give feedback to authors and supported decision making on
paper acceptance. Finally, deep thanks to the local organizers, Marco Kalz and Joshua
Weidlich, who have worked very hard to host this conference online with the same
quality as in previous years.

July 2020 Carlos Alario-Hoyos

María Jesús Rodríguez-Triana
Maren Scheffel
Inmaculada Arnedillo-Sánchez
Sebastian Dennerlein
Tracie Farrell Frey
Tom Broos
Zacharoula Papamitsiou
Adolfo Ruiz Calleja
Kairit Tammets
Christian Glahn
Marco Kalz
Joshua Weidlich
EC-TEL 2020 Chairs
 Organization

Program Committee
Marie-Helene Abel Université de Technologie de Compiègne, France
Andrea Adamoli Università della Svizzera Italiana, Switzerland
Nora Ayu Ahmad Uzir University of Edinburgh, UK
Gokce Akcayir University of Alberta, Canada
Carlos Alario-Hoyos Universidad Carlos III de Madrid, Spain
Patricia Albacete University of Pittsburgh, USA
Dietrich Albert University of Graz, Austria
Laia Albó Universitat Pompeu Fabra, Spain
Liaqat Ali Simon Fraser University, Canada
Ishari Amarasinghe Universitat Pompeu Fabra, Spain
Boyer Anne LORIA – KIWI, France
Alessandra Antonaci European Association of Distance Teaching
Universities, The Netherlands
Roberto Araya Universidad de Chile, Chile
Inmaculada Trinity College Dublin, Ireland
Arnedillo-Sanchez
Juan I. Asensio-Pérez Universidad de Valladolid, Spain
Antonio Balderas University of Cádiz, Spain
Nicolas Ballier Université de Paris, France
Jordan Barria-Pineda University of Pittsburgh, USA
Jason Bernard University of Saskatchewan, Canada
Anis Bey University of Paul Sabatier, IRIT, France
Miguel L. Bote-Lorenzo Universidad de Valladolid, Spain
François Bouchet Sorbonne Université, France
Yolaine Bourda LRI, CentraleSupélec, France
Bert Bredeweg University of Amsterdam, The Netherlands
Andreas Breiter Universität Bremen, Germany
Julien Broisin Université Toulouse 3 Paul Sabatier, IRIT, France
Tom Broos KU Leuven, Belgium
Armelle Brun LORIA, Université Nancy 2, France
Daniela Caballero Universidad de Chile, Chile
Manuel Caeiro Rodríguez University of Vigo, Spain
Sven Charleer KU Leuven, Belgium
Mohamed Amine Chatti University of Duisburg-Essen, Germany
John Cook University of West of England, UK
Audrey Cooke Curtin University, Australia
Alessia Coppi SFIVET, Switzerland
Mihai Dascalu University Politehnica of Bucharest, Romania
viii Organization

Peter de Lange RWTH Aachen University, Germany

Felipe de Morais UNISINOS, Brazil
Carrie Demmans Epp University of Alberta, Canada
Sebastian Dennerlein Know-Center, Austria
Michael Derntl University of Tübingen, Germany
Philippe Dessus Université Grenoble Alpes, LaRAC, France
Daniele Di Mitri Open Universiteit, The Netherlands
Yannis Dimitriadis Universidad de Valladolid, Spain
Monica Divitini Norwegian University of Science and Technology,
Norway
Juan Manuel Dodero Universidad de Cádiz, Spain
Raymond Elferink RayCom BV, The Netherlands
Erkan Er Universidad de Valladolid, Spain
Maka Eradze University of Modena and Reggio Emilia, Italy
Iria Estévez-Ayres Universidad Carlos III de Madrid, Spain
Juan Carlos Farah École Polytechnique Fédérale de Lausanne,
Switzerland
Tracie Farrell The Open University, UK
Louis Faucon École Polytechnique Fédérale de Lausanne,
Switzerland
Baltasar Fernandez-Manjon Universidad Complutense de Madrid, Spain
Carmen Universidad Carlos III de Madrid, Spain
Fernández-Panadero
Angela Fessl Know-Center, Austria
Anna Filighera Technical University of Darmstadt, Multimedia
Communications Lab, Germany
Olga Firssova WELTEN Institute, Open Universiteit,
The Netherlands
Mikhail Fominykh Norwegian University of Science and Technology,
Norway
Felix J. Garcia Clemente Universidad de Murcia, Spain
Jesús Miguel Universidad de la Sierra, Mexico
García-Gorrostieta
Serge Garlatti IMT Atlantique, France
Dragan Gasevic Monash University, Australia
Sébastien George LIUM, Le Mans Université, France
Carlo Giovannella University of Tor Vergata, Italy
Christian Glahn LDE CEL, The Netherlands
Samuel González-López Technological University of Nogales, Mexico
Sabine Graf Athabasca University, Canada
Monique Grandbastien LORIA, Université de Lorraine, France
Wolfgang Greller Vienna University of Education, Austria
David Griffiths University of Bolton, UK
Nathalie Guin Université de Lyon, France
Bernhard Göschlberger Research Studios, Austria
 Organization ix

Franziska Günther TU Dresden, Germany

Christian Gütl Graz University of Technology, Austria
Carolin Hahnel DIPF, Leibniz Institute for Research and Information
in Education, Germany
Stuart Hallifax Laboratoire d’InfoRmatique en Image et Systèmes
d'information, France
Bastiaan Heeren Open Universiteit, The Netherlands
Maartje Henderikx Open Universiteit, The Netherlands
Davinia Hernandez-Leo Universitat Pompeu Fabra, Spain
Ángel Hernández-García Universidad Politécnica de Madrid, Spain
Tore Hoel Oslo Metropolitan University, Norway
Adrian Holzer University of Neuchâtel, Switzerland
Pasquale Iero The Open University, UK
Francisco Iniesto The Open University, UK
Zeeshan Jan The Open University, UK
Patricia Jaques PPGCA, UNISINOS, Brazil
Johan Jeuring Utrecht University and Open Universiteit,
The Netherlands
Ioana Jivet Open Universiteit, The Netherlands
Srecko Joksimovic University of South Australia, Australia
Ilkka Jormanainen University of Eastern Finland, Finland
Jelena Jovanovic University of Belgrade, Serbia
Ken Kahn University of Oxford, UK
Rogers Kaliisa University of Oslo, Norway
Marco Kalz PH Heidelberg, Germany
Anastasios Karakostas Aristotle University of Thessaloniki, Greece
Reet Kasepalu Tallinn University, Estonia
Mohammad Khalil University of Bergen, Norway
Michael Kickmeier-Rust Graz University of Technology, Austria
Ralf Klamma RWTH Aachen University, Germany
Roland Klemke Open Universiteit, The Netherlands
Tomaž Klobučar Jozef Stefan Institute, Slovenia
Carolien Knoop-Van Radboud University, The Netherlands
Campen
Johannes Konert Hochschule Fulda, Germay
Külli Kori Tallinn University, Estonia
Panagiotis Kosmas Cyprus University of Technology, Cyprus
Vitomir Kovanovic University of South Australia, Australia
Dominik Kowald Know-Center, Austria
Milos Kravcik DFKI GmbH, Germay
Elise Lavoué Université Jean Moulin Lyon 3, France
Marie Lefevre Université Lyon 1, France
Dominique Lenne Université de Technologie de Compiègne, France
Marina Lepp University of Tartu, Estonia
Amna Liaqat University of Toronto, Canada
Paul Libbrecht IUBH Fernstudium, Germany
x Organization

Andreas Lingnau Ruhr West University of Applied Science, Germany

Martin Llamas-Nistal Universidad de Vigo, Spain
Aurelio Lopez-Lopez INAOE, Mexico
Domitile Lourdeaux CNRS, France
Margarida Lucas University of Aveiro, Portugal
Ulrike Lucke University of Potsdam, Germany
Vanda Luengo Sorbonne Université, France
George Magoulas University of London, UK
Jorge Maldonado-Mahauad Universidad de Cuenca, Ecuador, Pontificia
Universidad Católica de Chile, Chile
Nils Malzahn Rhine-Ruhr Institute for Applied System Innovation
e.V., Germany
Carlos Martínez-Gaitero Escuelas Universitarias Gimbernat, Spain
Alejandra Martínez-Monés Universidad de Valladolid, Spain
Wannisa Matcha University of Edinburgh, UK
Manolis Mavrikis UCL Knowledge Lab, UK
Agathe Merceron Beuth University of Applied Sciences, Germany
Vasileios Mezaris Information Technologies Institute, CERT, Greece
Christine Michel Université de Lyon, France
Konstantinos Michos Universidad de Valladolid, Spain
Alexander Mikroyannidis The Open University, UK
Tanja Mitrovic University of Canterbury, UK
Riichiro Mizoguchi Japan Advanced Institute of Science and Technology,
Japan
Inge Molenaar Radboud University, The Netherlands
Anders Morch University of Oslo, Norway
Pedro Manuel Universidad Carlos III de Madrid, Spain
Moreno-Marcos
Mathieu Muratet Sorbonne Université, France
Pedro J. Muñoz-Merino Universidad Carlos III de Madrid, Spain
Rob Nadolski Open Universiteit, The Netherlands
Petru Nicolaescu RWTH Aachen University, Germany
Stavros Nikou University of Strathclyde, UK
Nicolae Nistor Ludwig Maximilian University of Munich, Germany
Alexander Nussbaumer Graz University of Technology, Austria
Jennifer Olsen École Polytechnique Fédérale de Lausanne,
Switzerland
Alejandro Ortega-Arranz Universidad de Valladolid, Spain
Lahcen Oubahssi LIUM, Le Mans Université, France
Viktoria Pammer-Schindler Graz University of Technology, Austria
Sofia Papavlasopoulou Norwegian University of Science and Technology,
Norway
Abelardo Pardo University of South Australia, Australia
Cesare Pautasso University of Lugano, Switzerland
Maxime Pedrotti Verein zur Förderung eines Deutschen
Forschungsnetzes e.V., Germany
 Organization xi

Mar Perez-Sanagustin Université Paul Sabatier Toulouse III, France

Donatella Persico Istituto Tecnologie Didattiche, CNR, Italy
Yvan Peter Université de Lille, France
Niels Pinkwart Humboldt-Universität zu Berlin, Germany
Gerti Pishtari Tallinn University, Estonia
Elvira Popescu University of Craiova, Romania
Francesca Pozzi Istituto Tecnologie Didattiche, CNR, Italy
Luis P. Prieto Tallinn University, Estonia
Ronald Pérez Álvarez Universidad de Costa Rica, Costa Rica
Hans Põldoja Tallinn University, Estonia
Eyal Rabin The Open University of Israel, Israel, and Open
Universiteit, The Netherlands
Juliana Elisa Raffaghelli University of Florence, Italy
Gustavo Ramirez-Gonzalez Universidad del Cauca, Colombia
Marc Rittberger DIPF, Leibniz Institute for Researach and Information
in Education, Germany
Tiago Roberto Kautzmann Universidade do Vale do Rio dos Sinos, Brazil
Covadonga Rodrigo Universidad Nacional de Educación a Distancia, Spain
Maria Jesus Rodriguez Tallinn University, Estonia
Triana
Jeremy Roschelle Digital Promise, USA
José A. Ruipérez Valiente Universidad de Murcia, Spain
Ellen Rusman Open Universiteit, The Netherlands
Merike Saar Tallinn University, Estonia
Demetrios Sampson Curtin University, Australia
Eric Sanchez Université Fribourg, Switzerland
Patricia Santos Universitat Pompeu Fabra, Spain
Mohammed Saqr University of Eastern Finland, Finland
Petra Sauer Beuth University of Applied Sciences, Germany
Maren Scheffel Open Universiteit, The Netherlands
Daniel Schiffner DIPF, Leibniz Institute for Research and Information
in Education, Germany
Andreas Schmidt Karlsruhe University of Applied Sciences, Germany
Marcel Schmitz Zuyd Hogeschool, The Netherlands
Jan Schneider DIPF, Leibniz Institute for Research and Information
in Education, Germany
Ulrik Schroeder RWTH Aachen University, Germany
Stefan Schweiger Bongfish GmbH, Austria
Yann Secq Université de Lille, France
Karim Sehaba Laboratoire d’InfoRmatique en Image et Systèmes
d’information, Université Lumière Lyon 2, France
Paul Seitlinger Tallinn University, Estonia
Audrey Serna Laboratoire d’InfoRmatique en Image et Systèmes
d’information, France
Sergio Serrano-Iglesias Universidad de Valladolid, Spain
Shashi Kant Shankar Tallinn University, Estonia
xii Organization

Kshitij Sharma Norwegian University of Science and Technology,

Norway
Tanmay Sinha ETH Zurich, Switzerland
Sergey Sosnovsky Utrecht University, The Netherlands
Marcus Specht Delft University of Technology and Open Universiteit,
The Netherlands
Srinath Srinivasa International Institute of Information Technology,
Bangalore, India
Tim Steuer Technical University of Darmstadt, Germany
Slavi Stoyanov Open University, The Netherlands
Alexander Streicher Fraunhofer IOSB, Germany
Bernardo Tabuenca Universidad Politécnica de Madrid, Spain
Kairit Tammets Tallinn University, Estonia
Stefaan Ternier Open Universiteit, The Netherlands
Stefan Thalmann University of Graz, Austria
Paraskevi Topali Universidad de Valladolid, Spain
Richard Tortorella University of North Texas, USA
Lucia Uguina IMDEA Networks, Spain
Peter Van Rosmalen Maastricht University, The Netherlands
Olga Viberg KTH Royal Institute of Technology, Sweden
Markel Vigo The University of Manchester, UK
Sara Villagrá-Sobrino Universidad de Valladolid, Spain
Cathrin Vogel FernUniversität in Hagen, Germany
Joshua Weidlich Heidelberg University of Education, Germany
Armin Weinberger Saarland University, Germany
Professor Denise Whitelock The Open University, UK
Fridolin Wild The Open University, UK
Nikoletta Xenofontos University of Cyprus, Cyprus
Amel Yessad Sorbonne Université, France

Additional Reviewers

Anwar, Muhammad
Berns, Anke
Ebner, Markus
Ehlenz, Matthias
Halbherr, Tobias
Koenigstorfer, Florian
Kothiyal, Aditi
Liaqat, Daniyal
Liaqat, Salaar
Müllner, Peter
Ponce Mendoza, Ulises
Raffaghelli, Juliana
Rodriguez, Indelfonso
Zeiringer, Johannes
 Contents

Exploring Artificial Jabbering for Automatic Text Comprehension

Question Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Tim Steuer, Anna Filighera, and Christoph Rensing

Digital Value-Adding Chains in Vocational Education: Automatic

Keyword Extraction from Learning Videos to Provide Learning
Resource Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Cleo Schulten, Sven Manske, Angela Langner-Thiele,
and H. Ulrich Hoppe

Human-Centered Design of a Dashboard on Students’ Revisions

During Writing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Rianne Conijn, Luuk Van Waes, and Menno van Zaanen

An Operational Framework for Evaluating the Performance of Learning

Record Stores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Chahrazed Labba, Azim Roussanaly, and Anne Boyer

Does an E-mail Reminder Intervention with Learning Analytics Reduce

Procrastination in a Blended University Course? . . . . . . . . . . . . . . . . . . . . . 60
Iryna Nikolayeva, Amel Yessad, Bertrand Laforge, and Vanda Luengo

Designing an Online Self-assessment for Informed Study Decisions:

The User Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
L. E. C. Delnoij, J. P. W. Janssen, K. J. H. Dirkx, and R. L. Martens

What Teachers Need for Orchestrating Robotic Classrooms . . . . . . . . . . . . . 87

Sina Shahmoradi, Aditi Kothiyal, Jennifer K. Olsen, Barbara Bruno,
and Pierre Dillenbourg

Assessing Teacher’s Discourse Effect on Students’ Learning: A Keyword

Centrality Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
Danner Schlotterbeck, Roberto Araya, Daniela Caballero,
Abelino Jimenez, Sami Lehesvuori, and Jouni Viiri

For Learners, with Learners: Identifying Indicators for an Academic

Advising Dashboard for Students . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Isabel Hilliger, Tinne De Laet, Valeria Henríquez, Julio Guerra,
Margarita Ortiz-Rojas, Miguel Ángel Zuñiga, Jorge Baier,
and Mar Pérez-Sanagustín
xiv Contents

Living with Learning Difficulties: Two Case Studies Exploring

the Relationship Between Emotion and Performance in Students
with Learning Difficulties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
Styliani Siouli, Stylianos Makris, Evangelia Romanopoulou,
and Panagiotis P. D. Bamidis

Learnersourcing Quality Assessment of Explanations for Peer Instruction . . . 144

Sameer Bhatnagar, Amal Zouaq, Michel C. Desmarais,
and Elizabeth Charles

Using Diffusion Network Analytics to Examine and Support Knowledge

Construction in CSCL Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
Mohammed Saqr and Olga Viberg

Supporting Second Language Learners’ Development of Affective

Self-regulated Learning Skills Through the Use and Design
of Mobile Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Olga Viberg, Anna Mavroudi, and Yanwen Ma

We Know What You Did Last Semester: Learners’ Perspectives on Screen

Recordings as a Long-Term Data Source for Learning Analytics. . . . . . . . . . 187
Philipp Krieter, Michael Viertel, and Andreas Breiter

Teaching Simulation Literacy with Evacuations: Concept, Technology,

and Material for a Novel Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
Andre Greubel, Hans-Stefan Siller, and Martin Hennecke

Design of Conversational Agents for CSCL: Comparing Two Types

of Agent Intervention Strategies in a University Classroom . . . . . . . . . . . . . 215
Konstantinos Michos, Juan I. Asensio-Pérez, Yannis Dimitriadis,
Sara García-Sastre, Sara Villagrá-Sobrino, Alejandro Ortega-Arranz,
Eduardo Gómez-Sánchez, and Paraskevi Topali

Exploring Human–AI Control Over Dynamic Transitions Between

Individual and Collaborative Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
Vanessa Echeverria, Kenneth Holstein, Jennifer Huang,
Jonathan Sewall, Nikol Rummel, and Vincent Aleven

Exploring Student-Controlled Social Comparison . . . . . . . . . . . . . . . . . . . . 244

Kamil Akhuseyinoglu, Jordan Barria-Pineda, Sergey Sosnovsky,
Anna-Lena Lamprecht, Julio Guerra, and Peter Brusilovsky

New Measures for Offline Evaluation of Learning Path Recommenders . . . . . 259

Zhao Zhang, Armelle Brun, and Anne Boyer
 Contents xv

Assessing the Impact of the Combination of Self-directed Learning,

Immediate Feedback and Visualizations on Student Engagement
in Online Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
Bilal Yousuf, Owen Conlan, and Vincent Wade

CGVis: A Visualization-Based Learning Platform for Computational

Geometry Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
Athanasios Voulodimos, Paraskevas Karagiannopoulos,
Ifigenia Drosouli, and Georgios Miaoulis

How to Design Effective Learning Analytics Indicators? A Human-

Centered Design Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
Mohamed Amine Chatti, Arham Muslim, Mouadh Guesmi,
Florian Richtscheid, Dawood Nasimi, Amin Shahin, and Ritesh Damera

Emergency Remote Teaching: Capturing Teacher Experiences in Spain

with SELFIE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
Laia Albó, Marc Beardsley, Judit Martínez-Moreno, Patricia Santos,
and Davinia Hernández-Leo

Utilising Learnersourcing to Inform Design Loop Adaptivity . . . . . . . . . . . . 332

Ali Darvishi, Hassan Khosravi, and Shazia Sadiq

Fooling It - Student Attacks on Automatic Short Answer Grading. . . . . . . . . 347

Anna Filighera, Tim Steuer, and Christoph Rensing

Beyond Indicators: A Scoping Review of the Academic Literature

Related to SDG4 and Educational Technology . . . . . . . . . . . . . . . . . . . . . . 353
Katy Jordan

Pedagogical Underpinnings of Open Science, Citizen Science

and Open Innovation Activities: A State-of-the-Art Analysis. . . . . . . . . . . . . 358
Elisha Anne Teo and Evangelia Triantafyllou

Knowledge-Driven Wikipedia Article Recommendation

for Electronic Textbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
Behnam Rahdari, Peter Brusilovsky, Khushboo Thaker,
and Jordan Barria-Pineda

InfoBiTS: A Mobile Application to Foster Digital Competencies

of Senior Citizens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
Svenja Noichl and Ulrik Schroeder

Student Awareness and Privacy Perception of Learning Analytics

in Higher Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374
Stian Botnevik, Mohammad Khalil, and Barbara Wasson
xvi Contents

User Assistance for Serious Games Using Hidden Markov Model . . . . . . . . . 380
Vivek Yadav, Alexander Streicher, and Ajinkya Prabhune

Guiding Socio-Technical Reflection of Ethical Principles in TEL Software

Development: The SREP Framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
Sebastian Dennerlein, Christof Wolf-Brenner, Robert Gutounig,
Stefan Schweiger, and Viktoria Pammer-Schindler

Git4School: A Dashboard for Supporting Teacher Interventions in Software

Engineering Courses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392
Jean-Baptiste Raclet and Franck Silvestre

Exploring the Design and Impact of Online Exercises for Teacher Training
About Dynamic Models in Mathematics. . . . . . . . . . . . . . . . . . . . . . . . . . . 398
Charlie ter Horst, Laura Kubbe, Bart van de Rotten, Koen Peters,
Anders Bouwer, and Bert Bredeweg

Interactive Concept Cartoons: Exploring an Instrument for Developing

Scientific Literacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404
Patricia Kruit and Bert Bredeweg

Quality Evaluation of Open Educational Resources . . . . . . . . . . . . . . . . . . . 410

Mirette Elias, Allard Oelen, Mohammadreza Tavakoli, Gábor Kismihok,
and Sören Auer

Designing Digital Activities to Screen Locomotor Skills

in Developing Children . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416
Benoit Bossavit and Inmaculada Arnedillo-Sánchez

Towards Adaptive Social Comparison for Education . . . . . . . . . . . . . . . . . . 421

Sergey Sosnovsky, Qixiang Fang, Benjamin de Vries, Sven Luehof,
and Fred Wiegant

Simulation Based Assessment of Epistemological Beliefs About Science . . . . 427

Melanie E. Peffer and Tessa Youmans

An Approach to Support Interactive Activities in Live Stream Lectures . . . . . 432

Tommy Kubica, Tenshi Hara, Iris Braun, and Alexander Schill

Educational Escape Games for Mixed Reality . . . . . . . . . . . . . . . . . . . . . . . 437

Ralf Klamma, Daniel Sous, Benedikt Hensen, and István Koren

Measuring Learning Progress for Serving Immediate Feedback Needs:

Learning Process Quantification Framework (LPQF) . . . . . . . . . . . . . . . . . . 443
Gayane Sedrakyan, Sebastian Dannerlein, Viktoria Pammer-Schindler,
and Stefanie Lindstaedt
 Contents xvii

Data-Driven Game Design: The Case of Difficulty in Educational Games . . . 449

Yoon Jeon Kim and Jose A. Ruipérez-Valiente

Extracting Topics from Open Educational Resources . . . . . . . . . . . . . . . . . . 455

Mohammadreza Molavi, Mohammadreza Tavakoli, and Gábor Kismihók

Supporting Gamification with an Interactive Gamification Analytics

Tool (IGAT). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461
Nadja Zaric, Manuel Gottschlich, Rene Roepke, and Ulrik Schroeder

OpenLAIR an Open Learning Analytics Indicator Repository Dashboard . . . . 467

Atezaz Ahmad, Jan Schneider, and Hendrik Drachsler

CasualLearn: A Smart Application to Learn History of Art. . . . . . . . . . . . . . 472

Adolfo Ruiz-Calleja, Miguel L. Bote-Lorenzo, Guillermo Vega-Gorgojo,
Sergio Serrano-Iglesias, Pablo García-Zarza, Juan I. Asensio-Pérez,
and Eduardo Gómez-Sánchez

Applying Instructional Design Principles on Augmented Reality Cards

for Computer Science Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 477
Josef Buchner and Michael Kerres

Extending Patient Education with CLAIRE: An Interactive Virtual Reality

and Voice User Interface Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482
Richard May and Kerstin Denecke

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487

 Exploring Artificial Jabbering
for Automatic Text Comprehension
Question Generation

Tim Steuer(B) , Anna Filighera , and Christoph Rensing

Technical University of Darmstadt, Darmstadt, Germany

{tim.steuer,anna.filighera,christoph.rensing}@kom.tu-darmstadt.de

Abstract. Many educational texts lack comprehension questions and

authoring them consumes time and money. Thus, in this article, we ask
ourselves to what extent artificial jabbering text generation systems can
be used to generate textbook comprehension questions. Novel machine
learning-based text generation systems jabber on a wide variety of top-
ics with deceptively good performance. To expose the generated texts as
such, one often has to understand the actual topic the systems jabbers
about. Hence, confronting learners with generated texts may cause them
to question their level of knowledge. We built a novel prototype that
generates comprehension questions given arbitrary textbook passages.
We discuss the strengths and weaknesses of the prototype quantitatively
and qualitatively. While our prototype is not perfect, we provide evidence
that such systems have great potential as question generators and iden-
tify the most promising starting points may leading to (semi) automated
generators that support textbook authors and self-studying.

Keywords: Text comprehension · Language models · Automatic

question generation · Educational technology

1 Motivation

Reading, alongside direct verbal communication, is one of the most prevalent

forms of learning. For every new subject, we encounter in our educational careers,
highly motivated educators publish textbooks to help us understand. Even after
we finish our formal education, the modern knowledge society is based on life-
long informal learning in which learners in the absence of teachers, also often
devote themselves to textual learning resources. In both, the formal and informal
scenarios only gaining surface-level understanding is likely not enough. If we e.g.
study a physics or history textbook to pass an exam deeper understanding of
the topic is crucial. However, reading is difficult and to deeply comprehend a
text, passive consumption is insufficient [7,25].
Instead, readers need to actively reflect the information provided in the text
to reach a deep understanding [7,25]. A well-explored method to actively engage
c Springer Nature Switzerland AG 2020
C. Alario-Hoyos et al. (Eds.): EC-TEL 2020, LNCS 12315, pp. 1–14, 2020.
https://doi.org/10.1007/978-3-030-57717-9_1
2 T. Steuer et al.

Suppose you want to connect to your workplace network from home. Your workplace,
however, has a security policy that does not allow “outside” IP addresses to access essenƟal
internal resources. How do you proceed, without leasing a dedicated telecommunicaƟons
Textbook line to your workplace?
chapter
A virtual private network, or VPN, provides a soluƟon; it supports creaƟon of virtual links
that join far-flung nodes via the Internet. Your home computer creates an ordinary Internet
connecƟon (TCP or UDP) …

Generated Is the following statement true/false? Please discuss briefly why it is true or false: Vpns
Discussion have the disadvantage of requiring the VPN tunnel to be established before the Internet can
Prompt be accessed.
(italic)

Student
discusses
answer

Fig. 1. Example usage of the proposed system.

readers is posing questioning about what they have read [1,25]. Yet, posing
good questions consumes time and money and thus many texts encountered
by learners either contain only a few questions at the end of a chapter or lack
questions.
Educational automatic question generation investigates approaches to gen-
erate meaningful questions about texts automatically, reducing the necessity for
manually generated questions. It hereby relies either on machine learning-based
approaches that excel in question variety and expressiveness but pose mostly
factual questions [6] or on rule-based approaches that lack expressiveness and
variety [32] but have limited capability to pose comprehension questions depend-
ing on their purpose (e.g. [17]).
This article investigates a novel machine learning-based question generation
approach seeking to generate comprehension questions with a high variety and
expressiveness. We hereby rely on two main ideas. First, research in the educa-
tional domain has investigated learning from errors [19] indicating that explain-
ing why a statement or solution is faulty may foster learning, conceptual under-
standing, and far transfer [10]. Second, we rely on the artificial jabbering of
state-of-the-art neural text generators that are capable of extrapolating a given
text with high structural consistency and in a way that often looks deceptively
real for humans. We seek to explore whether this jabbering can be conditioned
in such a way that it generates erroneous examples from textbook paragraphs.
Presented with such a statement, learners need to justify if a given statement is
true or false (see Fig. 1). This work comprises three main contributions:

1. We present the idea of leveraging artificial jabbering for automatic text com-
prehension question generation and introduce a prototypical generator.
2. We provide a quantitative and qualitative evaluation of the strengths and
weaknesses of such an approach.
3. We distill the main challenges for future work based on an in-depth error
analysis of our prototypical generator.
 Automatic Question Generation via Neural Text Generators 3

2 Related Work
2.1 Learning from Erroneous Examples

When learning with erroneous examples, students are confronted with a task and
its faulty solution and have to explain why it is wrong (e.g. [30]). The underlying
theoretical assumptions are that erroneous examples induce a cognitive conflict
in students and thus support conceptual change [24] e.g. by pointing out typical
misconceptions [29]. It has been shown that erroneous examples are beneficial
for learning in a variety of domains such as mathematics [10], computer science
[4] or medicine [14]. Also, learners confronted with erroneous examples especially
improve deeper measures of learning such as conceptual understanding and far
transfer [24]. However, some studies have found that erroneous examples only
foster learning when learners receive enough feedback [14,30] and have sufficient
prior knowledge [30].

2.2 Neural Text and Question Generation

With the rise of high capacity machine-learning models, language generation has
shifted towards pretraining [27]. Trained on huge datasets, these models provide
state-of-the-art results on a wide variety of natural language generation tasks
[5,23] such as dialog response generation tasks [22] or abstractive summariza-
tion tasks [26]. Novel models like GPT-2 [23] are capable of extrapolating a given
text with high structural consistency and in a way that looks deceptively real for
humans. They copy the given text’s writing style and compose texts which seem
to make sense at first glance. Fine-tuning the model even increased the human-
ness of the generated texts [28]. Research in the credibility of such generated
texts found that hand-picked generated news texts were found to be credible
around 66% of the time, even when the model was not fine-tuned on news arti-
cles [28]. Another study found that human raters could detect generated texts in
71.4% of the cases with two raters often disagreeing if the text is fake or not [13].
These findings started a debate in the natural language generation community if
the model’s generation capabilities are to easy to misuse and therefore the mod-
els should not be released anymore [28]. Furthermore, such models are able to
generate poems [16] and to rewrite stories to incorporate counterfactual events
[21]. Besides of these open text generation models, special models for question
generation exist. They evolved from baseline sequence to sequence architectures
[6] into several advanced neural architectures (e.g. [5,33]) with different facets
such as taking the desired answers into account [34] or being difficulty-aware [8].
Although these systems work well in the general case they are mainly focusing
on the generation of factual questions [6,20,35]. Thus, although their expressive-
ness and domain independence is impressive, the educational domain still most
often uses template-based generators [15]. These template-based approaches are
often able to generate comprehension questions but lack expressiveness and rely
on expert rules limiting them to a specific purpose in a specific domain.
4 T. Steuer et al.

3 An Experimental Automatic Erroneous Example

Generator

To experiment with the idea of using artificial jabbering for improving text
comprehension, we propose the following text generation task. The input is a
text passage of a learning resource from an arbitrary domain, having a length
of 500–1000 words as this has been used in psychological studies that found
text accompanying questions to be helpful [1,31]. The output is a generated
text comprehension question about the given text passage, asking learners to
explain why a given statement is true or false. We aim to generate high-quality
questions of good grammaticality, containing educational valuable claims and
having the right difficulty for discussion. Some technical challenges are inherent
in the described task. Every approach must tackle discussion candidate selection
as this determines what the main subject of the generated text will be. Also,
every approach must provide the neural text generator with a conditioning con-
text to ensure that the generated text is in the intended domain. Finally, every
approach must render the actual text with some sort of open domain genera-
tor. These subtasks are active fields of research and a huge variety of possible
approaches with different strengths and weaknesses exists. Yet, our first aim is
to evaluate the general viability of such an approach. Thus, we do not exper-
iment with different combinations of sub-components but our generator relies
on well-tested domain-independent general-purpose algorithms for the different
subtasks (see Fig. 2).

Fig. 2. Architecture of the automatic text comprehension question generator. The final
output is a justification statement that is combined with a prompt to form the actual
text comprehension question.
 Automatic Question Generation via Neural Text Generators 5

First, for the discussion candidate selection, we make the simplifying assump-
tion that good discussion candidates are the concepts that are characteristic of
the text. To understand why this assumption is simplified consider a text about
Newtonian physics where a few sentences discuss the common misconception
that heavier objects fall faster than lighter objects. This discussion is unlikely to
involve any special keywords and thus will not be selected as input to the gen-
erator. Yet, it might be very fruitful to generate erroneous examples based on
these misconceptions. However, to test our general idea of generating erroneous
examples the simplification should be sufficient because we might select fewer
inputs but the one we select should be important. Furthermore, this assumption
allows us to rely on state-of-the-art keyphrase extraction algorithms. Considering
that the inputs are texts from a variety of domains, the keyphrase selection step
needs to be unsupervised and relatively robust to domain changes. Therefore,
we apply the YAKE keyphrase extraction algorithm [3] which has been shown
to perform consistently on a large variety of different datasets and domains [2].
Stopwords are removed before running keyphrase extraction and the algorithm’s
configured windows size is two.
Second, for selecting the conditioning context, a short text that already com-
prises statements about the subject is needed. Suppose the discussion subject
is “Thermal Equilibrium” in a text about physics. For the generator to pro-
duce interesting statements it must receive sentences from the text, discussing
thermal equilibria. Thus, we extract up to three sentences in the text compris-
ing the keyphrase, by sentence tokenizing the text1 and concatenating sentences
containing the keyphrase.
Third, we need to generate a justification statement as the core for the text
comprehension question. We use the pretrained GPT-2 774M2 parameter model
and apply it similar to Radford et al. [23] by using plain text for the model
conditioning. The plain text starts with the sentences from the conditioning
context and to generate the actual justification statement, a discussion starter is
appended. It begins with the pluralized discussion subject followed by a prede-
fined phrase allowing us to choose the type of justification statement the model
will generate. For instance, let “Thermal Equilibrium” be our discussion subject,
our to be completed discussion starter may be “Thermal equilibria are defined
as” or “Thermal equilibria can be used to” depending on the type of faulty state-
ment we aim for. The resulting plain text is given to GPT-2 for completion. To
prevent the model from sampling degenerated text, we apply nucleus sampling
[12] with top-p=0.55 and restrict the output length to 70 words. Finally, we
extract the justification statement from the generated text and combine it with
a generic prompt to discuss it, resulting in the final text comprehension question.
Note that we do not know, if the generated question is actually comprising a
true or false justification statement.

1
Using NLTK-3.4.5.
2
https://github.com/openai/gpt-2.
6 T. Steuer et al.

4 Research Question and Methodology

4.1 Research Question
We evaluate our generation approach on educational texts from a variety of
domains focusing on the following research question:
RQ: To what extent are we able to generate useful text comprehension statements
in a variety of domains given short textbook passages?
Looking at the related work, a fraction of the generated statements should
already be usable without any adjustments, while many other statements need
adjustment. We conduct a quantitative evaluation and qualitative evaluation.
Our procedure includes a quantitative expert survey, a qualitative error analysis
to determine useful error categories and a qualitative analysis of the already
usable results to better describe their features.

4.2 Methodology
Quantitatively, a total of 120 text comprehension questions coming from ten
educational texts are annotated by ten domain experts who have been teaching
at least one university lecture in a similar domain. Texts are equally distributed
across five different domains: Computer Science, Machine Learning, Networking,
Physics and Psychology. Twelve text comprehension questions are generated for
every text. They are based on three extracted discussion candidates and four
different discussion starters, of which we hypothesized that they represent inter-
mediate or deep questions according to Graesser et al. [9]. The discussion starters
are: “X has the disadvantage”, “X has the advantage”, “X is defined as” and “X
is used to” where X is the discussion candidate. This Every question is rated by
two experts who first read the educational text that was used to generate the
question and then rate it on five five-point Likert items regarding grammatical
correctness, relatedness to the source material, factual knowledge involved when
answering the question, conceptual knowledge involved when answering the ques-
tions and overall usefulness for learning. Before annotating every expert saw a
short definition of every scale, clarifying their meaning. Additionally, experts
can provide qualitative remarks for every question through a free-text field. For
the quantitative analysis the ratings where averaged across experts.
We use the quantitatively collected data to guide our qualitative analysis of
the research questions. To carry out our in-depth error analysis, we consider a
statement useless for learning if it scores lower than three on the usefulness scale.
This choice was made after qualitatively reviewing a number of examples. We
use the inductive qualitative content analysis [18] to deduce meaningful error
categories for the statements and to categorize the statements accordingly. Our
search for meaningful error categories is hereby guided by the given task for-
mulation and its sub-components. Furthermore, the useful generated statements
(usefulness ≥ 3) are analyzed. We look at the effects of the different discus-
sion starters and how they influence the knowledge involved in answering the
generated questions.
 Automatic Question Generation via Neural Text Generators 7

5 Results
5.1 Quantitative Overview

The quantitative survey results indicate that many of the statements generated
are of good grammar, are connected to the text but are only slightly useful for
learning (see Fig. 3). Furthermore, most questions involve some factual knowl-
edge and deeper comprehension, yet both scores vary greatly. Breaking down the
different rating scores by domain or discussion starter does not revealed no large
differences. By looking at various examples of different ratings (see Table 1) we
found that a usefulness score of three or larger is indicative of some pedagogi-
cal value. With minor changes, such questions could be answered and discussed
by experts, although their discussion is probably often not the perfect learning
opportunity. In total, 39 of the 120 statements have a usefulness rating of 3 or
larger (32.5%), in contrast to 81 statements rated lower (67.5%).

Fig. 3. Overview of the quantitative ratings for the generated statements without any
human filtering. Scores are between 1 and 5 where 5 is the best achievable rating. The
whiskers indicate 1.5 Interquartile range and the black bar is the median.

5.2 Qualitative Error Analysis

While conducting the qualitative error analysis, the following main error cate-
gories where deduced. Keyword inappropriate means that the discussion candi-
date was not appropriate for the text because the keyword extraction algorithm
selected a misleading or very general key term. Keyword incomplete means that
8 T. Steuer et al.

Table 1. Examples of differently rated generated statements (higher = better).

Usefulness Example statement

ranking
1 Fastest possible machines have the disadvantage of being more
expensive to build and maintain
2 Prior knowledges have the advantage that they are easy to
measure and easy to measure the causal role of
3 Knowledge bases can be used to test the performance of models,
and to improve the performance of inference engines
4 Von neumann architectures have the advantage of being able to
process a wide range of instructions at the same time, making
them highly scalable
5 Vpns have the disadvantage of being difficult to set up and
maintain, and they can be compromised by bad actors

the discussion candidate would be good if it would comprise additional terms.

For example, in physics, the discussion candidate sometimes was “Equilibrium”
instead of “Thermal Equilibrium”. Platitude means that the generated state-
ment was a generic platitude and thus not helpful. Hardly discussable means
that the statement was either too vague or too convoluted therefore making it
hard to write a good justification. Finally too easy means that the students could
answer by just relying on common sense.

Table 2. The different error categories and their distribution

Inappropriate Incomplete Platitude Hardly Statement

keyword keyword discussable too easy
43 6 9 11 12

The distribution of the different error categories can be seen is heavily skewed
towards keyword errors (see Table 2). The two keyword-based errors account for
49 or roughly 60% of the errors. Furthermore, statements generated by faulty
keyword selection mostly have a usefulness rating of one. The other error cate-
gories are almost equally distributed and are most often rated with a usefulness
score of two. The platitude case mostly comes from unnaturally combining a dis-
cussion candidate with a discussion starter resulting in very generic completion of
the sentence inside the generator. For instance, if the generator has to complete
the sentence “Classical conditionings have the disadvantage ...” it continues with
“...of being costly and slow to develop”. The remaining error categories have no
clear cause.
Besides the error analysis, annotators left some remarks about the erroneous
statements. Two annotators remarked on various occasions that the first part of
 Automatic Question Generation via Neural Text Generators 9

the sentence (discussion candidate + discussion starter) is incomprehensible and

thus the whole statement is worthless. One annotator remarked that there are
missing words in the keyword leading to a bad rating for the statement. The key-
word was for example “knowledge” instead of “knowledge base”. Furthermore,
one annotator remarked that the statement has not enough discussion potential.
Those comments are in line with our deduced error categories for keyword errors.

Fig. 4. Overview of the quantitative ratings for the generated statements with an
usefulness rating larger or equal three. Scores are between 1 and 5 where 5 is the best
achievable rating. The whiskers indicate 1.5 Interquartile range and the black bar is
the median.

5.3 Quality Characteristics of the Useful Statements

The 39 statements with a usefulness rating larger three also score well in the
other factors (see Fig. 4). Especially, the involved factual knowledge and deeper
comprehension clearly increase. Reviewing the generated statements reveals that
the generated statements are not simply a paraphrase of a fact stated in the text.
Thus, learners answering the corresponding question cannot simply do a keyword
spotting but need to think about the actual content of the text. Furthermore, the
generated statements adequately use technical terminology. Moreover, the dif-
ferent discussion starters play an important role as they lead to different types of
statements. When generating with the definition starter, the generator rephrases
the definition of a discussion candidate in “its own words”. As a result, these
definitions often lack important aspects or contain faulty claims (see Table 3).
Thus, to explain why the definition is wrong, learners have to compare and con-
trast their previous knowledge with the generated definition. The usage starter
10 T. Steuer et al.

leads to statements that force learners to transfer the knowledge learnt into new
situations (see Table 3). The usage that is described in the generated statements
is normally not mentioned in the text, but can often be deduced by the knowl-
edge provided in the text. The advantage and disadvantage discussion starter
requires learners to think about the discussion candidate but also about similar
concepts and solutions and to compare them (see Table 3). Otherwise, learners
cannot tell if the stated advantage or disadvantage is one that is specific to the
discussed concept.

Table 3. Highly rated examples of different types of statements resulting from different
discussion starter.

Discussion starter Domain Example statement

Definition Machine learning Knowledge bases are defined as data
structures that store knowledge and act
as a long-term memory
(Dis)advantage Psychology Conditionings have the advantage of
being simple and universal, which makes
them ideal for studies of complex
behavior
Usage Networking Hosts can be used to forward packets
between hosts

Finally, one annotator provided qualitative remarks for the good statements
as well. This includes remarks that the generated statements are helpful but
often could be improved by using different discussion starters depending on the
domain (e.g. speaking of the advantage of a physical concept is odd). Also,
it was highlighted that the statements cannot simply be answered by copying
information from the text and that thinking about the definition discussion
starter sometimes resulted in the annotator checking a textbook to refresh some
rusty knowledge.

6 Discussion and Future Work

Concerning our research question, we can say that roughly a third of the state-
ments have some educational value. This is in line with the related work that
reports between 29% and 66% deceptively real statements [13,28]. Yet, even
lower numbers of valuable statements can be beneficial. If we do not generate
questions directly for the reader, but for textbook authors for further review, it
can be a source of creative ideas and may reduce the authoring effort. In partic-
ular, such systems could be combined with question ranking approaches similar
to Heilman et al. [11] to only recommend the most promising candidates.
Furthermore, there is more to our research question than just this quantita-
tive view and looking at our qualitative results reveals interesting characteristics
 Automatic Question Generation via Neural Text Generators 11

of the well-generated statements. First, they are not the typical factual Wh-
questions that ask for a simple fact or connection directly stated in the text.
Therefore, they often need a deeper understanding of the subject matter to be
answered correctly. While this can be a benefit, we have to keep in mind that our
annotators were experts and thus drawing connections between the text inherent
knowledge and previously learned subject knowledge might be too difficult for
some learners as also remarked by the annotators. Second, depending on the used
discussion starter, we can generate different kinds of useful questions. Our four
different discussion starters generate questions requiring three different types of
thinking. Depending on the discussion starter, the text comprehension questions
involve comparison with previous knowledge, transfer of learned knowledge to
new situations or implicit differentiation from similar concepts. An encouraging
result, because it shows that the generator’s expressiveness can be harnessed
to create different types of tasks. Moreover it provides evidence for the remark
of the annotators, that the questions in some domains could be improved by
using different discussion starters and that this is a worthwhile direction for
future research. Third, although we work with a variety of domains and input
text from different authors we were able to generate some valuable questions in
every domain. Furthermore, the distribution of the different quality scores did
not change much from domain to domain. Hence, our approach seems, at least
to a degree, domain-independent. Yet, as currently only a third of the generated
statements are usable this should be reevaluated as soon as the general quality of
the statements becomes better because it might be a trade-off between domain-
independence and statement quality. In summary, our qualitative analysis of the
well-generated questions provided evidence for their adaptability through dif-
ferent discussion starters and that they are well suited for text comprehension
below the surface level when learners have to think not only about facts but also
have to integrate knowledge.
Our error analysis allowed us to identify why we fail to generate interesting
questions. The five different error categories are promising starting points for
future work. Most often, the approach failed because the keyword extraction
step did not find a meaningful discussion candidate or extracted only parts of
it. This is not surprising as our goal was to test the general idea without fine-
tuning any of the intermediate steps. General-purpose keyword extraction is
similar but not identical to discussion candidate extraction. Hence, future work
might explore specific educational keyword extraction algorithms and their effect
on the generation approach. We assume that a fine-tuned educational keyword
extraction algorithm will yield much more valuable statements if adaptable to
different domains. Furthermore, as discussed in the results section the platitude
errors can be alleviated by not combining discussion starters and discussion can-
didates in an odd manner. Future work should, therefore, investigate the optimal
use of discussion starters taking into account different domains and discussion
candidates. Finally, we have the hardly discussable and statement too easy error
categories. While no clear cause of these errors could be identified, we assume
that a fine-tuning of the neural generator with discussion specific texts would
Another random document with
no related content on Scribd:
Carpenter Bees 36
Catnip 232, 240
Chaff Hives 251
Chrysalids 69
Circulatory System 57
Class 28
insecta 28
of the honey-bee 28
Cleome—see Rocky M't'n bee plant 238
Clover 228
Alsike 228
figure of 229
sweet 228
figure of 230
white 228
figure of 228
Clustering Outside the Hive 153
cause of 153
how prevented 153
adding room 176
extracting 188
shading 153
Cocoons 69
of bees 98
College Course 118
Colonies,
always strong 119
how moved 187
Columella 44
Comb 108
cells in 110
worker 110
drone 110
figure of 109
for guide 208
how fastened 157, 158
how made 108, 110
 transparency of 110
use of 110
what determines kind 110
Comb Foundation 203
American 204
figure of 203
history of 203
how cut 207
how fastened 209
how made 206
use of 207
Comb Foundation Machine 205
figure of 205
inventor of 205
Comb Honey 215
apparatus to secure 141
care of 216
in boxes 142
in frames 144
in what form 144, 215
marketing 215
when to secure 215
Conventions 19
Corn 235
Cotton 236
figure of 236
Cover for Frames 129
Cover for Hives 129
figure of 130, 131
Crates,
section 149
market 216
Cyprian Bee 43

De Geer 45
Digestive Organs 60
Diseases 259
 dysentery 247, 259
foul brood 259
Dissection 50, 65
Dissecting Instruments 51, 65
lenses 51, 65
needle points 51
dividers—see separators 146
Dividing Colonies—see artificial colonies 171, 177
Division-board 137
figure of 137
use of 138
Dollar Queens 186
Dorsata Bee 40
Dress for Ladies 197
Drones 86
development of 87
eggs of 87
eyes of 86
function of 83
influence of, on drone progeny 89
jaws of 86
figure of 92
leg of 86
figure of 87
longevity of 88
number of 86
tongue of 86
when in hive 86, 88
why so numerous 89
Dysentery 247, 259

Egg 67
of insects 67
of bee 96
Egyptian Bee 43
Empty Cells 188
importance of 188
 how to secure 188
Entrance to Hive 128
Epicranium 48
Extractor,
of honey 188
figure of 189
Everett's 190
history of 188
how to use 194
knives for 191
figure of 191
rack for 189
figure of 190
use of 191
when to use 192
wire comb baskets for 189
figure of 189
of wax 212
figure of 213
Extracted Honey 214
market for 214
Extracting Honey 191
how done 194
why done 191
when done 192
Eyes of Insects 53
compound 54
simple 54

Fabricius 46
Family 34
apidæ 34
of the honey-bee 34
Feeder 160
figure of 160, 161
Feeding 159
amount to feed 159
 use of 159
what to feed 160
honey 160
sugar 160
flour 163
Female Organs 64
Fertile Workers 77
Fertilization of Flowers by Bees 220
Figwort 238
figure of 238
Fitch's Report 47
Foot-power Saw 151
Foul Brood 259
cause 260
cure for 200
symptoms of 259
Foundation 203
figure of 203
history of 203
use of 203, 207
how cut 207
how fastened 209
how made 206
Frames 132
arrangement for surplus 147
block for making 134
figure of 135
cover for 136
figure of 133, 134
form of 132
Gallup 133
gauge for construction 135
figure of 135
inventor of 123
Langstroth 132
number of 132
section 148
 small—see sections 144
space about 136
space between 136
Fruit trees 225

Gallup Frame 133

Geoffroy 45
Genus,
apis 38
of the honey-bee 38
German or Black Bee 31
Gleanings in Bee Culture 20
Gloves 197
Golden-rod 242
figure of 243
Grapes Injured by Bees 220
Grape Vines for Shade 153
Gunther 12

Handling Bees 195

Harris' Injurious Insects 47
Harvey 44
Head of Insects 48
organs of 43
Heart of Insects 57
Hexapods—see Insects 30
Hives 122
alighting-board of 127
Bingham 140
figure of 140
bottom-board of 127
figure of 128
box not good 122
chaff 251
cover of 129
division-board for 137
entrance to 128
 figure of 124, 130, 155
frames for 132
Huber 138
joints of 126
square 126
figure of 125
bevel 126
figure of 130
Langstroth 123
figure of 124
lumber for 124
movable comb 123
movable frame 122
near the ground 128
nucleus 165
position of 154
figure of 115
Quinby 139
figure of 139
rabbet of 125
size of 124
Honey 104
collected, not secreted 104
defined 104
extracted 193
for food 17
granulated, how dissolved 193
how collected 105
how deposited 105
how transported 105
marketing of 213
natural use of 106
source of 105
bark lice 105, 218
honey-dew 105, 219
plants 105, 210
plant lice 105, 218
 other sources 105, 219
Honey-Comb—see comb 108
Honey Extractor—see extractor 188
figure of 189
importance of 188
requisites of 189
use of 191
when to use 192
Honey Knives 191
figure of 191
Honey Plants—see plants 218
for April 223
for May 225
for July 237
for June 228
for August 242
importance of 218
list of 221
House Apiary 255
advantages of 256
are they desirable? 256
objections to 257
Huber 71
Huber Hive 138
kinds of 133
Hunter's Manual 23
Hymenopterous Insects 31
the highest 32
parasitic 32

Imago 70
Insecta 28
animals of 30
class 28
Insects, or Hexapods 30
abdomen of 30
head of 30
 imago of 30
larva of 30
pupa of 30
thorax of 30
transformations of 66
transformations, complete 66
transformations, incomplete 70
Introduction of Cell 185
figure of 167
Introduction of Queen 183
Intestines 61
Italian Bees 41, 180
description of 42, 181
figure of Frontispiece
history of 41
superiority of 181

Jaws 50
figure of 92
Judas Tree 225
figure of 224

King Bird 27?

King's Text-Book 22
Kirby & Spence's Entomology 47, 113

Labium 48
Labrum 48
Ladies' Bee Dress 197
Langstroth, Rev. L. L. 123
Langstroth Frame 132
figure of 124
Langstroth Hive 123
figure of 124
Langstroth on the Honey-Bee 21
Larva 68
Latreille 45
Leaf-Cutting Bee 36
Legs of Insects 90
Linnæus 45
Ligula 49
figure of 91
Location of Apiary 120
Locust Trees 236
Lyonnet 46

Male Organs 62
figure of 63
Mandibles 50
figure of 92
Maple 224, 225
figure of 222
Market—for honey 213
crate for 216
figures of 216, 217
for comb 215
for extracted 214
how to stimulate 213
rules for 215
Mason Bees 36, 37
Maxillæ 50
Megachile 36
Melipona 35
Mice 272
remedy for 272
Mignonette 231
figure of 231
Milk-Weed 232
pollen masses of 233
figure of 233
Mimicry 31
Mouth Parts 48
figure of 49
variation of 50
Movable-Comb Hives 123
two types 123
Moving Colonies 187
Multiplying Colonies 171
Muscles of Insects 56
Mustard 233
figure of 233

Natural History of the Honey-Bee 27

Natural Method of Increase 171
Natural Swarms 171
means to save 173
implements required 173
not desirable 171
second swarms prevented 175
Neighbour, The Apiary 23
Nerves of Insects 57
figure of 58
Neuters 90
cocoon of 98
development of 96
eggs of 96
eyes of 92
figure of 90
function of 99
old workers 99
young workers 99
honey stomach of 92
figure of 60
jaws of 92
figure of 92
larva of 97
figure of 97
longevity of 99
number of 90
pollen baskets of 93
figure of 93
 pupa of 98
figure of 97
size of 90
sting of 95
figure of 95
tarsi of 93
figure of 93, 94
tibia of 93
tongue of 92
figure of 91
wings of 92
figure of 38
Nymphs 69

Order 30
of insects 30
of the honey-bee 30
Osmia 37
Ovaries 64
figure of 64

Packard's Entomology 47
Palpi 49
Papers 19
American Bee Journal 19
Bee-Keepers' Magazine 21
Gleanings in Bee Culture 20
Paraglossæ 49
Parasitic Insects 32
Parasitic Bees 37
Parthenogenesis 80
in bees 80
in other insects 81
Plants 220
asters 243
figure of 243
April 223
August 242
barberry 225
figure of 226
basswood 237
figure of 237
beggar-ticks 244
bergamot 238
blackberry 236
boneset 238
figure of 241
buckwheat 243
figure of 243
button-ball 238
figure of 240
catnip 232, 240
clover 228
Alsike 228
figure of 229
sweet 228
figure of 230
white 228
figure of 228
coffee berry 226
corn 235
cotton 236
figure of 236
figwort 238
figure of 238
fruit trees 225
golden-rod 242
figure of 243
Judas tree 225
figure of 224
July 237
June 228
list of 221
locust 236
maples 221, 225
figure of 222
milk-weed 232
pollen-masses 232
figure of 233
mints 232
figure of 232
mignonette 231
figure of 231
mustard 233
figure of 233
okra 232
figure of 231
poplar 225
rape 234
figure of 234
Rocky Mountain bee 238
figure of 239
sage 232
white 226
figure of 227
sour-wood 240
Spanish needles 244
St. John's wort 240
sumac 226
teasel 235
figure of 236
tick-seed 244
tulip tree 234
figure of 235
willow 224
figure of 223
wistaria 225
American 225
figure of 225
Chinese 225
figure of 226
Pliny 44
Poison from Sting 12
innoculation of 12
Poison Sack 95
Pollen 111
function of 112
how carried 111
nature of 111
source of 111
where deposited 112
Preparation for Apiculture 117
college course 118
plan 118
read 117
visit 117
Products of Bees 104
comb 108
figure of 109
honey 104
pollen or bee-bread 111
propolis or bee-glue 112
wax 106
Products of Insects 104
Propolis or Bee-Glue 112
function of 113
nature of 112
source of 112
Publications 19
American Bee Journal 19
Bee-Keepers' Magazine 21
Gleanings in Bee Culture 20
Pupa 68
figure of 69

Queen 71
brood from eggs 78, 164
cages 184
cell 75
figure of 109, 167
introduction of 167
figure of 167
when started 164
where built 164
figure of 109
clipping wing of 168
how done 169
not injurious 168
why done 169
cocoon of 77
development of 75
eggs of 80, 81
how impregnated 81
Wagner's theory 81
fecundity of 83, 84
figure of 72
food of larvæ 76
function of 83
how procured 185
importance of 163
impregnation of 78
only on the wing 79
introduction of 183
laying of 82
longevity of 83
must have empty cells 188
never to be wanting 163, 176
never to be poor 186
no sovereign 85
ovaries of 72
figure of 64
oviduct of 64
piping of 102
rearing of 78, 163, 186
sex of 71
 shipping 186
size of 72
spermatheca of 72
sterility of 83
sting of 71
tongue of 73
figure of 73
wings of 73
Queen Cells 75
figure of 109, 167
how secured 164
introduction of 167
figure of 167
Queen Rearing 163, 186
Queen Shipping 186
cage for 186
figure of 187
Queen White Ant 84
fecundity of 84
Quilt 136
Quinby, M. 138
Quinby Hive 139
figure of 139
Quinby's Mysteries of Bee-keeping 22
Quinby Smoker 198
figure of 199

Rabbets for Hive 125

of tin 125
Races of the Honey-Bee 41
Egyptian 43
German or black 41
Italian or Ligurian 41
history of 41
characters of 42
superiority of 181
other 43
Ray 44
Réaumur 45
Respiration 59
Riley's Reports 47
Robbing 258
how checked 258
how prevented 259
when to fear 258
Rocky Mountain Bee Plant 238
figure of 239
Royal Jelly 76
Russell Hive 141

Salicylic Acid 261

use of 261
Sage 232
white 226
figure of 227
Sawdust 154
Saws 151
Barnes' 151
foot-power 151
Second Swarms 102
Secretion 62
Secretory Organs 61
Sections 147
dove-tail 147
figure of 146
Hetherington 146
glassing 146
Phelps-Wheeler-Betsinger 147
figure of 147
veneer 144
glassing 145
Section Block 145
figure of 145
Section Frame 147
 figure of 148
where placed 148
Section Rack 149
Doolittle 151
figure of 150, 151
Southard & Ranney's 150
use of 149
Wheeler 151
Senses of Insects 51
hearing 51
seeing 54
smelling 52
feeling 51
Separators 146, 150
figure of 146
tin 148
figure of 149, 150
wooden 146
figure of 146
Shade for Hives 153
ever-greens 154
grape-vines 153
houses 153
use of 153
prevents idleness 153
prevents melting of comb 153
Smokers 198
bellows 198
how used 201
Bingham 199
figure of 199
Quinby 193
figure of 199
Sour-wood 240
Spanish Needles 244
Specialists 11
Species of the Honey-Bee 41
Spermatheca 65
Spiders 271
Spiracles 59
Spring Dwindling 254
Starting an Apiary 117
Sting 95
figure of 95
Stingless Bees 35
Stings 201
cure of 201
St. John's Wort 240
Stomach 60
sucking 60
true 60
Sub-Order 31
Hymenoptera 31
of the honey-bee 31
Sumac 226
Sun-Flower 243
Swammerdam 44
Swarming 101, 171
after-swarms 103
clustering 103
drone-brood started 101
old colony—how known 102
preparation for 101
drone-brood 101
queen cells 101
prevented 176
when to expect 176
Swarms 172
hiving 173
easy method 172
second 172
how prevented 172

Tachina Fly 270