research-article

Reflecting on reflective writing analytics: assessment challenges and iterative evaluation of a prototype tool

Authors:

Simon Buckingham Shum,

Ágnes Sándor,

Rosalie Goldsmith,

Mindy McWilliamsAuthors Info & Claims

LAK '16: Proceedings of the Sixth International Conference on Learning Analytics & Knowledge

Pages 213 - 222

https://doi.org/10.1145/2883851.2883955

Published: 25 April 2016 Publication History

Abstract

When used effectively, reflective writing tasks can deepen learners' understanding of key concepts, help them critically appraise their developing professional identity, and build qualities for lifelong learning. As such, reflecting writing is attracting substantial interest from universities concerned with experiential learning, reflective practice, and developing a holistic conception of the learner. However, reflective writing is for many students a novel genre to compose in, and tutors may be inexperienced in its assessment. While these conditions set a challenging context for automated solutions, natural language processing may also help address the challenge of providing real time, formative feedback on draft writing. This paper reports progress in designing a writing analytics application, detailing the methodology by which informally expressed rubrics are modelled as formal rhetorical patterns, a capability delivered by a novel web application. This has been through iterative evaluation on an independently human-annotated corpus, showing improvements from the first to second version. We conclude by discussing the reasons why classifying reflective writing has proven complex, and reflect on the design processes enabling work across disciplinary boundaries to develop the prototype to its current state.

References

[1]

Aït-Mokhtar, S., Chanod, J-P., and Roux, C. 2002. Robustness beyond shallowness: incremental dependency parsing. Natural Language Engineering, 8, 2/3 (Aug. 2002), 121--144. DOI= http://dx.doi.org/10.1017/S1351324902002887

Digital Library

[2]

Boud, D., Keogh, R., and Walker, D. 2013. Reflection: Turning Experience into Learning. Routledge, Abingdon, Oxon.

[3]

Boud, D. and Walker, D. 1998. Promoting reflection in professional courses: the challenge of context. Studies in Higher Education, 23, 2 (Aug. 2006), 191--206. DOI= http://dx.doi.org/10.1080/03075079812331380384

[4]

Brun, C., and Hagège, C. 2003. Normalization and paraphrasing using symbolic methods. Proceedings of the Second International Workshop on Paraphrasing, Volume 16. Association for Computational Linguistics, 2003. DOI= http://dx.doi.org/10.3115/1118984.1118990

Digital Library

[5]

Brun, C., and Hagège, C. 2004. Intertwining deep syntactic processing and named entity detection. Advances in Natural Language Processing. Springer: Berlin, 2004. 195-206. DOI= http://dx.doi.org/10.1007/978-3-540-30228-5_18

[6]

Brun, C., Popa, D. N., and Roux, C. 2014. XRCE: Hybrid classification for aspect-based sentiment analysis. Proc. 8^th International Workshop on Semantic Evaluation (SemEval 2014), 838--842. (Dublin, Ireland, August 23-24, 2014).

[7]

Burnett, M. M., and Scaffidi, C. 2011. End-User Development. The Encyclopedia of Human-Computer Interaction, 2nd Edition. (Eds. Mads Soegaard and Rikke Friis Dam). Interaction Design Foundation: https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed

[8]

De Liddo, A., Sándor. Á., and Buckingham Shum, S. 2012. Contested collective intelligence: Rationale, technologies, and a human-machine annotation study. Computer Supported Cooperative Work (CSCW) 21.4-5 (2012): 417-448. DOI= http://dx.doi.org/10.1007/s10606-011-9155-x

Digital Library

[9]

Gibson, A. and Kitto, K. 2015. Analysing reflective text for learning analytics: an approach using anomaly recontextualisation. Proceedings of the Fifth International Conference on Learning Analytics & Knowledge. ACM: NY. DOI= http://dx.doi.org/10.1145/2723576.2723635

Digital Library

[10]

Hatton N. and Smith, D. 1995. Reflection in teacher education: Towards definition and implementation. Teaching & Teacher Education. 11, 1 (Jan. 1995), 33--49. DOI= http://dx.doi.org/10.1016/0742-051X(94)00012-U

[11]

Huang, Z., ten Teije, A., van Harmelen, F., and Aït-Mokhtar, S. 2014. Semantic representation of evidence-based clinical guidelines. Proc. 6th International Workshop on Knowledge Representation for Health Care. 2014: Vienna (21 July 2014). p. 78--94. DOI= http://doi.org/10.1007/978-3-319-13281-5_6

[12]

Lisacek, F., Chichester, C., Kaplan, A., and Sandor, Á. 2005. Discovering paradigm shift patterns in biomedical abstracts: application to neurodegenerative diseases. In Proceedings of the First International Symposium on Semantic Mining in Biomedicine (SMBM) (41-50).

[13]

MacLean, M., Kathleen Carter, Lövstrand, L., and Moran, T. 1990. User-tailorable systems: pressing the issues with buttons. Proc. Conference on Human Factors in Computing Systems. ACM, New York, 175--182. DOI= http://dx.doi.org/10.1145/97243.97271

Digital Library

[14]

Moon, J. 2010. Reflective Learning Workshop (Handout 10/07), University of Worcester, UK. http://worc.ac.uk/edu/documents/Jenny_Moon_RefLearnlong07.doc

[15]

Reidsema, C., Goldsmith, R., & Mort, P. 2010. Writing to learn: reflective practice in engineering design, Proceedings of ASEE Symposium Singapore, October 18-21

[16]

Rodgers, C. 2002. Defining reflection: Another look at John Dewey and reflective thinking. Teachers College Record. 104.4 (2002): 842--866. http://www.tcrecord.org/content.asp?contentid=10890

[17]

Russell, R. 2001. Reflection in higher education: A concept analysis. Innovative Higher Education. 21.6 (2001). DOI= http://dx.doi.org/10.1023/A:1010986404527

[18]

Ryan, M. 2011. Improving reflective writing in higher education: a social semiotic perspective, Teaching in Higher Education, 16, 1 (Jan. 2011), 99--111. DOI= http://doi.org/10.1080/13562517.2010.507311

[19]

Ryan, M. 2010. The 4 Rs Model of Reflective Thinking, Version 1.5. Developing Reflective Approaches to Writing (DRAW) Project, Queensland University of Technology. http://www.citewrite.qut.edu.au/write/4Rs-for-students-page1-v1.5.pdf

[20]

Ryan, M. & Ryan, M. 2012. Developing a systematic, cross-faculty approach to teaching and assessing reflection in higher education. Office of Learning and Teaching. http://www.olt.gov.au/system/files/resources/PP9_1327_Ryan_report_2012.pdf

[21]

Sándor, Á., Kaplan, A., and Rondeau, R. 2006. Discourse and citation analysis with concept-matching. International Symposium: Discourse and Document. (Caen, 15-17 June 2006).

[22]

Sándor, Á. 2007. Modeling metadiscourse conveying the author's rhetorical strategy in biomedical research abstracts. Revue Française de Linguistique Appliquée, 12, 2, 97--108.

[23]

Sándor, Á., and Vorndran, A. 2009. Detecting key sentences for automatic assistance in peer reviewing research articles in educational sciences. Proceedings of the Workshop on Text and Citation Analysis for Scholarly Digital Libraries, 47th Annual Meeting of the Association for Computational Linguistics (Singapore, 2-7 Aug, 2009).

Digital Library

[24]

Scouller, K. 1998. The influence of assessment methods on students' learning approaches: multiple choice question examination versus assignment essay. Higher Education, 35, 4, 453--472. DOI= http://dx.doi.org/10.1023/A:1003196224280

[25]

Simsek, D., Buckingham Shum, S., Sándor, Á., Liddo, A. D., and Ferguson, R. 2013. XIP Dashboard: Visual analytics from automated rhetorical parsing of scientific metadiscourse. 1^st International Workshop on Discourse-Centric Learning Analytics, 3^rd International Conference on Learning Analytics & Knowledge (Leuven, BE, Apr. 8-12, 2013).

[26]

Sumsion, J. and Fleet, A. 1996. Reflection: can we assess it? Should we assess it?. Assessment & Evaluation in Higher Education, 21, 2 (July. 2006), 121--130. DOI= http://dx.doi.org/10.1080/0260293960210202

[27]

Ullmann, T. D., Wild, F. and Scott, P. 2012. Comparing automatically detected reflective texts with human judgements. In 2nd Workshop on Awareness and Reflection in Technology-Enhanced Learning, Saarbruecken, Germany, September 2012, 101--116.

[28]

Webster-Wright, A. 2013. The eye of the storm: a mindful inquiry into reflective practices in higher education. Reflective Practice, 14, 4 (May. 2013) 556-567. DOI= http://dx.doi.org/10.1080/14623943.2013.810618

[29]

Xerox Incremental Parser. Open Xerox Documentation: https://open.xerox.com/Services/XIPParser

Cited By

Tang LShen KLe HShen YTan SZhao YJuelich TLi XGašević DFan Y(2024)Facilitating learners' self‐assessment during formative writing tasks using writing analytics toolkitJournal of Computer Assisted Learning10.1111/jcal.1303640:6(2822-2839)Online publication date: 11-Jul-2024
https://doi.org/10.1111/jcal.13036
Sembey RHoda RGrundy J(2024)Emerging technologies in higher education assessment and feedback practices: A systematic literature reviewJournal of Systems and Software10.1016/j.jss.2024.111988211(111988)Online publication date: May-2024
https://doi.org/10.1016/j.jss.2024.111988
Alwahaby HCukurova M(2024)Navigating the ethical landscape of multimodal learning analytics: a guiding frameworkEthics in Online AI-based Systems10.1016/B978-0-443-18851-0.00014-7(25-53)Online publication date: 2024
https://doi.org/10.1016/B978-0-443-18851-0.00014-7
Show More Cited By

Index Terms

Reflecting on reflective writing analytics: assessment challenges and iterative evaluation of a prototype tool
1. Applied computing
  1. Education

Recommendations

Reflective writing analytics for actionable feedback
LAK '17: Proceedings of the Seventh International Learning Analytics & Knowledge Conference

Reflective writing can provide a powerful way for students to integrate professional experience and academic learning. However, writing reflectively requires high quality actionable feedback, which is time-consuming to provide at scale. This paper ...
Learning Analytics as a Metacognitive Tool
CSEDU 2014: Proceedings of the 6th International Conference on Computer Supported Education - Volume 1

The use of learning analytics is entering in the field of research in education as a promising way to support learning. However, in many cases data are not transparent for the learner. In this regard, Educational institutions shouldnâ t escape the need ...
Effects of Learning Analytics on Students' Self-Regulated Learning in Flipped Classroom

The present article is aimed at analyzing the effects of learning analytics on students' self-regulated learning in a flipped classroom. An experiment was conducted with 96 engineering students, enrolled in a subject offered in the Flipped Classroom ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

LAK '16: Proceedings of the Sixth International Conference on Learning Analytics & Knowledge

April 2016

567 pages

ISBN:9781450341905

DOI:10.1145/2883851

General Chairs:
Dragan Gašević
University of Edinburgh, United Kingdom
,
Grace Lynch
Society for Learning Analytics Research
,
Program Chairs:
Shane Dawson
University of South Australia
,
Hendrik Drachsler
University of the Netherlands
,
Carolyn Penstein Rosé
Carnegie Mellon University

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 April 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

LAK '16

LAK '16: 6th International Conference on Learning Analytics and Knowledge

April 25 - 29, 2016

Edinburgh, United Kingdom

Acceptance Rates

LAK '16 Paper Acceptance Rate 36 of 116 submissions, 31%;

Overall Acceptance Rate 236 of 782 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

27
Total Citations
View Citations
805
Total Downloads

Downloads (Last 12 months)79
Downloads (Last 6 weeks)9

Reflects downloads up to 24 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tang LShen KLe HShen YTan SZhao YJuelich TLi XGašević DFan Y(2024)Facilitating learners' self‐assessment during formative writing tasks using writing analytics toolkitJournal of Computer Assisted Learning10.1111/jcal.1303640:6(2822-2839)Online publication date: 11-Jul-2024
https://doi.org/10.1111/jcal.13036
Sembey RHoda RGrundy J(2024)Emerging technologies in higher education assessment and feedback practices: A systematic literature reviewJournal of Systems and Software10.1016/j.jss.2024.111988211(111988)Online publication date: May-2024
https://doi.org/10.1016/j.jss.2024.111988
Alwahaby HCukurova M(2024)Navigating the ethical landscape of multimodal learning analytics: a guiding frameworkEthics in Online AI-based Systems10.1016/B978-0-443-18851-0.00014-7(25-53)Online publication date: 2024
https://doi.org/10.1016/B978-0-443-18851-0.00014-7
Messina CJones CPoe M(2023)Prompting Reflection: Using Corpus Linguistic Methods in the Local Assessment of Reflective WritingWritten Communication10.1177/0741088322114942540:2(620-650)Online publication date: 9-Mar-2023
https://doi.org/10.1177/07410883221149425
Adilkhanova BZhumadilova ATripathi S(2023)Supervised Learning Algorithm: Enhancing Reflective Writing Skills within the Context of the International Project ‘Out of Eden Walk’ and the Principles of ‘Slow’ Journalism2023 4th International Conference on Computation, Automation and Knowledge Management (ICCAKM)10.1109/ICCAKM58659.2023.10449604(1-5)Online publication date: 12-Dec-2023
https://doi.org/10.1109/ICCAKM58659.2023.10449604
Ahadi ASingh ABower MGarrett M(2022)Text Mining in Education—A Bibliometrics-Based Systematic ReviewEducation Sciences10.3390/educsci1203021012:3(210)Online publication date: 15-Mar-2022
https://doi.org/10.3390/educsci12030210
Ahmad ASchneider JGriffiths DBiedermann DSchiffner DGreller WDrachsler H(2022)Connecting the dots – A literature review on learning analytics indicators from a learning design perspectiveJournal of Computer Assisted Learning10.1111/jcal.1271640:6(2432-2470)Online publication date: 26-Jul-2022
https://doi.org/10.1111/jcal.12716
Alwahaby HCukurova MPapamitsiou ZGiannakos M(2022)The Evidence of Impact and Ethical Considerations of Multimodal Learning Analytics: A Systematic Literature ReviewThe Multimodal Learning Analytics Handbook10.1007/978-3-031-08076-0_12(289-325)Online publication date: 9-Oct-2022
https://doi.org/10.1007/978-3-031-08076-0_12
Falcão TMello RRodrigues RDiniz JTsai YGašević DRensing CDrachsler HKovanović VPinkwart NScheffel MVerbert K(2020)Perceptions and expectations about learning analytics from a brazilian higher education institutionProceedings of the Tenth International Conference on Learning Analytics & Knowledge10.1145/3375462.3375478(240-249)Online publication date: 23-Mar-2020
https://dl.acm.org/doi/10.1145/3375462.3375478
Pontual Falcão TFerreira Mello RLins Rodrigues R(2020)Applications of learning analytics in Latin AmericaBritish Journal of Educational Technology10.1111/bjet.1297851:4(871-874)Online publication date: 19-Jun-2020
https://doi.org/10.1111/bjet.12978
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents