Demonstration Session 2

Session Page

Session Information

This page shows the session details and the presentations assigned to this session.

Back to programme overview
Add this session to calendar

From Ratings to Formative Feedback: An AI-Based System for Automated Essay Scoring

Co-author
Debelak, Rudolf , Universität Zürich, Switzerland
Keywords
evaluation and assessment of writing competence;feedback;Generative AI;natural language processing (NLP)
Domain
Assessment and Evaluation

Abstract

Feedback is widely recognised as one of the most powerful influences on learning, particularly in the development of writing competence. However, in everyday classroom practice, the provision of detailed and timely feedback on student texts is constrained by limited time resources. Automated essay scoring (AES) has the potential to mitigate this tension, provided that it is pedagogically sound and sensitive to the complexity of writing.This demonstration introduces an AI-based AES system developed for primary and lower secondary education. The system generates structured feedback within seconds, addressing four core dimensions of writing: content quality, coherence and cohesion, language accuracy, and stylistic appropriateness. In addition to score-based ratings across eight criteria, the system provides qualitative, dimension-specific feedback designed to support formative learning processes.The development of the system builds on a large empirical foundation of 36,739 digitised student essays that were evaluated by trained human raters. By combining large language models with targeted natural language processing techniques and educational assessment frameworks, the system aims to produce automated feedback that is more consistent, transparent, and pedagogically grounded than that of general-purpose AI applications. The demonstration briefly outlines these design principles and explains the rationale underlying the selected feedback dimensions.The demonstration then focuses on how these principles are operationalised in practice. Participants are shown how the system structures multi-dimensional feedback, generates qualitative comments from textual features, and presents feedback in an interpretable manner for educational use. Particular attention is given to interface and feedback design choices that support formative use in the classroom and clearly differentiate the system from generic AI-based writing tools.Overall, the demonstration contributes to current discussions on AI in writing education by illustrating how automated feedback systems can be designed to augment instructional practice and support learning in classroom contexts.

Retrieval-Augmented Generation for Formative Thesis Writing Assessment

Keywords
academic writing;AI writing;assessment;evaluation and assessment of writing competence
Domain
Learning and Instructional Technology

Abstract

The development of academic writing skills in higher education presents numerous challenges. Instructors face high workloads alongside the ongoing need to provide focused and pedagogically sound feedback. While AI tools can assist in this area, current solutions show limitations. Automated writing assessment tools tend to focus on surface‑level language features (Zhao, 2025), and generative AI feedback may suffer from hallucinations, fail to address specific criteria, or lack alignment with teaching content (Gautschi, 2025). In addition, fine‑tuning large language models for specialized purposes—and many paid solutions—may be cost‑prohibitive.Recent developments in GenAI, particularly retrieval‑augmented generation (RAG) systems, offer a promising alternative (Li et al., 2025; Swacha & Gracel, 2025). Although RAG‑based architectures have been applied to academic writing support, to date they have not been applied to the specific problem of academic writing assessment for thesis writing. Existing tools such as CorpusChat (see Cheung & Crosthwaite, 2025) demonstrate RAG‑based support for student writers but do not include an assessment component. To our knowledge, no existing tool integrates RAG for assessment with the goal of providing feedback aligned with instructor or writing program specifications.To address these issues, we have developed a RAG‑based system for generating formative feedback. This approach allows for reduced hallucination, greater focus, and improved flexibility and control over generated feedback. Our SaaS‑independent multi‑chat, multi‑context RAG application (Node.js server, React frontend) incorporates user and persistence management, full handling of multiple RAG document contexts, recursive splitting, vector storage (Qdrant), and query rewriting to optimize similarity searches. Local context folders include target structures for feedback, target criteria, and samples of evaluated texts. The system provides feedback based on a modified IMRD‑based structure model for thesis writing. This demonstration session showcases the system’s potential to promote academic writing skills in higher education, benefiting both students and lecturers through a flexible, pedagogically grounded formative feedback ecosystem.

Studiying writing dynamics of students of dyslexia: the DYSTRACKER setup

Keywords
eye tracking;handwriting;writing and linguistics;writing impairment
Domain
Learning and Special Education

Abstract

During this demonstration, we aim to present an innovative experimental setup for collecting both offline (linguistic choices) and real-time (online, pauses, speed, duration, etc.) data, including eye-tracking data, for the same individual during both reading and writing tasks: DYSTRACKER device (Anonymisation). This device linked to a research project with the same name is the result of a transdisciplinary collaboration implying several disciplines (psycholinguistic, linguistic, speech therapy, neuroscience, computer science and orthoptics) and a company (Sierra Neurovision, France). Sierra Neurovision designs and develops eye-trackers to improve screening for neurovisual disorders in adolescents and young adults.Obtaining all these indicators for the same person in both reading and writing was a technical and scientific challenge. Data were collected using this innovative setup, which integrates a pen tablet, an eye-tracker, and their associated software. The written data will be collected using high-resolution pen tablets (Wacom One or similar) with Eye and Pen© software (Chesnet and Alamargot, 2005). This software records writing and eye activity. For eye activity, we will use the Eya S360 eye-tracker (SIERRA Neurovision, CE standard - ISO 62471), which records and displays eye movements. We will obtain data (enabling us to analyze lexical choices (off-line analysis), real-time processes (on-line analysis - pauses, flow, revisions, etc.), including ocular data (saccades, rhythms, etc.)) from written texts and readings.As said before, the device was developed for a previous project (a pilot study funded by a laboratory of excellence and the École Normale Supérieure of Lyon). It is also fully operational and has enabled the collection of these types of data for 44 students with and without dyslexia (Mazur, Quignard and Bigarnet, accepted). This setup indeed was therefore implemented to study the impact of dyslexia/dysorthographia on the reading and writing processes of students, contributing to a better understanding of this disorder and its impact.Chesnet, D. & Alamargot, D. (2005). Analyses en temps réel des activités oculaires et graphomotrices du scripteur : intérêt du dispositif 'Eye and Pen'. L'Année Psychologique, 105, 477-520.This demonstration is combined with a paper submission (30645).

The challenge of creating a coherent text: A Coherence-Focused AI Chatbot for Academic Writing

Keywords
academic writing;AI writing;EFL and ESL writing;writing tools and writing technology
Domain
Learning and Instructional Technology

Abstract

The challenge of creating a coherent text: A Coherence-Focused AI Chatbot for Academic Writing Despite growing interest in AI-supported writing instruction, relatively little is known about how interactive AI tools affect higher-order writing skills, such as textual coherence. To address this gap, we present the development of a coherence-focused feedback chatbot designed explicitly for research-based writing. The tool aims to translate writing research on cohesion and coherence into practical, real-time guidance in academic writing for students and early career academics.The tool is grounded in cohesion theory, which explains how linguistic features create unity and continuity across a text (Crossley et al., 2016; Halliday & Hasan, 2014; Morris et al., 2025). The coherence-focused chatbot was developed focusing on these cohesion theory frameworks, through iterative prompt engineering, and integrated into the free, online De-jargonizer platform (Rakedzon et al., 2017). The chatbot provides individualized, question-driven prompts that guide students in identifying aspects of coherence, such as lexical overlap across sentences/paragraphs, semantic similarity between ideas, and use of transitions. At each stage, they receive AI-generated suggestions and revise their texts across iterations. A reflection and a questionnaire on the process follow this. During the demonstration, attendees will receive an overview of the pedagogical rationale, tool development, and use in research design, followed by a demonstration of the chatbot. Participants will be invited to test the tool on their own writing samples and explore how coherence indices are employed to generate tailored feedback. The project demonstrates how research can be translated into classroom-ready practice, advancing understanding of AI’s potential to support higher-order writing skills in multilingual contexts. More broadly, it highlights how coherence-aware AI tools can contribute to more inclusive, adaptive, and discipline-responsive academic writing instruction.