Home // IARIA Congress 2024, The 2024 IARIA Annual Congress on Frontiers in Science, Technology, Services, and Applications // View article

Automatic Assessment of Student Answers using Large Language Models: Decoding Didactic Concepts

Authors:
Daniel Schönle
Christoph Reich
Djaffar Ould Abdeslam
Daniela Fiedler
Ute Harms
Johannes Poser

Keywords: machine learning; efficiency; linguistic; text classification; assessment.

Abstract:
This study evaluates machine learning for automating the evaluation of textual responses in virtual learning environments, particularly by applying advanced linguistic enhancement techniques. Techniques such as Transformer-based data augmentation, Part-of-Speech enhanced feature selection, and LinPair tokenisation were employed. The evaluation focused on classification quality and training efficiency using a synthetically created question-and-answer dataset, characterised by its limited sample size, extensive class range, and the complexity of identifying didactical elements. The findings indicate that while the Support Vector Machine (SVM) consistently outperforms DistilBERT in quality metrics, the integration of linguistic elements improved DistilBERT's performance significantly—achieving a 7.62% increase in F1-Score and a 17.02% rise in Hamming-Score. Despite these gains, DistilBERT recorded lower efficiency scores compared to SVM. This suggests that while SVM excels with synthetic data, Large Language Models demonstrate substantial potential in processing complex linguistic data when provided with linguistic information. These insights confirm the viability of both approaches as effective tools for automated assessment in educational settings.

Pages: 158 to 167

Copyright: Copyright (c) IARIA, 2024

Publication date: June 30, 2024

Published in: conference

ISBN: 978-1-68558-180-0

Location: Porto, Portugal

Dates: from June 30, 2024 to July 4, 2024