Improving LLM-based Automatic Essay Scoring with Linguistic Features

Zhaoyi Hou, Alejandro Ciuba, Xiang Li
Proceedings of the Innovation and Responsibility in AI-Supported Education Workshop, PMLR 273:41-65, 2025.

Abstract

Automatic Essay Scoring (AES) assigns scores to student essays, reducing the grading workload for instructors. Developing a scoring system capable of handling essays across diverse prompts is challenging due to the flexibility and diverse nature of the writing task. Previous work has shown promising results in AES by prompting large language models (LLMs). While prompting LLM is data efficient, it does not surpass supervised methods trained with extracted linguistic features Li and Ng (2024). In this paper, we combines both approaches by incorporating linguistic features into LLM-based scoring. Experiments show promising results from this hybrid method for both in-domain and out-of-domain essay prompts.

Cite this Paper

BibTeX
@InProceedings{pmlr-v273-hou25a, title = {Improving LLM-based Automatic Essay Scoring with Linguistic Features}, author = {Hou, Zhaoyi and Ciuba, Alejandro and Li, Xiang}, booktitle = {Proceedings of the Innovation and Responsibility in AI-Supported Education Workshop}, pages = {41--65}, year = {2025}, editor = {Wang, Zichao and Woodhead, Simon and Ananda, Muktha and Mallick, Debshila Basu and Sharpnack, James and Burstein, Jill}, volume = {273}, series = {Proceedings of Machine Learning Research}, month = {03 Mar}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v273/main/assets/hou25a/hou25a.pdf}, url = {https://proceedings.mlr.press/v273/hou25a.html}, abstract = {Automatic Essay Scoring (AES) assigns scores to student essays, reducing the grading workload for instructors. Developing a scoring system capable of handling essays across diverse prompts is challenging due to the flexibility and diverse nature of the writing task. Previous work has shown promising results in AES by prompting large language models (LLMs). While prompting LLM is data efficient, it does not surpass supervised methods trained with extracted linguistic features Li and Ng (2024). In this paper, we combines both approaches by incorporating linguistic features into LLM-based scoring. Experiments show promising results from this hybrid method for both in-domain and out-of-domain essay prompts.} }
Endnote
%0 Conference Paper %T Improving LLM-based Automatic Essay Scoring with Linguistic Features %A Zhaoyi Hou %A Alejandro Ciuba %A Xiang Li %B Proceedings of the Innovation and Responsibility in AI-Supported Education Workshop %C Proceedings of Machine Learning Research %D 2025 %E Zichao Wang %E Simon Woodhead %E Muktha Ananda %E Debshila Basu Mallick %E James Sharpnack %E Jill Burstein %F pmlr-v273-hou25a %I PMLR %P 41--65 %U https://proceedings.mlr.press/v273/hou25a.html %V 273 %X Automatic Essay Scoring (AES) assigns scores to student essays, reducing the grading workload for instructors. Developing a scoring system capable of handling essays across diverse prompts is challenging due to the flexibility and diverse nature of the writing task. Previous work has shown promising results in AES by prompting large language models (LLMs). While prompting LLM is data efficient, it does not surpass supervised methods trained with extracted linguistic features Li and Ng (2024). In this paper, we combines both approaches by incorporating linguistic features into LLM-based scoring. Experiments show promising results from this hybrid method for both in-domain and out-of-domain essay prompts.
APA
Hou, Z., Ciuba, A. & Li, X.. (2025). Improving LLM-based Automatic Essay Scoring with Linguistic Features. Proceedings of the Innovation and Responsibility in AI-Supported Education Workshop, in Proceedings of Machine Learning Research 273:41-65 Available from https://proceedings.mlr.press/v273/hou25a.html.

Related Material