Proceedings of Large Foundation Models for Educational Assessment Held in Vancouver, BC, Canada on 15-16 December 2024 Published as Volume 264 by the Proceedings of Machine Learning Research on 28 January 2025. Volume Edited by: Sheng Li Zhongmin Cui Jiasen Lu Deborah Harris Shumin Jing Series Editors: Neil D. Lawrence https://proceedings.mlr.press/v264/ Tue, 28 Jan 2025 09:10:02 +0000 Tue, 28 Jan 2025 09:10:02 +0000 Jekyll v3.10.0 This study explores the application of GPT-3.5-turbo for item generation and translation in non-cognitive educational assessments, specifically focusing on the University Belonging Questionnaire (UBQ). The UBQ, designed to measure university students’ sense of belonging across three dimensions, was expanded to include a new dimension on peer relationships, and translated into Chinese using GPT-3.5-turbo. A total of 25 new items, including those for the new dimension were generated and translated into Chinese, out of which 14 items passed the expert review. Psychometric analyses of the expanded and translated UBQ were conducted to evaluate reliability, internal structure, and external validity. The results demonstrate that the UBQ, with its new and translated items, maintains strong reliability and satisfactory internal structure, although the new items introduced some noise. Correlation analyses with the general belongingness scale revealed moderate associations with the acceptance dimension but weak associations with the overall scale. The study highlights GPT’s potential in efficiently expanding and translating non-cognitive assessment tools. This work addresses crucial needs in educational assessments and provides a foundation for future advancements in item development and translation. Tue, 28 Jan 2025 00:00:00 +0000 https://proceedings.mlr.press/v264/xue25a.html https://proceedings.mlr.press/v264/xue25a.html Large Language Models (LLMs) have shown great potential in education, which may significantly facilitate course preparation from making quiz questions to automatically evaluating student answers. By helping educators quickly generate high-quality educational content, LLMs enable an increased focus on student engagement, lesson planning, and personalized instruction, ultimately enhancing the overall learning experience. While slide preparation is a crucial step in education, which helps instructors present the course in an organized way, there have been few attempts at using LLMs for slide generation. Due to the hallucination problem of LLMs and the requirement of accurate knowledge in education, there is a distinct lack of LLM tools that generate presentations tailored for education, especially in specific domains such as biomedicine. To address this gap, we design a new framework to accelerate and automate the slide preparation step in biomedical education using knowledge-enhanced LLMs. Specifically, we leverage the code generation capabilities of LLMs to bridge the gap between modalities of texts and slides in presentation. The retrieval-augmented generation (RAG) is also incorporated into our framework to enhance the slide generation with external knowledge bases and ground the generated content with traceable sources. Our experiments demonstrate the utility of our framework in terms of relevance and depth, which reflect the potential of LLMs in facilitating slide preparation for education. Tue, 28 Jan 2025 00:00:00 +0000 https://proceedings.mlr.press/v264/xie25a.html https://proceedings.mlr.press/v264/xie25a.html The automatic generation of hints by Large Language Models (LLMs) within Intelligent Tutoring Systems (ITSs) has shown potential to enhance student learning. However, generating pedagogically sound hints that address student misconceptions and adhere to specific educational objectives remains challenging. This work explores using LLMs (GPT-4o and Llama-3-8B-instruct) as teachers to generate effective hints for students simulated through LLMs (GPT-3.5-turbo, Llama-3-8B-Instruct, or Mistral-7B-instruct-v0.3) tackling math exercises designed for human high-school students, and designed using cognitive science principles. We present here the study of several dimensions: 1) identifying error patterns made by simulated students on secondary-level math exercises; 2) developing various prompts for GPT-4o as a teacher and evaluating their effectiveness in generating hints that enable simulated students to self-correct; and 3) testing the best-performing prompts, based on their ability to produce relevant hints and facilitate error correction, with Llama-3-8B-Instruct as the teacher, allowing for a performance comparison with GPT-4o. The results show that model errors increase with higher temperature settings. Notably, when hints are generated by GPT-4o, the most effective prompts include prompts tailored to specific errors as well as prompts providing general hints based on common mathematical errors. Interestingly, Llama-3-8B-Instruct as a teacher showed better overall performance than GPT-4o. Also the problem-solving and response revision capabilities of the LLMs as students, particularly GPT-3.5-turbo, improved significantly after receiving hints, especially at lower temperature settings. However, models like Mistral-7B-Instruct demonstrated a decline in performance as the temperature increased. This study advances our understanding of the potential and limitations of LLMs in educational contexts, towards integrating these models into pedagogically grounded. Tue, 28 Jan 2025 00:00:00 +0000 https://proceedings.mlr.press/v264/tonga25a.html https://proceedings.mlr.press/v264/tonga25a.html In this paper, we present a complete framework for quickly calibrating and administering a robust large-scale computerized adaptive test (CAT) with a small number of responses. Calibration—learning item parameters in a test—is done using AutoIRT, a new method that uses automated machine learning (AutoML) in combination with item response theory (IRT), originally proposed in (Sharpnack et al., 2024). AutoIRT trains a non-parametric AutoML grading model using item features, followed by an item-specific parametric model, which results in an explanatory IRT model. In our work, we use tabular AutoML tools (AutoGluon.tabular) along with BERT embeddings and linguistically motivated NLP features. In this framework, we use Bayesian updating to obtain test taker ability posterior distributions for administration and scoring. For administration of our adaptive test, we propose BanditCAT, a method motivated by casting the problem in the contextual bandit framework and utilizing item response theory (IRT). The key insight lies in defining the bandit reward as the Fisher information for the selected item, given the latent test taker ability ($In this paper, we present a complete framework for quickly calibrating and administering a robust large-scale computerized adaptive test (CAT) with a small number of responses. Calibration—learning item parameters in a test—is done using AutoIRT, a new method that uses automated machine learning (AutoML) in combination with item response theory (IRT), originally proposed in (Sharpnack et al., 2024). AutoIRT trains a non-parametric AutoML grading model using item features, followed by an item-specific parametric model, which results in an explanatory IRT model. In our work, we use tabular AutoML tools (AutoGluon.tabular) along with BERT embeddings and linguistically motivated NLP features. In this framework, we use Bayesian updating to obtain test taker ability posterior distributions for administration and scoring. For administration of our adaptive test, we propose BanditCAT, a method motivated by casting the problem in the contextual bandit framework and utilizing item response theory (IRT). The key insight lies in defining the bandit reward as the Fisher information for the selected item, given the latent test taker ability ($\theta$) from IRT assumptions. We use Thompson sampling to balance between exploring items with different psychometric characteristics and selecting highly discriminative items that give more precise information about $\theta$. To control item exposure, we inject noise through an additional randomization step before computing the Fisher information. This framework was used to initially launch two new item types on the DET practice test using limited training data. We outline some validity, reliability, and exposure metrics for the 5 practice test experiments that utilized this framework. Tue, 28 Jan 2025 00:00:00 +0000 https://proceedings.mlr.press/v264/sharpnack25a.html https://proceedings.mlr.press/v264/sharpnack25a.html Timely, personalized, and actionable feedback is essential for effective learning but challenging to deliver at scale. Automated feedback generation (AFG) using large language models (LLMs) can be a promising solution to address this challenge. While existing studies using out-of-the-box LLMs and prompting strategies have shown promise, there is room for improvement. This study investigates the fine-tuning of OpenAI’s GPT-3.5-turbo for AFG. We developed feedback for open-ended situational judgment questions, and this small set of hand-crafted feedback examples was used to fine-tune the pre-trained LLM using specific prompting strategies. Our evaluation, conducted by independent judges and test experts, found that the feedback generated by our fine-tuned GPT-3.5-turbo model achieved high user satisfaction (84.8Timely, personalized, and actionable feedback is essential for effective learning but challenging to deliver at scale. Automated feedback generation (AFG) using large language models (LLMs) can be a promising solution to address this challenge. While existing studies using out-of-the-box LLMs and prompting strategies have shown promise, there is room for improvement. This study investigates the fine-tuning of OpenAI’s GPT-3.5-turbo for AFG. We developed feedback for open-ended situational judgment questions, and this small set of hand-crafted feedback examples was used to fine-tune the pre-trained LLM using specific prompting strategies. Our evaluation, conducted by independent judges and test experts, found that the feedback generated by our fine-tuned GPT-3.5-turbo model achieved high user satisfaction (84.8%) and met key structural quality criteria (72.9%). Also, the model generalized effectively across different items, providing feedback consistent with instructions, regardless of the respondent’s performance level, English proficiency, or student status. However, some feedback statements still contained linguistic errors, lacked focused suggestions, or seemed generic. We discuss potential solutions to these issues, along with implications for developing LLM-supported AFG systems and their adoption in high-stakes settings. Tue, 28 Jan 2025 00:00:00 +0000 https://proceedings.mlr.press/v264/mazzullo25a.html https://proceedings.mlr.press/v264/mazzullo25a.html Tue, 28 Jan 2025 00:00:00 +0000 https://proceedings.mlr.press/v264/li25a.html https://proceedings.mlr.press/v264/li25a.html Generating accurate and consistent visual aids is a critical challenge in mathematics education, where visual representations like geometric shapes and functions play a pivotal role in enhancing student comprehension. This paper introduces a novel multi-agent framework that leverages Large Language Models (LLMs) to automate the creation of complex mathematical visualizations alongside coherent problem text. Our approach not only simplifies the generation of precise visual aids but also aligns these aids with the problem’s core mathematical concepts, improving both problem creation and assessment. By integrating multiple agents, each responsible for distinct tasks—such as numeric calculation, geometry validation, and visualization—our system delivers mathematically accurate and contextually relevant problems with visual aids. Evaluation across Geometry and Function problem types shows that our method significantly outperforms a baseline system in terms of text coherence, consistency, and relevance, while maintaining the essential geometrical and functional integrity of the original problems. Although some challenges remain in ensuring consistent visual outputs, our framework demonstrates the immense potential of LLMs in transforming the way educators generate and utilize visual aids in math education. Tue, 28 Jan 2025 00:00:00 +0000 https://proceedings.mlr.press/v264/lee25b.html https://proceedings.mlr.press/v264/lee25b.html This study compared the classification performance of Gemini Pro and GPT-4V in educational settings. Employing visual question-answering (VQA) techniques, the study examined both models’ ability to read text-based rubrics and automatically score student-drawn models in science education. We employed quantitative and qualitative analyses using a dataset derived from student-drawn scientific models and NERIF (Notation-Enhanced Rubrics for Image Feedback) prompting methods. The findings reveal that GPT-4V significantly outperforms Gemini Pro regarding scoring accuracy and quadratic weighted kappa. The qualitative analysis shows that the differences may be due to the models’ ability to process fine-grained texts in images and overall image classification performance. Even adapting the NERIF approach by further de-sizing the input images, Gemini Pro seems unable to perform as well as GPT-4V. The findings suggest GPT-4V’s superior capability in handling complex multimodal educational tasks. The study concludes that while both models represent advancements in AI, GPT-4V’s higher performance makes it a more suitable tool for educational applications involving multimodal data interpretation. Tue, 28 Jan 2025 00:00:00 +0000 https://proceedings.mlr.press/v264/lee25a.html https://proceedings.mlr.press/v264/lee25a.html This study explores the feasibility of using large language models (LLMs), specifically GPT-4o (ChatGPT), for automated grading of conceptual questions in an undergraduate Mechanical Engineering course. We compared the grading performance of GPT-4o with that of human teaching assistants (TAs) on ten quiz problems from the MEEN 361 course at Texas AThis study explores the feasibility of using large language models (LLMs), specifically GPT-4o (ChatGPT), for automated grading of conceptual questions in an undergraduate Mechanical Engineering course. We compared the grading performance of GPT-4o with that of human teaching assistants (TAs) on ten quiz problems from the MEEN 361 course at Texas A&M University, each answered by approximately 225 students. Both the LLM and TAs followed the same instructor-provided rubric to ensure grading consistency. We evaluated performance using Spearman’s rank correlation coefficient and Root Mean Square Error (RMSE) to assess the alignment between rankings and the accuracy of scores assigned by GPT-4o and TAs under zero- and few-shot grading settings. In the zero-shot setting, GPT-4o demonstrated a strong correlation with TA grading, with Spearman’s rank correlation coefficient exceeding 0.6 in seven out of ten datasets and reaching a high of 0.9387. Our analysis reveals that GPT-4o performs well when grading criteria are straightforward but struggles with nuanced answers, particularly those involving synonyms not present in the rubric. The model also tends to grade more stringently in ambiguous cases compared to human TAs. Overall, ChatGPT shows promise as a tool for grading conceptual questions, offering scalability and consistency. Tue, 28 Jan 2025 00:00:00 +0000 https://proceedings.mlr.press/v264/gao25a.html https://proceedings.mlr.press/v264/gao25a.html Automatic question generation is a critical task that involves evaluating question quality by considering factors such as engagement, pedagogical value, and the ability to stimulate critical thinking. These aspects require human-like understanding and judgment, which automated systems currently lack. However, human evaluations are costly and impractical for large-scale samples of generated questions. Therefore, we propose a novel system, MIRROR (Multi-LLM Iterative Review and Response for Optimized Rating), which leverages large language models (LLMs) to automate the evaluation process for questions generated by automated question generation systems. We experimented with several state-of-the-art LLMs, such as GPT-4, Gemini, and Llama2-70b. We observed that the scores of human evaluation metrics, namely relevance, appropriateness, novelty, complexity, and grammaticality, improved when using the feedback-based approach called MIRROR, tending to be closer to the human baseline scores. Furthermore, we observed that Pearson’s correlation coefficient between GPT-4 and human experts improved when using our proposed feedback-based approach, MIRROR, compared to direct prompting for evaluation. Error analysis shows that our proposed approach, MIRROR, significantly helps to improve relevance and appropriateness. Tue, 28 Jan 2025 00:00:00 +0000 https://proceedings.mlr.press/v264/deroy25a.html https://proceedings.mlr.press/v264/deroy25a.html We explored emulating textually encoded personality information in a large language model. Given its predominant empirical validation, we chose the five-factor model of personality compiled for a broad range of natural languages. Our study assessed personality traits from a multicultural viewpoint over a diverse set of thirty universal contexts. Thus, contributing to the wider comprehension of generalizing relationships among personality traits across cultures. We administered psychometric tests to the language model, examined links between location and personality, and cross validated measures at various levels of trait hierarchy. Tue, 28 Jan 2025 00:00:00 +0000 https://proceedings.mlr.press/v264/bleiweiss25a.html https://proceedings.mlr.press/v264/bleiweiss25a.html