New study shows AI falls short in reliable grading of student essays

A new study conducted by the University of Cardiff and the University of Melbourne has found that current generative artificial intelligence (GenAI) systems are not yet reliable in grading student essays as effectively as human teachers. Researchers tested the ability of two versions of ChatGPT to assess 50 undergraduate biology essays using seven specific criteria. Four different prompts were used to guide the AI’s evaluation, and the results were compared to human grading averages and score variations. Dr. William Kay from the University of Cardiff’s School of Biological Sciences noted that the AI-generated scores showed significant variability and were not accurate in predicting human assessments. While the overall average scores from AI were generally higher than those from human graders, with the largest gap reaching 16.1 points, individual essay scores varied by as much as 40 points. Kay also observed that AI tended to lower the scores of high-performing essays and inflate the scores of lower-performing ones, leading to a systematic clustering of scores around the middle. This pattern suggests that AI is not yet capable of providing reliable, subjective assessments of written work. The study highlights that, despite extensive training, AI remains unsuitable for replacing human teachers in grading student work. While higher education institutions are interested in using AI to improve grading objectivity and efficiency, this research indicates that such a shift is not currently feasible. Additionally, the ethical concerns of submitting student work to AI without consent were also raised. The researchers concluded that while AI may improve over time, achieving consistent grading that matches human evaluations remains a challenge.