Exploring the Potential of Using AI Language Models in Democratising Global Language Test Preparation (75554)

Session Information: Design, Implementation & Assessment of Innovative Technologies in Education
Session Chair: Ed Sykes

Saturday, 25 November 2023 10:00
Session: Session 1
Room: Room C (Live Stream)
Presentation Type: Live-Stream Presentation

All presentation times are UTC + 9 (Asia/Tokyo)

This paper delves into the potential of AI language models for democratizing global language test preparation, focusing on the accuracy and consistency of assessment in the context of IELTS writing essays. Through a quantitative study, we compare the assessment scores generated by a Human Examiner (HE) and four AI Language Models: ChatGPT, Google Bard, Writing9.com, and Upscore.ai. The former two models are generative NLP systems, while the latter two are tailored specifically for evaluating IELTS essays. Evaluation is conducted using Mean Absolute Errors (MEA) and Bland Altman analysis. The findings reveal varying levels of accuracy, with Upscore.ai showcasing the lowest MEA of 0.5, followed by Google Bard at 0.85, ChatGPT at 0.9, and Writing9.com at 1.9. Bland Altman Plots provide a visual representation of the agreements between each alternative evaluation system and the Human Examiner, shedding light on their alignment. These results hold significant implications for assisting IELTS test takers in their preparation efforts and advancing the democratization of IELTS and global language assessment by harnessing AI technology to provide more accessible and consistent evaluation methods.

Authors:
Amalia Novita Sari, Sagara Abhipraya (SA) Edu Lab, Indonesia


About the Presenter(s)
Ms Amalia Novita Sari is a Business Owner/Executive at Sagara Abhipraya (SA) Edu Lab in Indonesia

See this presentation on the full scheduleSaturday Schedule



Conference Comments & Feedback

Place a comment using your LinkedIn profile

Comments

Share on activity feed

Powered by WP LinkPress

Share this Presentation

Posted by Clive Staples Lewis

Last updated: 2023-02-23 23:45:00