The Intersection of AI and Language Assessment: A Study on the Reliability of ChatGPT in Grading IELTS Writing Task 2

Koraishi O.

Language Teaching Research Quarterly, cilt.43, ss.22-42, 2024 (Scopus) identifier

  • Yayın Türü: Makale / Tam Makale
  • Cilt numarası: 43
  • Basım Tarihi: 2024
  • Doi Numarası: 10.32038/ltrq.2024.43.02
  • Dergi Adı: Language Teaching Research Quarterly
  • Derginin Tarandığı İndeksler: Scopus, Educational research abstracts (ERA), MLA - Modern Language Association Database
  • Sayfa Sayıları: ss.22-42
  • Anahtar Kelimeler: Artificial Intelligence (AI), Artificial Intelligence in Education (AIEd), CALL, ChatGPT, IELTS, Natural Language Processing (NLP)
  • Bursa Uludağ Üniversitesi Adresli: Evet

Özet

This study conducts a comprehensive quantitative evaluation of OpenAI's language model, ChatGPT 4, for grading Task 2 writing of the IELTS exam. The objective is to assess the alignment between ChatGPT's grading and that of official human raters. The analysis encompassed a multifaceted approach, including a comparison of means and reliability measures such as Cohen's weighted kappa and intraclass correlation. The results revealed a high agreement in means and substantial reliability between the two grading methods on the level of the majority of texts. However, individual discrepancies and outliers were also identified, underscoring the nuanced nature of grading. While ChatGPT demonstrated efficiency and general alignment with human grading, the study concludes that it should not replace human judgment, particularly due to these observed inconsistencies. The findings contribute valuable insights into the potential and limitations of AI in educational grading and emphasize the importance of a comprehensive quantitative evaluation.