Sign in Subscribe
Artificial intelligence

Artificial Intelligence in Foreign Language Teaching: Evaluating the Reliability of Large Language Models

Strategic Insights

— 2 min read

Researchers from the University of Belgrade have conducted a study on the use of artificial intelligence in foreign language teaching, with a focus on the Serbian language. The study aimed to explore the possibilities of using AI for higher-level foreign language teaching, where language competence implies understanding dialectisms, phraseology, idioms, proverbs, and gnomes. The researchers used three large language models, Claude, ChatGPT, and Gemini, to translate texts from various Serbian dialects into the standard Serbian language. The results showed that Claude and ChatGPT can be considered reliable in translating texts written in dialects, making it easier for students to understand the language.

Key Takeaways:

  • The study used three large language models, Claude, ChatGPT, and Gemini, to evaluate their reliability in translating texts from various Serbian dialects into the standard Serbian language.
  • The research sample consisted of 113 respondents, including non-philologists, foreign language teachers, and teachers of Serbian as a mother tongue or a foreign language.
  • The researchers used descriptive statistics and one-way ANOVA with post-hoc tests (Bonferroni test) to process the obtained statistical data.
  • The results show that Claude and ChatGPT can be considered reliable in translating texts written in dialects, making it easier for students to understand dialects, phraseology, idioms, and proverbs in the Serbian language.
  • The study offers didactical implications for using artificial intelligence in foreign language teaching, suggesting that AI can be a valuable tool for improving language proficiency.
  • The researchers used an anonymous questionnaire created on Google Forms to gather data from the research participants.
  • The study provides evidence that AI can be used to improve language teaching, particularly in areas where traditional methods may struggle, such as understanding dialectisms and phraseology.

Statistics:

  • 113 respondents from various backgrounds participated in the study, including non-philologists, foreign language teachers, and teachers of Serbian as a mother tongue or a foreign language.
  • The study used three large language models, Claude, ChatGPT, and Gemini, to evaluate their reliability in translating texts from various Serbian dialects into the standard Serbian language.
  • The results show that 85% of the respondents evaluated the texts generated using Claude and ChatGPT as very reliable, while 90% of the respondents evaluated the texts generated using Gemini as reliable.
  • The study demonstrates the potential of AI in improving language proficiency, particularly in areas where traditional methods may struggle.

Sources:

  • Artificial intelligence in foreign language teaching: Evaluating the reliability of large language models with a focus on Serbian as a foreign language. Inovacije u Nastavi, 2025,38(2):96-110.
  • University of Belgrade, Teacher Education Faculty.
  • Journal of Engineering. August 4, 2025; p 2702.

Read more