Evaluating the Effectiveness of Artificial Intelligence in Enhancing Teaching Effectiveness

Strategic Insights -

Research at the University of Southern Queensland suggests that artificial intelligence (AI) offers potential solutions to identify learning gaps and provide targeted improvements in online learning, but its reliability and effectiveness in educational contexts must be rigorously evaluated. A study aimed to evaluate the performance and reliability of an AI model designed to identify the characteristics and indicators of engaging teaching videos. The findings indicate that the AI model requires continuous updates to improve its accuracy and effectiveness, highlighting the need for ongoing evaluation and development.

Key Takeaways:

  • The growing popularity of online learning brings inherent challenges that must be addressed, particularly in enhancing teaching effectiveness, which AI can potentially solve.
  • AI models designed for education must be rigorously evaluated to ensure their reliability and effectiveness, requiring continuous updates and improvements.
  • The study employed a design-based approach, incorporating statistical analysis to evaluate the AI model's accuracy by comparing its assessments with expert evaluations of teaching videos.
  • Multiple metrics were employed, including Cohen's Kappa, Bland-Altman analysis, the Intraclass Correlation Coefficient (ICC), and Pearson/Spearman correlation coefficients, to compare the AI model's results with those of the experts.
  • The findings indicated low agreement between the AI model's assessments and those of the experts, with low Cohen's Kappa values, moderate variability in Bland-Altman analysis, and weak relationships in Pearson and Spearman correlations.
  • The ICC indicated moderate reliability in quantitative measurements, suggesting that the AI model has some limitations in its ability to accurately identify engaging teaching videos.
  • The research concluded that future work should focus on expanding the dataset and utilizing continual learning methods to enhance the model's ability to learn from new data and improve its performance over time.
  • The study highlights the importance of ongoing evaluation and development of AI models in education to ensure their effectiveness in enhancing teaching effectiveness.

Statistics:

  • The AI model analyzed 100 teaching videos to evaluate its accuracy and reliability.
  • The findings indicated low agreement between the AI model's assessments and those of the experts, with a Cohen's Kappa value of 0.23, indicating minimal categorical agreement.
  • The Bland-Altman analysis showed moderate variability, with a mean difference of 0.45 and a standard deviation of 0.23.
  • The ICC indicated moderate reliability in quantitative measurements, with a value of 0.67.
  • The Pearson and Spearman correlations revealed weak relationships, with values of -0.12 and -0.16, respectively.

Sources:

  • Evaluating an Artificial Intelligence (AI) Model Designed for Education to Identify Its Accuracy: Establishing the Need for Continuous AI Model Updates. Education Sciences, 2025,15(4):403. (Education Sciences - http://www.mdpi.com/journal/education)
  • NewsRx. University of Southern Queensland Researchers Highlight Research in Artificial Intelligence [Evaluating an Artificial Intelligence (AI) Model Designed for Education to Identify Its Accuracy: Establishing the Need for Continuous AI Model Updates]. Education Letter. May 14, 2025; p 784.