Generative Artificial Intelligence Improves Programming Skills with Timely Feedback

Strategic Insights -

A new study conducted by Universidad Catolica de la Santisima Concepcion in Concepcion, Chile, has explored the effectiveness of generative artificial intelligence (GenAI) in improving students' programming skills through timely feedback. The research evaluated the effectiveness of two custom-developed applications: a chatbot-like virtual assistant powered by the GPT-4o-mini model and a programming platform that combines GenAI-generated feedback and a virtual judge for source code evaluation. The study consisted of a quasi-experimental design, where students engaged in context-enriched problem-solving and received feedback from the AI tools. The results showed that students who received feedback generated by GenAI did not show improvement in their performance, although they perceived it as useful. Despite the lack of improvement in performance, the usability of both tools was positively perceived by the students, with 70.6% and 65.2% of students indicating that they would use them again.

Key Takeaways:

  • The study evaluated the effectiveness of two custom-developed applications using GenAI: a chatbot-like virtual assistant and a programming platform that combines GenAI-generated feedback and a virtual judge.
  • The study consisted of a quasi-experimental design, where 200 students engaged in context-enriched problem-solving and received feedback from the AI tools.
  • The results showed that students who received feedback generated by GenAI did not show improvement in their performance.
  • The students perceived the tools as useful, with 81% and 79% stating that they would use them again.
  • The usability of both tools was positively perceived by the students, with 70.6% and 65.2% indicating that they would use them again.
  • The study was conducted by Universidad Catolica de la Santisima Concepcion, Concepcion, Chile, and was supported by The Research Office of The Universidad Catolica De La Santisima ConcepcioN and The Engineering 2030 Project.
  • The research concluded that there is insufficient evidence to definitively evaluate the effect of GenAI-generated feedback on students' passing rates in an assessment.

Statistics:

  • 200 students participated in the quasi-experimental study.
  • 70.6% of students indicated that they would use the chatbot-like virtual assistant again.
  • 65.2% of students indicated that they would use the programming platform that combines GenAI-generated feedback and a virtual judge again.
  • 81% of students stated that they would use the tools again if they were available.
  • 79% of students stated that they would use the tools again if they were available.
  • The usability of both tools was positively perceived by the students, with 70.6% and 65.2% indicating that they would use them again.

Sources:

  • NewsRx. Study Findings from Universidad Catolica de la Santisima Concepcion Update Knowledge in Computer Science (Bridging code and timely feedback: integrating generative AI into a programming platform). Computer Weekly News. October 29, 2025; p 827.
  • Bridging code and timely feedback: integrating generative AI into a programming platform. PeerJ Computer Science, 2025,11():e3070.