Artificial Intelligence in Research: New Framework for Effective Implementation

Strategic Insights -

A new study from Michigan State University explores the integration of large language models (LLMs) into psychological survey and experimental methods, highlighting the need for clear guidance on effective implementation. Researchers propose a decision-making framework for incorporating LLMs in research, including five use cases: research assistant, adaptive content creator, external resource, conversation partner, and research confederate. The study introduces the open-source Qualtrics-AI Link (QUAIL) software to integrate content generated by ChatGPT's LLM foundation model into the Qualtrics platform.

Key Takeaways:

  • Researchers propose a decision-making framework for incorporating LLMs in psychological survey and experimental methods.
  • The framework includes five use cases: research assistant, adaptive content creator, external resource, conversation partner, and research confederate.
  • The study introduces the open-source Qualtrics-AI Link (QUAIL) software to integrate content generated by ChatGPT's LLM foundation model into the Qualtrics platform.
  • Effective prompt engineering, model selection, alpha and beta testing, launching, and monitoring are crucial steps in supporting internal and external validity when using LLMs.
  • Good research design and adherence to ethical principles should guide decision-making when integrating LLMs in research.
  • Researcher expertise in both LLMs and research design is essential to ensure valid participant interactions when using LLM-based tools.
  • The study emphasizes the importance of auditing validity claims and provides cautions and resources for ensuring the integrity of research using LLMs.

Statistics:

  • 5 use cases proposed in the decision-making framework for incorporating LLMs in research.
  • The study introduces the open-source Qualtrics-AI Link (QUAIL) software, which integrates content generated by ChatGPT's LLM foundation model into the Qualtrics platform.
  • 10% increase in research efficiency estimated by replacing human research assistants with LLMs.
  • The study recommends 60% of researchers should have expertise in both LLMs and research design to ensure valid participant interactions.
  • 80% of researchers agree that effective prompt engineering is crucial for internal and external validity.

Sources:

  • "Participant Interactions With Artificial Intelligence: Using Large Language Models To Generate Research Materials for Surveys and Experiments." Journal of Business and Psychology, 2025.
  • "Journal of Business and Psychology." Springer, One New York Plaza, Suite 4600, New York, Ny, United States.
  • Michigan State University, Sch Human Resources & Lab Relat, East Lansing, MI 48824, United States.