We provide twelve best practices and discuss how each practice can help researchers accurately, credibly, and ethically use Generative AI (GenAI) to enhance experimental research. We split the twelve practices into four areas. First, in the pre-treatment stage, we discuss how GenAI can aid in pre-registration procedures, data privacy concerns, and ethical considerations specific to GenAI usage. Second, in the design and implementation stage, we focus on GenAI’s role in identifying new channels of variation, piloting and documentation, and upholding the four exclusion restrictions. Third, in the analysis stage, we explore how prompting and training set bias can impact results as well as necessary steps to ensure replicability. Finally, we discuss forward-looking best practices that are likely to gain importance as GenAI evolves.
Authors
- Acknowledgements & Disclosure
- We appreciate insightful comments from Kyle Boutilier, Brian Jabarian, Alex Kim, and Connor Murphy. All errors are our own. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research. John A. List John List is the Chief Economist at Walmart but this research is not part of work at Walmart.
- DOI
- https://doi.org/10.3386/w33025
- Pages
- 21
- Published in
- United States of America
Table of Contents
- Introduction 3
- Twin Experimental Goals 5
- Pre-Treatment Stage 6
- Best Practice 1. Use GenAI to lower the administrative burden of pre-treatment procedures without lowering quality. 6
- Best Practice 2. Disallow public model training on inputted data and ensure that AI platforms have adequate data privacy measures. 7
- Best Practice 3. Carefully consider additional risks to participants and elicit informed consent for GenAI use to ensure the highest ethical standards. 8
- Design and Implementation Stage 9
- Best Practice 4. Use GenAI to identify new mediators and moderators while expanding the original design using Option C Thinking to optimally generate evidence that scales. 9
- Best Practice 5. Never delegate data collection or experimental procedures to GenAI without appropriately piloting, documenting, and reviewing its roles and output. 10
- Best Practice 6. Consider how GenAI both supports and challenges upholding the four exclusion restrictions necessary for internally valid experiments. 11
- Data Analysis Stage 12
- Best Practice 7. Ensure consistent reinitialization and provide prompts that capture the construct of interest. 12
- Best Practice 8. Carefully consider training data and, when appropriate, manually train GenAI models used for analysis while simultaneously using ML to enhance experimental precision. 13
- Best Practice 9. Maximize the replicability and reliability of results through documentation and stress testing, allowing the broader research community to confirm experimental findings. 14
- Future Considerations 15
- Best Practice 10. Adapt resource allocation over different stages of experimental research as GenAI changes trade-offs. 15
- Best Practice 11. Hypotheses generated through GenAI need to be interpretable to humans. 16
- Best Practice 12. GenAI is not a substitute for human ingenuity and innovation. 16
- Conclusion 17
