How will I adjust my analysis and the write-up of the research results to reflect my findings on the validity and reliability of my measures? The initial responses to these measurement questions at the design stage, along with considerations of the importance of having valid and reliable measures of a particular skill to answer the research questions in the RCT, can inform decisions about the pilo. [...] As described in the online appendix B of Laajaj and Macours (2021), if the factor analysis does not clearly align with the theoretical expected grouping of variables in a certain scale, it brings into question the validity of the interpretation of the scale in accurately capturing what it is supposed to measure. [...] Referencing the validation of the interpretation of the tool can provide a greater signal to the reader of its validity and reliability than simply indicating the tool has been used in the past, as frequently used tools are not necessarily the most valid. [...] Especially in the aftermath of the pandemic, the global education system has not only had to cope with the disruption to academic learning, but also the social isolation and the stunting of children’s social, emotional, and cognitive development. [...] This resource is not meant to be a prescriptive document on which tools to use when conducting research around holistic skills but aims to provide guidance to researchers on the questions to consider when designing RCT studies with an important focus on skill measurement, to point to the relevance of establishing validity and reliability of the measures for the context of study, and to provide exa.
Authors
- Pages
- 36
- Published in
- United States of America
Table of Contents
- prepared by 1
- 1. Motivation 2
- 2. Questions on Skill Measures to Consider at the RCT Design Stage 4
- Guiding questions: 4
- 3. Skill Measurement Methods 6
- Introduction 6
- Self-reported measures 6
- Measures reported by others 7
- Observed/Direct Assessment 7
- Combined measures 9
- 4. Validity and reliability of Skill Measures in RCT studies 10
- Introduction 10
- Validation techniques employed 10
- Example of reliability checks 11
- Examples of intra-measure content and construct validity checks 12
- Examples of predictive validity checks: 12
- Examples of piloting work 14
- Additional approaches 15
- Some additional thoughts on validity and reliability checks 15
- 5. Measurement validation based on the citations and references 17
- 6. Takeaways 18
- References 19
- Appendix 1: Detailed Findings from Review of RCT Literature on Holistic Skills based on AEA registry 26
- 1.1 BROAD CHARACTERIZATION OF STUDIES COVERED BY THE REVIEW 26
- 1.2 SKILLS MEASURED 27
- 1.3 MEASUREMENT TOOLS 30
- 1.4 VALIDATION TECHNIQUES EMPLOYED 32
- 1.5 REFERENCING VALIDATION AND PRECEDENT PAPERS 34
- 1.6 RESULTS AND IMPACT DIRECTIONS 35
- Appendix 2: Measures Observed in the Review 36
- Appendix 3: Papers Observed in the Review 36
