The purpose of this study was to describe the measures used to evaluate principals in New Jersey in the first (pilot) year of the new principal evaluation system and examine three of the statistical properties of the measures: their variation among principals, their year-to-year stability, and the associations between these measures and the characteristics of students in the schools. The study reviewed information that developers of principal practice instruments provided about their instruments and examined principals' performance ratings using data from 14 districts in New Jersey that piloted the principal evaluation system in the 2012/13 school year. The study had four key findings: First, the developers of principal practice instruments provided partial information about their instruments' reliability (consistency across raters and observations) and validity (accurate measurement of true principal performance). Second, principal practice ratings and schoolwide student growth percentiles have the potential to differentiate among principals. Third, school median student growth percentiles, which measure student achievement growth during the school year, exhibit year-to-year stability even when the school changes principals. This may reflect persistent school characteristics, suggesting a need to investigate whether other evaluation measures could more closely gauge principals' contributions to student achievement growth. Finally, school median student growth percentiles correlate with student disadvantage, a relationship that warrants further investigation using statewide evaluation data. Results show a mix of strengths and weaknesses in the statistical properties of the measures used to evaluate principals in New Jersey. Future research could provide more evidence on the accuracy of measures used to evaluate principals. The following are appended: (1) Description of districts participating in the pilot; (2) Data used in the study; (3) Design of the principal evaluation system and component measures selected by pilot districts; and (4) Variation in ratings on the component measures.
Authors
- Authorizing Institution
- ['Regional Educational Laboratory Mid-Atlantic (ED)', 'ICF International', 'National Center for Education Evaluation and Regional Assistance (ED)']
- Education Level
- Elementary Secondary Education
- Location
- New Jersey
- Peer Reviewed
- T
- Publication Type
- Reports - Research
- Published in
- United States of America
Table of Contents
- Measuring principals’ effectiveness: Results from New Jersey’s principal evaluation pilot 1
- Key findings 1
- Summary 3
- Contents 4
- Boxes 4
- Figures 4
- Tables 5
- Why this study? 6
- Box 1. Component measures of New Jersey’s principal evaluation system 7
- Pilot measures of principal practice 7
- Pilot measures of student achievement 7
- Measures developed or refined during the pilot year 7
- What the study examined 8
- Box 2. Data and methods 9
- Data 9
- Methods 9
- What the study found 11
- The developers of the principal practice instruments provided partial information on reliability and validity 11
- Variation in ratings on the component measures 12
- Changes in school median student growth percentiles and school median student growth percentile ratings across years 17
- School median student growth percentiles statewide were more stable for larger schools. 17
- Correlations between ratings and student characteristics 20
- Implications of the study findings 21
- Limitations of the study 22
- Appendix A. Description of districts participating in the pilot 24
- Pilot districts were diverse in size, school composition, and student demographic composition 24
- Pilot districts included a larger proportion of principals with school median student growth percentile ratings of ineffective or partially effective than principals statewide 25
- Appendix B. Data used in the study 27
- Implementation data 27
- Evaluation ratings 27
- Principals’ job assignments 28
- School-level student achievement growth 28
- School-level student background characteristics 29
- Appendix C. Design of the principal evaluation system and component measures selected by pilot districts 30
- Box C1. New Jersey Department of Education criteria for principal practice instruments 30
- Districts selected six commercially available principal practice instruments and developed one instrument approved by the New Jersey Department of Education 31
- Principal practice instruments contain four to seven domains or areas or practice 33
- Developers recommend multiple observations to evaluate principals on the practice instruments 33
- Developers recommend one to three days of training on the principal practice instruments 35
- Half the pilot districts reported that they received 11–30 hours of training on the principal practice instruments from the developers 35
- Pilot districts reported training successes and challenges 36
- Human capital management was highlighted as a critical responsibility of principals, requiring its own measure 37
- The majority of pilot districts reported measuring human capital management responsibilities by using their selected principal practice instrument or by designing their own rubric 38
- Pilot districts provided limited information on principal goals 38
- Box C2. Example of guidance for setting principal goals 39
- Rationale 39
- Administrator goal 39
- Students included in goal 39
- The method of converting school median student growth percentiles into school median student growth percentile ratings compresses the variation in school median student growth percentiles, especially for percentiles in the middle of the distribution 40
- Summative ratings were not calculated uniformly for the pilot year 41
- Appendix D. Variation in ratings on the component measures 43
- Notes 45
- References 46
