Assessing the Impact of Community-Level Initiatives: A Literature Review

Addressing many of the most difficult social problems requires an approach that combines the efforts of multiple organizations, often operating in different sectors, coordinating across an entire community. Numerous programs overseen by the Administration for Children and Families (ACF) engage in these community-level initiatives, or community-level change efforts. These initiatives, however, are challenging to evaluate. To inform ACF on how to better capture these initiatives’ impact, the Office of Planning, Research, and Evaluation (OPRE) at ACF commissioned this literature review to see what can be learned from recent efforts that have not only attempted to achieve community-level change, but have also attempted to produce credible evidence about the impact of these efforts. To draw out these lessons, this review begins by providing an overview of the methodological challenges community-level initiatives pose for assessing causal impacts. It then highlights some key lessons gleaned from recent evaluations of community-level change efforts. Summary of Key Findings and Considerations

• Credibly assessing the causal impact of initiatives on outcomes of interest is a long-standing challenge in the evaluation literature. The most credible way to determine causality is through random-assignment studies. However, such studies are often infeasible or impractical for evaluating community-level initiatives.
• Impact evaluations for community-level initiatives involve special challenges. Evaluation designs must account for these challenges, including the difficulty of defining the “treatment,” the importance of spillover effects within communities, the interactions between different initiative programs and services, and the difficulties of detecting effects in community-level outcomes.
• New methods are strengthening our ability to measure the effects of community-level initiatives. Recent advances in quasi-experimental methods suitable for community-level evaluation are promising. In addition, theory-based or mechanism evaluations—i.e., evaluations focused on uncovering the casual links between an initiative’s activities and its outcomes—can be especially helpful for learning about an initiative’s impacts. Moreover, mixing methods to include qualitative and quantitative approaches can strengthen causal claims.
• Recent evaluations of community-level initiatives such as the Community Development Block Grant, Promise Neighborhoods, and the Urban Health initiative, among others, offer particularly useful lessons for how to approach community-level evaluations.