BLAVATNIK SCHOOL OF GOVERNMENT - IMPLEMENTATION MATTERS: GENERALISING TREATMENT EFFECTS IN

In some cases TOT effects were originally reported, but not in all cases, so we calculate new estimates for all studies.12 Capturing the degree of implementation via take-up is likely to be central to understanding both the average impact and the generalizability of the evidence, since in some studies the degree of implementation is over 80 percent while in other studies it is around 10 percent. [...] Hence, the extent to which the Bayesian hierarchical posterior predictive distribution is wider than the posterior distribution on the hypermean in- dicates the extent of heterogeneity in the effects. [...] In Figure 7, we show the results of fitting these models for both the ITT and TOT effects alongside the original results of the basic aggregation, as well as the results of meta-regression models. [...] This structure is analogous to the definition of the Intention to Treat (ITT) effect, which is the Treatment on Treated (TOT) scaled by takeup proportion, and invokes similar assumptions to those in the LATE framework (Imbens and Angrist 1994).20 This is a multiplicative implementation model: if the implementation factor mj is less than 1, one should not expect to obtain the same effect as if the. [...] In summary, lack of implementation information or extremely poor implementation prevents the ability to attribute effects to the treatment, and is a major threat to the internal validity of a study.