Abstract:

      Multisite studies are a commonly used way to assess how a treatment works across contexts. In multisite random controlled trials (RCT), cross-site treatment effect variance is a way to quantify treatment effect variation. However, there are no standard methods for estimating cross-site treatment effect variation designed to be used in multisite regression discontinuity designs (RDD). In this research, we rectify this gap in the literature by developing and evaluating two methods for estimate cross-site treatment effect variance in RDDs. The first method combines a fixed intercepts/random coefficients (FIRC) model with a local linear RDD analysis. The second method borrows techniques from random effects meta-analysis and employs them with the RDD model. We find that although the FIRC model may look appealing ex-post to a researcher because it has a smaller confidence interval than the random effects meta-analysis model, simulations show the FIRC model estimates of the cross-site treatment effect standard deviation have substantial bias, poor coverage, and lack well defined confidence intervals. In contrast, the random effects meta-analysis estimates of the cross-site treatment effect standard deviation have good coverage across a range of conditions. We then apply these models to a high school exit exam policy in Massachusetts that required students who passed the high school exit exam but were still determined to be nonproficient to complete a Education Proficiency Plan". We find that students on the margin of proficiency required to complete an Education Proficiency Plan in math were seven percentage points more likely to complete a math course their senior year. However, if we assume normality, the cross-high school treatment effect standard deviation was high enough in three cohorts for the treatment effect to have been negative in more than a third of high schools.