Listen: https://serve.buildtall.systems/c2b0ea5c29801127113936ecb13c7014a092eafcbb6ac28ce711d6ec9edbfa15.mp3
marginal revolution read out. This episode covers the post Are RCTs underpowered? The post opens with a question that matters for development economics and for empirical social science more broadly: are the randomized controlled trials that we rely on actually powerful enough to detect the effects they claim? The author's answer is a cautious maybe so. The post points to a new paper by Qingyuan Chai, Kevin Lang, and Liang Zhong, published as National Bureau of Economic Research Working Paper 35999 in November 2026. The author's take is that the paper offers a serious empirical case that many development RCTs are underpowered.
Randomized controlled trials have become a major method in development economics, often treated as the gold standard for causal evidence about what works in poverty reduction, health, education, and other areas. Their credibility depends on being able to distinguish real effects from noise. That is why the question of statistical power matters so much.
The linked paper is titled (In)Credible Development RCTs: False Discoveries, Low Power, and a Solution. It is a detailed statistical study of 745 principal-hypothesis tests drawn from 190 articles in leading economics journals. The authors look at the distribution of t-statistics and compare it to the theoretical benchmark expected under 80 percent power, which is about 2.8. What they find is that actual t-statistics peak well below that benchmark, but still above the values typically associated with p-hacking. That pattern matters because it suggests the problem is not primarily that researchers are gaming their results. Instead, many trials may simply be too small to detect the effects they are intended to measure.
From this starting point, the authors build an empirical Bayes mixture model that accounts for selective publication. Their estimate is striking: approximately 30 percent of published rejections are false discoveries, and average directional power is only about 40 percent. At conventional significance thresholds, the posterior probability of a false discovery reaches 66 percent. That means a published statistically significant result, at standard thresholds, may be more likely false than true. That is a sharp challenge to the credibility of RCT evidence in development economics.
The paper then asks what could be done. Doubling the original sample sizes lowers the average false discovery rate by about five percentage points. Tightening the significance threshold to 0.005 lowers it more, but at a cost: there are fewer total discoveries. This tradeoff is a central concern for funders. The authors characterize a funder's optimal replication policy under a fixed budget and an evidence standard. Under standard regularity conditions, replications concentrate in an intermediate interval of original test statistics. The optimal replication sample size is single-peaked in the original t-statistic. In illustrative calibrations, successful replications reduce aggregate false discovery rates to between 1.8 and 7.8 percent, depending on the evidence standard.
The author's commentary is brief, but the choice of the paper is pointed. The question Are RCTs underpowered? is answered with a maybe so, and the linked paper supplies a rigorous basis for that hesitation. The larger theme is that underpowered trials plus selective publication can make published rejections less reliable than they appear. The paper suggests that targeted replication and stricter evidence standards could recover much of that lost credibility.
