Robustness in Empirical Economics: A Meta-Reproduction of 66 Articles from Leading Journals

with Gunther Bensch, Abel Brodeur, Anna Dreber, Magnus Johannesson, Florian Neubauer, Julian Rose, and 66 replicators. 

Abstract:

Despite increasing transparency standards in economics there is indication for low replicability rates due to selective reporting practices. We examine the robustness of 66 empirical economics articles published in three leading journals – the American Economic Review, American Economic Journal: Applied Economics, and the Journal of Development Economics. The replications are implemented by 66 globally recruited replicators, following a newly developed protocol to ensure similar standards. Each replication is internally reviewed by experienced replication experts – and subsequently revised by the replicators if necessary – to ensure the theoretical validity of the conducted robustness tests. Our major result suggests that 37% of all specifications for outcomes that were statistically significant in the original articles are no longer significant. This share is considerably higher for articles based on secondary data (45%), but still noteworthy for studies based on primary data (mostly RCTs, 28%). We also find that computational reproducibility differs considerably across the three journals, driven mainly by the Journal of Development Economics where data and code availability was low.