A Systematic Reassessment and Meta-Analysis of Wellbeing Research

with Caspar Kaiser (University of Warwick)

Fierce debate over the feasibility of cardinally measuring utility – or ‘wellbeing’ – with surveys has recently resurfaced. Several prominent papers claimed that when interpreting survey data as strictly ordinal, most of the literature’s results are easily reversed. We systematically assess this claim. To do so, we replicate the universe of wellbeing research published in top economics journals since 2010. In total, we replicate 35 studies, containing 9,183 coefficients. For all coefficients, we assess whether signs of regression coefficients are invariant under all positive monotonic transformations of the scale with which wellbeing is recorded. About 40% of results cannot be reversed with any monotonic transformation of the scale. Comparatively low reversal risks are observed for the effects of income (19%) and unemployment (8%) as key wellbeing determinants. Once we allow for a mild degree of heterogeneity in mean wellbeing within response categories, these figures increase. To aid the robustness of future wellbeing research, we also estimate models of reversal risk. Generally, reversal risk decreases drastically with the statistical significance of the original estimates. Keeping everything else constant, the risk of reversal of an estimate that is statistically significant at the 1% level is 10 percentage points lower than that of an estimate that is significant at only the 5% level. Likewise, estimates with a clear exogenous and causal identification strategy also have a significantly lower risk of reversibility.