Evidence for behavioural interventions looks increasingly shaky

WHEN economists at the University of Toronto started to tell undergraduates in 2014 how many hours’ extra work they needed to put in to boost their grades, they hoped it would encourage the students to work harder.

They didn’t. Instead, the students just began to expect the lower grades they received.

The university’s experience is frequently quoted as an example of “nudge” theory backfiring. Nudge, the fashionable face of behavioural economics that launched a thousand light-touch government policies, has soared in popularity since the 2008 book of the same name by Richard Thaler, an economist, and Cass Sunstein, a legal scholar.

There are now more than 200 “nudge” units around the world, teams that specialise in applying behavioural science to everyday life. Nudges seek to persuade rather than compel behaviour change, through a series of psychological strategies, from presenting information in a different way to offering alternatives. As Thaler and Sunstein put it: “Putting fruit at eye level counts as a nudge. Banning junk food does not.”

Nudge theory did not have a great pandemic. Nudge-friendly behavioural scientists were blamed by some for the British government’s initial embrace of soft messaging — appeals to personal responsibility such as the slogan “stay home, save lives” — over strict measures including lockdowns, while a scheme using lotteries with prizes up to US$50,000 did little to boost vaccination uptake in Philadelphia.

Nudge fans received some better news at the end of 2021. The first attempt to pool and judge the academic foundations of the theory, more formally known in the field as choice architecture interventions, reported in glowing terms. Psychologists at the University of Geneva analysed some 200 nudge studies and concluded that not only did nudges work overall, but that they did so impressively.

Behavioural scientists judge effect size with a measure called “Cohen’s d”, which shows the difference between the average results in a control and treatment group. A score of zero means that the nudge has made no difference, while anything over 0.8 is considered to indicate a very large effect. The Geneva team said that nudge measures promoted behaviour change with an overall Cohen’s d of 0.43. That places it firmly in the medium-sized category, and is more than enough to make most policymakers sit up and take notice.

Other psychologists saw the claims too, and many were unhappy with them. Last week, three separate academic groups, from Britain, Hungary and America, published critiques in the Proceedings of the National Academy of Sciences, which also published the Swiss team’s initial analysis.

The complainants make two similar points. First, the academic trials take such different approaches and report such wildly different effect sizes that it does not make sense to bundle them together in the same analysis. A “medium-sized” effect for nudges overall grossly exaggerates the impact of those that are useless and underestimates the benefits of those that work.

Second, nudge research is highly susceptible to publication bias — academic journals tend to favour publishing studies that report the largest effect sizes. After using statistical tools to account for the distorting effect of publication bias, the critics point out that the average effect size from the original analysis collapses, to as low as 0.04, which is effectively useless.

That does not mean that all nudges are ineffective, says Barnabas Szaszi, a psychologist at the University of Budapest and one of the critics. Famously, images of everything from flies to golf flags placed in urinals have been shown to improve men’s aim and reduce cleaning costs. But in such a heterogeneous meta-analysis, those trials of nudge policies that do show significant effects are essentially swamped by the mass of those that don’t.

Such meta-analyses, says Szaszi, should break down nudges into smaller distinct groups by type. Only then will policymakers have better ways to measure the effectiveness of different nudge tactics and so learn lessons from the best ones.

“We agree with most of the points raised,” says Ulf Hahnel, one of the Geneva group that carried out the original meta-analysis in 2021. That work did not intend to portray nudges as a silver bullet, he says, and did include caveats about heterogeneity and publication bias.

The controversy emphasises that psychologists and other social scientists need to do more to combat publication bias, the critics say. One ongoing effort asks researchers to pre-register studies before they start, and for journals to agree to publish the results however they fall. Take-up is patchy. Time for a nudge?

©2022  The Economist Newspaper Limited. All rights reserved