The Broad View
·
SUBSCRIBERS

The folly of data mining

The problem we face today is not data scarcity but surfeit – and big data sets can easily be mined and tortured to identify patterns that are utterly useless

    • The rush to find statistical significance in pools of data has led to discredited studies that are the tip of a statistical iceberg.
    • The rush to find statistical significance in pools of data has led to discredited studies that are the tip of a statistical iceberg. Photo: Pixabay
    Published Sat, Apr 30, 2022 · 05:19 AM — Updated Mon, Feb 3, 2025 · 06:36 PM

    GARY SMITH

    COFFEE was wildly popular in Sweden in the 17th century — and also illegal. King Gustav III believed that it was a slow poison and devised a clever experiment to prove it. He commuted the sentences of murderous twin brothers who were waiting to be beheaded, on one condition: One brother had to drink three pots of coffee every day while the other drank three pots of tea. The early death of the coffee-drinker would prove that coffee was poison.

    It turned out that the coffee-drinking twin outlived the tea drinker, but it wasn’t until the 1820s that Swedes were finally legally permitted to do what they had been doing all along — drink coffee, lots of coffee.

    Share with us your feedback on BT's products and services