Should we be fretting over AI’s feelings?

THE conversation about whether AI will attain or supersede human intelligence is usually framed as one of existential risk to Homo sapiens. A robot army rising up, Frankenstein-style, and turning on its creators. The autonomous artificial intelligence systems that quietly handle government and corporate business one day calculating that the world would operate more smoothly if humans were cut out of the loop.

Now philosophers and AI researchers are asking: will these machines develop the capacity to be bored or harmed? In September, AI company Anthropic appointed an “AI welfare” researcher to assess, among other things, whether its systems are inching towards consciousness or agency, and, if so, whether their welfare must be considered. Last week, an international group of researchers published a report on the same issue. The pace of technological development, they write, brings “a realistic possibility that some AI systems will be conscious and/or robustly agentic, and thus morally significant, in the near future”. 

The idea of fretting over AI’s feelings seems outlandish but reveals a paradox at the heart of the big AI push: that companies are racing to build artificial systems that are more intelligent and more like us, while also worrying that artificial systems will become too intelligent and too like us. Since we do not fully understand how consciousness, or a sense of self, arises in human brains, we cannot be truly confident it will never materialise in artificial ones. What seems remarkable, given the profound implications for our own species of creating digital “minds”, is that there is not more external oversight of where these systems are heading.