How human should an AI agent seem?

[SINGAPORE] Meta’s Muse, OpenAI’s dots and Anthropic’s Claude Duet signal a shift from artificial intelligence assistants that answer questions to agents that remember context and work on our behalf.

Singapore’s moves in October – the release of AI Risk Management Guidelines by the Monetary Authority of Singapore and three agentic-AI testing initiatives by the Infocomm Media Development Authority – underscore the importance of governing these systems.

As agents become more capable, ubiquitous and familiar, we need to address what they can do and the actor they seem to be as they interact with people.

When an assistant becomes an actor

An agent that acts on our behalf raises questions about authority, oversight and accountability.

Singapore’s Model AI Governance Framework for Agentic AI asks organisations to set boundaries on what agents can access and do; it preserves meaningful human responsibility, tests these agents before and after deployment, and crucially, tells users about their capabilities and limits.

The framework forms a strong governance foundation, and addresses what an AI agent may do.

However, users also respond to the actor an agent seems to be. Once an agent speaks in a warm voice, helpfully recalls an earlier conversation or checks on the user’s mental health during a discussion, it starts building a relationship.

And that side of the agent requires a stronger guardrail.

Even when users know it is AI, how human-like should the agent be designed to seem? And what happens when the model underneath the agent changes and its persona disappears?

The relationship needs restraint

The first question sits at the centre of research I published with Andrew Prahl of Nanyang Technological University. We mapped 2,685 authenticity articles and synthesised 111 peer-reviewed studies of AI personas in 23 national contexts.

The analysis revealed a basic mistake in the way organisations think about human-like design.

Human cues – such as a name, emotional expression and cultural fit – can help build trust, but adding more of them to an AI agent does not necessarily make it more trusted.

Why do human signals in AI agents foster trust up to a point, then then backfire? Our proposed Threshold Model of Synthetic Authenticity offers an explanation.

When interacting with agents, people perform two tasks at once: they assess what the agent says while quietly monitoring what it is. Every lifelike feature or claim of emotion is one more thing to reconcile with the fact that the actor is not human.

Beyond a point, more realism may prompt scrutiny from the human user rather than trust. That point depends on the task and relationship.

The practical implication for organisations is to disclose the use of AI personas clearly and add only the human cues that serve a role.

A text assistant may need clarity and a consistent voice, a health-support agent may need patience and careful acknowledgement. Neither needs a realistic face or simulated vulnerability. An agent can acknowledge distress without claiming to feel empathy.

Organisation should honestly inform what the agent can do (and cannot feel), and clarify in advance how an exchange can be routed to a human if needed.

The relationship must survive the upgrade

As model prices continue to fall and capabilities improve quickly, swapping the model underneath an agent could become routine.

This is why organisations should also consider and manage possible changes to the agent’s persona when the underlying model is replaced.

The AI agent’s persona is the relationship layer through which people learn how far to trust the AI actor. Human users learn what the AI agent remembers, how it behaves and how it exercises judgment.

With a new model, the agent could become, for instance, more emotionally expressive or less willing to escalate an exchange to a human. A design that sat below the human-likeness threshold yesterday can cross it after an upgrade that is invisible to users.

A faster, more sophisticated model can still produce a far worse relationship.

The fix here is a model-independent “Persona Charter”. It should set out what the agent does, how human it may sound, what it may remember, and when a human must take over.

Before a major release, teams should replay representative user histories and test whether the new system stays within the charter. If the new behaviour differs materially from before, the agent’s permissions should be reviewed instead of rolled over automatically.

Govern the action as well as the actor

AI assurance then works in two layers: The first is action assurance, which covers what the agent may access, decide and do. The second is actor assurance, which covers what it seems to be, the number of human qualities that suit its role and whether its behaviour remains within set boundaries after an upgrade.

Singapore’s pragmatic approach to AI governance provides a base for both. The next step is congruence. Disclosure, design, role and authority should tell users the same story. None of this means agents must come across synthetic.

A plain-speaking AI agent can still be unsafe, and a human-like one can still be distrusted. The better principle is proportionality: Give an agent enough humanity to serve its task, never so much that the performance outruns reality, and always keep a clear path back to a person.

The writer is vice-president of revenue and insights for South-east Asia at Isentia

AI was used for research support, drafting and critiquing the draft before submission. The author remains fully accountable for the article’s accuracy and originality.