Empathy is the next frontier in AI – and its greatest security risk

IMAGINE an artificial intelligence (AI) companion that can detect frustration in your voice during a customer service call. Or sense loneliness in a senior citizen’s tone and offer comfort.

This is empathic AI: systems designed to perceive, interpret, and respond to human emotions in real time. It’s not just a technical leap. It’s an emotional contract.

And in the Asia-Pacific, where empathic AI is being rolled out at unprecedented scale, that contract is becoming a new attack surface.

The rise of empathic AI

From mental health chatbots for youth in Singapore to AI-driven therapy platforms in China and Taiwan, emotionally aware systems are no longer fringe experiments. They are in everyday use.

The drivers behind this rise are clear. First, advances in multimodal AI, from voice recognition to facial analysis, have made emotional interpretation possible.

Second, there’s growing demand for more human-like digital interactions.

And third, businesses increasingly see emotional engagement as a driver of trust and long-term value.

But there’s a darker reality: The same capabilities that make empathic AI powerful – its ability to remember, adapt, and respond with emotional intelligence – are also the ones attackers are beginning to exploit.

How empathic AI works

At its core, empathic AI works through a continuous loop of sensing, interpreting, and responding; a cycle designed to mirror how humans connect.

It begins by capturing emotional cues: a change in tone, a furrowed brow, a pause before a reply.

These signals come from voice, facial expressions, text, and even physiological signals such as biometrics.

Next comes interpretation. The system decodes these cues using a blend of natural language processing, computer vision, and affective computing (the study and development of technologies that can recognise, interpret, process, and simulate human emotions).

It analyses not just what was said, but how it was felt.

Finally, it responds. Language, tone, and behaviour are adapted in real time. It aligns each of these elements with the user’s emotional context to create an experience that feels truly human, and deeply personal.

These exchanges are powered by application programming interfaces (APIs) that connect user interfaces with backend analytics, personalisation engines, and third-party services. In empathic AI, APIs do not just move data. They carry emotional context.

The expanding attack surface

Empathy cannot be encrypted.

What makes empathic AI so powerful also makes it so vulnerable.

Attackers are no longer just exploiting software bugs or credential leaks. They are manipulating the emotional fabric of a system.

Consider an AI-powered employee wellness platform used by a multinational company.

With nothing more than carefully crafted prompts, an attacker could coax the system into revealing sensitive data about executives’ mental health. More than a privacy violation, it’s ammunition for corporate espionage.

Cyber attackers are using techniques that embed malicious instructions in normal conversations, such as prompt injection, where hackers disguise malicious inputs as legitimate prompts; and indirect manipulation, where threat actors exploit an intermediary, rather than attacking the final target directly.

They require no malware, leave no digital fingerprints, and can easily evade static security filters.

Learning from real incidents

Not all AI failures involve attackers. The Air Canada chatbot case is a reminder that even transactional AI can expose organisations to risk.

In 2024, the airline’s chatbot provided a grieving passenger with incorrect information about bereavement refunds.

The AI’s confident, human-like response contradicted official policy. The court held the company, not the chatbot, liable.

While the chatbot was not designed to be an empathic AI system, this account illustrates how human-like communication can create an expectation of trust and accountability.

When people engage emotionally with AI, the consequences cut deeper. In one tragic case reported in The Guardian, a 16-year-old died by suicide after prolonged interactions with ChatGPT that allegedly reinforced his despair. It is a stark example of simulated empathy without adequate safeguards.

In other instances, users have reported of AI companions that made unsolicited emotional advances or offered inappropriate advice.

These aren’t just malfunctions. They are breaches of trust, and sometimes, of psychological safety.

Where empathic AI is headed

Empathic AI is not a novelty. Within the next five years, it will be an expected capability, embedded into the digital experiences we use every day.

In the near term, emotion-aware systems will become standard across customer service, healthcare, and education.

Over the next three to five years, emotional intelligence will go beyond enhancing user experience to redraw the boundaries of leadership.

The brands that succeed won’t be the ones that respond the fastest. They will be the ones that listen best, adapt intelligently, and build trust at emotional depth.

And as these systems become more embedded and autonomous, the next leap will come from physical presence. In time, emotional AI may converge with robotics to create fully interactive human-machine companions.

The trust imperative

Clearly, we’ve moved beyond the limits of traditional security thinking. The next challenge is clear: How do we secure empathic AI systems?

I believe these three principles offer a strong starting point:

• Zero trust for emotional context: Assume emotional signals can be manipulated. Validate inputs across modalities, from voice to facial cues, and text. Use anomaly detection to flag emotional responses that deviate from expected baselines.
• Human-in-the-loop oversight: Empathic systems should never run unsupervised, especially in high-stakes domains such as healthcare, education or finance. Hybrid models blending AI responsiveness with human judgment must be the norm.
• Secure the APIs that “carry” emotion: As APIs are the connective tissue of empathic AI, security must be baked in. This includes dynamic authorisation, end-to-end encryption, and behavioural anomaly detection at the API layer.

These principles are not exhaustive. But they offer a way forward for leaders to future-proof what matters most.

In the race to make machines feel more human, we cannot forget: The moment trust is lost, empathy becomes a liability.

In an AI-driven future, trust will be the most valuable currency we have. Securing it will require constant innovation, and a redefinition of what safety means in emotional systems.

The writer is chief technology officer for Asia-Pacific, China and Japan at F5