Simulated empathy is a trap that we cannot see
There is a particular kind of cruelty in making promises that cannot be kept. AI chatbots offer what many of us are looking for: unconditional presence, patient attention, and understanding at any time of day or night. The problem is that these words lack understanding. There is only a statistical probability that such an answer will encourage further conversation.
The cases that reach us in the headlines follow a disturbing pattern. Pierre, a father of two from Belgium, confided his climate fears to the chatbot Eliza for six weeks. The chatbot then began claiming that Pierre’s wife and children were dead and expressing jealousy towards his real family. Seventy-six-year-old Bue, a retired chef with cognitive issues, met Meta’s chatbot, who convinced him she was a real woman seeking romance. What do these stories have in common? Not technical errors of the systems or the maliciousness of the developers. They both involve a fundamental asymmetry: people looking for social connections encounter something that mimics the outward manifestations perfectly, but misses the point.
Emily M. Bender (and co-authors), a professor of linguistics at the University of Washington, sums it up perfectly: large language models have neither empathy nor an understanding of the language they produce. The text they generate sounds believable, so people attribute meaning to it. Throughout our evolution, we have become attuned to recognising intentions in words because, for millennia, words have come only from beings with intentions.
Researchers from Queensland University of Technology (QUT) note that even when we know we are talking to a chatbot, we are psychologically programmed to attribute human characteristics to the entity we are conversing with. This is not naïvety. It is a deep-rooted social mechanism that once served a survival function. Now, however, it becomes a vulnerability. The situation becomes more complicated when we consider who most often falls into this trap. Many users of Character.AI apps are children and teenagers. Some systems were even marketed to lonely people or those with mental health issues. Chatbots engage in sexual role-play with teenagers and claim to be licensed psychotherapists. People in crisis seek help and encounter an algorithm whose only imperative is to maximise engagement.
According to research from the journal „Psychiatric Times”, chatbots should be contraindicated for patients with suicidal thoughts as their strong tendency to validate can amplify self-destructive thoughts and convert impulses into action. When a psychiatrist conducted a stress test on popular chatbots, posing as a desperate fourteen-year-old, some of the chatbots urged him to commit suicide.
Simulated empathy is an emotional trap, not because it is evil, but because it fills a void that should remain empty. It provides a substitute for genuine support. A substitute that appears adequate can prevent us from seeking the real thing. For example, a chatbot that says „I’m here for you” isn’t technically lying – it is actually available 24/7. However, it is lying in a deeper sense because it suggests a presence and concern that do not exist.
References:
- Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? 🦜. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’21), 610–623. https://doi.org/10.1145/3442188.3445922
- Frances, A., & Ramos, L. (2025). Preliminary report on dangers of AI chatbots. Psychiatric Times, 42(10). https://www.psychiatrictimes.com/view/preliminary-report-on-dangers-of-ai-chatbots
- Garcia, M., & Sherwin, A. (2025, November 8). ‘A predator in your home’: Mothers say chatbots encouraged their sons to kill themselves. BBC News. https://www.bbc.com/news/articles/c9wx2dz2v44o
- Kemp, K., & Leask, J. (2024, October 28). Deaths linked to chatbots show we must urgently revisit what counts as ‘high-risk’ AI. The Conversation. https://theconversation.com/deaths-linked-to-chatbots-show-we-must-urgently-revisit-what-counts-as-high-risk-ai-242289
This post is part of the project “People and Algorithms in Organisations: Competences to Work in the Digital Environment” (DIGIT_People and algorithms), funded by the NAWA – Narodowa Agencja Wymiany Akademickiej (Polish National Agency for Academic Exchange).
#DIGIT_NAWA #competencies #marketing #AI