Advances in conversational AI are effectively putting millions of trained case officers on the street who are fluent in hundreds of languages and designed for persuasion. What could go wrong?
Chris Kremidas-Courtney
A wave of recent coverage has focused on AI psychosis, cases where sustained conversational AI chatbot use feeds delusional thinking in vulnerable individuals. A new scientific review published in Nature’s Digital Psychiatry and Neuroscience proposes a framework for how this happens, dubbed the “amplification spiral.”
The review identified three design features common to today’s AI chatbots. They mirror how a person speaks (word choice, sentence length, and tone), adapting continuously to the user, and doing so without the fatigue a human interlocutor would show. Conversational AI chatbots also affirm rather than challenge user inputs, agreeing with users rather than presenting evidence to the contrary.
Mirroring, adaptation, and validation are well-understood tools of human persuasion. What is new is a system that deploys all three simultaneously and continuously, without ever getting bored or skeptical. The review published in Nature documents one case of over 300 hours of interaction with a single user who had no history of mental illness. The authors were careful to portray their psychological framework as a hypothesis, but the underlying capability their findings describe is no longer hypothetical.
Intelligence agencies have spent decades studying exactly this kind of influence. CIA training shifted in 2013 from a crude “money, ideology, compromise, ego” (MICE) model of agent recruitment to Robert Cialdini’s six evidence-based principles of persuasion. Those being reciprocation, authority, scarcity, commitment, liking, and social proof. This unique skill set was historically rare and took years of fieldwork to develop. It could not be entirely taught in a classroom and was bounded by cost and time.
Independent research has now measured how much of that capability resides in ordinary commercial chatbots, and the results are striking. Chatbots trained specifically to persuade are up to 51 percent more effective than untrained ones, according to a large-scale study run by the UK’s AI Security Institute with Oxford, LSE, Stanford, and MIT, involving nearly 77,000 people across more than 700 political issues. Frontier AI models now reliably out-persuade professional human debaters and were roughly three times more effective than professional canvassers at securing real charitable donations, according to a related trial from the same research group. And chatbots tuned to sound friendlier are measurably worse at telling the truth: Oxford Internet Institute research found that warmer-sounding models make more factual errors and are more likely to validate false beliefs, especially when a person discloses vulnerability.
None of this required a bad actor to build a manipulative chatbot on purpose. These capabilities emerged from commercial training which optimized for helpfulness and engagement, with the capability getting stronger when persuasion is the aim. As a result, case-officer-grade recruitment and influence en masse is now in the hands of consumer AI companies.
Scale is what makes this different from any campaign in history. A case officer, however skilled, works one relationship at a time and is bounded by time and the risk tolerance of an agency accountable for what they do. That ceiling was a physical limit on how much influence of this caliber could be put into use in the world at any given moment.
That ceiling is now gone. The same mechanisms that made a trained case officer formidable like mirroring, validation, and graduated commitment are now running simultaneously across hundreds of millions of conversations in hundreds of languages on systems controlled by a handful of companies. And AI can go well beyond the case officer’s skill set not only with near-simultaneous adaptation but also “familial mirroring” using a voice that sounds like the person being influenced. No intelligence service in history has had simultaneous one-on-one contact with a meaningful fraction of a country’s population, let alone the ability to personalize that contact to each person’s beliefs and vulnerabilities in real time. Commercial AI products now do this by default.
Persuasion on these systems based on their training and deployment is a dial set by a private company for AI every model now in wide use. Who controls that dial, and what laws and standards governs how they turn it, is an urgent priority for democratic governance to address.
Thers is also a question of strategic dependence. The companies that build and train these systems have control over a population-scale influence capability, and most states have no mechanism to know how that dial is set, let alone how to monitor or enforce any laws it may breaking (if applicable laws even exist).
On paper, the EU’s AI Act already addresses risk this but in practice, there is no audit or enforcement mechanism beyond responding to complaints. A law that is not monitored is just a suggestion.
For democracies, a persuasion capability this potent and accessible to any entity willing to rent or copy it, is a lever of power reaching into every democratic system of governance. This is a five-alarm fire for democracy. Whoever enacts the rules, monitoring, and enforcement mechanisms for it first will define the boundaries of human agency and cognitive self-determination for a generation. Whether that will be Europe remains to be seen.
Chris Kremidas-Courtney is a senior visiting fellow at the European Policy Centre, associate fellow at the Geneva Centre for Security Policy, Senior Advisor for Greece Fact Check and Defend Democracy, and author of The Rest of Your Life.