The experiment deployed AI-powered bots that posted over 1,000-1,700 comments to test whether large language models (LLMs) could persuade users to change their minds on controversial topics. The study tested three approaches: generic AI responses, community-aligned responses using a fine-tuned model, and personalized responses that analyzed users' post histories to infer demographic details like age, gender, political orientation, and ethnicity. This personalized approach didn't just edge out humans—it crushed them. AI proved three to six times more persuasive than human commenters and achieved persuasion rates that placed it in the 99th percentile of all users.
A few provisos: nothing in this study should be taken as gospel. Given that the Reddit users agreed to have others attempt to change their minds, there's certainly the possibility for opt-in bias at play here. And the results would need to be replicated before drawing definitive conclusions. Finally, let's be clear: much of the study was downright unethical. The bots assumed fabricated identities including a rape victim, a Black man opposed to Black Lives Matter, and a domestic violence counselor. But the results are striking nonetheless and have enormous implications for any form of persuasion.
Throughout the entire four-month experiment, not a single user suspected they were interacting with AI rather than humans. The bots blended seamlessly into authentic online discourse, racking up over 10,000 "karma points" and 137 "delta awards"—the subreddit's gold standard for successfully changing someone's mind.
After moderators discovered the unauthorized experiment and filed ethics complaints, Reddit's response was swift and unequivocal. The Chief Legal Officer called the research "deeply wrong on both a moral and legal level," banned every account associated with the study, and initiated formal legal proceedings against the University of Zurich. The researchers, facing the backlash, ultimately decided not to publish their findings. But the AI genie was already out of the bottle.