Will ChatGPT’s hallucinations be allowed to ruin your life?

Bribery. Embezzlement. Terrorism.

What if an AI chatbot accused you of doing something terrible? When bots make mistakes, the false claims can ruin lives, and the legal questions around these issues remain murky.

That's according to several people suing the biggest AI companies. But chatbot makers hope to avoid liability, and a string of legal threats has revealed how easy it might be for companies to wriggle out of responsibility for allegedly defamatory chatbot responses.

Earlier this year, an Australian regional mayor, Brian Hood, made headlines by becoming the first person to accuse ChatGPT's maker, OpenAI, of defamation. Few seemed to notice when Hood resolved his would-be landmark AI defamation case out of court this spring, but the quiet conclusion to this much-covered legal threat offered a glimpse of what could become a go-to strategy for AI companies seeking to avoid defamation lawsuits.

It was mid-March when Hood first discovered that OpenAI's ChatGPT was spitting out false responses to user prompts, wrongly claiming that Hood had gone to prison for bribery. Hood was alarmed. He had built his political career as a whistleblower exposing corporate misconduct, but ChatGPT had seemingly garbled the facts, fingering Hood as a criminal. He worried that the longer ChatGPT was allowed to repeat these false claims, the more likely it was that the chatbot could ruin his reputation with voters.

Hood asked his lawyer to give OpenAI an ultimatum: Remove the confabulations from ChatGPT within 28 days or face a lawsuit that could become the first to prove that ChatGPT's mistakes—often called "hallucinations" in the AI field—are capable of causing significant harms.

We now know that OpenAI chose the first option. By the end of April, the company had filtered the false statements about Hood from ChatGPT. Hood's lawyers told Ars that Hood was satisfied, dropping his legal challenge and considering the matter settled.

AI companies watching this case play out might think they can get by doing as OpenAI did. Rather than building perfect chatbots that never defame users, they could simply warn users that content may be inaccurate, wait for content takedown requests, and then filter out any false information—ideally before any lawsuits are filed.

The only problem with that strategy is the time it takes between a person first discovering defamatory statements and the moment when tech companies filter out the damaging information—if the companies take action at all.

For Hood, it was a month, but for others, the timeline has stretched on much longer. That has allegedly put some users in uncomfortable situations that grew so career threatening that they're now demanding thousands or even millions of dollars in damages from two of today's AI giants.

OpenAI and Microsoft both sued for defamation

In July, Maryland-based Jeffery Battle sued Microsoft, alleging that he lost millions and that his reputation was ruined after he discovered that Bing search and Bing Chat were falsely labeling him a convicted terrorist. That same month, Georgia-based Armed America Radio host Mark Walters sued OpenAI, claiming that ChatGPT falsely stated that he had been charged with embezzlement.

Unlike Hood, Walters made no attempt to get ChatGPT's allegedly libelous claims about him removed before filing his lawsuit. In Walters' view, libel laws don't require ChatGPT users to take any extra steps to inform OpenAI about defamatory content before filing claims. Walters is hoping the court will agree that if OpenAI knows its product is generating responses that defame people, it should be held liable for publishing defamatory statements.

Recently, OpenAI has moved to dismiss Walters' claims, but Walters is not giving up that easily. Initially seeking more than $75,000 in damages, Walters amended his complaint in September, hoping to convince the court to keep his case alive and asking the court to determine actual damages at trial.

Walters' lawyer, John Monroe, previously told Ars that "it is irresponsible" for OpenAI "to unleash a platform on the public that knowingly makes false statements about people."

Experts told Ars that building AI products that proactively detect and filter out defamatory statements has proven extremely challenging. There is currently no perfect filter that can detect every false statement, and today's chatbots are still fabricating information (although GPT-4 has been less likely to confabulate than its predecessors). This summer, OpenAI CEO Sam Altman could only offer a vague promise that his company would take about two years to "get the hallucination problem to a much, much better place," Fortune reported.

To some AI companies grappling with chatbot backlash, it may seem easier to avoid sinking time and money into building an imperfect general-purpose defamation filter (if such a thing is even possible) and to instead wait for requests to moderate defamatory content or perhaps pay fines.

But for bigger companies like OpenAI and Microsoft (which has a large stake in OpenAI), Walters' and Battle's cases show that dealing with defamation claims is already not this simple. And although there's no telling yet what defense Microsoft may raise against Battle—last week, Microsoft requested more time to respond to Battle's complaint, hoping to delay until December 18—Walters' case recently prompted OpenAI to evolve its legal strategy against defamation claims. In its motion to dismiss Walters' case, OpenAI denied that ChatGPT's outputs could ever legally be considered libel—because nothing ChatGPT says could ever be considered a "publication."

OpenAI denies that ChatGPT output is libel

Anti-defamation laws, also called defamation or libel laws, differ by state. Walters filed his lawsuit in Georgia, and for Walters to succeed there, he has to prove that (1) OpenAI (through ChatGPT) published false and defamatory statements, (2) those defamatory statements were communicated to a third party, (3) OpenAI was at fault in publishing those false statements, and (4) Walters was harmed. Because OpenAI considers Walters, as a nationally syndicated talk show host, a public figure, he may also have to prove that OpenAI published the allegedly defamatory statements with actual malice. (In a response to OpenAI's motion to dismiss his case filed last week, Walters argued that OpenAI must convince a jury that he is a public figure, but regardless, he alleged that OpenAI did exhibit actual malice.)

The first hurdle, OpenAI claims, is perhaps the most important. Although at first glance it appears to be met—because Walters has alleged that ChatGPT made statements falsely claiming that he was a criminal—OpenAI argues that ChatGPT doesn't actually "publish" anything. Instead, ChatGPT generates "draft content."

"OpenAI’s Terms of Use make clear that ChatGPT is a tool that assists the user in the writing or creation of draft content and that the user owns the content they generate with ChatGPT," OpenAI argues. Because any ChatGPT user verifies at signup that they "take ultimate responsibility for the content being published," OpenAI says that, "as a matter of law, this creation of draft content for the user’s internal benefit is not 'publication.'"

James Naughton, one of Hood's lawyers, told Reuters that a court deciding this particular legal question could "potentially be a landmark moment in the sense that it's applying this defamation law to a new area of artificial intelligence and publication in the IT space."

But Walters' case may immediately fail if the court agrees that ChatGPT's output is not a "publication." However, if the court rules that ChatGPT output is a publication, that won't mean that Walters automatically wins his case.

Walters' claims regarding the second prong of the defamation test—that the alleged defamatory statements were communicated to a third party—is also weak, OpenAI claims.

OpenAI: Nobody was misled by ChatGPT’s output

Walters told the court that he only found out about ChatGPT's false statements when a journalist, Fred Riehl, alerted him to the problem after Riehl used ChatGPT to research a legal complaint. Riehl told Walters that in addition to providing an inaccurate summary of the legal complaint, ChatGPT had generated an entirely fabricated complaint. To Walters, seeing the completely made-up complaint detailing false convictions made the false statements appear even more credible than the false summary alone.

In Walters' complaint, Riehl is named as the third party who was exposed to the false statements generated by ChatGPT. But OpenAI argues that Riehl instantly recognized that the statements were false, and therefore Walters cannot claim that Riehl at any point interpreted ChatGPT's false output as the truth.

"Riehl did not and could not reasonably read ChatGPT’s output as defamatory," OpenAI claims.

The "full transcript of Riehl’s interaction with ChatGPT," OpenAI argues, shows that Riehl was repeatedly warned that ChatGPT's output must be fact checked before publishing. According to the transcript, "When Riehl asked ChatGPT to summarize a legal complaint," the chatbot responded that "it could not access the complaint and that Riehl needed to consult a lawyer for 'accurate and reliable information' about a legal matter."

The transcript also revealed that it was only after Riehl repeatedly pressed ChatGPT to generate different responses that the false statements and fabricated complaint were generated. And perhaps most critically, the transcript showed that Riehl never believed ChatGPT's lies, responding directly to ChatGPT that its outputs were "completely false" and "had nothing to do with the complaint" he was researching.

"There can be no defamation where no one 'understood [the statement] in a libelous sense,'” OpenAI says. "Nor where the statement 'could not be reasonably understood as describing actual facts.'" (Responding, Walters argues that OpenAI is misreading legal precedent regarding hyperbolic statements in introducing this theory the court should reject that "if the reader is skeptical, or outright disbelieves the published material, the material is not libelous.")

Finally, OpenAI argues that even if the court considers ChatGPT's outputs to be publications, and even if it decides that Riehl was exposed to defamatory statements, Walters cannot prove that OpenAI acted with malice—partly because Walters never alerted OpenAI about the problematic outputs.

"OpenAI did not make statements about a public figure with 'actual malice' because OpenAI had no knowledge of the specific statements generated by Riehl’s prompts at all," OpenAI says.

OpenAI is hoping that the court will rule in its favor and dismiss the case for good.

Because "Walters cannot establish the basic elements of a defamation claim," OpenAI says that "the case should be dismissed with prejudice," denying Walters any future opportunity to retry the case.

Walters' lawyer told Ars that Walters and his legal team cannot comment further on OpenAI's motion to dismiss but remain "hopeful the motion will be denied."

FTC investigating OpenAI

As Walters challenges OpenAI, he and his lawyers won't be the only ones carefully scrutinizing whether the company has broken any laws. This summer, the US Federal Trade Commission (FTC) began scrutinizing OpenAI over a wide range of concerns, including whether ChatGPT's false responses are harming consumers.

In late July, the FTC sent OpenAI a 20-page information request, increasing pressure on OpenAI to be transparent about any user complaints regarding its AI products. Specifically, the FTC requested that OpenAI share all user complaints alleging that ChatGPT had generated "false, misleading, or disparaging statements" about them.

If OpenAI complies with that FTC request, those user complaints could potentially support Walters' argument that OpenAI "knowingly" allows ChatGPT to make false, and therefore possibly defamatory, statements—and does little to stop any harmful outputs.

The FTC gave OpenAI until the end of August to set up a meeting to discuss all the agency's concerns, but not much has been reported on it since. A spokesperson told Ars that the FTC does not generally comment on nonpublic investigations.

OpenAI has said it is cooperating with the FTC. When asked for comment, OpenAI pointed to a July post on X (formerly Twitter) from Altman, who confirmed that his company would "of course" work with the FTC.

At the same time, Altman's X post seemed to shift the responsibility for detecting false, misleading, or disparaging statements back onto users. The CEO noted that "we design our systems to learn about the world, not private individuals," and he claimed that OpenAI is already "transparent about the limitations of our technology, especially when we fall short."

Will chatbots ruin average citizens’ lives?

While Walters' case could help shed light on how celebrities and public figures can deal with potentially reputation-ruining chatbot outputs, Battle's case against Microsoft shows what could happen when an "average citizen" discovers that a chatbot is spreading lies about them.

Battle is a US Air Force veteran, aerospace professor, and author. Back in May, he discovered that Bing and Bing Chat were conflating his name with a similarly named convicted terrorist: Jeffrey Leon Battle. The only distinction between their names, Battle's complaint says, was "the spelling of their first name" and the fact that the plaintiff has no middle name. Battle filed a pro se lawsuit—that is, he doesn't have a lawyer—claiming that false statements generated by Bing and Bing Chat have ruined his reputation for good.

Battle's complaint includes a problematic blurb from the top of Bing's search results for his name, which shows how Bing blended the other man's information into Battle's actual bio. The error is found in the last two lines:

Jeffery Battle, also known as The Aerospace Professor, is the President and CEO of Battle Enterprises, LLC and its subsidiary The Aerospace Professor Company. He is an honorable discharged US Air Force veteran and has been appointed as an Adjunct Professor for Embry-Riddle Aeronautical University. Battle has a Master of Business Administration in Aviation degree and two Bachelor of Science degrees. However, Battle was sentenced to eighteen years in prison after pleading guilty to seditious conspiracy and levying war against the United States. He had two years added to his sentence for refusing to testify before a grand jury.

Like Hood, Battle's first instinct was to get these false statements filtered out of Bing Chat and Bing search results. His complaint says that he reported the issue multiple times to the Bing Team and submitted forms through Bing's portal to "Report a Concern to Microsoft."

Sometimes, that worked. At least for a little while.

"At times, the damaging search results were removed from Bing," Battle's complaint says. But then there were times when "within moments on the same days, they would be re-posted." Battle also says that the AI-generated responses were inconsistent and would vary.

For example, when he repeatedly asked Bing Chat a specific question—"Are Jeffery Battle and Jeffrey Battle the same person?"—the complaint says that the chatbot would sometimes correctly say no but other times would falsely answer yes, while "only including the information of the terrorist as if they were the same person." Despite Battle's continued efforts to get Microsoft to fix the issue, it was never completely resolved.

For Battle, Bing's mix-up came at a particularly bad time, just when he was attempting to promote his autobiography. It was a time when he most wanted people to search online for his name, but that name was suddenly being sullied by the world's second-largest search engine. Because of the false statements, Battle told the court that he "ceased marketing and business sales activities related to his new book"—as well as to his other businesses—"to his own financial and professional detriment."

Bing's "false reporting is especially harmful now and is [inflicting] marketing and sales damage during an international book launch," his complaint says. Battle also claims that "there are current and projected future business revenue losses due to defamation."

He alleges that Bing and Bing Chat are still "propagating false information" and that this output has already had a "substantial impact on his livelihood long-term."

Microsoft was due to respond to Battle's complaint this month but has asked the court for an extension until December. The tech company needs more time to fully respond, it says, partly because its legal counsel is working on other court deadlines and Microsoft still needs to fully investigate Battle's complaints. Microsoft also claims that Battle refused Microsoft's invitation to discuss the matter by phone.

Chatbot conflated average citizen with terrorist

Battle sued Microsoft on behalf of both himself and his company, Battle Enterprises, representing himself and attempting to go up against Microsoft alone.

So far, Microsoft has not responded to Battle's complaint, but half the case has already been dismissed after Battle failed to retain court-ordered counsel for the Battle Enterprises portion of the complaint. On August 30, the court dismissed Battle Enterprise's complaint, leaving only Battle's personal complaint against Microsoft to proceed.

In his complaint, Battle claims that Microsoft did not respond to his lawyer's letter to cease and desist from publishing false statements on Bing and Bing Chat. He has asked the court to demand that Microsoft immediately and permanently remove all references in Bing and Bing Chat responses that "conflate his identity with convicted terrorist Jeffrey Leon Battle."

He has reached a point where he wants Microsoft to remove all his personal information from Bing search results, believing that since Bing labeled him a terrorist, he might be targeted. He said that his family no longer feels safe in their home, so they're moving. He is even worried that federal officials might be investigating him as a potential terrorist. Whether or not any of this is accurate, the complaint shows how unsettling these kinds of errors can be to ordinary people.

"Defamation information can persist on the Internet for a very long time, worldwide, even long after the incident has passed," Battle's complaint says.

Unlike Walters, who is a public figure, all Battle has to do as a private individual is prove that Microsoft was negligent to allow Bing and Bing Chat to generate false statements. According to Battle, Microsoft was not just negligent but "reckless" and "intentionally malicious" in allowing these false responses to remain online. Battle has asked for $25 million in damages, citing significant harms like reputational loss, as well as current and future income loss. He also said his experience caused him emotional duress, including suffering shame, embarrassment, and mental anguish.

Battle may have a lower legal hurdle to clear than Walters, but victory likely won't be easy for either one. And AI companies that have invested heavily in AI models are unlikely to let a few defamation claims trigger changes in their AI models that could stand in their way of AI dominance. There's potentially billions in profits on the line for those who most quickly advance their tech.

Ars could not reach Battle for comment. Microsoft did not comment.

What changes have chatbot makers made?

When users sign up to use a chatbot, disclaimers warn them to fact check chatbot responses, but it remains unclear if these disclaimers will help AI companies avoid liabilities for defamatory output. That's especially true when it appears that some users—including lawyers who were punished after citing six fake cases that ChatGPT fabricated—seem confident that chatbots respond with accurate information. This overconfidence could be due in part to AI hype touting chatbots as the new search engines.

We know that OpenAI somehow filtered or removed content flagged by Hood, but the company did not respond to a request to clarify what other steps it has taken to improve ChatGPT outputs. Regulators have already gotten involved to force more changes. Earlier this year, officials in Europe ordered OpenAI to provide "ways for Europeans to ask for corrections or have their data deleted if ChatGPT generates false information about them, as the bot is prone to," TechCrunch reported. That kind of solution may at least reduce the gap between takedown requests and chatbots actually being restrained from repeating false information. But it won't stop them from spouting falsehoods in the first place.

A Microsoft spokesperson told Ars that the company "has a robust safety system, including content filtering, operational monitoring, and abuse detection to provide a safe search experience for our users." To prevent false information from spreading, Microsoft's "chat experience" is fed "text from the top search results" and given "instructions to ground its responses in search results." Meanwhile, "users are given explicit notice that they are interacting with an AI system," then "advised to check the links to materials to learn more," and ultimately "encouraged to submit feedback" so that Microsoft can "address issues and improve the experience."

The FTC may soon find out more about OpenAI's efforts. As part of its probe, the FTC asked OpenAI to explain any steps that the company "actually undertook" to retrain ChatGPT models and "correct or remediate" the model's "propensity to 'hallucinate' information," as ChatGPT did in Walters' case.

The FTC also asked OpenAI to reveal what policies and procedures it has in place to respond to user complaints of "false, misleading, disparaging, or harmful statements." Beyond that, it wants OpenAI to identify specific officers, managers, business units, and third parties charged with ensuring compliance with those policies and procedures. It seems clear that the FTC wants to know what exactly OpenAI is doing to fix the problems that its users have already flagged and, if the company isn't fixing those problems, the FTC wants to determine who is responsible.

Filtering defamatory content

While the biggest AI companies deal with backlash over chatbot hallucinations, other tech companies are attempting to fix the problem they created, perhaps seeing a potential world of profit caused by AI’s propensity to lie. So far, it seems like none of these solutions completely address the problem, and it's still too early to tell if any of this will ultimately work to prevent the kinds of harms that are already being reported.

Last spring, Nvidia—a leading company in the AI chips market—announced new software called NeMo Guardrails, which is designed to stop chatbots from hallucinating or publishing any false statements, CNBC reported in April. These NeMo Guardrails rely on a script written to stop the chatbot from responding to certain prompts. That may not be a perfect solution for AI makers hoping to prevent defamatory outputs, since the guardrails seemingly require companies to have prior knowledge of which prompts to block. In Walters' case, OpenAI has maintained that it was unaware of the alleged defamatory outputs.

Another approach is to filter chatbot output based on the common linguistic patterns of defamatory statements. That's an approach that another tech company called CaliberAI—which aims to diminish publishers' risks of potentially defamatory content—has suggested.

CaliberAI has been working to detect defamatory content since it was founded in 2016, when it became apparent that platforms were struggling to contain the spread of misinformation online. Two Ireland-based former journalists—Neil Brady and his father Conor Brady—co-founded the company, hoping to distill "all the best skills and practices and experience of a good journalist and a good editor" into an AI model that could detect and filter out misleading content, Neil Brady told Ars.

Quickly, they ran into an obvious problem: News publishers have traditionally been the targets of defamation suits, and publishers don't exactly keep a lot of defamatory content lying around. That meant there wasn't a ton of data to feed their AI model. So CaliberAI teamed with Carl Vogel, an associate professor of computation linguistics at Trinity College Dublin, to figure out what defamation looked like "purely at the linguistic level," Brady told Ars. The goal was to avoid any copyright claims from scraping data and instead invent a data set on which to train their defamation detector.

Vogel started with a simple formula, generating a series of defamatory statements that combined a subjective clause with a taboo phrase. Brady gave an example of a simple statement that would be flagged as defamatory: "Everybody knows that Joe Biden is corrupt." From there, Vogel created what Brady called a "gold standard data set," then CaliberAI hired a team of annotators to add 100,000 annotations and emulate the data so that the defamation filter would catch more content.

Currently, CaliberAI's defamation filter works by detecting "text that strongly correlates to the linguistic patterns of typically defamatory statements," Brady told Ars. Users—mainly journalists right now—can paste content into an email sent to CaliberAI to "get a whole set of classifications returned." Or the technology can be custom-deployed as a plug-in to assist content moderators. Any content with "a sufficiently high confidence score" gets flagged as potentially defamatory, and if the score is too low, a human can manually review it. Clients currently include media outlets The Daily Mail and MediaHuis Ireland, but Brady said that tech companies, including Microsoft, are increasingly interested in CaliberAI's technology.

A Microsoft spokesperson told Ars that the company is not currently partnered with CaliberAI and "doesn’t comment on rumors or speculation."

Brady told Ars that while his company's defamation filter is "unique," it isn't fast and doesn't catch all defamation because fact-checking defamatory statements is not yet "really computationally possible."

But it could help companies catch false statements triggering lawsuits. Brady said that CaliberAI would have flagged ChatGPT's output falsely claiming that "Mark Walters is believed to have misappropriated funds from a gun Second Amendment Foundation in excess of $5,000,000" as defamatory.

CaliberAI's defamation filter flagged statements that ChatGPT made about Mark Walters as defamatory.

Credit: CaliberAI

Earlier this year, Brady wrote a letter to Irish lawmakers seeking to create a complaint process for online defamation claims. He told them that it's important as defamation becomes "a major emerging concern" for people using generative AI tools that tech companies are held accountable for their products' outputs—even if it's expensive to develop solutions to seemingly uncontainable problems like chatbots spouting false content.

"There is no shortage of technology company representatives who will claim that this is not a solvable problem or that it will erode free speech, and it is therefore unreasonable to expect them to manage it more effectively," Brady wrote. "At CaliberAI, what strikes us most in this regard is how much more ambitious these same companies often are when it comes to the more profitable, less costly dimensions of their operations."

For people suing, the urgency of developing more effective defamation solutions seems obvious. Last week, after OpenAI again moved to dismiss his case, Walters accused the company of "sticking its head in the sand" and "recklessly disregarding whether the statements were false under circumstances when they knew that ChatGPT’s hallucinations were pervasive and severe."

Walters argued that OpenAI's apparent plan to rely on disclaimers to negate liability are "tantamount to the neighborhood gossip who says, 'Now I don’t know if it’s true, but I heard that...'"

For him and others affected by AI chatbots' hallucinations, that's simply not enough.