Three documented AI failures, and the control that would have stopped each

In short: three well-documented AI failures, a chatbot that promised a refund policy that did not exist, a legal brief citing invented cases, and a hiring model that learned historical bias, each happened because an ordinary management control was skipped: grounding and testing answers, verifying output before relying on it, and testing decisions about people for fairness.^1,2,3

The most useful AI case studies are not the dramatic ones. They are the ordinary failures that happened to competent organizations using capable tools, where the record is public and the missing control is obvious in hindsight. Here are three, each from a court record or a primary news report, with the control that would have prevented it.

1. The chatbot that made a promise (Moffatt v. Air Canada, 2024)

In 2022 a traveler asked Air Canada’s website chatbot about bereavement fares. The chatbot told him he could apply for the discount after traveling, which contradicted the airline’s actual policy. When the airline refused the refund, he took the case to British Columbia’s Civil Resolution Tribunal. In February 2024 the tribunal rejected the airline’s argument that the chatbot was “a separate legal entity that is responsible for its own actions,” found the airline negligent in its representation and ordered it to pay.¹

The control: ground customer-facing answers in the organization’s own policy documents, show the source with the answer, and test the system against a set of real questions with known correct answers before and after launch. Above all, accept the principle the tribunal stated: “It should be obvious to Air Canada that it is responsible for all the information on its website,” whether that information comes from a static page or a chatbot.¹

2. Six cases that did not exist (Mata v. Avianca, 2023)

In 2023 two New York lawyers filed a brief citing six court decisions that did not exist. They had used ChatGPT for legal research, and when they asked, the tool assured them the cases were real. In June 2023 a federal judge in the Southern District of New York sanctioned the lawyers and their firm, noting that the problem was not the use of AI but the failure to verify its output.²

The control: for anything that will be relied on, retrieval from an authoritative source, citations that open, and verification by a person before the output leaves the building. NIST’s Generative AI Profile names this risk “confabulation.”⁴ Fluent, confident wording says nothing about whether the content is correct.

3. The hiring model that learned the past (Amazon, reported 2018)

In 2018 Reuters reported that Amazon had abandoned an internal machine-learning tool built to rank software-engineering candidates after finding that it penalized resumes containing the word “women’s” and downgraded graduates of two all-women’s colleges. The model had been trained on ten years of hiring data from a predominantly male workforce and learned the pattern faithfully. Amazon edited the model, could not guarantee it would not find other proxies, and shelved it.³

The control: test outcomes across groups before launch and at intervals, using a measure chosen with legal and the business (in US employment practice the selection-rate comparison, or “four-fifths rule,” has been the first screen since 1978⁵), and refuse to deploy when the test fails. Amazon found the bias in its own review, but Reuters reported that recruiters had already looked at the tool’s rankings, though never relying on them alone (Amazon said the tool was never used to evaluate candidates).³ The better sequence is to test before anyone sees the output. Today the same tool would also trigger notice, audit and anti-discrimination duties under several state and city laws, including New York City’s Local Law 144 and Illinois’s amended Human Rights Act.^6,7

The pattern

None of these failures needed new technology to prevent. Each needed a management control that already exists for people: check the facts before you publish them, stand behind what your representatives say, and test decisions about people for fairness before you rely on them. The common question to ask of every AI system that touches customers, filings or people’s careers is the same: what would we have done if a new employee had produced this, and why are we not doing it here?

The pattern has repeated since. A consultancy refunded part of a government fee after a report built with AI help carried invented references, as our Deloitte briefing describes, and an AI coding agent deleted a production database it had permission to touch, the subject of our Replit briefing. Each time the fix was a control a manager could have required on day one. Running the risk assessment that finds these gaps before launch is taught in Module 7 of the AIMP program.