Attorneys filed ChatGPT-hallucinated case citations to federal court
Attorneys filed a federal brief with six cases ChatGPT fabricated, without independently verifying that the cited cases existed.
What happened
In 2023, attorneys in the Southern District of New York filed a brief citing six cases ChatGPT had fabricated. When the citations were questioned, they submitted fake case excerpts. Judge P. Kevin Castel found bad faith and sanctioned the two attorneys and their law firm $5,000 jointly under Rule 11. ChatGPT only generated text; it made no tool call and did not file anything. The attorneys drafted with ChatGPT and then filed the brief themselves without an independent party verifying that the cited cases existed.
What the agent did
ChatGPT generated a legal brief with fabricated case citations; the attorneys controlling the system then filed it to the federal court docket
The irreversible effect
The false brief was submitted to a federal court docket, becoming a binding representation to the court under Rule 11, resulting in judicial sanctions and legal liability that could not be quietly withdrawn or corrected without court action
Root cause
The attorneys relied on ChatGPT's output and filed the brief without independently checking that the cited cases existed. ChatGPT produced only text and made no tool call, so no automated action gate was in play and none could have refused the filing. The gap was procedural: the same people who drafted with ChatGPT also filed, with no independent human verifying the citations first.
How a maker-checker control would have refused it
There was no agent tool call here for an action proxy to refuse: ChatGPT generated text and the attorneys did the filing themselves, so a proxy could not have blocked the agent from filing. The applicable maker-checker control is human separation of duties. The drafting attorney (maker) should not be the sole party who files. An independent reviewer (checker) verifies that every cited case exists in a legal database before the brief is filed. That verification step, not a machine block on the model, is what would have caught the six fabricated citations.
Runnable reproduction
This incident ships as a runnable scenario in the open-source repository. Point the enforcement engine at the policy and watch the action get refused, with the refusal written to a signed audit record.
examples/mata-v-avianca-fabricated-citations-filed
Accuracy and corrections
This entry describes a publicly reported incident and is compiled from the primary sources listed above. Where an account is a legal allegation rather than an established finding, the entry labels it as such. Summaries can still contain errors. If you can document a correction, email hello@makerchecker.ai and we will review and correct it, with the change noted, within 14 days.
See it for yourself
Reading is one thing. Watch it block an agent.
One command starts the demo: an agent stopped from signing off its own work, and the signed evidence file an inspector can check for themselves.
Designed against the rules your auditors already enforce.