A Lawyer Cited AI-Hallucinated Murder Witnesses and Got Fined 5,000 Dollars

A New Mexico lawyer has been fined 5,000 dollars by the state's Supreme Court after submitting an appeal for a murder conviction that included AI-fabricated witnesses and fake police testimony. The case is one of the most serious legal consequences handed down in the US for AI misuse in a courtroom, and honestly, it probably should have been worse.

The lawyer used ChatGPT to help construct an appeal, and the model did what language models sometimes do when pushed to produce specific information they don't actually have: it made things up. Convincingly. The fabricated witnesses and testimony were apparently plausible enough to make it into a formal legal document submitted to a supreme court. That's not a small thing. This is a murder case. Someone's freedom depends on the quality of that appeal.

This Is Not a New Problem, But the Stakes Keep Getting Higher

Lawyers citing fake AI-generated cases have been caught in US courts since at least 2023, when a New York attorney famously submitted a brief full of entirely fictional case citations. That lawyer was fined 5,000 dollars too, which makes you wonder whether the punishment is actually proportionate to the harm. In that instance, nobody's liberty was directly on the line in the same way. A murder conviction appeal is a different category of seriousness entirely.

What's changed since 2023 is that AI tools have become dramatically more capable and far more embedded in professional workflows. GPT-6 Astra, the model currently dominating headlines, is being used by companies like Perplexity and Cognition to run production systems autonomously. The pressure on lawyers, especially those at smaller firms without large research teams, to use AI to stay competitive is real. But capability and reliability are not the same thing, and courts are the last place you want to discover that gap.

The New Mexico Supreme Court's ruling made clear that it doesn't matter whether the lawyer intended to deceive the court. Using AI to generate legal citations or witness information without independently verifying every single claim is professional negligence. Full stop.

What This Means for the Legal Profession

Bar associations across the US have been scrambling to update their guidance on AI use since 2023, with varying degrees of urgency. Some states now require lawyers to disclose when AI tools were used in preparing documents. Others are still in consultation. The American Bar Association issued guidance in 2023 urging competence and supervision but stopped short of mandating disclosure.

The problem is that 5,000 dollars, while embarrassing, is not necessarily a strong enough deterrent for a profession where AI tools can save dozens of billable hours per case. The math on whether to risk it still works out favourably for some practitioners, particularly if they believe they can catch hallucinations before they cause damage. Clearly, that confidence is sometimes misplaced.

In the UK, the Solicitors Regulation Authority has been similarly cautious, issuing warnings rather than firm rules. The Law Society has called for clearer standards. But courts in both countries are increasingly losing patience with the argument that lawyers didn't know their AI tools could fabricate information. That excuse wore thin about two years ago.

The Bigger Picture

There's a broader question here about who bears responsibility when professional AI use causes harm. The lawyer is being punished, rightly. But the tools themselves present information with a confidence that can be genuinely misleading. OpenAI and other labs have worked to reduce hallucination rates significantly, and GPT-6 Astra is substantially more accurate than earlier models. But substantially more accurate is not the same as reliable enough to use without verification in a legal document.

For anyone in a profession where errors have real consequences, whether that's law, medicine, engineering, or finance, the rule remains the same as it was in 2023: AI can draft, research, and suggest, but a qualified human has to check the work. Every time. That's not a criticism of AI. It's just an honest accounting of where the technology currently sits.

The 5,000 dollar fine will be paid and largely forgotten. The murder conviction appeal will presumably be resubmitted correctly. But the underlying tension between professional reliance on AI and professional accountability for errors is only going to intensify as these tools become more deeply embedded in how lawyers actually work.