Recent news stories highlight the remarkable extent to which artificial intelligence (AI) and chatbot technology can already assist in the practice of law. If progress continues at the current rate, chatbots may soon be able to function just like real lawyers. That would be both good news and bad news.
A team of University of Minnesota law professors has demonstrated that a chatbot, ChatGPT (Generative Pre-trained Transformer), can pass exams from actual law school courses. The chatbot answered essay and multiple-choice questions in final exams in constitutional law, employee benefits, tax, and torts.
The chatbot passed all four finals, averaging a C+. Performance at that level throughout law school would keep a student on probation but would be sufficient to earn a J.D. See Choi, “Chatbot Goes to Law School,” J. of Legal Education, 2023.
What about actual practice? You might ask New York lawyer Steven Schwartz. He relied on ChatGPT in a personal injury action against Avianca airline. Mata v. Avianca, S.D.N.Y.
When Avianca moved to dismiss, Steve responded with an impressive 10-page brief citing over half a dozen decisions on point, including Martinez v. Delta, Zicherman v. Korean Air, and Varghese v. China Southern Airlines.
So far, so good. ChatGPT can pass law school exams well enough to earn a J.D., and it can conduct legal research and find helpful legal precedents in a real lawsuit.
Now for the bad news. It turns out that chatbots share some of the less attractive characteristics of real lawyers. Maybe we should have suspected as much, knowing that in law school ChatGPT was content to coast along on probation, with a gentleman’s C average.
It turns out that in law practice when chatbots can’t find helpful authorities, they may be tempted to invent them. How do we know that? Because when the opposing lawyers tried to look up the decisions cited in Steve’s brief, they couldn’t find them. And no wonder. The decisions don’t exist. ChatGPT made them up.
Judge Kevin Castel asked Steve why he had submitted a brief chock-full of nonexistent precedents. Steve’s response will not sound unfamiliar to experienced lawyers and judges. He shifted the blame elsewhere, namely to ChatGPT. He lamented, in effect, I relied on ChatGPT, and alas, ChatGPT let me down.