Understanding the Risks of Generative AI—What Does GAI Have to Say about It?
When asked about the number one risk to lawyers who use GAI for legal work, the GAI program I use (ChatGPT) told me:
The number one risk associated with lawyers using ChatGPT is providing inaccurate or misleading legal advice, often due to the AI's limitations in understanding legal context and nuances.
[Emphasis added.] We have all heard about lawyers who blindly took flawed or completely unfounded legal analysis or “pretend” caselaw generated by GAI and plugged it into a pleading and filed it with the court. One would think by now that all lawyers would understand that GAI is not a substitute for actual legal work and analysis—the work that lawyers are trained in and get paid to provide to their clients. Nonetheless, at least a few attorneys have recently been referred to their local disciplinary authority for citing a nonexistent case generated by ChatGPT in a legal brief, which the court found violated Federal Rule of Civil Procedure 11 and amounted to the submission of false statement to the court. It also likely violated the attorneys’ duty of candor to the tribunal under ABA Model Rule 3.3 or its equivalent.
The GAI program then went on to remind me:
AI tools can generate convincing-sounding responses that might seem factually or legally correct but can include factual inaccuracies, outdated information, or what’s known as “AI hallucinations” — where the AI confidently produces false or fabricated information.
This statement is true—and is part of the allure of using GAI. The “convincing-sounding” responses are so tempting to simply cut and paste. Consider, however, that soon—if it has not already happened—everyone in the legal system (i.e. judges, lawyers, paraprofessionals, clients, and professors) will be able to recognize the difference between GAI-generated text and actual legal analysis and argument written by a skilled, smart, inciteful, and intelligent lawyer. When that happens, anyone who chooses to continue to use lightly edited GAI text even in everyday correspondence will lose credibility with their colleagues and the courts in which they practice.
Competence in the AI Age
So, what does competence in this AI-driven era require? First and foremost, it involves independent verification of AI output. Simply copy-pasting what the tool produces and sending it off to a client or court is not enough. And trying to mask your AI-generated text by running it through a program that “humanizes” it is not a solution. Try it once and you will see why. There are no shortcuts when it comes to the actual practice of law. Lawyers must apply their legal knowledge and judgment to review the AI’s work and modify it accordingly.
The degree of verification needed will depend on the task. But never forget that clients usually face complex, emotional, and high-stakes situations, and they rely on lawyers for more than just legal knowledge—they need understanding, legal guidance, strategic thinking, and human empathy. And they deserve (and lawyers get paid for) far more than just the output of a machine. No amount of technology can substitute for the tailored advice that comes from years of training, real-world experience, and the trust built through personal relationships. As technology reshapes how lawyers work, understanding how to navigate these changes ethically and responsibly is vital for keeping clients happy, maintaining competence, and complying with our rules of professional conduct.