LTRC Roundtable Discussion: ChatGPT

Our Panelists:

Alexander Paykin (AP), Allison Johs* (AJ), Chris Fortier (CF), Debra Bruce (DB), Joann L. Hathaway (JH), Lance Johnson (LJ), Reid Trautz (RT), Sharon Nelson (SN)

*Allison used ChatGPT to answer the roundtable questions in addition to her own answers. 

1. You can’t turn around without hearing about ChatGPT and Open.ai these days. Have you tried ChatGPT or other similar products, and if so, what was your experience with it/how did you evaluate the quality of what it produced?

AP: At first glance, the tool seems incredibly competent.  It is certainly eloquent and writes in a way that is both engaging and compelling.  Playing around with it, I was able to produce some very powerful monologues, funny short stories, compelling poems and even intelligent discussion and what appeared to be well thought out views on social issues.  Much like a good litigator, it was able to take either position on an issue and put together coherent arguments for that position.  However, as I played with it further, it became clear that its underlying understanding of the universe was limited and it was not able to grasp the nuance of many questions.  It was also completely unable to answer any legal questions that related to a particular statute or citation and was generally unable to grasp the substance of the underlying issue.

AJ: (ChatGPT) As an AI language model, I am ChatGPT, so I don't have personal experience using myself. However, I have been extensively trained and tested by OpenAI to ensure high-quality performance.

Regarding evaluating the quality of language models, there are several metrics that are commonly used, including perplexity, accuracy, fluency, and coherence. Perplexity is a measure of how well a language model predicts a sequence of words, and lower perplexity indicates better performance. Accuracy measures how well a language model can generate correct answers to specific questions or tasks. Fluency measures how well a language model can produce coherent and grammatically correct sentences, while coherence measures how well a language model can maintain a logical flow of ideas within a text.

In addition to these quantitative metrics, the quality of a language model can also be evaluated qualitatively by assessing its ability to produce human-like responses and engage in natural-sounding conversation. This is often done through subjective evaluations, such as asking humans to rate the quality of the language model's responses.

Overall, language models like ChatGPT are continuously evaluated and improved to ensure high-quality performance, and their quality can be assessed using a combination of quantitative and qualitative measures.

(Allison)  Although like everyone else, I had heard a lot about ChatGPT and other similar AI tools ad nauseum recently, and had even seen some examples and demonstrations, I hadn’t had much of a chance to check any of them out until these Roundtable questions were circulated. I decided to dump our questions into ChatGPT to see what the responses were (knowing that some of the questions would yield responses like “I am ChatGPT so I haven’t used myself”). You can judge for yourself what you think of the quality of those responses, but the tool was very easy to use.

After having ChatGPT generate answers to these questions, I gave it another assignment to write a short article about a specific topic. Then I asked it for some statistics to add to the article. The speed of the response was much slower for the statistics, but it did come up with statistics from a reliable source. But when I asked for the source material or links, none of the links were still working. Overall, what was generated was fairly generic, which wasn’t too much of a surprise.

CF: I have only tried ChatGPT and like with anything new, you have to go in with managed expectations. Knowing that it had a knowledge base that ended in 2021, I knew its answers about recent events would be limited. As always, I try to break the program with some curveballs. ChatGPT had some updates (like who is the monarch of the UK) but could not answer others (such as who is the pastor of my church).  I asked some medical and legal questions and there was some accuracy there.

DB: Have not tried ChatGPT yet

JH: I have not tried them, although am anxious to do so.  There is currently an influx of shared information about ChatGPT results and what users are experiencing. I am ready to test the waters.

LJ: I haven’t been able to get in. The site has been slammed.

RT: I’ve used it to help me write an upcoming Future Proofing column in Law Practice magazine. For general information I’ve found it very helpful to create a first draft that helps organize my thoughts.

SN: I was pleased and surprised by the quality of what it produced. It’s not ready for writing poetry yet, though the poems can be unintentionally hilarious.

It is important to keep in mind that it was fed data only until some point in 2021, so some questions on current topics will not be answered – or not answered appropriately. Also, it still makes errors, although the developers seem to be updating the AI.

I asked ChatGPT if lawyers were required to be competent with their technology and it replied (initially) that lawyers had no duty of technical competence under U.S. law. Several days later, it seemed to have learned about legal ethics and I got a different (and correct) answer citing legal ethics rules.

2. If you haven’t tried ChatGPT (or other similar technologies) yet, why not? Do you have any plans to try it, and if so, for what purpose?

AJ: (Chat GPT) As an AI language model, I am ChatGPT, so I don't have a personal experience of trying myself. However, I am constantly being updated and improved by OpenAI to ensure high-quality performance.

Regarding using language models for specific purposes, I'm designed to be a general-purpose language model, meaning I can generate human-like responses to a wide range of topics and prompts. My primary purpose is to provide users with a conversational interface to engage with and ask questions, provide information and support, and generate human-like text for various applications, such as chatbots, virtual assistants, and automated customer service systems.

As an AI language model, I'm always available for anyone to use, and I'm integrated into various applications and platforms, including messaging apps, social media platforms, and websites. If you have a specific purpose or application in mind, you can explore various tools and APIs that allow you to integrate my capabilities into your products and services.

Overall, my goal is to assist users in generating high-quality and natural-sounding text for various purposes, and I'm continually being improved to enhance my performance and expand my capabilities.

(Allison) As I mentioned above, I hadn’t had a chance to try out ChatGPT until I had to write responses for this post, mostly because I hadn’t had the time and I wasn’t sure how I would use it. I think for me, based on my two short experiences, I would be most likely to use ChatGPT to help me generate some ideas if I have writer’s block, or as a jumping-off point for something I was writing. I’m not sure it’s better than just doing some research online, but that remains to be seen. At the moment, I don’t see much of a use for me personally in my consulting practice right now, but that could change in the future as I learn more and the technology matures.

DB: I am interested in trying out ChatGPT, but by the time I got a little space in my schedule to do that, ChatGPT has been “at capacity.” you have any plans to try it, and if so, for what purpose? I thought a reasonable initial experiment would be to ask ChatGPT to write a blog post with a defined subject and structure. Another experiment would be to have it draft a fairly standardized legal document like a Power of Attorney or incorporation documents compliant with the laws of a particular state.

LJ: I did try Dall-E2 to make some drawings of a patent attorney in a steampunk style. I was pleasantly surprised.

3. Do you think that ChatGPT and its AI successors will help meet the access to justice challenge? Why or why not?

AP: Absolutely not.  These tools do not fact-check but merely use predictive algorithms to put words together in what appears to be a coherent manner.  However, they are unable to distinguish between facts and "alternative facts" if both are encountered on the internet.  Regularly, the tool provides responses that sound legitimate but are clearly wrong to anyone who knows the underlying substantive facts or law.  If anything, these tools will provide a false sense of security as they guide the user in the absolutely incorrect direction.  These tools may be useful in drafting eloquent argument sections and compelling oratory, but they should only be used by someone with a good independent knowledge of the facts and law, as otherwise, there is no safeguard against a completely incorrect work product.

AJ: (ChatGPT) ChatGPT and other AI language models have the potential to help meet the access to justice challenge by providing affordable and accessible legal information and support to people who might not have had access to it otherwise. With the help of AI-powered chatbots and virtual assistants, people can obtain legal information, guidance, and support without having to consult with an expensive lawyer or navigate the complex legal system on their own.

Language models like ChatGPT can also help bridge the communication gap between legal professionals and non-lawyers by providing natural language processing capabilities that allow people to ask questions and receive answers in plain language, making the legal system more accessible and understandable to the general public.

However, while AI language models can provide significant benefits, they are not a complete solution to the access to justice challenge. They cannot replace the expertise and judgment of a human lawyer, and there are many legal issues that require a human touch and cannot be fully addressed by AI alone.

Furthermore, the use of AI in the legal system raises ethical and privacy concerns that must be carefully addressed. For example, the use of AI-powered chatbots and virtual assistants may raise issues of bias, transparency, and accountability, and data privacy concerns may arise when personal information is collected and processed through these systems.

Overall, AI language models like ChatGPT have the potential to help address the access to justice challenge by providing affordable and accessible legal information and support to people, but they should be used in conjunction with human expertise and judgment, and ethical and privacy concerns must be carefully addressed.

(Allison) I think there is a place for ChatGPT and other similar AI tools to help people navigate through the court system, administrative agencies, and the like, especially with things that involve static information, such as helping fill out forms, guiding people on which agency to contact, or providing the steps one must go through to complete a simple task or transaction. I look forward to seeing what kinds of innovations in this area can help improve access to justice.

CF: Maybe. As with all AI enabled technology, the data behind the tech will determine how useful the tech becomes. If ChatGPT and its successors struggle with accuracy and filtering misinformation, judges and lawyers will not take any of work product seriously, even if used properly. 

DB: I think that will depend on how the operating parameters can be established. On the negative side, members of the public may try to create their own documents without hiring a lawyer and create problems for themselves. They do that already, however, by using forms downloaded from the internet. Lawyers straying beyond their area of expertise may also make mistakes. On the positive side, the ChatGPT power may permit lawyers to produce documents and perhaps even legal research more efficiently, making legal services for the underserved less expensive. And we have already seen efforts to have bots advise “clients” on run-of-the-mill legal representation issues in court.

JH: I do believe there is enormous potential, but there are so many scenarios that come into play this is certainly not an “always” or “never answer.  It is certainly a leap in helping meet access to justice, but it depends on the accuracy of the information sought and provided, the depth of the guidance requested, and jurisdictional and ethical issues that come into play.

LJ: I did try Dall-E2 to make some drawings of a patent attorney in a steampunk style. I was pleasantly surprised.

RT: There are many issues that impede access to justice, but this may go a long way to make lawyers more efficient in certain tasks, possibly resulting in lower costs to deliver legal services.

SN: Yes, they may prove useful. Andy Perlman, the Dean of Suffolk University Law School, was blown away (his words) by ChatGPT. He recognizes, as most people do, that it is not entirely ready for prime time. But he thinks it may be not far off.

He especially stressed that AI could be helpful in the routine legal issues that people of limited means often face. ChatGPT can already offer sensible legal advice about correcting a Social Security payment or what to do if you disagree with your child’s school district about the creation of an Individualized Education Program. Those are just two examples – there are many, many more where ChatGPT can offer sound advice.

4. Do you envision law firms becoming less dependent on newer attorneys and legal assistants as ChatGPT and its successors mature?

AP: Not really.  Much of the work handled by junior associates consists of factual and legal research to identify the facts and relevant law.  A reliance on a ChatGPT-like tool is not a substitute for this, but rather, is something that can only be used to supplement that work.  Firms that correctly embrace the use of these tools to improve on work products that has otherwise been drafted/researched manually is a possibility, but that will only improve the quality of the work product - not eliminate any of the human work involved.

AJ: (Chat GPT) While AI language models like ChatGPT can provide valuable support and assistance to legal professionals, they are unlikely to replace the need for newer attorneys and legal assistants in the legal industry entirely.

Newer attorneys and legal assistants bring unique skills, expertise, and perspectives to the legal profession that cannot be fully replicated by AI technology. They can perform tasks that require human judgment, such as evaluating evidence, drafting legal briefs, negotiating settlements, and providing personal attention to clients. Furthermore, legal assistants can provide support to attorneys by performing tasks such as scheduling, document management, and legal research.

AI language models can provide significant benefits to law firms by increasing efficiency, reducing costs, and improving the quality of legal work. However, they are best used as a tool to support and enhance the work of human attorneys and legal assistants, rather than replacing them entirely.

Overall, AI language models like ChatGPT are unlikely to completely replace the need for newer attorneys and legal assistants in the legal industry, but they can provide valuable support and assistance to legal professionals, leading to greater efficiency and improved quality of legal work.

(Allison) I think ChatGPT and other AI programs may help law firms utilize all of their professional and support staff better and more efficiently to provide better and more expansive services to clients without eliminating those professional and staff positions. Newer attorneys and legal assistants  have valuable human and life experiences that AI cannot replicate that help them to connect and collaborate with clients and colleagues, understand nuances of the law and of a client’s situation. What law firms may need to do is to adapt how they are providing services to clients and how they are maximizing all of the strengths of their lawyers and professional staff.

CF: No. Work evolves with technology and we still have a substantial part of the population who do not have meaningful access to a lawyer.  Research and drafting may take less time but proofreading, counseling, answering questions, strategizing, and delivering on a strategy is still a human dependent task. With all of these tasks, you have to deliver a client’s product tailored to that client. AI will get you most of the way to a deliverable product but humans will still need to get that product across the finish line.

DB: Perhaps, but it might actually make it easier to use less experienced and less skilled lawyers and legal assistants.

JH: I think that will happen, but not in the immediate future.  Rather, i foresee associates and legal assistants still being relied upon, but they, themselves, becoming more efficient in their jobs by using ChatGPT and its successors. I believe this will transpire as they and those attorneys they report to become confident with the accuracy of query output.

LJ: Nope. Nothing can synthesize legal concepts quite like humans for now.

RT: My best guess is this technology will allow for fewer paralegals and legal assistants in a law office over the next decade but won’t replace new lawyers quite yet.

SN: Sure. As it is possible for ChatGPT to do tasks which do not require a high level of legal skill, it appears to me that the AI itself may become a research and legal assistant. And much more. Here is a snippet of a conversation I had with ChatGPT.

Question: Is it possible that AI will one day replace some lawyers?

Answer:  “It is possible that AI could eventually replace some aspects of a lawyer’s job, such as document review, legal research and contract analysis.” This, of course, would put some lawyers/legal assistants out of a job. Perhaps to make us feel better, it offered its opinion that “it is unlikely that AI will completely replace the role of lawyers as the legal profession requires a high degree of critical thinking, problem-solving and decision-making skills that are currently difficult for AI to replicate.”  Note the word “currently.”

Dean Perlman says the AI is getting better and better – he tested ChatGPT, asking it to write briefs, explain legal concepts, create a contract for the sale of real estate and develop a list of deposition questions in a motor vehicle accident, among other tasks. The results were, as he put it, “not bad.”  He’d like to see how it has progressed in a few years and is clearly expecting major advancement.

5. How do you envision ChatGPT and other similar technologies disrupting the legal market?

AP: In no particular way.  After the hype dies down, these tools will simply be used for what they are.  A good comparison would be how Siri, Alexa and Google Assistant have totally failed to reduce our need for assistants and secretaries, but have made certain tasks (making lists, setting reminders, et cetera) slightly less human focused.

AJ: (Chat GPT) AI language models like ChatGPT and other similar technologies have the potential to disrupt the legal market in several ways.

Firstly, AI language models can provide cost-effective and accessible legal support to individuals and small businesses who might not have had access to legal services otherwise. By using AI-powered chatbots and virtual assistants, people can obtain legal information and support without the need to consult with an expensive lawyer or navigate the complex legal system on their own.

Secondly, AI language models can enhance the efficiency and quality of legal work, leading to improved outcomes for clients. For example, AI-powered legal research tools can quickly and accurately sift through vast amounts of legal data and provide insights that might be difficult or time-consuming for human lawyers to uncover.

Thirdly, AI language models can assist in automating routine and repetitive legal tasks, such as document review, contract analysis, and legal document drafting. This can increase efficiency and reduce costs for law firms, allowing them to focus on more complex and high-value legal work.

However, there are also potential risks associated with the use of AI in the legal industry, including issues related to bias, transparency, and accountability. For example, AI language models may generate biased or incorrect results if the training data used to develop the model is biased or incomplete. Moreover, there is a need to ensure that AI language models are transparent and accountable so that their decision-making processes can be understood and challenged when necessary.

Overall, AI language models like ChatGPT have the potential to disrupt the legal market by increasing efficiency, reducing costs, and enhancing the accessibility of legal services. However, there is a need to carefully address the ethical and legal issues associated with their use to ensure that they are used in a fair and responsible manner.

(Allison) For me, it’s a bit too early to tell. I am sure that there will be AI disruption in legal in the next 10 years in ways that we cannot even conceive of now. Lawyers, law firms, bar associations, and legal organizations should be thinking about ways that they can use AI to their advantage to improve the services that they provide. But generally, I agree with ChatGPT on this one – the main disruptors, at least in the short term, will be in improving efficiency, reducing costs, and providing access and information to those who may traditionally have had fewer resources and less access to the legal help they need.

CF: I see these products changing the way we research and filter information. These products can also help us set up our deliverables for clients.  The lawyers who emerge from this change will be the ones who understand  who know how to get the correct information to the client and (most importantly) explain how that information impacts the client legally (along with the other aspects).

DB: I can imagine ChatGPT features being licensed to companies like LegalZoom, practice management software providers and legal drafting software providers. Using the data those companies already have access to, plus data otherwise available on the internet, could facilitate drafting documents cheaply and efficiently. That could cause more commoditization of legal services. Similarly, frequently asked legal questions and common legal processes may be able to be handled by bots. Lawyers will need to highly develop their emotional intelligence skills to different themselves from the services provided by bots.

JH: Sometimes my crystal ball works and other times not, but at this point in time, i see it more educational for its users, helping to point them in the right direction to seek legal counsel.  In the more distant future, as its functionality matures, i do think it will be a legal market disrupter.

LJ: They might make intraoffice emails more intelligible, but truly useful technology for conflict resolution is a ways off yet, IMHO.

RT: This technology has the power to reduce the drafting time for lawyers and their staff to produce briefs, pleadings, correspondence, and other legal documents. We are already getting acceptable first drafts of these. The process will only get better, much like Google has gotten better with consumer use. If ChatGPT consumes even more legal specific information the quality of the output should only improve. The real issue right now is not to rely on the veracity of the content, but to use your knowledge as a lawyer to confirm the written results carefully.

SN: Those who use it efficiently and effectively are more likely to thrive. Those who ignore it are not likely to thrive. The AI can do some legal tasks with amazing accuracy and speed – to be competitive, law firms will need AI.

Having answered the questions at hand, my caveat to AI enthusiasts is highlighted by my question and ChatGPT’s answer below:

Q: Is it possible that AI will to a dystopian future?

A: It is possible that AI could lead to a dystopian future if it is not developed and used responsibly.

We’ve all seen “The Terminator” – and we know how that turned out. It worries me.