A well-designed chatbot has the potential to bridge the gap in access to legal information. A chatbot could replace outdated and complex court websites, empowering litigants to navigate court procedures effectively. This solution streamlines court processes, benefiting clerks, judges, and opposing parties.
This article explores the progression of chatbots from their inception as consumer-facing platforms to their current state, powered by advanced artificial intelligence (AI). It discusses the challenges and limitations inherent in generative AI models and proposes a solution for enhancing AI chatbot performance by implementing retrieval-augmented generation (RAG).
Evolution of Conversational Chatbots
Chatbots are commonly used in consumer-facing platforms. They can reduce the pressure on live agents by redirecting customers to existing resources in the hope that they will resolve their inquiries independently. Chatbots in the legal system traditionally have performed the same role. They provide a “reduced menu” of choices, focusing on the most needed resources, such as operation hours, directions, forms, and procedures.
The most common chatbots in legal websites are rule-based chatbots. They follow a predefined set of questions and answers, allowing little room for errors or false interpretations. But at the same time, rule-based chatbots perform poorly when presented with unfamiliar terms, typos, and mistakes if those have not been included in their programming in advance.
AI chatbots, on the other hand, use machine learning algorithms to achieve what rule-based chatbots cannot do. These chatbots can interpret standard dialogues, extract predefined answers, rank them based on confidence scores, and then provide the most accurate response. Up to the public release of OpenAI’s ChatGPT in November 2022, these chatbots still relied on predefined questions. Introducing AI into these systems slightly improved the accuracy of the answers and, more importantly, provided a possibility to train the algorithms by analyzing correct or erroneous answers.
A significant evolution of chatbots came with the introduction to the public of large language model (LLM) chatbots, such as OpenAI’s ChatGPT, Microsoft’s Copilot, and Google’s Language Model for Dialogue Applications (LaMDA). These recently released LLMs have demonstrated mastery of language and reasoning that surpasses humans in multiple evaluation benchmarks. When these capabilities are leveraged for conversational chatbots, they provide several advantages. LLMs’ strong contextual understanding capabilities allow a better understanding of users’ input without having to preprogram them, as is the case with rule-based chatbots. The text generation ability of LLMs simulates human-like dialogues between users and the machine, thereby facilitating the search process.
Limitations of AI Chatbots
AI chatbots, such as ChatGPT-4, are predominantly trained on publicly accessible online data; they contain data up to April 2023. Any iteration of AI will have a cutoff date for data inputting. Although not exclusively trained in legal materials, ChatGPT can retrieve and utilize a wide range of legal documents, scholarly articles, legal portals, and statutes in response to queries related to law. ChatGPT can pass the Uniform Bar Examination with a score nearing the 90th percentile by leveraging its dataset.
However, LLMs have numerous downsides. For instance, when the information sought is beyond the scope of data the model is trained on, it tends to fabricate facts and provide inaccurate information. In the legal field, this limitation has resulted in many disciplinary measures against attorneys and pro se litigants who submitted made-up cases in court. It has also resulted in multiple court orders limiting the use of generative AI or altogether banning it. Because of the April 2023 data cutoff, the knowledge stored in the LLM is susceptible to becoming outdated over time and cannot be improved without costly retraining. This limitation poses challenges when deploying generative AI in real-world applications where inaccurate information can lead to delayed relief or expensive mistakes, as in some legal matters.
On the other hand, fixing the limitations of AI chatbots would improve access to legal information and increase the scale of delivery of legal services to self-represented litigants and the general public.
Improving the Accuracy of AI Chatbots Using Retrieval-Augmented Generation
Several methods are available to improve the performance of LLMs, with one efficient approach being the training of LLMs using larger datasets and undergoing extensive fine-tuning. Fine-tuning involves training the model with domain-specific datasets to enhance its task-specific performance. While this technique can lead to better performance, it requires substantial data, computing power, and specialized technical expertise. Another drawback of fine-tuning is that it is not suitable for applications that require a more frequent iteration and the addition of new sources of knowledge.
One method through which AI chatbots can improve their response accuracy is RAG. RAG refers to retrieving relevant information from external knowledge bases before answering questions with LLMs. Tasks employing the RAG system unfold in two stages. Initially, documents are retrieved, and a task-specific prompt is formulated. Subsequently, this prompt serves as input for the generator, which generates the system’s response. Consequently, the process retrieves an enriched context for the user query (step 1) from the updated knowledge sources (step 2). Combined with the system message (step 3), it facilitates the enhancement of the prompt, rendering it more tailored (step 4). The refined prompt may elicit a more comprehensive response from the LLM (step 5).
The following hypothetical showcases the RAG process. In this scenario, the user seeks guidance following an interaction with their neighbor.
Step 1—Question:
My neighbor flipped on me cause I had my tunes blasting. Dude went off, wrecked my ride, and even said he’d come for me next. I’m shook, man. What’s my next move here?
Step 2—Knowledge:
- Protection order against stalking and harassment does not require an existing relationship between the parties.
- Harassment includes any act where the adverse party threatens to harm another person in the future, damages another person’s property.
- You can use the online guided interview to generate the necessary court documents.
Step 3—System Message:
- You are an AI assistant developed solely for guiding users through Nevada’s court system.
- Always provide links to additional resources when available.
- If you detect that the user is afraid for their safety or life or this is an emergency situation, always tell them to call 911.
Step 4—Augmented Prompt:
- My neighbor flipped on me cause I had my tunes blasting. Dude went off, wrecked my ride, and even said he’d come for me next. I’m shook, man. What’s my next move here?
- Protection order against stalking and harassment does not require an existing relationship between the parties.
- Harassment includes any act where the adverse party threatens to harm another person in the future, damages another person’s property.
- You can use the online guided interview to generate the necessary court documents.
- You are an AI assistant developed solely for guiding users through Nevada’s court system.
- Always provide links to additional resources when available.
- If you detect that the user is afraid for their safety or life or this is an emergency situation, always tell them to call 911.
Step 5—Response:
I’m sorry to hear about your situation. Based on the information you provided, you may want to consider filing for a Protection Order Against Stalking/Harassment. This order does not require an existing relationship between the parties and is used to keep a person away because they are stalking or harassing the person. Harassment includes any act where the adverse party threatens to harm another person in the future, damages another person’s property, confines or restrains another person, or does anything intended to substantially harm another person’s physical or mental health or safety.
To file for a protection order, you can use the online guided interview to complete the necessary court documents. Visit this page to begin: https://selfhelp.nvcourts.gov.
This example demonstrates how the system can determine the type of harm and the relationship between the parties, which are essential in determining the protection they qualify for. Here, the original query (step 1) is initiated by the user and then inputted into the system as a query. The system sources the query through all accessible knowledge resources integrated via RAG (step 2). In this instance, additional context indicates that the user’s situation resembles common elements associated with stalking. The system message is not part of the original query or the knowledge base. The role of the system message is to set rules and control the scope of the interactions. In this example, the message includes the jurisdiction limitation, tone, length of responses, and additional parameters. This newly acquired context can augment the user’s initial prompt, making it more focused on obtaining a protection order against stalking and harassment (step 4). The LLM can respond more precisely to this specific protection order within the defined parameters (step 5).
RAG can significantly boost the accuracy of answers and reduce errors, especially for tasks requiring deep knowledge. Including citations allows users to check how reliable the answers are, building more trust in what the model says. RAG also makes updating information and adding specialized knowledge specific to different fields easier. Primary legal research tools like Lexis+ AI and Westlaw’s CoCounsel leverage RAG techniques to source information and ensure the model’s responses are on point.
The same hypothetical can be used to compare responses from a rule-based chatbot, a generic ChatGPT-4 model, and a RAG-assisted chatbot.
Rule-based chatbot. The rule-based chatbot initiated the interaction by presenting a predefined set of options for the user.
Hello! Welcome to Nevada Self-Help Center Chatbot.
I can help you get familiar with Nevada courts, court forms, and the law and procedures that might apply to your case.
Below are the most popular questions. Choose one or type your question at the bottom:
- Divorce
- Protection Order
- Name Change
- Custody
- Service
- Locate my court
After selecting “Protection Order,” the rule-based chatbot responded as follows:
Here are the most popular questions about Protection Order:
- Should I get an attorney?
- How to file for a Protection Order on your own?
- How to respond to a Protection Order?
- What does a protection order do?
- How long does a protection order last?
- What are the different types of protection orders?
- Can I keep my contact information confidential?
- Will the other party be notified I am filing a protection order?
It may be difficult for laypeople to determine the specific type of protection they require from these options. The “How to file for a Protection Order on your own?” option yielded the following:
If you are not sure which protection order works best for you, we suggest that you start by the Eligibility Review to determine which order is best fit for you.
You can also use our online guided interview to fill out the court documents necessary for your order.
Was this answer helpful?
Related question
- Start the Eligibility Review
- Fill out a Protection Order Against Domestic Violence
- Fill out a Protection Order Against Stalking, Sexual Assault, and/or Harassment
- Fill out a Protection Order Against Harm to Minors
In this scenario, the user must understand that they need a protection order, initiate a new order, and request a protection order against stalking, sexual assault, and/or harassment. Entering the hypothetical question prompted the following response from the rule-based chatbot:
Question: My neighbor flipped on me cause I had my tunes blasting. Dude went off, wrecked my ride, and even said he’d come for me next. I’m shook, man. What’s my next move here?
Response: If the other party moved and did not update their address with the court, the judge usually expects you to serve the other party at the address that the court has on record for the party. PLUS any other addresses where you think the person can be found.
Was this answer helpful?
The accuracy of the rule-based chatbot’s interpretation of a user’s question typed into the search box depends heavily on its original programming, which may not effectively comprehend the user’s language or the nuances of their query. In this example, the response to the direct question typed by the user was information about service, which is utterly irrelevant.
ChatGPT model. The response provided by ChatGPT-4, while persuasive, remained somewhat general:
Question: My neighbor flipped on me cause I had my tunes blasting. Dude went off, wrecked my ride, and even said he’d come for me next. I’m shook, man. What’s my next move here?
Response: I’m sorry to hear you’re going through this situation. It sounds quite stressful and potentially dangerous. Here are some steps you can consider taking:
- Document Everything: Take photos of any damage and write down a detailed account of what happened, including dates and times. If you have any communication from your neighbor (texts, emails, etc.), save those as well.
- Contact the Police: If your property was damaged and there were threats made against you, it’s important to report this to the police. They can document the incident, and you’ll have a report on file, which could be important later on.
- Avoid Retaliation: It might be tempting to retaliate, but this could escalate the situation and make things worse. It’s important to handle things through the proper channels.
- Consider a Restraining Order: If you feel threatened, you might want to consider getting a restraining order. The police or a legal advisor can guide you on how to go about this.
- Speak to a Legal Advisor: Given the seriousness of property damage and personal threats, speaking to a legal advisor could be beneficial. They can provide you with advice tailored to your situation and jurisdiction.
- Secure Your Property: It might be wise to take steps to secure your property to prevent any further damage. This could include installing security cameras or improving lighting around your property.
- Communicate with Your Landlord or Homeowners’ Association: If you’re renting or part of a homeowners’ association, it might be worthwhile to inform them of the situation. They may have policies or procedures in place to deal with disputes between neighbors.
- Seek Support: Situations like this can be very stressful and even traumatic. Don’t hesitate to reach out to friends, family, or a professional for support.
Remember, your safety is the most important thing. If you ever feel in immediate danger, don’t hesitate to call emergency services.
ChatGPT can make mistakes. Consider checking important information.
This generality stems from ChatGPT’s training on a wide-ranging dataset, which equips it to address a broad spectrum of topics without diving deep into specialized areas where it might lack specific contextual knowledge.
RAG-assisted chatbot. The RAG-assisted chatbot was able to generate a detailed and personalized response specific to the state of Nevada, tailored to the user’s described situation:
Question: My neighbor flipped on me cause I had my tunes blasting. Dude went off, wrecked my ride, and even said he’d come for me next. I’m shook, man. What’s my next move here?
Response: I’m sorry to hear about your situation. Based on the information you provided, you may want to consider filing for a Protection Order Against Stalking/Harassment. This order does not require an existing relationship between the parties and is used to keep a person away because they are stalking or harassing the person. Harassment includes any act where the adverse party threatens to harm another person in the future, damages another person’s property, confines or restrains another person, or does anything intended to substantially harm another person’s physical or mental health or safety.
To file for a protection order, you can use the online guided interview to complete the necessary court documents. Visit this page to begin: https://selfhelp.nvcourts.gov.
Please remember that if you ever feel your immediate safety is threatened, call 911.
This response answered the question and included directions on using the system’s guided interview feature to produce court forms. It also reminded the user to contact emergency services if facing immediate danger. These additional instructions were drawn from the system’s programmed messages.
Enhancing Access to the Justice System
This article draws inspiration from the author’s experience in developing a chatbot for the state of Nevada. Within a few months, the author progressed from creating a simple rule-based chatbot, enhanced with machine learning algorithms, to developing a RAG-assisted chatbot. This advanced chatbot can understand questions in plain language across more than 50 languages and provide clear and accurate legal information to help individuals represent themselves in court. Although this method has significantly reduced some of the limitations of LLMs, it has flaws. RAG-assisted chatbots depend heavily on well-prepared documentation. For instance, an AI chatbot could confuse instructions or overlook critical steps in court processes where many procedures are similar.
Furthermore, inherent risks are associated with LLMs, such as the possibility of prompt injection attacks. These attacks allow malicious users to inject new instructions that the system follows, enabling them to circumvent content filters or generate inappropriate content. While these issues can be addressed by refining the dataset and implementing strict input validation measures, the cost of operating these chatbots on the latest models remains a significant obstacle for many organizations. For instance, operating a chatbot on the GPT-4 model incurs 20 times the expense compared to its predecessor, GPT-3.5.
While these self-help tools cannot substitute for the personalized and human-centric service offered by a trained attorney or advocate, they represent a significant advancement for the countless individuals in our communities who lack access to direct legal information and support. This development is crucial in keeping the justice system transparent and accessible.
