Court Planning During a Technology Explosion

The legal world got off to a slow start in realizing the importance of generative AI because of the early case from the U.S. District Court in New York in which a lawyer submitted a brief composed by ChatGPT3.5 that contained fabricated case citations. This clownish blunder was the initial frame through which we viewed the most powerful technology of our lifetime. (There is that old saying that “you never get a second chance to make a good first impression.”) A lot of early bar association and court policy guidance has focused primarily on the ethical responsibility for a lawyer’s submissions to a court—hardly a new concept. But the article cited calls on courts to recognize the full potential for generative AI to “democratize the law” and describes the efforts of the New Jersey court system to set policy in that broader framework.

In that larger context, the AI-lead process of technological change has major implications for our ability to plan responsibly. In 1980, it was not unusual to find “court futures” reports with titles like “Our Courts in 2000.” Then we became a bit less bold and rolled out five-year “strategic plans.” Ten years ago, Tom Clarke of the National Center for State Courts was advising state court leaders to limit their planning horizon to two years. Brittany Kaufman, CEO of the Institute for the Advancement of the American Legal System, now reports that she revises her organization’s strategic plan every six months.

“The Great One” Wayne Gretzky told us that we need to skate not to where the puck is but to where the puck will be. But we can’t do that in this IT environment. We know that the puck is moving out ahead of us on the ice and that it is going fast. But it is changing directions on its own, and we have no idea where it will be when we get to the far end of the hockey rink.

We are facing a pace of IT change in the environment within which we operate that poses the absurdity of launching IT initiatives that have a high likelihood of becoming obsolete before we can get them implemented. How can court leaders make plans in such times? Should we simply pull the covers over our heads and go to sleep like Rip Van Winkle until we can awaken to a less chaotic world? Would that we had that luxury. But as court leaders, we have no choice but to make the best possible decisions for the use of our IT resources and personnel in these turbulent times.

Here are a few suggestions. They focus on investing heavily in proven AI tech and developing a “learning environment” for testing generative or large language model (LLM) AI tech as it develops rapidly in its capabilities. (It is already incorporated into many of the products that we use daily. As I draft this article, Word is using generative AI to predict what words I will type next. It is disconcerting when Word inserts the word that I am still searching for in my mind.) But first, we have to face the reality that the IT procurement process is clearly unworkable in this environment. We can’t prepare a request for proposal (RFP), get responses, make an award, and agree on a contract with a vendor before the project to be developed and implemented has crossed the potential obsolescence threshold. This is a real problem because if there is one durable truth in our court world (both state and federal), it is the immutability of the procurement rules. It is my understanding that major corporations and institutions (such as universities) are making bets on one of the “big three” vendors—Amazon, Google, or Microsoft—adopting one of these basic technology platforms. All three have complete software “suites” to support most business processes, and they have robust security and personal information protection features. All are investing heavily in their own generative AI products. Following this strategy will reduce the burden of recurring procurement processes while leaving courts and court systems with the flexibility to add AI-based applications designed for our special needs. And vendors developing these AI-based applications can depend on a stable IT environment on which to build.

Perhaps we can develop a similar “innovation experimentation” contract with a single court-oriented IT vendor that calls for the design, deployment, and assessment of multiple “to be defined” generative AI–based initiatives over a five- to 10-year period—in short, creating an ongoing public/private partnership to jointly explore and implement in our court new technologies as they arise in the marketplace. Each project can begin with a definition of “what will success look like?” It will be up to the court to insist that each new effort define the new technology to be tested, ensuring that it goes beyond already tried and tested IT applications.

We also need to have a strategy for funding this ongoing body of work. One approach that has worked in the past is to invest in labor-saving technology that generates savings that can then be reinvested in a righteous improvement cycle. A starting point is investing in the already-proven technology referred to as “robotics,” which developed in recent years before the advent of generative AI. In the court world, the robots are not physical objects; they are simply computer programs that parse digital or scanned documents, make docketing decisions (such as the correct docket code based on the contents of the document rather than on the code suggested by the filer), and automatically docket the filing, placing it in the document management system and executing next steps (such as scheduling a first hearing and generating and sending out a summons). Such applications can be configured to measure the likelihood of uncertainty in the correctness of their conclusions (e.g., based on finding contradictory information in the text of the document) and routing decisions to a human when there is a significant possibility of error. These systems have been tried and proven in a number of courts, with substantial reductions in document processing time, and very significant savings of clerk time. These savings can then be used to generate capital to invest in more risky Generative AI innovations.

Finally, we need an approach to testing new applications of LLM (or generative) AI. This is an altogether new world for us. Expectations for the positive impact of LLM AIs are enormous. They are expected to contribute from $2.6 to $4.4 trillion to the world economy in the course of the next few years. But LLM AI is truly a black box. Once it is trained on multiple billions of documents, its algorithms enable it to create linkages and relationships within its data segments. It begins augmenting its knowledge and capabilities on its own. Faced with a prompt that it cannot answer with the information at hand, it will search the internet for more information. It will add its own products to its information base. We are unable to see or record its internal processes.

The spontaneous creativity within an LLM AI is astounding. The creators of ChatGPT never provided any training on writing computer code or even imagined code generation to be one of its capabilities. Suddenly it was responding to prompts asking for computer code with code that it generated—code that works (but not always; there are reports of it invoking functions that don’t exist in the software product for which the code was generated).

The result is that LLM AIs can accomplish stupendous feats. They can analyze large data sets, report the results and their statistical significance, and make recommendations for actions based on the results.

But if anything is found to be wrong with the analysis, software engineers are unable to trace the code that the LLM AI used to produce the results or to make changes to the LLM AI algorithms to correct the problem(s) identified. All they can do is revise the prompt that initiated the process and run the process again, hoping for more accurate results.

The purpose of our early court LLM AI implementations should have a single objective—our own education concerning the use of this remarkable technology: how to take advantage of its capabilities; how to minimize its “hallucinations”; how to limit its analyses to “trusted” information sources; how to measure its cost and benefits for our own processes. Big law firms are reportedly training their summer associates (who will be their new hires in another year) in this new technology, realizing that the future of the firm will be determined by its ability to use this new tech.

We can learn best from narrowly constrained experiments—for instance, to a small set of cases, to a single judge or small group of judges, or to a limited task, all for a short time period. Our objective should be to invest enough resources to accomplish our learning objective—not to revolutionize a court process.

If we fail, we will want to fail small and fast; negative consequences and lost investment will be minimized. And even if we fail, we will have accomplished our learning objective. We will know more about how LLM AI works in our environment and what it can and cannot do than we understood at the beginning of the experiment.

If we succeed, we can decide whether to expand the effort or, instead, to move to another experiment. Our decision about whether to expand a successful project will take into account the overall direction in which technology innovation has moved while we have been conducting our experiment. If our effort appears to still be in the mainstream of tech innovation, we can pursue it. If not, we can test some other use that capitalizes on the more recent tech trends.

We should pursue this work as a community, not as isolated courts. The Joint Technology Committee (sponsored by the Conference of State Court Administrators, the National Association for Court Management, and the National Center for State Courts) recently released a guide on the use of AI, with a pledge to update it every six months. We should use the JTC AI guide committee as a clearinghouse of information and as a source of strategic direction—circulating up-to-the-minute information on ongoing court LLM AI projects, circulating news of new tech products and directions (including those that fail as well as those that succeed), and identifying gaps in the experiments being conducted to test the utility of those products and directions in the court environment.

Where would it make sense for the courts to start with LLM AI experiments? Below are a few suggestions. They are meant to be illustrative, neither prescriptive nor exhaustive.

• Preparing forms, checklists, and explanatory materials that are understandable by the public at large: It has become evident that lawyers are literally unable to do this. Legal concepts and legal terms are so ingrained in our thought processes that we cannot explain our reasoning or our processes without using them. Telling an LLM AI to produce a form understandable at a third-grade reading level accomplishes what lawyers, regardless of the seriousness of their good intentions, cannot.
• Using chatbots: A chatbot takes a question, either submitted as text or translated from spoken language, and prepares an answer. A chatbot can operate externally or internally. An external chatbot will respond to inquiries from outside the court; an internal chatbot will respond to questions from inside the court. Examples of external chatbots are:
• Answering the questions of self-represented litigants about the status of a matter, the next step in their cases, or steps they should take to prepare for a trial or hearing.
• Answers to requests from any external source for information from the court’s records, e.g., the status of a case, the existence of criminal records for a background check, or a copy of an order (e.g., a domestic violence restraining order).
• An internal chatbot could answer a new staff person’s question about how to handle a particular case document or how to answer a self-represented litigant’s question.
• Speeding court dispositions: When analyzing court case processing data to identify factors associated with prompt and protracted case disposition times, the LLM AI could be asked for recommendations to speed case dispositions, or the data could be used by court personnel to generate such recommendations.
• Improving courts’ understanding of their caseloads: Incoming court filings could be analyzed to identify case characteristics that the court could use to create more granular case categorizations for differentiated case management or simply for understanding the court’s workload at a more detailed level.
• Predicting court outcomes: Court processing data could be analyzed to generate information for lawyers and litigants on the likely outcomes of pursuing different courses of action in a case, e.g., the likelihood of success in seeking summary judgment in different case types and fact patterns. (This analysis could use the more granular case filing data to show the influence of various facts on outcomes.) Vendors are currently collecting data from courts to generate such predictive information that they market to the practicing bar. LLM AI–generated data drawing on the whole of the information available to the court could be more valid (and it could be presented with “confidence” statistics—the likelihood that this prediction will be correct in a particular case). These data could be focused on the needs of a different audience, e.g., self-represented litigants seeking insights into the likely impact of raising certain issues or pursuing particular procedural pathways.

We are entering unprecedented times. Our experience with courts is that they follow familiar paths no matter what the circumstances (short of a pandemic). We cannot let that happen in the turbulent tech times in our immediate future. There is no familiar path to follow here. Nor is there an agreed-upon roadmap to show us where the new technology is headed. It is clear that generative AI has both extraordinary potential and known risks. We must, therefore, move into that future as one walking through the Sonoran Desert in the light of a waning moon. It will be a “bucket list” level experience if we avoid intimate interactions with saguaro and barrel cacti and the occasional irritated reptile.