chevron-down Created with Sketch Beta.

Law Technology Today


Four Best Practices for Evaluating Legal AI Solutions

Nathan Alan Cemenska


  • AI-powered solutions can be a boon to any legal department as they continue explore ways to optimize their processes and service delivery.
Four Best Practices for Evaluating Legal AI Solutions

Jump to:

AI continues to capture the imagination of the legal industry – as well it should. Leveraging AI-enabled technology solutions can help corporate legal teams and their organizations generate boundless efficiencies and new insights into their existing business practices.

But those robust levels of interest and enthusiasm also means that AI products – or products claiming to be AI – will continue to flood the market at an escalating pace, making it increasingly difficult to tell the difference between the two. Even vendors who have not historically operated in the legal AI space may decide to try their luck.

While it’s always nice to have options, it does make it hard to separate the truly effective legal AI solutions from the ones backed by a terrific marketing campaign and not much else. Fortunately, there are some easy steps that corporate legal departments can take to help vet the effectiveness and utility of an AI solution before it ever reaches your processes or employees.

Grounding Expectations

If the marketing campaign for a new AI-powered legal tech solution echoes the ads you’ve seen for the latest sci-fi film playing at the multiplex, it may be time to stop and reassess. While there are many AI products on the legal market that already offer tremendous value – perhaps by automating a process-centric task or analyzing data to provide new business insights - others may claim to possess functionality that is still far too theoretical to be of much use anyone, let alone an attorney.

Look for AI solutions where the underlying concept or value proposition is relatively easy to understand. If a solution promises to execute a task previously overseen by very expensive humans, for example, then there’s a very good chance that you’re in the clear. Proper due diligence is also critical – talk to a prospective AI vendor’s salespeople and ask them any and every question the crosses your mind. Be sure to obtain concrete answers to what an AI solution can – or can’t – accomplish in order to avoid the sting of thwarted expectations.

Is There a Legal Professional in the House?

One of the core benefits of AI is its ability to make human-like decisions. In the legal profession, at least, many of those humans just so happen to be attorneys. So it stands to good reason that many of the most advanced or effective pieces of AI on the legal technology market just so happen to be designed with input from legal domain experts who can teach a fledgling piece of AI how to think like an attorney or law professional.

To be sure, a skilled team of data scientists are also a very important piece of the piece here. However, legal teams interested in procuring a new AI solution to assist with spend management or other process-centric tasks can save themselves a lot of heartache by confirming that a vendor also includes an equally savvy group of legal domain experts in their technology development process.

The inclusion of those domain experts helps to ensure that each AI-powered solution is not only technically sound, but has also been designed with the everyday workflows and needs of practicing attorneys in mind. Domain experts can also serve as a conduit between a legal department client and the tech vendor, collecting feedback from the user and relaying it back to data scientists for continuous improvement to the underlying model and services.

Double-down on Data

Data is the underlying foundation of any AI-enabled solution – and understanding how to evaluate the strength and quality of that data can make all the difference when it comes to determining how a solution will perform in a corporate legal or law firm environment. Unfortunately, this may also prove to be the most challenging aspect of the vetting process for general counsel, legal operations professionals or others who don’t hail from a strictly technical background.

It's important to remember that “good” data isn’t just a question of quantity (although that does help). The quality of a vendor’s data set will correlate to the quality of its AI-solution, so buyers should get some basic questions out of the way upfront – especially when it comes to verifying the source of said data. Data sourced from e-billing or matter management systems, for example, is typically clean because law firms comply with a standard billing format called Legal Electronic Data Exchange Standard (LEDES).

Other questions worth asking include: What quality controls are in place around the data? And how has the vendor leveraged the data internally?

Understand the Change Management Process

Poor change management practices can doom the implementation of an AI-powered legal solution before it’s even begun. After all, the impetus behind onboarding any new solution is to enhance your existing processes, not to create a situation where you need to endure substantial disruptions or even devise an entirely a new process out of whole cloth.

You’ll want to ensure that a prospective AI vendor provides ongoing support both during the initial implementation process and beyond. Corporate legal departments, for example, should verify that a vendor will provide training for its in-house teams as well as any outside counsel who may be asked to engage with the solution.

One way to quickly gauge the level of support a vendor may be willing to offer is to ask if they maintain personnel who can serve as a designated point of contact over the life of the solution. This isn’t simply a crisis hotline or glass to be broken in case of emergency. Instead, hold out for regular meetings to discuss any questions or problems that may have arisen.

AI-powered solutions can be a boon to any legal department as they continue explore ways to optimize their processes and service delivery. Still, as dazzling as the promise of AI may be, it’s crucial not to get caught up in the hype. Every department has its own needs, and general counsel, legal operations professionals and others in corporate leadership must invest the time necessary to find the AI solution that best satisfies those requirements and can turn promise into best practice.