Over the last 20 years, corporate legal departments have seen a marked increase in the number and type of providers available to handle their matters. In 2017, revenue for Alternative Legal Service Providers (“ALSPs”) was around $10.7 billion, and covered an entire expanse of legal staffing companies, Legal Process Outsourcers (“LPOs”), consulting companies, and legal technology providers.[1] Between new market entrants and the big firms, the pressure is on to be “cost efficient” and select the best provider for the work at hand.
Most of us are familiar with our annual trip to the dentist, and often you’ll see a sign at check-out asking how your service was that day. But a simple customer satisfaction survey provides only so much information about the underlying services. Perhaps you were angry because the dentist ran late due to traffic, or perhaps she discovered a cavity and you had to pay extra for that visit. For customer feedback to have greater impact, the type of information gathered must be consistent, have diagnostic value, and be easily comparable. By developing internal metrics and analyzing them using statistical tools, in addition to driving cost savings, corporate counsel can share this rich information with service providers to improve mutual understanding and drive better performance.
First is knowing what to measure and how to measure it. Many lawyers discuss selecting for the highest “quality” providers, but Noel Semple points out that any approach that leaves out cost-effectiveness risks creating an “all-Cadillac” service market, not to mention raising eyebrows when the legal spend budget is reviewed. Instead, the focus must be on “value.” This includes the traditional focuses of effectiveness and cost, but also client experience (such as timeliness and communication) and third-party effects (including pro-bono programs, whether the provider meets the Mansfield Rule[2], etc.). Additionally, you must ensure that the metrics are part of a consistent methodology. Many departments consider the time to complete a matter as a crucial data point, but it's important to ensure that you and outside counsel are on the same page as to when that clock starts. Inconsistent metrics can hamper the intention of measuring and driving performance.[3]
Next is understanding what types of data you will be working with. If you have data about the time it has taken several different service providers to complete matters they’ve been assigned, there are a few ways to view the data. One method is a cross-section, where you view the data for different providers at the same point in time (e.g., all matters in March). Another is a time-series, where the data for a single provider is examined over time (e.g., their matters from January-March). Panel data is another that is like a time series, but with multiple providers instead of just one. And finally, there is experimental data, where a “treatment” may be randomly assigned (such as a single attorney who is assigned to supervise some matters, but not others).[4]
Next is utilizing different types of statistical tests. One common statistical test is linear regression, where the relationship between an independent variable and a dependent variable is explored.[5] For instance, what’s the correlation between the amount of money spent on a matter and whether that matter is resolved successfully? In this instance, you can run a linear regression to see how related they may be. Other types of test can also be used, such as Sample T-tests.[6] There are multiple online courses that teach how to use these tests, such as at Udacity or edX.
The application of your new metrics and KPIs can be extremely beneficial.[7] For example, Toyota Motors was able to review its historical invoice data to approximate how much the different phases of a matter should cost. Utilizing metrics to better forecast its budget, the legal department saw an average of 17% cost savings on outside counsel spend yearly. And lest you think that this is only for large legal departments, Firtree’s small legal team was able to use a more data-driven evaluation of service providers in order to reduce outside counsel spend by 20%.
There are external resources to begin evaluating your service providers more methodically, including the ACC’s Value Challenge[8] and CLOC resources.[9] But creating metrics to evaluate your service providers is only the first step. By using these metrics to ask “why,” you can begin to understand root causes and how you can most effectively manage your outside counsel and legal spend.
[1] For more information on growth in alternative legal service providers, see Legal Executive Institute, Alternative Legal Service Providers 2019: Fast Growth, Expanding Use and Increasing Opportunity (2019).
[2] Diversity Lab, https://www.diversitylab.com/mansfield-rule-4-0/ (last visited August 12, 2020).
[3] For additional resources on assessing value and quality in legal services, see Daniel W. Linna Jr.’s forthcoming book chapter in Research Handbook on Big Data Law titled “Evaluating Legal Services: The Need for a Quality Movement and Standard Measures of Quality and Value” (Roland Vogel ed. Edward Elgar Publishing 2020), https://www.legaltechlever.com/2020/03/evaluating-legal-services-the-need-for-a-quality-movement-and-standard-measures-of-quality-and-value-chapter-in-research-handbook-on-big-data-law/; Noel Semple, Measuring Legal Service Value, 52 U.B.C. L. Rev. 943 (2019); William H. Simon, Where is the “Quality Movement” in Law Practice?, 2012 Wis. L. Rev., 387 (2012).
[4] Omar Robles, Statistical Methods 3, ABA Business Law Section Annual Meeting (2020).
[5] Supra Omar Robles at 7.
[6] Supra Omar Robles at 6.
[7] Jamal Stockton of Fidelity Investments will discuss his company’s use of metrics and statistical techniques in evaluating and managing outside counsel. Catherine Krow, CEO and founder of Digitory Legal, will discuss how she helps lawyers and legal departments use data to solve cost-management challenges.
[8] Association of Corporate Counsel, Value Challenge, https://www.acc.com/services-initiatives/value-challenge (last visited August 12, 2020).
[9] Corporate Legal Operations Consortium, https://cloc.org/ (last visited August 12, 2020).