Da Silva Moore was the first case to address the use of TAR. The court approved the parties’ joint proposed protocol for the use of TAR, in which the parties agreed that the defendants would allow the plaintiffs to review the nonprivileged documents used to train the TAR system, including documents not responsive to the action. Judge Peck acknowledged that “not all experienced ESI counsel believe it necessary to be as transparent as MSL was willing to be” but noted that this level of transparency allowed the court to be “more comfortable with computer-assisted review, reducing fears about the so-called ‘black box’ of the technology.” Da Silva Moore, 287 F.R.D. at 192. Judge Peck encouraged a similar level of transparency in future cases but stated that this “does not mean . . . that the exact ESI protocol approved here will be appropriate in all future cases.” Id. at 193.
In re Actos (Pioglitazone) Products Liability Litigation, the second case to address a TAR protocol in depth, was similar to Da Silva Moore in that the court entered an order concerning the production of ESI agreed to by both parties. No. 6:11-md-2299, 2012 WL 7861249 (W.D. La. July 27, 2012). This protocol may be the most transparent entered to date in published cases. The plaintiffs were allowed not only to view the documents used to train the TAR system but also to collaborate directly with the defendant in the classification of relevance of the training documents. The parties also agreed to meet and confer after the system was trained to determine which relevance score would provide a cutoff point for documents to either be discarded or be manually reviewed by the defendant.
Shortly after In re Actos, the court in In re Biomet M2a Magnum Hip Implant Products Liability Litigation stepped back from a broad emphasis on transparency and focused on the scope of discoverable material required by the Federal Rules of Civil Procedure. No. 3:12-MD-2391, 2013 WL 6405156 (N.D. Ind. Aug. 21, 2013). Initially, the defendants used keyword searches and de-duplication to cull 19.5 million documents; the defendants then applied predictive coding to identify the relevant documents to be produced from the remaining documents. The plaintiffs’ steering committee later requested that the defendants produce the documents used in the training of the TAR system. The court denied the plaintiffs’ request for access to the entire seed set used to train the TAR system, stating, “That request reaches well beyond the scope of any permissible discovery by seeking irrelevant or privileged documents used to tell the algorithm what not to find.” Id. at *1. The court noted that Federal Rule of Civil Procedure 26(b)(1) only makes nonprivileged, relevant information discoverable. The court also observed that although Sedona Conference principles and Seventh Circuit rules encourage parties to cooperate in discovery, they do not provide the court with the authority to compel discovery of information that is not discoverable.
While the In re Biomet court was not willing to require Biomet to reveal the irrelevant training documents, the court did note that Biomet’s opposition to disclosure was “troubling” and suggested that Biomet rethink its position. The court agreed with the plaintiffs that Biomet’s cooperation fell below the Sedona Conference standards and stated that “[a]n unexplained lack of cooperation in discovery can lead a court to question why the uncooperative party is hiding something, and such questions can affect the exercise of discretion.” Id. at *2.
Transparency and Timing
The Progressive Casualty Insurance Co. v. Delaney case focused on transparency and timing. No. 2:11-cv-00678-LRH-PAL, 2014 WL 3563467 (D. Nev. July 18, 2014). In Progressive, the parties jointly proposed an ESI protocol, which was approved by the court, for search terms to cull documents for review. The approved ESI protocol did not provide a TAR process. Subsequently, the plaintiff decided that a manual review would be too time intensive and costly and unilaterally implemented a TAR process. The plaintiff then requested that the court modify the stipulated ESI protocol to permit TAR.
The court denied the request, stating that the use of TAR “requires an unprecedented degree of transparency and cooperation” between counsel in the review and production of ESI. Id. at *10. The court cited Da Silva Moore and In re Actos, noting that these courts required a producing party to provide full disclosure regarding the technology used, the process, the methodology, and the documents used to train the model. The court found that the plaintiff was unwilling to engage in cooperation and transparency. The court stated that if the parties worked collaboratively and agreed at the onset to the use of TAR, the court “would not hesitate to approve a transparent, mutually agreed upon ESI protocol.” Id. at *9.
Similarly, in Bridgestone Americas, Inc. v. International Business Machines Corp., the plaintiff sought to use TAR in reviewing documents for responsiveness. No. 3:13-1196, 2014 WL 4923014 (M.D. Tenn. July 22, 2014). The defendant opposed this request as it was an unwarranted change in the original case-management order. The defendant further argued that it was unfair to allow TAR after an initial screening was already completed with search terms. Unlike in Progressive, however, the Bridgestone court allowed the plaintiff to use TAR midreview despite acknowledging that it was, to some extent, allowing the plaintiff to “switch horses in midstream.” Similar to previous opinions regarding TAR, the court advocated transparency: “openness and transparency . . . will be of critical importance,” and “[t]he Magistrate Judge expects full openness in this matter.” Id. at *1.
Seed Sets and Sampling
In the years after Bridgestone, courts distinguished between the different TAR processes and found that transparency and disclosure regarding seed sets is neither necessary nor required.
In Rio Tinto v. Vale S.A., the court entered an order addressing the use of TAR after both parties reached an agreement. 306 F.R.D. 125 (S.D.N.Y. 2015). In the order, Judge Peck noted that “where the parties do not agree to transparency, the decisions are split and the debate in the discovery literature is robust.” Id. at 128. Judge Peck found that if a party’s TAR methodology opts to use continuous active learning as opposed to simple passive learning or simple active learning, the content of a seed set “is much less significant.” Id.
Though Judge Peck expressed a preference for disclosure and transparency, he recognized that seed set transparency is not necessary to ensure that the TAR model has been properly trained. Instead, Judge Peck suggested that there are alternate means for evaluating the effectiveness of the TAR training and review processes—for example, statistical estimation of recall and quality-control review of nonresponsive documents. Judge Peck concluded by cautioning that it is “inappropriate to hold TAR to a higher standard than keywords or manual review.” Id. at 129.
In Winfield v. City of New York, the defendant used TAR to identify and produce responsive documents. 15-CV-05236 (LTS) (KHP), 2017 WL 5664852 (S.D.N.Y. Nov. 27, 2017). After the TAR process was complete, the defendant agreed to run additional search terms but wanted to apply the established TAR system to the additional documents resulting from these terms. The plaintiffs objected, citing concerns over the reliability of the defendant’s TAR process: the plaintiffs pointed to certain documents inadvertently produced by the defendant that, though designated as nonresponsive and produced with a slip sheet, were viewable in extracted text and found to be responsive to the plaintiffs’ document requests. To increase transparency and confidence in the defendant’s process, the plaintiffs sought the production of random samples of documents used in the TAR process and requested documentation of the defendant’s TAR process and ranking system.
In analyzing the issues, the court felt that
there is nothing so exceptional about ESI production that should cause courts to insert themselves as super-managers of the parties’ internal review processes, including training of TAR software, or to permit discovery about such process, in the absence of evidence of good cause such as a showing of gross negligence in the review and production process, the failure to produce relevant specific documents known to exist or that are likely to exist, or other malfeasance.
Id. at *9.
The court found no evidence of gross negligence or unreasonableness in the defendant’s TAR processes. The defendant’s in camera submissions showed that the defendant appropriately trained and utilized its TAR model, provided detailed training to its document-review team, used a large sample size for its training tool, included multiple rounds of iterative training, and conducted a validation process at the conclusion.
Though the court rejected the notion that the TAR process as a whole was defective, the court still granted the plaintiffs’ request for sample sets of nonprivileged documents because the “Plaintiffs have presented sufficient evidence to justify their request for sample sets.” Id. at *11. The court found that sample sets would increase transparency in light of the volume of documents collected, low responsiveness rate, and examples presented by the plaintiffs suggesting possible human error in categorization.
Interestingly, the court did not allow the plaintiffs access to detailed information about the TAR process, for example, what cutoff point was used. Similar to other cases, the court did encourage the defendant to disclose this information voluntarily in the interests of transparency and cooperation.
In contrast to Winfield, In Re Broiler Chicken Antitrust Litigation established an order advocating “reasonable” transparency between parties about the TAR process. No. 1:16-cv-08637, 2018 WL 1146371 (N.D. Ill. Jan. 3, 2018). Special Master Grossman issued an ESI order outlining the search and validation methods to be used by the parties.
With respect to search methods, the order stated that if a party uses TAR, the producing party must disclose (a) the name of the software and vendor; (b) how the TAR process will work, including how it will train the algorithm; (c) a description of the categories of documents that will be included and excluded from the TAR process; and (d) the quality-control measures that will be taken. Moreover, in developing the process for TAR and keyword searching, the parties should work together in good faith to resolve any differences.
Lastly, the order noted that the review process should incorporate quality-control procedures for both TAR and manual review. The validation sample should be reviewed and coded by a subject matter expert (SME) who is familiar with the issues of the case but is not provided with any information concerning the collection from which the document was derived or the prior coding of any document. After the coding is complete, the producing party must prepare a table of the validation sample, including the sample it was drawn from and the SME’s responsiveness and privilege designations. The parties should then meet to determine if they agree that the recall estimate, as well as the quantity and nature of the responsive documents identified through the sampling process, indicates that the review is substantially complete.
Transparency Not Required?
More recently, the Entrata, Inc. v. Yardi System, Inc., court rejected the notion that transparency is a requirement attendant to a party’s use of TAR. No. 2:15-cv-00102, 2018 WL 5470454 (D. Utah Oct. 29, 2018). In Entrata, the parties conferred over the terms of an ESI search protocol that addressed the use of TAR. Though no final protocol was submitted, the plaintiff began using TAR in its document productions. The defendants filed a motion to compel the plaintiff to disclose the complete methodology and results of the plaintiff’s TAR process. The magistrate court denied the defendants’ motion, stating that the defendants have not provided examples of deficiencies in the plaintiff’s document review.
The defendants objected to the order, arguing that the plaintiff must provide transparent disclosures as a requirement attendant to its use of TAR. The Entrata court determined that while the Federal Rules support cooperation, there is no requirement for a responding party to disclose its TAR processes, nor is there an obligation of transparency.
The court also noted that the case law cited by the defendants (Progressive, Bridgestone, and Da Silva Moore) does not impose an obligation of transparency before using TAR. In addition, the court observed that unlike the three cited cases, the parties here never reached any agreement regarding search methodology. Furthermore, unlike in Progressive and Bridgestone, the plaintiff in this case did not switch to TAR midstream.
In sum, it is clear that some degree of transparency will be required when implementing a TAR protocol to minimize the volume of documents to be reviewed. The required degree of transparency will first depend on whether an agreement is reached between the parties. If the parties agree to a TAR protocol, reported case law suggests that the court likely will approve it. Conversely, if the parties do not agree, it is not crystal clear what degree of transparency the court may require; but it is likely that, at a minimum, a party will be encouraged to be as transparent as possible in disclosing its TAR process.
Rose J. Hunter Jones is a partner and Priya Kolli Funderburk is an attorney at King & Spalding in Atlanta, Georgia.