Judge Scheindlin Addresses Keyword Searches

Background
In February 2010, the plaintiffs submitted a FOIA request to the defendants—five government agencies—asking for information regarding Secure Communities, a federal immigration-enforcement program launched in 2008. Subsequently, the parties agreed to several categories of documents that the defendants would produce, and in December 2010, Judge Scheindlin ordered the defendants to comply with that agreement. Pursuant to that order, the defendants searched the documents of hundreds of government employees and produced tens of thousands of responsive documents.

In July 2011, the defendants moved for summary judgment on the adequacy of their searches, claiming they had fulfilled their obligations under FOIA. In support of their summary-judgment motions, the five agencies submitted affidavits describing how the agencies’ custodians had searched for documents. For some searches, the affidavits listed the search terms that the custodians had used, while for other searches, the affidavits failed to identify the search terms.

The plaintiffs opposed all but one of the agencies’ summary-judgment motions, asking Judge Scheindlin to find that the agencies had failed to meet their FOIA obligations. The plaintiffs argued that: (1) the agencies had failed to conduct any searches of the files of certain custodians who were likely to have relevant documents; and (2) the agencies had not established that the searches they conducted were adequate because, for certain searches, the affidavits did not provide sufficient detail to determine the adequacy of the searches, and because the searches that were sufficiently described were inadequate.

On July 13, 2012, Judge Scheindlin granted two of the defendant agencies’ summary-judgment motions. She granted one agency’s motion for summary judgment because the plaintiffs did not oppose it, and granted a second agency’s summary-judgment motion because the agency consisted of a single, small office of about 20 attorneys and had conducted an extensive search that indicated that no members of the agency had even worked on the Secure Communities project. She denied in part the other three defendant agencies’ summary-judgment motions and ordered them to conduct further searches, finding that they had not searched all necessary custodians and had not identified the search terms that certain other custodians had used.

Legal Standard for Summary Judgment under FOIA
Judge Scheindlin held that the legal standard for granting summary judgment under FOIA requires that a defendant agency demonstrate that its search of its records was reasonably calculated to uncover all relevant records. She noted that an agency could meet this burden by presenting affidavits containing facts showing that the agency performed an adequate search. Judge Scheindlin held that to carry an agency’s burden on summary judgment, an affidavit must establish that the agency “searched all custodians who were reasonably likely to possess responsive documents,” and “must set forth the search terms and the type of search performed.”

Failure to Identify Search Terms and Follow Best Practices
Judge Scheindlin found that three of the defendant agencies had failed to search certain custodians who were reasonably likely to possess responsive documents. In addition, she analyzed the adequacy of the searches the three defendant agencies had conducted. With respect to searches for which the defendants’ affidavits failed to provide the search terms the custodians had used, the three defendant agencies argued that these searches could be deemed adequate without knowing the search terms because running such searches is something that office workers do every day.

Judge Scheindlin rejected that argument and held that an agency cannot meet its burden under FOIA to establish that its searches were adequate “if it does not record and report the search terms that it used, how it combined them, and whether it searched the full text of documents.” She also found that creating legally sufficient electronic searches is not part of most custodians’ daily job duties. She noted that searching for an answer to a question through keyword searches on a search engine—as office employees regularly do—is a much simpler and different task from searching for all documents responsive to a FOIA or discovery request. This is because keyword searches often find only some, not all, responsive documents.

Judge Scheindlin then described the emerging best practices for dealing with the pitfalls of simple keyword searches. She cited to recent articles and opinions on e-discovery that described best practices that include “careful thought, quality control, testing, and cooperation with opposing counsel in designing . . . keywords to be used to produce emails or other electronically stored information,” and “beyond the use of keyword search, . . . latent semantic indexing, statistical probability models, and machine learning tools to find responsive documents.”

In light of the defendants’ failure to identify their custodians’ search terms or present evidence showing that those custodians followed the above best practices, Judge Scheindlin held that she could not accept the three defendant agencies’ “unsupported assertions” that their custodians had created and conducted adequate searches, and therefore held that the three defendant agencies had failed to establish the adequacy of the searches for which they had not specified search terms. Id. at 110–11.

Adequacy of Defendants’ Search Terms
Judge Scheindlin next turned to those searches for which the three defendant agencies provided the terms the custodians had used. In addressing the search terms’ adequacy, she first observed that evaluating search terms is “highly context-specific” and that “even courts that have carefully considered the defendants’ search terms have generally not grappled with the research showing that, in many contexts, the use of keywords without testing and refinement (or more sophisticated techniques) will in fact not be reasonably calculated to uncover all responsive material.”

Judge Scheindlin observed that making a determination of the adequacy of the three defendant agencies’ search terms was difficult because, although the agencies had used search terms that reflected the subject matter the plaintiffs sought, they had not tested the efficacy of their search terms. Judge Scheindlin held that due to the lack of evidence regarding the search terms’ efficacy, it was impossible for her to determine whether most of the keyword searches the three defendant agencies had conducted were adequate. She warned the agencies that they must move beyond simple untested keyword searches and learn to use modern technologies to meet their FOIA obligations.

Judge Scheindlin ordered the agencies whose summary-judgment motions she denied to conduct further searches that included (1) complete, fully-documented searches by custodians who should have but had not yet conducted searches, and (2) new, fully documented searches for a sample set “of the custodians who conducted searches but failed to provide the Court with any details about those searches” and of “the custodians who listed the search terms that they used but provided no evidence about the efficacy of those terms.” Judge Scheindlin required the three defendant agencies to work with the plaintiffs to develop search terms and protocols to comply with her order.

The Application of National Day Laborer to Typical Civil Discovery
Civil litigators and judges will look to Judge Scheindlin’s National Day Laborer opinion for guidance because of her reputation as a leading authority on e-discovery, despite the opinion being limited to FOIA disputes. Lawyers and judges are likely to use Judge Scheindlin’s guidance on three issues regarding keyword searches.

First, to demonstrate the adequacy of a custodian’s search, the party conducting the search must be able to show the search terms that the custodian used and how it combined those terms. In fact, at least one case has already cited National Day Laborer for this point. See Brown v. West Corp., 287 F.R.D. 494, 499 (D. Neb. 2012) (noting that National Day Laborer “caution[s] . . . that the reliability of a self-search cannot be determined without examination of the parameters of the search.”)

Second, a mere keyword search is not adequate without some testing of the efficacy of the keyword search terms. As indicated by Judge Scheindlin’s order instructing certain custodians to conduct new searches, parties can test the efficacy of their search terms by running subsequent, modified searches and comparing the results of those modified searches with the results their original search terms obtained.

Third, e-discovery best practices include careful thought and cooperation with counsel in devising keyword search terms and going beyond keyword searches to use computer-assisted methods like latent semantic indexing, statistical probability models, and machine learning tools to find responsive documents.

Keywords: litigation, pretrial practice, discovery, Scheindlin, e-discovery, search terms

Andrew J. Herink is an associate with Jenner & Block LLP in Chicago, Illinois.

^{Copyright © 2013, American Bar Association. All rights reserved. This information or any portion thereof may not be copied or disseminated in any form or by any means or downloaded or stored in an electronic database or retrieval system without the express written consent of the American Bar Association. The views expressed in this article are those of the author(s) and do not necessarily reflect the positions or policies of the American Bar Association, the Section of Litigation, this committee, or the employer(s) of the author(s).}