The Impact of Defense Experts on Juror Perceptions of Firearms Examination Testimony

The median and modal responses are 1-in-1,000, though there is quite a deal of variability in the responses. Notably, 7% of participants thought that “such an error is impossible,” and 17% of participants thought the error rate was between 1 in 1 million and 1 in 1 billion.

After completing this question, participants were presented with the case materials. The case materials, which were over 1,600 words in length, are de­scribed in detail in the next section. After reading the case materials, participants received judicial instructions regarding the standard of proof (i.e., “Beyond a Reasonable Doubt”) and were asked whether, given the state’s burden to prove guilt beyond a reasonable doubt, they would convict the defendant (“guilty” or “not guilty”). Participants also rated the likelihood the defendant was the indi­vidual who discharged the firearm on a scale from 0 to 100% and rated the strength of the case against the defendant on a 9-point scale from “not at all strong” to “extremely strong.”

Participants subsequently responded to three items probing the validity of the firearm analysis presented in the instant case. These items used a 7-point Likert scale, with higher values indicating greater validity. These items were combined to create a composite score referred to as “Case-specific Reliabil­ity.” Participants also answered two questions (also rated on a 7-point Likert scale) about firearm examination analysis generally. These items were com­bined to create a composite score referred to as “General Reliability.” Finally, participants were asked whether guns leave unique markings on discharged bul­lets or casings and which trial error they believed causes more harm to society: “failing to convict a guilty person,” “erroneously convicting an innocent per­son,” or “both are equally bad.” Participants were then thanked for their effort and compensated.

C. Materials

Each participant read a synopsis of a criminal case involving a defendant charged with discharging a firearm. The case synopsis was adapted from an ac­tual criminal case, United States v. Driscoll (2003). In brief, the defendant, referred to as “Mr. Cole,” was charged with willfully firing a firearm during the commission of a felony and attempted armed robbery. During the attempted robbery of a convenience store, a firearm was discharged into the floor, and the masked culprit fled. A 9mm bullet was recovered from the floor. During a rou­tine traffic stop two days later, the police confiscated a 9mm handgun from the defendant.

At trial, the prosecution called a firearms examiner. Participants were pre­sented with transcripts of his testimony. In short, the examiner testified about his training and experience and gave a detailed analysis of how firearm exami­nation is conducted, including a description of class and individual characteris­tic evaluation and as his analysis of the case at hand. The portion of testimony in which he described his conclusion is as follows:

Q. What did you conclude?

A. I observed sufficient agreement between the individual characteristics on the surfaces of the crime scene bullet and the surfaces of the test-fired exemplar bullets. Sufficient agreement means that the level of similarity I saw during my comparison exceeds the best agreement demonstrated between toolmarks known to have been produced by different tools and is consistent with agree­ment demonstrated by toolmarks known to have been produced by the same tool. Therefore, I concluded that the crime scene bullet was fired by the De­fendant’s gun.

Q. Is there a term for that conclusion?

A. Yes, it is called an identification.

Q. Can you explain what that term means?

A. It means that the likelihood that another gun besides the Defendant’s gun could have fired the crime scene bullet is so remote as to be considered a prac­tical impossibility.

Q. Does that mean you are absolutely certain that the Defendant’s gun fired the bullet recovered from the convenience store?

A. My conclusions are not made with 100% certainty. But repeated studies show that errors, especially false identifications are incredibly rare, they hap­pen in less than 1% of cases.

There were four experimental conditions in this study. Participants were randomly assigned to a single condition. In the control condition (hereafter de­scribed as the “Identification condition”), the defense did not call a witness nor cross-examine the findings of the prosecution expert. In two conditions meant to capture the impact of firearms examination’s limited reproducibility, the defense called its own examiner who had reached a different conclusion than the prosecution’s expert. In both conditions, the defense’s examiner described his training and experience (akin to that of the prosecution’s expert), explained that he used “the same basic methods” as the prosecution’s examiner to conduct his own comparisons, and provided context for how a disagreement between examiners could occur as follows:

Q. Are those methods foolproof?

A. Of course not. Every scientific method has some degree of uncertainty when used in practice. The amount of agreement that each examiner considers suffi­cient is also somewhat subjective, there is no numerical standard, it is what each individual examiner has built up in their mind’s eye. That means, although it is not common, firearms examiners can disagree about whether two bullets were or were not fired by the same gun.

In the first of these conditions, the defense’s examiner then testified that the defendant’s gun did not fire the 9mm bullet recovered at the crime scene (here­after described as the “Elimination condition”). In the second of these condi­tions, the defense’s examiner reached an inconclusive decision about whether the defendant’s gun fired the bullet recovered (hereafter described as the “In­conclusive condition”).

In a fourth and final condition, the defense called a statistics professor from a major research university to testify about the scientific foundation of firearm and toolmark examination (hereafter described as the “Methods Expert condi­tion”). The methods expert explained that he was a research scientist, not a prac­titioner, and he had reviewed the scientific studies of firearm examination. He explained that the National Academy of Sciences and other prestigious scientific organizations had reviewed the discipline of firearm examination and found it had not been studied adequately. The methods expert explained that he reviewed twenty studies of firearm examination and reached the same conclusion. He also noted that inconclusive responses in the studies made the results ambiguous. A portion of his testimony appears below, while the complete version appears in Appendix A:

Q. Let’s start with this; what is your general conclusion from these studies?

A. Generally speaking, I reached the same conclusion as the committees of research scientists. The studies of firearms examination report error rates of around 1–2% but these error rates are not trustworthy because of weaknesses in the design of these studies: they do not cover a wide enough range of guns, they use volunteers, and they do not make sufficient efforts to include chal­lenging comparisons. But there is an even bigger problem involving how the authors of these studies calculate error rates.

Q. What is that problem?

A. First of all, note that these studies are like an exam that I give to students. The firearm examiners who participate in the studies know they are being tested, and they know that if they fail the test, so to speak, there could be major problems for their field. So, the biggest problem with the studies, in my opin­ion, is that the examiners call most of the comparisons “inconclusive.” They are basically skipping the questions with this response.

We note the case materials did not include elements such as cross-exami­nation, opening or closing arguments, or additional evidence aside from the fire­arm expert’s testimony. The absence of these features limits the ability to extrapolate the findings without additional testing. However, we also note that many of the issues that would be brought up on cross-examination were pre­sented during the direct examination of the rebuttal witness. Given that this study is the first of its kind, we deemed the sacrifices to the realism of the study appropriate and acknowledged that the findings should be replicated under more realistic conditions before definitive pronouncements are made.

III. Study Results

A. Jury Verdicts

Overall, 35% of participants voted to convict the defendant at the conclu­sion of the trial. The percentage of participants who voted to convict in each experimental condition appears in Figure 2 below.

Figure 2. Guilty Verdicts as a Function of Experimental Condition

As is apparent from Figure 2, 66% of participants in the Identification condition voted to convict. Recall that participants in this condition were exposed only to the testimony of the state’s firearms examiner; the defense did not call a witness in this condition. The conviction rates decreased when the expert called by the defense reached an Inconclusive (28%) or Elimination (14%) conclusion. The conviction rate in the Methods Expert condition was 34%. The conviction rate differed statistically across the experimental conditions.

B. Perceptions of the Strength of the Case and Evidence

In addition to providing a binary verdict, participants also rated the likeli­hood that the defendant was the individual who discharged the firearm, the strength of the case against the defendant, the case-specific reliability of the firearm examination, and the general reliability of firearm examination. The im­pact of the experimental conditions on each of these measures is reported in Table 1 below.

Table 1. Impact of the Experimental Conditions

For the strength of the case measure, statistically significant differences ex­ist across the experimental conditions. The case was perceived as weaker for all conditions compared to the Identification condition. This is not surprising given that the prosecution’s firearms examination evidence went unchallenged in the Identification condition. Compared to the Methods Expert condition, there was no difference in the strength of case ratings for the Inconclusive condition but the Elimination condition was deemed significantly less strong. In other words, the case was deemed weaker when the defense expert concluded “elim­ination” compared to a methods expert testifying about discipline, but there was no difference when the defense expert concluded “inconclusive” compared to a methods expert.

The findings are consistent for the probability the defendant fired the gun measure: while all experimental conditions yielded lower probability ratings than the Identification condition, the Methods Expert and Inconclusive condi­tions did not differ from one another but the Elimination condition was rated significantly lower than both. The findings are also quite similar for the relia­bility ratings (case specific and general). For the case-specific reliability, iden­tification yielded the highest rating, while the other conditions reduced the case-specific reliability. When a defense expert reached a different conclusion (ei­ther inconclusive or elimination) or when a methods expert testified about the weakness of the discipline, the identification made by the state’s examiner was deemed less reliable. While the ratings in the Methods Expert condition were not statistically different than the Elimination condition, the ratings in the Methods Expert condition were lower than the Inconclusive condition. All conditions involving a defense expert also reduced the General Reliability rat­ings compared to the Identification condition. The Methods Expert condition yielded lower ratings than the Inconclusive condition, though it was not statis­tically different than the Elimination condition. Aside from statistical signifi­cance, the method expert reduced the perceived reliability both in the specific case and of the discipline generally. In contrast, the Elimination condition yielded the lowest strength of case ratings and the lowest probability that the defendant fired the gun ratings.

IV. Discussion

As the United States Supreme Court made clear in its Daubert decision, both Rule 702 (governing expert witness testimony) and Rule 403 (calling on judges to weigh the probative value and prejudice of proffered evidence) impose gatekeeping responsibilities on judges to assess the reliability of forensic evi­dence. These responsibilities require judges to carefully “evaluate the evi­dence in light of the jury’s unique role,” to “assess not just how valid the data is, but how well the jury can understand it after direct and cross-examination, and legal instructions.” Although judges have cautioned that the safeguards of the adversarial process may not suffice to alert jurors to the limitations of meth­ods, as outlined above, judges have not often excluded questionable forensic expert evidence, including firearms experts. This study, therefore, meets the ju­diciary essentially where it lays and explores whether, in the absence of stringent gatekeeping, defendants might nonetheless succeed in highlighting method lim­itations and inconsistency using experts of their own.

As a caution to defendants and the courts, our results show that laypeople appear to assume a high degree of validity in firearms comparison and are likely to convict based on such evidence if left unrebutted. Initially, when we explored mock jurors’ intuitive sense of the false positive rate for firearms examination, our participants provided responses largely out of step with the empirical record underlying the method. Even discounting criticisms of the research exploring the accuracy of firearms examination described earlier and using calculations favored by the authors of much-cited studies, false positive rates for the field still fall in the general range of 1–2%, or put another way, a 1 in 100 or even 1 in 50 chance of a misidentification. But a sizable majority of our participants (72%) believed the false positive rate to be orders of magnitude smaller than such estimates. In fact, the percentage who placed the chance of a misidentifi­cation at 1 in a million or less (24%) was nearly equal to that of those who placed the chance of misidentification at levels equal to or greater than rates reported in studies (28%).

Our findings about the persuasiveness of firearms examination evidence are not limited to intuitions. A majority of our participants (66%) were also willing to convict on the basis of firearms examination evidence alone when it went unrebutted. Recall that our case synopsis included no evidence whatsoever cor­roborating the firearms examiner’s linking of the convenience store bullet to the firearm possessed by the defendant: no confession, eyewitness identification, or otherwise. These figures are consistent with prior work in which 55–62% of mock jurors voted to convict under most variations of the language used to de­scribe a bullet “match,” despite the complete lack of evidence corroborating the firearms examiner’s conclusion and when exposed to cross-examination. Thus, attorney preparation and strategy, as well as judicial gatekeeping, should begin from the premises that jurors may enter trial with an inflated sense of the reliability of firearms examination methods and may accord such evidence un­due weight if it is left unrebutted and uncontested.

However, when defense expert testimony of any kind was present, the rate of guilty verdicts fell substantially (along with our other measures of juror per­ceptions of reliability and case strength). These findings powerfully support the defense strategy (in the event a judge admits firearms examination evidence) of calling an expert of its own, even if that witness testifies solely regarding methods. Further, they should impel judges and legislatures to ensure that de­fendants have access to adequate funding to retain expert witnesses in cases in­volving bullet and cartridge case comparisons. Because of insufficient public defender budgets or below market expert rates set by states, defendants too often lack such funding. This reality must change if the accused are to have the opportunity to meaningfully contest forensic evidence.

Finally, when considering the impact of calling a rebuttal expert, defense attorneys and courts should account for the reality that different varieties of ex­pert witness conclusions appear to affect jurors unevenly. Guilty verdicts dropped precipitously when the defense expert concluded Elimination (14% voted to convict), compared to when the defense expert concluded Inconclusive (28% voted to convict), or to when a methods expert was called (24% voted to convict). The same pattern of results followed for the strength of case ratings and the estimated probability that the defendant fired the gun. However, ratings of the credibility of the firearm evidence—in general and in the specific case—were lowest for the Methods Expert condition. These findings suggest that mul­tiple factors affect juror decision making; the specific type of rebuttal evidence differentially affects at least some of these factors. Future research might ex­plore additional factors, such as the impact of experts presenting images of the examined bullets or cartridge cases or a second examiner verifying evidence.

In the meantime, we caution that, however predictable or reasonable, the lesser weight jurors appear to accord inconclusive and methods testimony by defense experts poses significant risks to the innocent. That jurors would privi­lege elimination over methods testimony is not surprising given prior research into jurors’ particular attentiveness to case specific information suggesting that an expert could have made an error. And, the same is true of elimination ver­sus inconclusive testimony; it is more exculpatory for a defense witness to opine that two bullets display “significant disagreement” than to merely suggest “an absence, insufficiency, or lack of reproducibility” of relevant characteristics. But the elimination conclusions jurors most value are likely to be unavailable even to innocent defendants. Across multiple studies of firearms examination, accuracy rates have been substantially lower on different-source comparisons than they are on same-source comparisons, with divides between sensitivity and specificity of up to 73.4% (96.2% sensitivity versus 22.8% specificity). Even if the bullets were not fired by the same gun, there might be only a 23% chance an examiner will report an Elimination condition compared to a 96% chance an examiner will call an Identification condition if the bullets were fired by the same gun.

The scientific community has raised concerns over this massively asym­metric production of inculpatory and exculpatory evidence and the potentially inappropriate use of the inconclusive category that drives it. But our results suggest that such concerns are not merely academic or related to the appropriate calculation of errors rates. Instead, because jurors do not perceive Inconclusive conclusions as having similar exculpatory value as an Elimination, the asym­metry of firearms examination is likely to exact a very real cost from defendants seeking to rebut bullet and cartridge case comparison testimony.

At present, laboratory policy and examiner practices cut against elimination conclusions. As other scholars have pointed out, examiners call Inconclusive “despite almost certainly knowing they were looking at actual nonmatches.” And even more problematic, some laboratories have an explicit policy that pro­hibits examiners from calling Elimination when class characteristics agree. Unless and until this ecosystem of policies and practices changes, even defend­ants whose firearms display significant, exculpatory differences are likely to be denied the expert testimony with the best chance of acquitting them. Adherents of firearms examination have long argued that issues concerning accuracy and reliability go to weight rather than admissibility because “unlike some DNA analysis, ballistic evidence is never consumed and is, therefore, always available to be reexamined.” But if all defendants can regularly hope for are inconclu­sive or methods experts, then our results show that they are losing critical ex­culpatory value, and this logic cannot stand.

Conclusion

Our study shows defendants will benefit from calling their own experts in cases involving bullet and cartridge case comparisons. But it simultaneously highlights the great weight jurors are likely to place on firearms examination evidence, and reinforces concerns about the field’s asymmetric production of inculpatory and exculpatory evidence. Some of these concerns might well be remedied by evolving reporting practices or the development of “objective” comparison algorithms untethered from current examiner biases against elimi­nations. But until then, and at a time when the rules of evidence have been amended to emphasize the role of judges as gatekeepers, we should continue to explore a wider range of tools to improve the litigation of forensic expert testi­mony.

Appendix A. Testimony of the Methods Expert

Q. Good afternoon. Can you please introduce yourself to the jury?

A. Good afternoon. My name is Jeffery Smith.

Q. How are you employed?

A. I am a professor at Northwestern University.

Q. What is your title and how long have you been employed there?

A. I am a professor of statistics, and I have been on the faculty for 15 years.

Q. Can you describe your educational background?

A. I have a bachelor’s degree in psychology from New York University, and a PhD in statistics from Georgetown University

Q. Dr. Smith, are you a firearms examiner?

A. No.

Q. Have you ever looked under a microscope at bullets or cartridges?

A. No.

Q. Are you aware that a firearm examiner—Mr. Patrick West—looked at bullets in this case and reached an identification?

A. Yes, I am aware of his findings.

Q. Ok, so you are not a firearm examiner and you have never conducted a fire­arm examination—what does your expertise have to do with firearm examina­tion?

A. I am a research scientist. I conduct research as part of my duties at North­western University. A basic principle of science is that hypotheses need to be tested by collecting data. My expertise allows me to conduct tests of firearm examiners to see if the methodology they follow produces valid results.

Q. Can you give us an analogy to help us understand that?

A. Sure. In medicine, there are scientists who work to develop a vaccine or drug and then they test the new drug with a randomized clinical trial. If the test vali­dates the drug, then nurses or doctors will administer the drug to patients. The nurses who administer the drug don’t actually conduct the research to see if the drug works or not. That requires a different skill set. That is where research scientists like me come in.

Q. Have research scientists examined whether firearm examination is a valid science?

A. Yes, a panel of renowned scientific experts reviewed the studies of firearm examiners in 2009. That panel found that the practice of firearm examination had yet to be studied adequately.

Q. What do you mean by “studied adequately”?

A. The panel reviewed the studies that existed at the time and found that they were not properly designed. The studies were created by firearm examiners with no specialized training in conducting research.

Q. That was over 10 years ago; have any other reviews been conducted?

A. Yes, another review was conducted in 2016. Again, that committee of preeminent research scientists found the studies were not properly designed.

Q. And who were these committees?

A. One committee was a group appointed by the National Academies of Sci­ence, which is the most prestigious group of scientists in the United States.

Q. Ok. Dr. Smith, have you conducted your own review of the scientific studies on firearm examination?

A. Yes, I have.

Q. How many studies did you review?

A. I would estimate 20 studies.

Q. Let’s start with this; what is your general conclusion from these studies?

A. Generally speaking, I reached the same conclusion as the committees of re­search scientists. The studies of firearms examination report error rates of around 1–2% but these error rates are not trustworthy because of weaknesses in the design of these studies: they do not cover a wide enough range of guns, they use volunteers, and they do not make sufficient efforts to include challenging comparisons. But there is an even bigger problem involving how the authors of these studies calculate error rates.

Q. What is that problem?

A. First of all, note that these studies are like an exam that I give to students. The firearm examiners who participate in the studies know they are being tested, and they know that if they fail the test, so to speak, there could be major prob­lems for their field. So, the biggest problem with the studies, in my opinion, is that the examiners call most of the comparisons “inconclusive.” They are basi­cally skipping the questions with this response.

Q. Can you explain that in more detail?

A. In fieldwork, fired bullets can be damaged such that they lack any infor­mation that an examiner could use. For example, when a bullet hits a concrete wall and flattens out. Comparing such a bullet to one from the defendant’s gun would be “inconclusive.” We can’t tell one way or another whether the two bul­lets were fired by the defendant’s gun. But that is not what happens in a study. In a study, bullets are excluded from the study if they don’t have good markings. So there should be no inconclusives in the studies.

Q. But are there inconclusives in the studies?

A. Yes, quite a bit of them actually. In one study, over 50% of all the responses were called inconclusive.

Q. And how did the study count those responses?

A. As correct responses.

Q. Do you agree with that approach?

A. No. In a study, there is only one correct answer. The bullets are either a match or not a match. When they say “inconclusive” that is not a correct response. So the error rate could be as high as 50%. That’s basically a coin flip.

Q. Can you give an analogy to help us understand why this is a problem?

A. Imagine I give my students an exam to test whether they understand statistics; if they respond “I don’t know” to half of the exam questions and then get the other half correct, I don’t conclude that my students are ready to be statisticians at the FDA. Obviously, they selected the easy questions and dodged the difficult questions. The test results do not tell me whether the students can actually do statistics problems. It is unknown.

Q. And is that what you are saying with respect to the studies of firearm exam­iners—we just don’t know if they can do what they claim or not?

A. Yes, precisely. The tests do not provide evidence that examiners can do what they claim to be able to do. Maybe they can do what they claim, but I have not seen a study that supports this conclusion. More studies are needed.

Q. Just to summarize, you do not believe the practice of firearm examination is scientifically valid?

A. Correct. It has not been proven valid.

Q. And that is the same conclusion researcher my many other committees of research scientists like yourself?

A. Yes.

Q. Thank you for your testimony today.