Criminal Justice Section
Criminal Justice Magazine
Volume 17 Issue 3
How Probative Is Comparative Bullet Lead Analysis?
By William A. Tobin and Wayne Duerfeldt
Comparative bullet lead analysis has been used in courtrooms for more than 35 years by expert witnesses who have testified that they can tell where a bullet or bullet fragment was manufactured and even from which box it originated—all based on an analysis of the composition of the lead. This article will argue that these conclusions are faulty and could call into question hundreds of convictions upon which such evidence has relied.
Comparative bullet lead analysis (CBLA) has been used to connect bullets or bullet fragments found at the scene of a crime with bullets in the possession of a defendant. In some court testimony—though not in official FBI laboratory reports—experts have gone so far as to claim to associate a crime scene bullet or fragment with a specific box of bullets. Witnesses have testified "to a reasonable scientific certainty" that the chemical composition of a crime scene bullet is "analytically indistinguishable from" or an "exact match to" bullets from the known source.
But the premises used to support the validity of CBLA appear to contravene well-established metallurgical principles and phenomena. A review of the practices reveals that CBLA was pioneered and promulgated by analytical chemists with no apparent cross-discipline input or publicly documented review by metallurgists or materials scientists. Consequently, both authors participated in a collaborative scientific study of comparative bullet lead analysis and the conclusions it presented at trial.
To understand CBLA, it’s necessary to know something about how the lead used for bullets is manufactured. Suppliers of lead for bullets are secondary refiners that use scrap lead that comes primarily from old automobile batteries. They first "charge" or smelt the scrap lead into lots or "molten sources" of up to 200,000 pounds (100 tons). They then refine the lot to ensure that the levels of the alloying elements (normally antimony, sometimes tin and arsenic) are within the bullet manufacturer’s specifications, and that trace elements, such as copper, bismuth, silver, cadmium, and sometimes tin and arsenic, are below specified levels (generally 0.1 percent, or 1000 parts per million).
Bullet manufacturers order the refined lead in the form of either ingots (also called "pigs") or billets that come from a much larger pot or lot of molten lead. A premium is paid for billets because they are already of a shape and size to fit in the first extrusion press in the bullet-making process. As a result, no remelting is needed, which is an important consideration for this article because significant changes in lead composition occur only during the melting process.
Premises underlying the practice
Several critical assumptions serve as the basis for CBLA as it is presented in judicial proceedings. The first premise, likely attributable to a subtle but significant difference between the disciplines of analytical chemistry and metallurgy, is that a small sample submitted for chemical analysis is compositionally representative of the source from which it was taken. Specifically, a bullet or bullet fragment is assumed to be compositionally representative of a molten source of lead. That the bullets or bullet fragments were submitted for forensic comparison also assumes that comparative compositional analyses are of significant forensic value.
The second premise is that the molten source from which the sample lead originated is compositionally uniform or homogeneous. CBLA experts have testified to this as fact in many trials. Among those experts is Dr. Vincent Guinn, an analytical chemist, considered to be the pioneer of the practice.
The third and most significant premise is the claim that each molten source of lead is unique in composition, establishing, in essence, a chemical and metallurgical "fingerprint."
CBLA expert witnesses frequently make much of the fact that they have visited the bullet manufacturing plants, yet the only manufacturing processes relevant to determining the composition of bullets are melting and solidification. Metallurgists and materials scientists understand that metal castings (molten metal solidified in a mold)—particularly very large castings—are not homogeneous or compositionally uniform. The composition of a very small metal sample derived from a large casting is not likely to be a representative sample of the overall or average casting composition because of metallurgical phenomena that occur during the casting process.
Because lead for bullets is not the principal product of secondary refiners, the quality control process is not geared toward the bullet lead customer; bullet lead is not "un-engineered" from its primary intended product—lead for automobile batteries—a product with stringent compositional specifications. Inasmuch as the composition of the lead supplied for bullets is not particularly mission-critical (except for "match ammo"), the bullet manufacturer is generally not concerned about localized variations in the lead’s composition as long as an average range is maintained per specification.
The average quantity of each chemical element present in the lead (intentionally or as unrefined constituents) can be determined, but determining an average is no assurance that each sample taken from the source will match the average composition. For example, if three samples are taken from a source that result in analysis values of 10, 20, and 30, the average is 20. Three other samples from the same lot may have values of 19.9, 20, and 20.1, and, again, the average is 20. Yet the range of values is much larger in the first sample than the second.
To get an idea of what this means statistically for the process of CBLA, consider that approximately 35 million .22 caliber bullets can be produced from one lot (100 tons) of lead. FBI laboratory protocol requires a sample weight of 60 milligrams of lead for bullet sample analysis through a process called inductively coupled plasma atomic emission spectroscopy (ICP). Using these figures, a single molten source (one lot) could theoretically contain 1.53 billion samples with different compositions. Because of metallurgical phenomena that occur during the casting process of the molten source—such as preferential oxidation, stratification, and segregation (both macrosegregation and microsegregation)—the smaller the sample that is removed from a 100-ton melt (source), the less likely it is to be representative of the average composition.
Homogeneity. CBLA is based, in part, on the premise that the source of the lead is homogeneous. That is, that the alloying constituents and trace elements that are present in the lead are distributed evenly throughout the batch. However, our studies of molten source composition data over two, two-year periods (1987–1988 and 1998–2000) revealed that compositions could be either fairly uniform or dramatically variant within each lot. The findings were based on samples extracted at the beginning, middle, and end of a 100-ton molten source pour at one refiner, and at the beginning and end of a pour at the second refiner. As an example of how much a single source may vary, data from one refiner revealed that a single lot could vary in antimony content by almost 12 percent, copper by 142 percent, tin by 1,871 percent, or arsenic by 31 percent, from the beginning to the end of the pour.
Our study demonstrates that a forensic examiner cannot know whether the molten source of the bullet or bullet fragment was homogeneous or dramatically variant in composition, nor does there exist any feasible way to track the compositions back to specific molten sources. In fact, compositional inhomogeneities in such large castings are not unique to the lead refining industry; they exist in all large casting foundry practices and for all metals.
Based on our studies and known metallurgical and empirical experience, it is our conclusion that the assumption of homogeneity as an underlying premise for the practice of CBLA is not scientifically valid.
Compositional uniqueness. Our study also evaluated whether or not molten sources (lots) are compositionally unique. Industry data from two different refiners revealed many instances in which the lead lots produced were compositionally indistinguishable—what we call "repeats"—indicating that the lead in bullets is not compositionally unique. This is attributable to the fact that battery lead—the source for bullet lead—is a product with relatively tight specifications because of electrical conductivity, corrosion, processing, and other considerations. Because all secondary lead refiners aim to produce compositionally uniform products, it is logical to expect a high probability of compositional "repeats" in the bullet lead industry. In fact, much of our data represents lead compositions that went directly from secondary refiner to bullets on retail shelves because bullet manufacturers opted for billets (rather than ingots or pigs) that required no remelting. Recall, it is only the remelting that changes the composition.
Our study concluded that compositional uniqueness as an underlying premise or universal statement for the practice of CBLA is not scientifically valid.
Scientific and forensic concerns
CBLA is comprised of three phases: (I) sample analysis by neutron activation analysis (NAA) or inductively coupled plasma atomic emission spectroscopy (ICP) (the latter exclusively used since approximately 1995), (II) data collation (associative "grouping" of the bullets into composition groups), and (III) interpretation of the forensic significance of the alleged associations.
In challenging CBLA, we assume the competent conduct of the Phase I examinations. With regard to the Phase II classifications of bullets into composition groups within which bullets are allegedly "analytically indistinguishable," although we find the practice subjective in practice and without written protocol, we have not generally challenged the Phase II activities except in case-specific instances where examiner bias was evident. It is Phase III, also without written protocol, that we find most objectionable.
Our first objection is based on qualifications. An analytical chemist’s expertise is in compositional characterization of the specimen. It is the metallurgist, materials scientist, or an experienced and properly trained industry specialist operating as a de facto metallurgist who is qualified by academic and empirical background to interpret such compositional data as they relate to the properties of metals, production, processing, and eventual product use. Much of a metallurgist’s or materials scientist’s academic training and practical experience focuses on the effects of composition, melting, and solidification on the mechanical processes involved in fabricating and processing the material (metal) in the interim and final stages of production. How and why a material changes with processing—mechanically, microstructurally, and compositionally—is the subject of many hours of undergraduate and graduate study. Recognition of this difference between disciplines is implied in Frye-Daubert "Plus" evaluations, where the qualifications of both the person conducting the procedure and the person interpreting the results are separately considered.
Reviews of judicial transcript reveal numerous ipse dixit assertions that the practice of CBLA is "generally accepted" within the scientific community and that the conclusions rendered are "to a reasonable degree of scientific certainty." Absent a definition of what comprises "the scientific community," we dispute this assertion, and also observe that the portion of the community "accepting" the underlying theory and hypothesis for CBLA is not generally qualified by academic background for materials data interpretation. Further, it is our position that only the judiciary "generally accepts" CBLA theory, not the scientific community.
In addition, there are other considerations. First, the FBI has the only forensic lab in the United States that offers comparative bullet lead analyses as a forensic service. During our studies, it came to our attention that other forensic laboratories had evaluated the practice of CBLA with the intention of offering the service, but rejected it because of the unreliability of the practice and its conclusions. The fact that the FBI is the only domestic forensic laboratory offering CBLA as a forensic service elicits serious concerns regarding the critical process of scientific peer review.
Another consideration in assessing adequacy of any peer review is the availability of equipment to perform or verify CBLA. For several decades, the equipment required for verifying the scientific reliability or testability of the practice, such as neutron activation analysis (NAA), eliminated participation by most of the forensic scientific community. Few, if any, interested parties had access to the nuclear reactor required for NAA. The technique now in use—inductively coupled plasma atomic (or optical) emission spectroscopy (ICP-AES or -OES)—is in its infancy and has been in use for CBLA only since about 1990.
Aside from analytical chemistry, the only technical disciplines with any practical interest in CBLA for issues of reliability, testability, and peer review are forensic metallurgy and statistics. The metallurgy unit at the FBI laboratory is the only such unit in any law enforcement agency in the United States. Yet comparative bullet lead analysis is not within the job responsibilities of the unit and it is not in the publication peer review chain.
The question arises as to what constitutes the "scientific community," and what is its capability for testability and peer review. The authors know of no significant peer reviews of CBLA by anyone other than practice advocates. Until our study, the only peer review has been of each other’s work. We do not believe this constitutes valid peer review in that each reviewer has a vested interest in the outcome of the review. Indeed, the pioneer of CBLA, Dr. Vincent Guinn, who is the only independent practitioner of CBLA, is in a biased position because of his public standing and relationship to the practice. As Judge Pollack in U.S. v. Plaza, 179 F. Supp. 2d 492 (E.D. Pa. 2000) stated:
[T]his is not "testing" of the "theory" or the "technique" . . . in the Daubert sense. With respect to "theory," the fact that a second examiner, following the same "technique" as a prior examiner, reaches the same (or, indeed, a different) result, would not seem to shed any light on the validity of the "theory" underlying that "technique."
Even peer review of published articles is suspect. Scientists outside the forensic community lack the financial incentive, time, and equipment to conduct a proper scientific review, including testability, of the theory and technique used in the published studies. In contrast, when new techniques or theories are espoused in the metallurgical field of corrosion, for example, there is widespread attention due to the financial interests of billion dollar industries such as oil and gas. No such financial or even professional incentive exists for detailed scientific review of CBLA practice, outside the FBI Laboratory.
However, even if CBLA articles were adequately peer reviewed, the court testimony that bullets can be associated with a unique lot and/or box of bullets has never been included in a published CBLA article. As FBI expert witness testimony and industry specialists concede, bullet manufacturers generally do not analyze the composition of the lead provided by the secondary lead refiners or that of their own lead. Thus, the only significant sources of such data are the secondary refiners; data we acquired for our study. Although some of the compositions in the data provided by the secondary refiners changed as a result of bullet manufacturers remelting ingots, roughly half of the data are of bullet compositions from billets (in which no remelting is required) that can be found on the shelves of retail outlets such as Wal-Mart and K-Mart.
The data supplied for our studies were produced by the secondary refiners using analytical chemistry techniques comparable to, or exceeding, those used by the FBI Laboratory, and were acquired in the normal course of business at the secondary refiners in the process of bullet lead production. They are data relied upon by their analytical chemists, metallurgists, and customers for product acceptance and technical troubleshooting. Billions of dollars of commerce and, indeed, the very existence of the secondary refiners, are dependent on the accuracy and precision of the data produced.
The fact that CBLA theory lacks data supporting the existence of homogeneity and compositional uniqueness in bullet lead, and that our data reveal numerous lots that are either compositionally inhomogeneous or analytically indistinguishable, impacts the most important Daubert consideration: testability.
As stated by Professor Edward J. Imwinkelried in Evidence Law Visits Jurassic Park: The Far-Reaching Implication of the Daubert Court’s Recognition of the Uncertainty of the Scientific Enterprise (81 Iowa L. Rev. 55, 62 (1995)), explaining the importance of falsifiability in scientific testing:
Attempts to disprove the hypothesis are more significant [than verification] in two respects. First, although a single outcome consistent with a hypothesis furnishes little proof of the truth of the hypothesis, a hypothesis phrased as a universal statement is disproved by even one singular inconsistent outcome. Second, even when there are an impressive number of consistent outcomes and no inconsistent outcomes, the hypothesis is not definitively confirmed because it is always possible that an empirical test will some day demonstrate the theory to be incorrect. The theoretical possibility of disproof remains.
Accordingly, even if CBLA was premised on good science, a single instance of finding that a small sample did not represent the composition of the molten source, or that regions within one molten source were analytically distinguishable, or that two or more molten sources that are analytically indistinguishable would disprove each respective premise. In addition, only one of the three underlying premises of CBLA theory need be falsified to disprove reliability or scientific validity of the entire practice; our studies have disproved all three.
Heightening suspicions of the practice are four occasions when FBI laboratory researchers and examiners inadvertently encountered implications of compositional repeats, but chose to ignore the significance. In one instance, a laboratory examiner found a sample from the FBI Laboratory’s reference collection to be analytically indistinguishable from a bullet in a case under examination. In two other instances, FBI researchers found sets of bullets manufactured months apart that were analytically indistinguishable—in one instance, the bullets were manufactured seven months apart, in another instance they were made 15 months apart—in a very small study of only four boxes from four manufacturers. ( See Ernest R. Peele and Donald G. Havekost et al., Comparison of Bullets Using the Elemental Composition of the Lead Component, Proceedings of the International Symposium on the Forensic Aspects of Trace Evidence, 61–2 (U.S. Dep’t of Justice, 1991).) In the fourth instance, bullets from a foreign country were found to be analytically indistinguishable from bullets in an unrelated case. CBLA proponents opted to ignore what should have been red flags concerning the reliability and testability of CBLA. Instead, they attributed all four occurrences to storage considerations. At the very least, the discoveries of unrelated, analytically indistinguishable compositions should be considered a direct reflection of a possible rate of error until the storage considerations can be scientifically proven to be a valid explanation. Of course, the first step in an objective scientific study should be the recognition that these chance encounters could be unrelated. There exists no proof that such coincidences are attributable to storage issues. Based on our studies, we conclude it’s equally or more probable that the four examples are the result of production repeats. Whatever the cause, the fact that they did occurr is of probative significance.
To date, decisions as to whether or not to admit the testimony of CBLA witnesses have focused primarily on the first two phases of the practice (data acquisition and data grouping) rather than on the third (inference) phase. Compositional comparisons of materials for forensic purposes have been admitted for decades. When challenged as to general acceptance of CBLA, which should include consideration of all three phases, practitioners have resorted to descriptions of the generally accepted Phase I portion of the practice, imputing acceptance of Phase III (interpretation) and of the foundational premises previously discussed. It’s also worth noting that to date there are no written laboratory protocols for either the Phase II (data grouping and associations) or Phase III (data interpretation) portions of CBLA, facts also conceded in FBI expert witness testimony.
CBLA witnesses frequently testify that, "in my experience, I have never [seen][found]" instances of compositional repeats in unrelated cases. Each witness has examined up to tens of thousands of bullets and fragments. There can be only two bases for such an empirical assertion of compositional uniqueness: the existence of a bullet database collected through years of police agency submissions, or superhuman memory. Considering that each bullet examination encompasses compositional characterization of trace and alloying elements of from three (in early years) to the present six or seven analytes, and each analysis is generally performed in triplicate (three samples), analyses of "thousands of bullets" literally requires the witness to recall hundreds of thousands to millions of compositions. The only credible manner of representing that the bullet in question is compositionally unique is for the witness to have queried a database of all bullets analyzed by the FBI Laboratory.
Such a database does exist, with more than 13,000 entries, and FBI examiners have confirmed in court testimonies that the use of the data is a foundation for representations of composition uniqueness. Studies of the data by metallurgists and statisticians, however, have revealed it to be seriously flawed—likely the reason that CBLA witnesses in recent years have disavowed the use of the database for assessing compositional prevalence.
One difficulty with accepting the database as a representative sample of the overall bullet lead population is that of selection bias. The majority of samples in the database are statistically considered self-selected, submitted primarily from state and local law enforcement agencies in connection with homicides or shooting incidents (since most homicides and shootings are investigated by local and state agencies rather than federal agencies). Because the samples submitted do not represent a scientifically designed selection process, the collection is overrepresentative of high crime areas and underrepresentative of low crime areas nationwide, resulting in subtle but important implications as to forensic significance of a compositional match within a region. The results can be quite prejudicial at trials, particularly in low crime areas where forensic witnesses often claim no compositional matches to bullets in the FBI’s "vast database."
In addition, the 13,000-plus entries in the FBI’s database do not represent 13,000-plus sources of molten lead, or even 13,000-plus different bullets. There are numerous entries of same-case (and even same-bullet) data. If the database entries were to consist of only two entries from each bullet or fragment analyzed, the number of unrelated bullet samples in the database would be reduced to approximately 6,500. Eliminating database entries of multiple bullets from the same case (a subset of data where there’s a great likelihood of questioned and known bullets having derived from the same molten source) further erodes the number of unrelated bullets representing separate molten sources of lead down to possibly a few thousand. In the database, there are also a large number of (a) specimens where only three constituent elements were analyzed (inadequate even according to the FBI’s own studies), (b) lead standards, and (c) a variety of bullet calibers. After eliminating all same-case multiple entries, same-bullet multiple entries, lead standards, samples with only three analytes evaluated, and calibers that are unlike the type under examination, the database is reduced to a near worthless number of molten sources, possibly as few as several hundred, to represent the overall bullet population or for any meaningful forensic significance.
In an effort to associate DNA-type statistical and probability assessments to their bullet database, the FBI Laboratory funded a study by the Department of Statistics at Iowa State University of the observational data collected. One of the principal investigators was Professor Alicia L. Carriquiry, who concluded:
[B]allistic [sic] evidence at this time may have little probative value. . . . Suppose, in fact, that a very close match has been established. But does this then imply that the fragments came from the same box, or even the same molten lead batch as the bullets in the suspect’s possession? No, it does not. . . . While establishing [that] a match is possible by looking at the fragments and the unused bullets, the significance of the match cannot be assessed from that information. . . . The fact that two bullets have similar chemical composition may not necessarily mean that both have the same origin. . . . The leap from a match to equal origin is enormous and not justified given the available information about bullet lead evidence. . . . Our conclusion: it simply does not follow that a match implies equal origin.
(E-mail from Alicia L. Carriquiry, Associate Provost and Professor of Statistics, Iowa State University (April 25, 2001) (revised June 2002) (on file with author).)
Our research, as well as that of Dr. Carriquiry and associates, is considered prelitigation in the Daubert sense with respect to specific criminal cases. It is interesting to note that neither the authors nor Dr. Carriquiry were aware of the others’ research projects. Both teams, however, arrived independently at the same conclusion based on different academic disciplines and experiential histories.
Author William Tobin conducted an in-depth evaluation of the FBI Laboratory database, comparing numerous actual and hypothetical compositions with the FBI’s data in an effort to see how many hits (analytically indistinguishable compositions from unrelated cases) could be triggered, and under more stringent search conditions than those claimed by CBLA witnesses. Contrary to testimony of CBLA witnesses, Tobin repeatedly found hits of compositions from unrelated cases, suggesting (1) the presence of multiple sources of lead in the bullet lead population with analytically indistinguishable compositions, (2) the existence of bullets from unrelated cases originating from the same molten source, or (3) both.
The change in the technique used to analyze bullet lead, from NAA to ICP, has subtle but significant implications. During the years that NAA was used—from the 1960s to approximately 1990—only three, sometimes four, analytes were used to characterize the lead. In the report Forensic Neutron Activation Analysis of Forensic Bullet Lead Specimens (H.R. Lukens et al., Gulf General Atomic Proj. Rep., San Diego, Cal., June 30, 1970), researchers determined that three or four analytes were an inadequate number from which to characterize a molten source of lead. "[I]t is not safe to say that the bullets come from the same lot" based on only three or four analytes, they stated. Yet such conclusions were admitted in the courtroom for several decades. The researchers recommended a minimum of six analytes be used for compositional characterization. Despite the fact that the report was published in 1970, the courts were never effectively apprised of those findings because of a subsequent dearth of challenges to CBLA. Only recently has the number of analytes used for characterization of bullet lead compositions been changed by CBLA practitioners to six or seven. We conclude that even the six or seven analytes selected for compositional discrimination are inadequate for uniquely characterizing a source of bullet lead.
Obviously, the fewer variables used to characterize a material, the greater the risk of analytical indistinguishability between or among dissimilar items. For example, if "metal" was a discriminant, few if any automobile bodies would be eliminated as analytically distinguishable. Add the variable of the generic color "blue" as a discriminant, and the field of analytically indistinguishable vehicles begins to narrow—and so on.
Conversely, using a large number of analytes for discrimination can be somewhat deceiving if not properly selected. For example, if all 103-plus elements in the periodic table were selected as discriminants, but 99 of them are neither present nor reasonably expected to be present, then those 99 should not be considered discriminants. This does not mean that the presence of the 99 elements shouldn’t be analyzed or sought, only that their absence should not be considered as discriminating.
The present CBLA practice uses the following elements as discriminants: antimony, arsenic, copper, bismuth, silver, tin, and sometimes cadmium (though cadmium is rarely present). Tin was not present in a significant number of lead analyses evaluated by the authors. Of the remaining elements, bismuth is almost always in a very narrow range of compositions for the caliber analyzed, typically between 90 to 110 parts per million (ppm) for .22 caliber bullets. Silver is almost always in a very narrow range between 20 to 30 ppm. Thus, the latter two elements are generally poor discriminants—a fact to which even CBLA experts have testified. This leaves the number of elements that are considered good to excellent discriminants at three, the very number that early CBLA research has indicated to be inadequate to uniquely characterize a source of lead.
Our final concern relates to what we call the "density and distribution demographics" of bullets. We know that once the manufacturers load cartridges with primer and powder, the bullets are distributed relatively soon after packaging. In our study, we found that K-Mart and Wal-Mart are the two largest purchasers of .22 caliber bullets, the most prevalent caliber in use. It is reasonable to assume that trucks deliver large numbers of boxes of ammunition to these customers. It is very likely that tens, perhaps hundreds of thousands of bullets from the same molten source are delivered to the same geographic region, resulting in geographic concentrations of specific compositions. As a result, we would expect to find an increase in the likelihood of bullets retrieved from crime scenes coming from the same molten source, particularly in many small- to midsized towns where there is a limited number of retail ammunition outlets.
This is not our concern alone. Dr. Vincent P. Guinn, considered the pioneer of CBLA, had serious misgivings about the forensic significance of bullet composition matches based on distribution as far back as 1988, when he testified in the postconviction relief hearing of Huffington v. Nuth, 140 F.3d 572 (4th Cir. 1988):
[Y]ou only look into these things in cases, in a criminal case. That’s the only time you get around to wondering about the composition of these [bullets]. If you just went around and visited farmhouses in the area and saw them [bullets] and gathered up a lot of them, you’d probably find a lot of those were from some of this same material. . . .
Observations and conclusions
We believe that the current practice of CBLA is scientifically flawed, and that no scientifically or statistically adequate data exist to support its foundation. Conclusions drawn from CBLA are demonstrably unreliable and have an unacceptable probable (and unproven) rate of error. In addition, individuals interpreting the compositional data derived from CBLA techniques are not qualified to render such conclusions. We also believe that conclusions rendered in courtroom testimony represent unjustifiable extrapolations from generally accepted practices. Based on current practices, conclusions as to what constitutes a "same molten source" that are drawn from ostensibly analytically indistinguishable compositional associations are also demonstrably unreliable.
As an analogy, consider blood samples. Blood is extracted from two individuals and analyses are conducted for sodium, chlorine, potassium, zinc, copper, and iron. This is a "generally accepted" practice. The analyses indicate that each analyte pair is present in analytically indistinguishable amounts. If the analyst were to state that the two individuals are from the same source—in other words, the same parents—the conclusion would be an unjustifiable extrapolation from a generally accepted practice resulting in a scientifically unfounded inference. CBLA, as it has been presented and admitted in the courtroom, is quite similar: unjustifiable extrapolation of data from a "generally accepted" practice into a scientifically invalid inference.
Compositional associations of bullets may demonstrate the possibility that a bullet or fragment has a common source. Our studies also suggest that there is an equal possibility that the bullet did not have a common source. It is difficult to see how expert testimony regarding "possibilities" could assist the trier of fact. In our view, risk of prejudicial effect is inordinate, given the aura of credibility and reliability attributed to expert witness testimony, particularly from the FBI, at least until studies can be conducted assessing probabilities relating to both coincidental "repeats" and geographical distribution issues.
As to distribution issues, an analogy to fiber evidence is helpful. Expert testimony on fibers is commonly admitted at trials, with challenges going to weight, even when the expert testifies as to a match between, for example, two blue denim cotton fibers. Most jurors can assess the significance of an alleged match of blue denim cotton fibers, given that most jurors are aware of the prevalence of blue jeans. Neither laypersons nor experts, however, are equipped to assess the probative value of a bullet lead match when there are thousands, even hundreds of thousands, of bullets of similar composition within a small geographical region. How, then, can such testimony assist the trier of fact in evaluating the likelihood that a defendant did or did not commit a crime? It is only possible to know that the defendant could have (as could almost any other local resident, based on ammunition availability).
The present state of scientific research does not support any inference beyond that the bullet or fragment could have derived from the same source as known bullets submitted for comparisons. Ironically, our data revealed that compositional matches are not only unreliable as to inclusion inference (due to repeats), but unreliable as to exclusion inference (due to inhomogeneities of ingots) as well.
The court in U.S. v. Hines, 55 F. Supp. 2d 62, 67 (D. Mass. 1999), opined that Kumho Tire Co. v. Carmichael, 526 U.S. 137 (1999), "plainly invit[es] a reexamination even of ‘generally accepted’ venerable, technical fields," and ruled that testimony regarding similarities and differences between a defendant’s handwriting exemplar and a holdup note was admissible but not expert testimony that the defendant was the author, because such an opinion lacked empirical validation. ( Id. at 70–71.)
Although papers have been published in the public domain relating to the conduct of the Phase I analyses, the absence of valid data, laboratory protocol, or proper scientific study of the theory and underlying premises to support Phases II and III, other than those of my colleagues and I, is glaring. As observed in U.S. v. Santillan, 1999 U.S. Dist. Lexis 21611, at * 14–15 (N.D. Cal. Dec. 3, 1999):
Nothing has been presented to the Court that the opinion of a handwriting "expert" as to the unique identity of the author of the questioned handwriting is a valid or reliable expert opinion. No tests or studies of the accuracy of such an opinion have as yet been conducted. (Emphasis added.)
Applying the court’s reasoning to bullet lead associations would result in a statement that adequately conveys our objection to the practice: "Nothing has been presented . . . that the opinion of a [bullet lead expert] as to the unique identity of the [molten source] is a valid or reliable expert opinion. No tests or studies of the accuracy of such an opinion have as yet been conducted."
William A. Tobin is a forensic metallurgical consultant. He is a retired FBI special agent, who was the de facto chief forensic metallurgist for the FBI laboratory, providing expertise on local, state, and federal cases, including the TWA 800 midair explosion. He has testified as an expert witness in more than 200 proceedings and at two U.S. Senate subcommittee hearings. Wayne Duerfeldt is an analytical chemist with 29 years’ experience in secondary lead refining and lead acid batteries. He has been involved in the design of lead smelters, alloy design for batteries and ammunition, specifications, installation, and calibration of spark source and inductively coupled plasma emission spectrometers. He is currently the laboratory manager at a large secondary lead smelter.
Impact in the Courtroom
Thousands of defendants have been convicted wholly or in part based on comparative bullet lead analysis (CBLA). One examiner alone claims to have testified more than 900 times, the significant majority of which were CBLA.
Courts have been misled or misguided into acceptance of the practice for more than three decades. CBLA has come under cross-discipline scientific scrutiny only within the past few years. Forensic metallurgical and statistical studies reveal the practice to be seriously flawed; courtroom testimony is without scientific foundation and often contradictory among witnesses from trial to trial. Although accepted in the courtroom, CBLA is not scientifically accepted outside the very narrow discipline practicing it.
Yet, only a few judges have granted motions for Daubert hearings, and each has deferred to its history of admissibility. However, jurors seem to understand the issues: every trial in which I’ve testified has resulted in either a hung jury or an acquittal, the last two I was the only witness for the defense.
— William A. Tobin
For more information . . . .
A more in-depth discussion of the authors’ research and studies is presented in "A Metallurgical Review of the Interpretation of Bullet Lead Compositional Analysis" by E. Randich, W. Duerfeldt, W. McClendon, and W. Tobin in Forensic Science International, volume 127, issue 3 (September 2002), pp. 174–91, Elsevier Science Publishing. You can also contact author William A. Tobin at (540) 854–7551 or by e-mail at firstname.lastname@example.org.
Case in Point
The chemical analysis of bullets to ascertain their elemental composition has been conducted for more than three decades. Courts have admitted evidence identifying bullets by their elemental composition using a variety of techniques, such as spectrographic analysis, neutron activation analysis, and inductively coupled plasma-atomic emission spectrometry (ICP) among others. ( State v. Noel, 723 A.2d 602, 606 (N.J. 1999).) In Noel, the state supreme court ruled that expert opinion testimony concerning similarity in the composition of bullets based on ICP was admissible even in the absence of statistical evidence. An FBI expert testified that bullets that come from the same box have the same composition of lead and bullets that come from different boxes having different compositions. He also acknowledged, however, that mixing may occur during storage and that 50,000 bullets may have the same composition.
It is one thing to say that two bullets have the same elemental composition (and in the same percentages), it is quite another thing to say how rare these findings are. ( See generally 1 Paul Giannelli & Edward Imwinkelried, Scientific Evidence § 14–11 (3d ed. 1999).)