Firearm & Toolmark Identification: What the Marks Can Support

Firearm and toolmark identification rests on a rule the discipline calls the theory of identification. When the marks on two items agree well enough, the examiner concludes they came from the same source. The governing standard, from the Association of Firearm and Tool Mark Examiners, is that agreement counts as an identification when it exceeds the best agreement seen between marks known to have been made by different tools, so that another tool making the mark is a practical impossibility. The theory states plainly that this judgement is subjective.

The difficulty is in the phrase "well enough." There is no number behind it. Ronald Nichols, who wrote the discipline's leading defence, concedes the point directly: there is no universal agreement as to how much correspondence exceeds the best known non-matching situation. So the threshold at which two marks become a match is not written down anywhere. It sits in the individual examiner's head, built from training and experience.

There is a second problem the examiner has to manage, and it cuts toward false identifications rather than missed ones. Tools that come off the same production line can leave the same marks. These are called subclass characteristics, and they are shared by many items made on the same machine. A single button used to rifle barrels can serve thousands of guns; a broach, hundreds. A 2024 study funded by the US National Institute of Standards and Technology drove the point home with numbers: on unfinished broached breech faces, an objective algorithm put 89% of comparisons between different guns above the threshold it uses to declare a match. The signature the examiner attributes to one gun can be shared by every gun made on the same tooling, none of which the examiner has ever seen.

None of this means the work is guesswork. A trained examiner distinguishes marks that untrained people cannot, and subclass is a known hazard that a careful examiner watches for. The honest reading of the foundation is narrower. The discipline can compare marks and describe how well they correspond. What it cannot do is point to a measured threshold, or a measured rate of shared marks across the world's guns, that would justify naming one source to the exclusion of the rest.

The discipline's answer to the validity question is a handful of black-box studies, in which examiners judge comparisons whose true source is known and their answers are scored. The headline numbers are low. The 2014 Ames study, which the President's Council of Advisors on Science and Technology relied on, reported a false-positive rate of about 1% on cartridge cases. The 2022 Ames II study reported false-positive rates under 1% for both bullets and cartridge cases. Taken alone, those numbers sound reassuring.

Two things sit underneath them. First, the errors were not spread evenly. In the 2014 study, most of the false identifications came from a small number of examiners. Second, and more important, the studies count only conclusive answers. When examiners in Ames II were shown cartridge cases fired by different guns, they reached the correct answer, elimination, only a third to a half of the time. The rest they called inconclusive, and inconclusives are left out of the error rate. A discipline that correctly clears a different gun half the time at best is not the same as one that is right 99% of the time.

There is a further result that speaks to reliability rather than accuracy. When the Ames II examiners re-examined the same comparisons months later, they disagreed with their own earlier conclusions about 35% of the time on non-matching cartridge cases. When two examiners judged the same non-matching comparison, they agreed only about 36% of the time. An opinion that the same examiner will not reproduce, and that a second examiner will often not share, is not a stable measurement.

The fair statement of the black-box work is this. These are the best data the discipline has, and on the reported basis the false-positive numbers are genuinely low. But the numbers describe specific examiners judging specific guns in a test they knew was a test, with every difficult non-answer set aside. They are, as the Ames II authors themselves say, an upper bound on accuracy, not a measurement of casework error.

The single most contested question in firearms is how to count an inconclusive answer. It sounds like bookkeeping. It changes the error rate by more than an order of magnitude.

A 2023 study in the Proceedings of the National Academy of Sciences by Guyll and colleagues called firearm examination highly valid, reporting accuracy near 99%. That figure counts only conclusive decisions. The same study's own numbers tell the other story. Put the inconclusives back in and the ability to correctly identify a same-source pair falls from 99.9% to 93.4%, and the ability to correctly clear a different-source pair, the specificity, falls from 99.1% to 63.5%. In the same data, examiners answered inconclusive six times more often on different-gun comparisons than on same-gun comparisons.

The other side of this argument is worth stating fairly, because it is coherent. An identification or an elimination asserts something about the world that is either right or wrong. An inconclusive asserts nothing about the source. Hal Arkes and Jonathan Koehler put it simply: an inconclusive is not an error, it is a pass. Alex Biedermann and Kyriakos Kotsoglou add that calling a non-assertion an error forces the field to abandon ground truth and substitute a majority vote for what the answer should have been, which brings its own problems. On this view an inconclusive on genuinely poor marks is the honest and cautious call, and forcing a conclusion would manufacture error.

Both things can be true at once. An inconclusive need not be scored as wrong, and counting inconclusives as correct still deflates the error rate and hides a one-directional failure to exclude. The conclusion for the witness box is not that the discipline is worthless. It is that the honest error rate is unknown, bounded well above the advertised figure, and that a low number on a proficiency test measures willingness to commit as much as it measures accuracy.

A firearms examiner rarely works on the marks alone. The case comes with a theory, a suspect, and often the conclusion of a first examiner. Cognitive science shows that context shapes a judgement without the person feeling it move, and the firearms-specific evidence is direct.

The clearest result is a field study inside real casework. Erwin Mattijssen and colleagues at the Netherlands Forensic Institute compared blind and non-blind peer review of firearm conclusions. When the second examiner could not see the first examiner's conclusion, they disagreed about the strength of the comparison 42.3% of the time. When they could see it, disagreement fell to 12.5%. The reviewer who already knows the answer drifts toward it. So the second signature on the report, the one presented in court as independent verification, was for the most part not an independent check at all.

The last link is the jury. Brandon Garrett and colleagues ran mock jurors through the same case with different conclusion language. Declaring a match raised the odds of conviction roughly five to six times over an inconclusive, and softening the wording made almost no difference: jurors treated "consistent with" much as they treated a flat identification. In the same study, conviction rates were 65% when the examiner said identification and 57% when the examiner said consistent with, a gap too small to matter. And cross-examination, the safeguard courts rely on to expose a weak opinion, did not significantly move jurors' verdicts.

The honest position on the stand is not a claim of immunity. It is a description of what was done: whether the analysis was recorded before the case theory was known, and whether the verifier was blind to the first conclusion. Those are procedures a lab either followed or did not. "I kept an open mind" is neither, and the evidence says the jury will over-weight the conclusion regardless.

If the trouble is a subjective threshold, the obvious answer is to make the comparison objective. Researchers have built methods that do exactly that: congruent matching cells, which divide a 3D surface scan into small regions and count how many agree; algorithmic likelihood ratios that score similarity numerically; and automated correlation systems that search image databases. This is real progress, and the numbers on the best datasets are strong. One 3D likelihood-ratio method reported values exceeding a billion for same-source pairs.

Two cautions travel with those numbers, and they are the ones a cross-examiner will use. The first concerns databases. The National Integrated Ballistic Information Network, NIBIN, searches images of cartridge cases and returns a ranked list of candidates. A candidate is not a match. As the network's own evaluation states, a high-confidence candidate must be manually confirmed by an examiner on a comparison microscope before it becomes a hit. The database narrows the field and generates an investigative lead; the identification, if there is one, is still the human examiner's, with all the limits the earlier sections describe.

So the objective methods change the picture without settling it. They replace a private judgement with a reproducible number, a shared data format, and a stated error rate, which is genuine progress the field should be credited for. But the strongest results come from small, mostly consecutively manufactured datasets that cannot themselves test the uniqueness assumption; the scores need calibration against the right population before they become weights of evidence; and the tool in daily use, the database, produces leads for a human to confirm rather than identifications. A courtroom-ready, validated, calibrated objective method is not yet here.

The same comparison logic is applied well beyond firearms, to the marks left by screwdrivers, bolt cutters, saws and knives, and often with far less validation behind it. The most sobering results come from marks on the body.

Consider cut marks in cartilage. Christian Crowder and colleagues, and separately a group led by Jennifer Love, tested whether trained analysts could read a knife's class, its type, from cuts in costal cartilage using the accepted method. The result was a misclassification rate above 65%. Three trained analysts, asked the comparatively easy question of what kind of blade made the cut, got it wrong roughly two-thirds of the time. And that is a coarser claim than naming one specific knife. On bone, the same pattern appears in a narrower form: the direction of the bevel on a cut was wrong more than half the time, against under 17% in wax.

The honest use of toolmark evidence on the body is real but narrow. The literature supports class-level statements and exclusions: this cut is consistent with a serrated blade, or a particular saw is excluded by kerf width. The founding text of saw-mark analysis says as much, calling a positive identification of one saw a rare occurrence, and the discipline's own guidance describes naming a specific tool from a mark as an unacceptable practice. What the marks cannot support is the confident individualisation, this saw and no other, that has still been offered in court. Where the discipline stays at the level of class and exclusion, and states the conditions it is assuming about angle and substrate, it is on defensible ground. Beyond that it is not.

How a firearms conclusion is worded is where cases are won and lost. The signature overstatement is source attribution to the exclusion of all other firearms in the world, and its relatives: practical impossibility, a hundred per cent certainty, a match to an exact statistical certainty, it is like a fingerprint. Since United States v. Green in 2005, a growing line of courts has ruled these phrases inadmissible, and the discipline's own governing language now bars them. The United States Department of Justice's uniform language tells its examiners they shall not claim a source identification to the exclusion of all other sources, shall not use the word individualise, and shall not assert that their examinations are infallible or have a zero error rate.

The gap between what the examiner means and what the jury hears is measurable. When Thomas Busey and colleagues surveyed jurors, 71% took the word identification to mean to the exclusion of all others. And a reanalysis of the error-rate studies by Aggadi and colleagues found that the strength implied by the word identification runs ahead of the measured strength of the evidence by up to five orders of magnitude. The word does work in the courtroom that the data cannot support.

Courts are moving, unevenly. Since 2005 a line of decisions has limited firearm testimony, some restricting the examiner to describing similarities, some barring the certainty language, and the Maryland high court in Abruquah v. State in 2023 pushed back on unqualified identification language. But most courts still admit the evidence, largely on precedent and long use rather than fresh findings of reliability, which is the very ground the bullet-lead story warns against. The examiner who wants to be believed, and to stay believed on appeal, states what the marks can support: that they correspond, how strongly, under what assumptions, and with what error rate, and not that one gun fired the round to the exclusion of the world.