Digital Forensics: The Evidence That Looks Like Maths

For twenty years, the Post Office prosecuted its own subpostmasters on the word of a computer. From 1999, branch after branch showed accounting shortfalls in a system called Horizon, and the Post Office took those shortfalls to court as proof of theft and fraud. Hundreds of people were convicted. Many were imprisoned, fined, or had their homes taken; some died still labelled thieves. The shortfalls were software bugs. In the words of the appeal courts now quashing conviction after conviction, it is likely the largest miscarriage of justice in British history.

It happened because of a legal default most examiners have never heard of and all of them rely on. In England and Wales, a court treats a computer, as a matter of law, as having worked correctly unless someone proves otherwise. Evidence from a computer is presumed reliable. Lawyers call it a rebuttable presumption. You can rebut it in theory. In practice it is close to impossible, especially when the system belongs to a "substantial institution" with the records and the lawyers on its side. The subpostmasters could never get inside Horizon to show it was broken, so the presumption held. It was wrong.

It was not always the law. Until 1999, section 69 of the Police and Criminal Evidence Act 1984 put the burden the other way. The prosecution had to show a computer was operating properly before its output could come in. The Law Commission decided that was a nuisance. Section 69, it said, served "no useful purpose," and had it repealed in favour of a presumption that "mechanical instruments were in order at the material time." Computers, the Commission reasoned, counted as mechanical instruments. It also assured everyone that "such a regime would work fairly." Horizon is what fair looked like.

Calling a computer a mechanical instrument is where it goes wrong. A set of scales fails in obvious, physical ways you can see and repeat. Software fails in ways that hide. In 2022, ten of Britain's most senior figures in computer reliability and evidence law (Ladkin, Littlewood, Thimbleby, Thomas, Murdoch and Mason among them) set out why. Every computer system contains bugs, and some, in their words, "rarely reveal themselves... because they can masquerade as normal behaviour." Even after a failure, it can be impossible to say whether the cause was a software defect or someone using the system wrongly. A machine that has run cleanly for years can still be running on a bug that surfaces for the first time today.

American courts built the same problem one level down, around the tools. In 2006, Van Buskirk and Liu gave it a name. Courts had granted forensic software a "presumption of reliability," treating the output of EnCase and its rivals as accurate because it usually is. There was, they argued, no scientific basis for it. The assumption then runs the whole way down. The system is sound, the tool that reads it is sound, the examiner who reports the tool is sound, and nobody had to prove any of it.

None of this is a reason to attack computers in general on the stand. It is a reason to know what you are standing on. When you present a device's output as fact, you invite the court to assume the one thing Horizon proves it should not. Bohm and his colleagues propose a better position: treat reliability as a claim you have to back, not a default you inherit. If anyone asks, you should be able to point to the system's own error logs, its audit trail, and its change records.

In 2010, the man who runs the testing lab published a confession for a title. James Lyle, the computer scientist who heads NIST's Computer Forensic Tool Testing program, called his paper "If error rate is such a simple concept, why don't I have one for my forensic tool yet?" He had been testing these tools since 2000. He still could not hand you a number.

The reason matters in the witness box. A soil test for some chemical has a stable error rate because its failures are random. They scatter, so you can model them. Digital tools don't fail like that. They fail systematically: the same drive, the same operating system, the same interface, and the tool drops the same data every time. Lyle's own example is the imaging tool SafeBack. On one test it copied 3,335,472 sectors and got 1,008 of them wrong, an error rate of 0.0003. On most other runs it was perfect: zero. Same tool. Whichever number you quote misrepresents all the others. A write blocker, as Lyle put it, does not have a 3% failure rate. It "either works or it fails." There is no average to give a jury.

A decade of that testing, written up by Barbara Guttman, Lyle and Rick Ayers in "Ten Years of Computer Forensic Tool Testing," reads as a catalogue of tools failing silently. Some skip the last sectors of an NTFS partition because "those sectors are not used to contain user data," and overwrite them with a block already acquired. When a bad sector turns up on a Linux ATA acquisition, seven readable sectors around it get replaced with zeros. Gone, no warning. The same tool over a different interface drops a different, variable number. None of it throws an error. The report still looks correct.

Then there are the people using the tools. In 2019 Graeme Horsman surveyed up to 100 working examiners. 76% were worried about the state of tool testing. 88% had personally hit erroneous results from a forensic tool. 79% admitted using a tool in a live investigation they had never tested themselves, on the vendor's word alone. The vendor's word is an EULA. EnCase's says the software "is not fault-tolerant" and the maker "does not warrant that the software is error-free." X-Ways is blunter: "the user must assume the entire risk of using the program." The usual fallback is dual-tool agreement. Run two tools, and if they match, call it confirmed. Horsman takes that apart. Two tools built on the same flawed code library will agree and both be wrong. Agreement is not proof. It is two witnesses who copied each other's homework.

That leads to the title of Horsman's other paper: "I couldn't find it your honour, it mustn't be there!" A tool returns nothing, and the examiner reads nothing as not there. But the tool may simply not parse that browser version, that compressed format, that unallocated region, a limit it never disclosed. Marshall and Paige put the deeper problem precisely: the inputs to a forensic exam are unknown. You have no ground truth to check the tool against. So when counsel asks how you know your tool found everything, the truthful answer is that you don't, and you can't, from the tool's output alone.

Radina Stoykova and her colleagues pulled 124 forensic reports out of the Norwegian police case-management system: 21 homicide and sexual-assault cases that all went to indictment, 187 seized devices. Then they checked something simple. Did the reports show the evidence had actually been verified? For 50 of the reported acquisitions, the report said nothing about whether integrity was checked at all. Eighteen said "MD5 was used" but printed no hash value. Three named both MD5 and SHA1, again with no value. In the whole dataset, exactly two acquisitions stated the algorithm and gave the number. Their conclusion was blunt. None of the 21 cases validated tool results or error rates, and it was "not possible to trace the digital forensic actions performed on each item or link the digital evidence to its source."

A hash is the one thing examiners are sure they got right, and most of the time it is not even in the report.

The deeper problem is the one counsel will press. Even a perfect, documented, matching MD5 proves a single thing: the copy is faithful to what was read off the device. It says nothing about whether you read off everything that mattered. Stoykova's framework separates the acquisition space (partitions, volumes, allocated versus unallocated) for exactly this reason. Manual extraction, she notes, "does not actually capture the digital data stored on the device." It captures only "the representation of data as provided by the device itself," and it changes the device while it does so. Selective extraction, a cloud-only pull, an encrypted partition you could not open: you hash the fraction you got, and the hash comes back immaculate. An immaculate hash of a fragment.

The hash itself is not the bedrock it is sold as either. Rasjid and colleagues reviewed the cryptographic collisions in the functions forensic tools rely on. MD5 collisions have been public since Wang's 2004 Eurocrypt result. SHA-1 fell to a full collision attack from Stevens and colleagues. Collisions have to exist: the message is far larger than the fixed-length digest, so by the pigeonhole principle two different files must share a value somewhere. Tools, Rasjid notes, "still use MD5 due to performance issues." A defence expert who knows this can ask whether your matching hash really proves the two objects are identical, and "yes, mathematically certain" is the wrong answer.

The live issue is rarely that someone forged your evidence with a chosen-prefix collision. That is exotic. The issue is rhetorical. "Verified MD5" sounds like proof that everything is correct. It is proof of copy fidelity for the data you happened to acquire, and nothing else. As Horsman puts it, the field has to verify its tools using its tools, "trapped in an infinite loop," and a matching hash does nothing to catch a tool that silently failed to parse a browser format or skipped a compressed region.

In the box, the mistake is letting "I verified the hash" stand in for "my extraction was complete and my interpretation correct." Those are three different claims, and the hash covers the first inch of the first one. When counsel asks how you know you recovered everything that mattered, the real answer is about scope and method and what the device, the cloud, or the encryption put out of your reach. It is not about a thirty-two-character string that matched.

In 2021, Nina Sunde and Itiel Dror handed the same 3 GB disk image to 53 working digital forensic examiners across eight countries: Norway, India, the UK, Denmark, Finland, the Netherlands, Kenya and Canada. Same file, an old Windows XP machine owned by "Jean," timestamps from 2008. Same question: what happened, and was Jean involved? The traces on the drive were not exotic. The eleven they scored, including emails, chat logs, a spreadsheet and a USB mount, were the kind of thing the authors say "any competent examiner would find." Not one examiner found all eleven. Most found between five and eight. Fourteen found four or fewer.

That should trouble anyone who tells a court digital evidence is objective, and the second half of the study is sharper. Sunde and Dror split the 53 into four groups. The control group got only the bare scenario. The others got a line of context first. One was told Jean had confessed. One got an ambiguous wage-dispute story. One was told the police now believed Jean was innocent, framed in a phishing attack. The drive was identical for all of them. The context was a sentence.

The examiners told innocence were the ones who looked and stopped. They observed the fewest traces, an average of 4.5 against 6.9 for the group nudged toward guilt. They did not find less because the data was different. They found less because they stopped searching once the file fit the story. Sunde and Dror state it directly: an examiner who believes the suspect is innocent observes fewer traces and "may have less information for developing explanations" of what actually happened. The bias did more than colour the conclusion. It set the moment the examination ended. One examiner reported it was "very likely" Jean was framed in a phishing attack, a confident finding built on a drive that, examined without that prompt, supported no such thing.

Then there is reliability, which the authors called their most important result. They measured agreement between examiners with Krippendorff's alpha, where 0.80 is strong and anything below 0.667 is inadequate. Across observations, interpretations and conclusions, every score came in low. The best was 0.51. Same evidence file, same context, and the chance that a second examiner would see, read and conclude the way the first did was poor. On individual traces, examiners split between "indicates guilt" and "indicates innocence." Opposite directions, same bytes.

The 2019 Sunde and Dror paper had already argued that digital forensics, alone among the major disciplines, had sidestepped the bias research that reshaped DNA and fingerprints, and that examiners meet case context as a matter of routine. In a side study they collected 30 case-submission forms from European and US units. 22 of them, 73%, had an open "information about the case" box, and not one warned against putting task-irrelevant detail in it. The pipeline that hands you the drive also hands you the theory.

In the box, the question is not whether you are truthful. It is whether the conclusion was yours or the file's. Did you read the suspect interview before you analysed the image? Did you know there was a confession? Would the examiner at the next desk, given your drive and nothing else, have written your report?

On 17 October 2003, at Southwark Crown Court, Aaron Caffrey walked free. The prosecution said he had flooded a computer server at the Port of Houston with data and shut it down. His machine held the attack tools. His machine held the connection. The jury acquitted anyway, because Caffrey said unknown hackers had taken control of his computer and launched the attack to frame him. The prosecution's own expert, Professor Neil Barrett, told the court he could find no Trojan on the machine. Caffrey's answer was simple. You cannot test every file, and a Trojan could delete itself and leave no trace. The CPS casenote records the detail that mattered: this was "the first case where a Trojan virus defence has been raised without a trace of a Trojan virus being found." No Trojan, and he still walked.

Six months earlier, in April 2003 at Reading, Karl Schofield walked free too. He was charged over 14 indecent images of children on his PC. A defence expert found a real Trojan on the machine, installed, Schofield told the Reading Evening Post, the day before the images were downloaded. The prosecution accepted it probably did the downloading. Two cases, one lesson for the box. The file was there, the connection was there, and it still was not enough. Susan Brenner called this the Trojan horse defence. Lawyers know its older cousin as SODDI, Some Other Dude Did It. The device is not the person.

It gets worse. Writing in 2007, Simson Garfinkel laid out anti-forensics as a field with goals, one of which, in words he quotes from Liu and Brown, is to "implicate an innocent party by planting data." The tools exist. Timestomp overwrites the NTFS create, modify, access and change timestamps. Transmogrify disguises a text file as an executable so EnCase skips it. Slacker hides data in slack space. An attacker can plant, forge, back-date and erase, and Garfinkel makes the obvious point that the examiner who does not account for anti-forensics is the one most likely to miss it.

The timestamps deserve their own warning. Céline Vanini, Christopher Hargreaves and colleagues titled their 2024 paper with the question you will be asked: "Was the clock correct?" Often you cannot assume it was. System time comes from the device's own clock, and that clock drifts, gets reset, runs down a failing battery, or is skewed on purpose. In a controlled experiment they backdated a virtual machine's clock by about three hours, and every file and history timestamp on it then lied by three hours. Their fix is a "time anchor," an artifact that carries both the local system time and an independent external time, such as a server timestamp baked into a browser cache header or a Google search URL. With one, you can say the clock was probably correct at that moment. Without one, a file-creation time is just a number the machine asserts about itself.

In a 2020 paper, Eoghan Casey works through a murder case and arrives at a likelihood ratio "on the order of 1,000,000." The phone's data is a million times more expected if the device was at the crime scene than if it was somewhere else. Then he stops and refuses to say it. The number, he writes, is "ill-conditioned": the alternative is so rare that a small wobble in one input probability swings the answer by orders of magnitude. Presenting that as a precise figure would "disguise the broad and subjective opinion... into a scientific-looking result." His recommended wording is not a number at all. The evidence supports the device being at Location X "so extremely strongly that it would be precarious or problematic to express a precise numerical likelihood ratio."

Truthful testimony about digital evidence almost never sounds like certainty. It sounds like strength of evidence, stated against a named alternative.

Casey's method is buildable. His standardisation paper sets out a seven-stage process: observe, form hypotheses, infer, predict, then run a second-phase search that actively looks for evidence contradicting each hypothesis, and only then assign strength. His worked example is an intellectual-property case. Messages missing from a phone could mean they never existed, were deleted, or were not recovered. A second tool then pulls back deleted message remnants, including an attachment named "customers.xlsx." Now he can assign strength, high for the deletion hypothesis and low for the others. You assign strength to the evidence given a hypothesis, never to the probability the hypothesis is true. That last step belongs to the court, and reaching for it is the transposed-conditional fallacy that gets experts taken apart.

All of that assumes the underlying work is sound. Stoykova's 2022 study of the Norwegian police asks whether it is, and the answer was hard. Across 21 real homicide and sexual-assault cases, the digital forensic actions could not be traced to their source, not one case validated tool results or error rates, and across every acquisition exactly two recorded a hash value. This is what NAS 2009 and PCAST 2016 mean by validity as applied: not the tool in theory, but what was actually done, documented, and able to be reproduced. Stoykova found the document trail often was not there.

There is also the tool that will not behave the same way twice. Gougherty's 2024 test of a large language model on scientific reports found it 50 times faster than a human and over 90% accurate on simple categories. It also invented pathogen-incidence figures for 53 of 100 reports where the real answer was "not stated," and returned the wrong records for some queries without flagging it. It does not check that its own output is internally consistent. A forensic duty is deterministic: same input, same result. A language model, even at "temperature zero," is a probability machine that will fill a gap with confidence. Bolt one into your workflow and you have imported the exact certainty-shaped error this whole reading warns against.