Digital Forensics: What the Output Does Not Tell You

From 1999, the Post Office treated accounting shortfalls produced by its Horizon system as evidence of theft or fraud by subpostmasters. Hundreds were convicted, and many were imprisoned, fined or lost their homes. Software defects were later shown to produce shortfalls. Appeal courts quashing convictions have described the matter as likely the largest miscarriage of justice in British history.

The cases operated against a legal presumption in England and Wales that a computer worked correctly unless evidence shows otherwise. Although rebuttable in principle, the presumption can be difficult to challenge when a substantial institution controls the system records. Subpostmasters lacked access to the material needed to demonstrate Horizon's failures.

Van Buskirk and Liu argued in 2006 that US courts had also given forensic software a "presumption of reliability," accepting output from tools such as EnCase without a scientific basis for assuming accuracy. The assumptions may accumulate: that the source system worked, the forensic tool read it correctly and the examiner interpreted the output correctly.

Reliability should be supported rather than assumed. Relevant evidence may include system error logs, audit trails, version and change records, tool validation and documentation of the examination.

James Lyle of NIST's Computer Forensic Tool Testing program asked in a 2010 paper: "If error rate is such a simple concept, why don't I have one for my forensic tool yet?" His program had tested tools since 2000.

Digital-tool failures are often systematic rather than randomly distributed. Under the same drive, operating system and interface, a tool may omit the same data each time. In Lyle's example, SafeBack copied 3,335,472 sectors and produced 1,008 errors on one test, a rate of 0.0003, while many other tests produced none. Neither figure describes performance across all conditions. A write blocker, Lyle writes, "either works or it fails." Its performance should be stated for the tested configuration rather than as a general average.

Guttman, Lyle and Ayers describe silent failures in Ten Years of Computer Forensic Tool Testing. Some tools omitted final NTFS sectors on the assumption that they contained no user data and replaced them with a previously acquired block. In one Linux ATA acquisition, encountering a bad sector caused seven readable neighbouring sectors to be replaced with zeros. A different interface produced a different number of omissions, sometimes without an error message.

Graeme Horsman's 2019 survey of up to 100 examiners found that 76% were concerned about tool testing, 88% had encountered erroneous tool results and 79% had used an untested tool in an investigation based on vendor claims. EnCase states that it "is not fault-tolerant" and does not warrant error-free software; X-Ways says "the user must assume the entire risk of using the program." Agreement between two tools is useful only if their failure modes are independent. Tools using the same defective library may agree and both be wrong.

Horsman's other paper is titled "I couldn't find it your honour, it mustn't be there!" A null result may mean the artifact was absent, or that the tool did not parse the browser version, compressed format or storage region. Marshall and Paige note that the inputs to a forensic examination are unknown, so there is no complete ground truth against which to check recovery. Tool output alone cannot establish that every relevant artifact was found.

Stoykova and colleagues reviewed 124 reports from 21 Norwegian homicide and sexual-assault cases involving 187 seized devices. Fifty acquisitions did not state whether integrity had been checked. Eighteen said "MD5 was used" but gave no value, and three named MD5 and SHA-1 without values. Only two acquisitions recorded both the algorithm and hash value. None of the 21 cases validated tool results or reported error rates. The authors concluded that it was "not possible to trace the digital forensic actions performed on each item or link the digital evidence to its source."

A hash assists only if the algorithm, values, stage of calculation and relationship between source and copy are documented.

A documented matching MD5 value shows that the acquired copy corresponds to the data the tool read. It does not show that every relevant partition, volume or unallocated region was acquired. Stoykova distinguishes acquisition scope for that reason. Manual extraction "does not actually capture the digital data stored on the device"; it captures "the representation of data as provided by the device itself" and may alter the device. A selective, cloud-only or partially decrypted extraction may have a matching hash while remaining incomplete.

Rasjid and colleagues also review cryptographic collisions. MD5 collision attacks have been public since Wang's 2004 Eurocrypt work, and SHA-1 has been subjected to a full collision attack by Stevens and colleagues. Because a fixed-length hash maps an unlimited number of files into a finite set of values, collisions necessarily exist. Rasjid notes that tools "still use MD5 due to performance issues." A matching hash provides extremely strong evidence of copy fidelity in ordinary use, but not mathematical certainty that arbitrary objects are identical.

An engineered collision is rarely the practical issue. The larger risk is treating "verified MD5" as proof that the extraction was complete and the interpretation correct. It verifies the copied data, not artifacts the tool failed to parse or regions outside the acquisition scope. Horsman describes the field as "trapped in an infinite loop" when tools are used to verify other tools.

In court, separate copy fidelity, completeness and interpretation. The latter two require evidence about acquisition scope, tool capability, inaccessible cloud or encrypted data, validation and the reasoning applied to recovered artifacts.

In 2021, Nina Sunde and Itiel Dror gave the same 3 GB Windows XP disk image to 53 digital forensic examiners in Norway, India, the UK, Denmark, Finland, the Netherlands, Kenya and Canada. They scored eleven artifacts, including emails, chats, a spreadsheet and USB activity, which the researchers had selected for scoring, some easier to find than others. No examiner found all eleven. Most found five to eight, and fourteen found four or fewer.

Participants were divided into four groups. One received only the scenario. Others were told that the user had confessed, given an ambiguous wage-dispute account, or told police believed the user was innocent and had been framed through phishing. The disk image was identical; only the context changed.

Examiners told the user was innocent observed an average of 4.5 artifacts, compared with 6.9 in the guilt-context group. Sunde and Dror say an examiner who believes a suspect is innocent may stop earlier and "may have less information for developing explanations." One participant described framing through phishing as "very likely," although that proposition was supplied in the context.

Agreement was measured with Krippendorff's alpha, where 0.80 indicates strong agreement and values below 0.667 are inadequate. All scores for observations, interpretations and conclusions were low; the highest was 0.51. Examiners sometimes interpreted the same artifact as indicating guilt or innocence. The findings show variation in both what examiners recover and how they interpret it.

Sunde and Dror argued that digital forensics had not adopted the bias research influencing DNA and fingerprint practice. In a 2021 study, of thirty submission forms collected from European and US units, twenty-two (73%) contained an unrestricted case-information field, and none warned against including task-irrelevant detail.

An examiner should be able to state what case information was received, when it was received, and what procedures ensured that another examiner could review the evidence without being directed by the same theory.

Aaron Caffrey was acquitted at Southwark Crown Court in October 2003 of an attack on a Port of Houston server. His computer contained attack tools and evidence of the connection, but he said unknown hackers had controlled it and framed him. Prosecution expert Professor Neil Barrett found no Trojan. Caffrey argued that not every file could be tested and a Trojan could remove itself. The CPS described it as "the first case where a Trojan virus defence has been raised without a trace of a Trojan virus being found."

Six months earlier, Karl Schofield was acquitted over fourteen indecent images on his computer after a defence expert found a Trojan, which the prosecution accepted probably downloaded them. Susan Brenner called this the Trojan-horse defence; the broader alternative is sometimes called SODDI, Some Other Dude Did It. Both cases illustrate the difference between activity on a device and attribution to a person.

In 2018, a mother in Madison, Wisconsin checked her twelve-year-old daughter's phone and found messages from an adult man. The extraction report showed 125 Kik messages from a user called fus__ro_dah, spelled with two underscores between "fus" and "ro". When Wisconsin police asked Kik for the account holder's details, the request named fus_ro_dah, with one underscore. That happened to be a real account. It belonged to Brandon Klayme of Halifax, Nova Scotia. Kik supplied his email address, Google records gave an IP address in Halifax, and a Bell Aliant production order gave his home address. In February 2021, police seized the phones and laptops from his bedroom. The devices confirmed he had a Kik account and nothing more: no messages with the girl, no images, no Kik activity at all during the offence period. He was convicted at trial in April 2023 and sentenced to eighteen months in prison, which he served in full.

His lawyer found the discrepancy while preparing the appeal. The two usernames had been in the disclosure since before the trial; the Court of Appeal said there was "no evidence confirming or explaining how it went unnoticed". The Crown conceded the appeal, and its own enquiries located the correct account: a man named Jay, apparently in California. When the acquittal was entered in July 2026, Klayme had finished the prison term and was still on probation.

No system failed. Kik answered the request it received, the Google and Bell Aliant records were accurate, and every agency trusted the identifier it was handed. The error was one character in the request, and nothing in the chain compared the username being investigated with the username on the victim's phone. Caffrey and Schofield illustrate the gap between a device and a person. Klayme illustrates the step before it: whether the identifier even points at the right account.

Vanini, Hargreaves and colleagues asked in their 2024 paper: "Was the clock correct?" Device time may drift, be reset, reflect a failing battery or be deliberately changed. They backdated a virtual machine by about three hours, causing file and history times to shift by the same amount.

A time anchor contains both local device time and an independent external time, for example a server timestamp in a browser-cache header or search URL. It may support the accuracy of the device clock at that moment. Without an anchor, a file-creation time is an assertion derived from the device's own clock.

In a 2020 murder-case example, Eoghan Casey calculated that phone data was about one million times more expected if the device was at the scene than if it was elsewhere. He did not recommend reporting that number because it depended heavily on an estimated probability for the alternative proposition. Changing that estimate from one in ten thousand to one in one hundred thousand changed the LR tenfold. Casey warned that the calculation could "disguise the broad and subjective opinion... into a scientific-looking result." He instead said the evidence supported the device being at the scene "so extremely strongly that it would be precarious or problematic to express a precise numerical likelihood ratio."

The example supports evaluative language tied to stated propositions when the data do not justify numerical precision.

Casey recommends setting out competing explanations before evaluating the artifacts and then searching for observations that would contradict each one. In his example, missing messages might never have existed, might have been deleted or might not have been recovered. A second tool recovered deleted-message fragments and a spreadsheet attachment, allowing the first and third explanations to be tested. The evidence then supported deletion.

That does not establish who deleted the messages. The evidence is expected if messages were deleted is an evaluative conclusion. The accused probably deleted them is a conclusion about a person that requires other case evidence. Moving from one to the other transposes the conditional.

Validity as applied requires a traceable and reproducible record of what was done in the case. Stoykova's 2022 review found that digital forensic actions in 21 homicide and sexual-assault cases could not be traced to source, none documented validation of tool results or error rates, and only two acquisitions recorded a hash value.

Gougherty's 2024 test of a large language model on scientific reports found it fifty times faster than a human and more than 90% accurate on simple categories. It also invented values in 53 of 100 reports where the correct answer was "not stated," and returned incorrect records for some queries without warning. A generative model is probabilistic rather than deterministic. If used in a forensic workflow, its outputs require independent verification, version and prompt documentation, and disclosure of its non-repeatable failure modes.