Folder Snapshot Differ and Duplicate Finder

Which files changed, moved or doubled up, proven by hash.

  • sha256sum-style lists
  • Moves found
  • Duplicates confirmed
Two plain rounded trays of soft pebbles on a clay workbench, a few matching pebbles joined across them by ribbons and one pebble sitting out of place.

The guide

A folder is what its bytes say: how to tell changed, moved and copied files apart by content

Names and dates describe how a file was handled, while a digest describes what it holds, and comparing whole folders honestly needs the second kind of evidence.

  • under 2^64 bits longest message SHA-256 accepts under FIPS 180-4
  • 23 February 2017 first public SHA-1 collision was published
  • 2^18 steps for a published MD5 collision attack
  • 1 MiB slice read at a time while hashing
Read the full guide Fold the guide 6 min · 20 sources

Two folders that look alike on a screen can differ in a single byte, and two that look different can hold exactly the same files under new names. A name records what someone called a file. Only the content records what the file is.

The usual way to check content is not to lay two copies side by side and compare them bit by bit. That needs both copies in the same place, and it can miss damage that struck both. The more popular approach is to compute a hash of the copy and compare it with the hash of the original. A snapshot of a folder is nothing more than a list of those facts: a path, a size and a digest for every file.

What a matching digest proves, and what it does not

A digest is a short, fixed-length value derived from input of any length. The SHA-2 family has six members, with digests of 224, 256, 384 or 512 bits. FIPS 180-4 states how each is computed and how long a message may be: under 2^64 bits for SHA-1, SHA-224 and SHA-256, and under 2^128 bits for the 384 and 512 bit members.

Published test values make the idea concrete. RFC 6234 lists the SHA-256 digest of the three letters abc as ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad, and it lists a value for a message of one million letters a, which starts cdc76e5c9914fb9281a1. RFC 1321 gives d41d8cd98f00b204e9800998ecf8427e as the MD5 digest of nothing at all. Any code that claims to compute these functions has to reproduce such values exactly.

When two files give the same SHA-256 digest, their bytes are the same, as far as anyone knows how to find out. That is a statement about the two files and nothing else. It does not say either one is the file you meant to get, or that it came from where you think. A digest copied from a tampered download matches the tampered download perfectly.

Why names and dates are poor evidence

The modified time of a browser file object is a count of milliseconds since the Unix epoch, and a file with no known date reports the current time. Browsers may also reduce its precision. A copy gets a new time, and any program can set an old time back onto changed content. So this page shows modified times and never lets them decide anything.

Names have their own habits. A folder picked in a browser reports each file by a path relative to the folder the visitor chose, so two folders called by different names compare cleanly once that top name is dropped. Paths are then compared exactly, one character after another, with a single switch for treating differently spelled but visually identical names as one.

Size is the cheapest evidence there is, and it is conclusive in one direction. If a path exists in both states and the sizes differ, the file changed, and nobody needs to read a byte to say so.

How a move or a rename is recognised

Take the paths that exist in only one state. A path that is gone from the earlier state is a removal candidate, and a path new in the later state is an addition candidate. If a removal candidate and an addition candidate carry the same digest and size, the file did not vanish and appear. It moved or was renamed.

When several candidates share one digest, they are paired one to one, in sorted path order, and any leftovers stay as plain additions or removals. Here is a worked example with five tiny files. In the earlier state a.txt holds alpha and a line break, b.txt holds beta, and docs/c.txt holds gamma. In the later state a.txt is unchanged, b.txt holds beta2, archive/c.txt holds gamma and d.txt holds alpha.

The tool gives exactly this answer. The file a.txt is unchanged. The file b.txt is changed, and it needs no hashing, because its size went from 5 bytes to 6. The two gamma files are 6 bytes each with the SHA-256 digest ae9a6306a205417afddd14316cc1d0d5e04a98f1be10865dce643925ee070ce2, so docs/c.txt moved to archive/c.txt. The file d.txt is added, and it is a copy of a.txt, whose digest begins b6a98d9c. Matching by name alone would have called the gamma file removed and added. You can reproduce every figure here by loading the Basics example on this page: one file unchanged, one changed, one moved, one added, none removed, and a single duplicate pair wasting 6 bytes.

Duplicates come from the same digests. Single-instance storage replaces several copies of content with one shared copy at the whole-file level, and the space a folder wastes is the size times one less than the number of copies. In the example that is 6 bytes. Files are grouped by size first, because two files of different sizes cannot match, and only groups of two or more need reading at all.

Two files of the same size and different content are never duplicates. Only the digests, and then the bytes, can say.

Retired algorithms, and why equal files still get their bytes compared

MD5 was designed by Ronald Rivest in 1991 and specified in 1992 as RFC 1321. It gives a 128-bit digest. In 2005 researchers built two certificates with different keys and the same MD5 hash, and in December 2008 a group faked the validity of web certificates with the same technique. A published attack from 2013 finds an MD5 collision in about 2^18 steps, in under a second on an ordinary computer.

SHA-1 followed. The first public collision was published on 23 February 2017. NIST deprecated SHA-1 in 2011, disallowed it for digital signatures in 2013 and said it should be phased out by 2030, and chosen-prefix attacks against it were practical by 2020. A chosen-prefix attack starts from two different prefixes and finds two suffixes that make the whole messages collide.

None of that makes these digests useless for what a folder comparison mostly does, which is to notice accidents. Nobody is trying to forge a collision in a folder of holiday photographs, and a hash that is cheap to compute and almost never wrong by chance does that job well. The GNU manual for md5sum says the chance of two files having the same MD5 by accident is vanishingly small, and also that it should not be considered secure against malicious tampering. So the page offers MD5 and SHA-1 only for matching a published value and for spotting accidents.

The remedy is old and cheap. Some deduplication systems validate a match bit for bit before they merge anything. This tool does the same for the weak algorithms: equal files are read side by side in aligned slices until the first difference, and a difference is reported with its offset. A pair with equal MD5 digests and different bytes is a collision and is shown as one, not as a duplicate.

A list the standard tools can read, and why big files are read in slices

A checksum file is a small file holding the checksums of other files, and utilities can verify a whole directory with it in one operation. In the untagged layout of the GNU utilities, each line has the digest, a space, a flag saying binary or text mode, and the file name. An asterisk marks binary mode.

File names are the awkward part. A name can contain a newline, and a newline would end the line early. So when a name holds a backslash, a newline or a carriage return, the line starts with a backslash and each of those characters is written as a backslash escape. Because the backslash itself is escaped, any other escape is reserved. The format stays unambiguous even for arbitrary names, and the tool writes no comment lines.

Big files raise a different problem. SubtleCrypto.digest takes its data as one buffer, so a browser cannot stream a file through it. The Blob.slice method returns a new blob holding a subset of the bytes, and it works in a worker, which is a background thread that keeps the page responsive. So the tool feeds its own incremental SHA-256, SHA-512, SHA-1 and MD5 code one slice at a time, 1 MiB per read. In a test run a 2 GiB file hashed this way grew the JavaScript heap by under one mebibyte.

There are limits on what a page can see. A folder chosen with the directory input includes its whole hierarchy, but the browser lists files and not empty folders, links, owners or permissions. A dropped folder has to be requested inside the drop handler itself, and the directory picker is not available in every browser. A snapshot also holds digests, not bytes, so a file that exists only in a snapshot can be matched by digest and never confirmed by comparing bytes.

Take a snapshot before a copy and another after it, and let the content say what happened.

Sources

Every source consulted for this guide. Links checked November 9, 2026.

  1. US Secure Hash Algorithms (SHA and SHA-based HMAC and HKDF) RFC 6234, RFC Editor (IETF)
  2. The MD5 Message-Digest Algorithm RFC 1321, RFC Editor (IETF)
  3. Secure Hash Standard (SHS), FIPS PUB 180-4 National Institute of Standards and Technology
  4. cksum output modes (GNU Coreutils 9.12) GNU Project, Free Software Foundation
  5. md5sum invocation (GNU Coreutils 9.12) GNU Project, Free Software Foundation
  6. File verification - Wikipedia Wikipedia
  7. Data deduplication - Wikipedia Wikipedia
  8. Cryptographic hash function - Wikipedia Wikipedia
  9. Collision attack - Wikipedia Wikipedia
  10. MD5 - Wikipedia Wikipedia
  11. SHA-1 - Wikipedia Wikipedia
  12. SHA-2 - Wikipedia Wikipedia
  13. Blob: slice() method MDN Web Docs
  14. HTMLInputElement: webkitdirectory property MDN Web Docs
  15. File: webkitRelativePath property MDN Web Docs
  16. File: lastModified property MDN Web Docs
  17. DataTransferItem: webkitGetAsEntry() method MDN Web Docs
  18. Window: showDirectoryPicker() method MDN Web Docs
  19. SubtleCrypto: digest() method MDN Web Docs
  20. Web Workers API MDN Web Docs