SHA-3 Hash Generator

In 2004 and 2005 the ground moved under hash functions. Several were successfully attacked, and serious attacks were published against SHA-1, the one the United States government had approved. NIST, the agency that writes those standards, held two public workshops and came away with a decision. It would find a new hash algorithm through an open contest, and it gave the winner a name before a single entry existed: SHA-3.

The strange part of the story is what happened next. The function that SHA-3 was meant to stand in for, SHA-2, never fell.

A contest for a spare

NIST announced the competition on November 2, 2007. By October 2008 it had 64 submissions from around the world. Fifty-one of them became first-round candidates that December, fourteen went through in July 2009, and five finalists were named in December 2010. Every entry was published, so that anyone could examine and criticize it. On October 2, 2012 the agency announced that an algorithm called Keccak had won.

Keccak came from Guido Bertoni, Joan Daemen, Gilles Van Assche and Michaël Peeters. NIST’s final report is candid that the choice was hard, and that all five finalists would have made acceptable standards. It weighed security, performance and one unusual wish: that the newcomer should complement SHA-2 instead of resembling it. On security the report notes that years of cryptanalysis had left Keccak with a huge margin, because only five of its 24 rounds had been broken by a near-collision attack. On performance Keccak was broadly competitive in software and clearly better in hardware for its size.

The wish for difference was the heart of it. “An attack that could work on SHA-2 most likely would not work on Keccak because the two algorithms are designed so differently,” a NIST computer security expert, Tim Polk, said on the day of the announcement. He called the winner an insurance policy.

The standard itself, FIPS 202, was released on August 5, 2015. NIST described it as nine years in the making. It defines four hash functions with fixed output lengths, SHA3-224, SHA3-256, SHA3-384 and SHA3-512, and two functions of a new kind, SHAKE128 and SHAKE256, whose output can be as long as the caller asks.

The sponge, in plain words

Every one of those six functions is the same machine with different settings. The machine keeps a state of 1,600 bits, which the standard arranges as a block five wide, five high and 64 deep. A line of 64 bits running through the block is called a lane, so the state is 25 lanes.

The state is divided into two parts with different jobs. The rate is where data goes in and comes out. The capacity is never touched directly by the message and never shown in the output; it is the part of the state an outsider does not get to see. For SHA3-256 the rate is 1,088 bits and the capacity is 512, twice the digest length.

Hashing starts with every bit at zero. The message is padded and cut into blocks the size of the rate. Each block is combined with the rate part of the state by exclusive or, and then the whole state, capacity included, is stirred by a fixed permutation. This is the absorbing phase, and it is why the construction is called a sponge.

The stirring is 24 rounds of the same five steps, which carry Greek letters. Theta mixes every bit with the parity of two nearby columns. Rho rotates each lane by its own fixed amount. Pi moves the lanes to new positions. Chi combines each bit with two neighbors in its row, and it is the only step that is not linear. Iota mixes a different constant into one lane each round, so that no two rounds are alike.

When the last block has gone in, the output is read straight from the rate part. That is the squeeze. If more output is wanted than the rate holds, the state is stirred again and read again. SHAKE128 and SHAKE256 simply keep going for as long as asked. That freedom has a catch that the standard spells out. Ask SHAKE for 32 bytes and then for 64, and the first 32 bytes of the longer answer are the shorter answer. A longer output extends a shorter one.

What changed from SHA-2, and what did not

SHA-2 is built on another plan. Its standard, FIPS 180-4, pads the message, appends the length of the message in bits, cuts the result into blocks of 512 or 1,024 bits, and expands each block into a message schedule that drives a compression step. For SHA-256 and SHA-512 the digest is the whole working state as it stands at the end.

SHA-2 has a known property called length extension, which NIST’s final report says the SHA-3 candidates were required to resist and earlier functions were not. The report adds the reason nobody panicked: essentially all Internet protocols, in particular those using HMAC, are designed to be secure even with a hash that has the property. HMAC is the keyed construction of RFC 2104, which wraps a hash function around a secret key. FIPS 202 says that the SHA-3 functions are designed to resist length-extension attacks, and a later NIST publication defines KMAC, a keyed function made directly from Keccak.

None of this retired SHA-2. NIST’s policy statement of August 5, 2015 says there is no need to move applications from SHA-2 to SHA-3, and that federal agencies may use either family wherever a secure hash is called for. “It doesn’t replace SHA-2, which has not shown any problem, but offers a backup,” NIST’s Shu-jen Chang said when the standard appeared.

The function on its way out is the older one. In December 2022 NIST announced that SHA-1 should be phased out by December 31, 2030, in favor of SHA-2 and SHA-3. Meanwhile the spare found work of its own. The key-establishment standard that NIST published on August 13, 2024, FIPS 203, uses SHA3-256, SHA3-512, SHAKE128 and SHAKE256 throughout.

One byte apart: Keccak and SHA-3

There is a trap between the contest and the standard, and it is one byte wide. FIPS 202 says that its four hash functions differ slightly from the versions of Keccak proposed for the competition: two extra bits, 0 and 1, are appended to every message before the padding. SHAKE gets four bits instead, all ones. The suffix lets the two kinds of function, and any future variants, give unrelated answers for the same input.

In bytes it looks like this. The standard fills each byte from its lowest bit, so the bits 0 and 1 followed by the first padding bit make the byte 0x06. For SHAKE the same sum gives 0x1F. The original Keccak, with no suffix, starts its padding with 0x01. In all three the block ends with 0x80. So the first padding byte is 0x06 or 0x01, and nothing else differs.

Not all software follows the standard. The best-known case is Ethereum, whose specification names the Keccak-256 hash function as per version 3 of the winning entry to the SHA-3 contest, not the final SHA-3 specification. A program that says Keccak-256 and a program that says SHA3-256 will disagree about every digest.

You can watch the byte matter. In the tool above, clear the text box so that the input is empty. SHA3-256 gives a digest that begins a7ffc6f8 and ends 434a, the value in NIST’s own example file. Now switch on the legacy Keccak-256 choice and select it. The digest begins c5d24601 and ends a470, the value an Ethereum proposal gives for the hash of empty data. Same state, same rate, same 24 rounds; one byte of padding.

Then type abc. The step view has 123 steps: the empty sponge, one absorb, 24 rounds of five mappings and the squeeze. Go to the second step and count. Absorbing sets 13 bits: ten in the three letters, two in the byte 0x06 and one in the closing 0x80. One hundred and twenty steps later 786 of the 1,600 bits are set, and the digest begins 3a985da7.

What a digest can and cannot prove

A digest answers one question well. If two files give the same SHA3-256 value, they are the same bytes, with a confidence the standard puts at 128 bits of resistance to collisions. Change one bit and the value changes beyond recognition. A match of the first few characters proves nothing, which is why a comparison should be all or nothing.

It answers no other question. A checksum cannot say who made a file. If it is published beside the download, whoever can replace the file can replace the checksum, so its worth depends on where you got it. It is also sensitive in ways that surprise people. The same paragraph saved with Windows line endings has one more byte per line than with Unix ones, and a different digest.

Nor is a fast hash a way to keep passwords. NIST’s identity guidelines require stored passwords to be salted and hashed with a password hashing scheme that takes a cost factor, so that each guess is expensive for an attacker. SHA-3 is designed to be quick.

Checksum files have a plain convention worth knowing: one line per file, the digest in hexadecimal, a space, a one-character flag and the file name. The GNU tools write that layout and read it back. Whether the numbers in it mean anything is still a matter of trust, and the hash was only ever half of that bargain: it can show that nothing changed, never that the original deserved belief.