your storage is lying to you

A four-state RAID model rebuilt an untouched block as 0xeb instead of its original 0x35. Two combinations of old or new data and old or new parity gave that wrong answer. The script used no drive and injected no physical faults.

I wanted to pin down what goes wrong between two writes that belong to one update. The model script uses D₁=0x35, D₂=0x9c, and D₃=0xa7. Their XOR parity is 0x0e. Changing D₂ to 0x42 requires new parity of 0xd0. Let each of those two writes persist or stay old, then reconstruct D₁ from the surviving blocks. Old data with old parity returns 0x35. New data with new parity also returns 0x35. Either mixed pair returns 0xeb.

The script says nothing about crash likelihood: it works with whole blocks and has no device timing or physical power-cut test. Even within those limits, preserving each completed write is too weak a promise for an array: recovery can need several writes to agree with one another. D₁ was never part of the update, yet its reconstruction depends on how that update finished.

rebuilding the block we never changed

Updating one block in a RAID4 or RAID5 stripe also means updating the parity P shared by its data blocks. The read-modify-write starts by reading the old D₂ and P. Calculate P' = P ⊕ D₂ ⊕ D₂', then write D₂' and P'. The new data and its parity reach storage in separate writes. A power failure can leave either one unfinished.

Suppose the new D₂' survives and the old P remains. All the data blocks can still read back successfully. Without a scrub or consistency check, the parity mismatch can stay hidden until a degraded read or rebuild uses it.

Now lose the disk holding D₁. The rebuild calculates D₁* = P ⊕ D₂' ⊕ D₃. Substituting the old parity gives D₁* = D₁ ⊕ D₂ ⊕ D₂'. The old and new second blocks don't cancel out. Their difference gets folded into the reconstructed first block. That is the RAID write hole, and it is exactly what the two mixed cases in the script expose.

What can recovery use to finish the update in those two bad states? Protected controller cache can retain unfinished writes, or a RAID journal can save the information needed to complete them. RAID-Z writes a new logical block as a full stripe with parity. ZFS copy-on-write leaves the old committed state available until the new blocks are committed, so it avoids overwriting a live stripe a piece at a time.

The phrase "copy-on-write" alone leaves this question unanswered. The Btrfs RAID56 documentation records write-hole limitations. I would read the parity update protocol before trusting either design. Btrfs and ZFS differ there, and those steps decide which old/new combinations a crash can leave behind.

what did the write acknowledge?

The four-state model assumes that a write marked persistent really survives. An application has to establish that through the stack. An ordinary buffered write() can return after copying bytes into the kernel's page cache. Reading them back may use that same RAM. A byte-for-byte comparison can pass before the drive has received anything, so it cannot by itself distinguish a cached write from a persistent one.

Finding a SQL injection in rice.edu at 17 got me thinking about trust boundaries. The question I find useful here is what the next layer actually agreed to do. Which failures will an acknowledged update survive? That is durability. Will a later read return the right bytes? That is integrity. Neither follows from a successful call to write.

A healthy drive and a correctly implemented filesystem can still lose bytes held only in the page cache. Sending them to the drive may leave them in its volatile write cache instead. To get past that cache, the OS needs a flush or force-unit-access operation that the device honors. A controller with protected cache can acknowledge persistence while the bytes are still cached: keeping them through power loss already fulfills that promise. Ordinary RAM cannot.

syncing the contents leaves the name to do

On Linux, the application normally asks for persistence with fsync(fd). It sends the file's data and metadata through to storage and waits for the device to report completion. A successful write earlier doesn't spare us checking this return value. The failure may only turn up during writeback, when the cached bytes finally have to leave RAM.

For a file replacement I would follow both objects that change. Write a temporary file, sync it, rename it over the destination, then sync the parent directory. At the file sync, the contents are covered but the later rename hasn't happened yet. Syncing the directory makes that name change durable. An atomic rename makes the name change indivisible for readers; it leaves the power-loss question to the sync protocol.

Follow that request below the filesystem and it still ends at a device's completion report. The filesystem and block layer have to arrange their writes and cache operations correctly. If a controller acknowledges a flush while its only copy sits in unprotected RAM, their correct ordering cannot save the update after power loss.

PostgreSQL's corruption guide lists disabled syncing and storage that ignores write barriers among the causes of corruption. That is why I would trace the sync calls and acknowledgments before deciding where a lost write went wrong. Delayed allocation and write ordering are filesystem questions. Whether the controller kept its promise depends on its cache policy. The damaged file alone doesn't identify the culprit.

FUA (Force Unit Access) attaches the persistence requirement to a particular write. Its completion must wait for that write to reach nonvolatile storage, which may include suitably protected cache. "Bypass every cache" would be the wrong requirement. A flush covers the relevant writes already cached and makes them persistent before acknowledging the command. A device's write-cache properties determine whether syncing a file needs that command.

If the hardware supports disabling its volatile write cache, that removes one place bytes can disappear and may cost performance. Faulty controllers and faults elsewhere remain possible. Waiting for persistence takes time; journals and group commit let updates share a flush, and a separate journal device can change the cost. I would check which recovery records the batch has persisted before counting any transaction in it as durably acknowledged.

0xeb is a perfectly readable byte

The script can call 0xeb wrong because it keeps 0x35 as the expected value. An actual reader needs its own evidence. A read can succeed with wrong bytes after a firmware defect, an ECC miscorrection, or damage elsewhere in the I/O path. Retrying only calls that report errors won't catch this silent corruption. A checksum stored outside the component that failed gives the reader something to compare against.

Magnetic media and flash develop errors, many of which the device's error-correcting codes recover. An unrecoverable read error at least tells the caller to seek another copy or reconstruct from the array. Silent corruption removes that warning. I care about the distinction because a recovery path that only runs after a reported error leaves successful-but-wrong reads alone.

Prabhakaran and colleagues tested partial failures in IRON File Systems (2005), injecting them beneath ext3, ReiserFS, JFS, and NTFS. The implementations they tested had inconsistent failure policies and bugs in some recovery paths. Those are findings about those tested implementations, but the question carries over: what happens to one bad block while the rest of the drive keeps working?

A drive specification won't answer that recovery question. Some quote an unrecoverable-error figure of 1 in 10^14 bits read, others 10^15 or better. I can convert 10^14 bits to about 12.5 TB to get a sense of scale. What I cannot get from that figure is a failure date, or the independent error probabilities I would need to calculate rebuild risk. If a RAID5 disk is already gone, a surviving disk that cannot read a needed block may leave the array unable to reconstruct the affected data.

where did the expected value come from?

For a checksum to expose the wrong value, it has to be computed while the value is still right. Flip a value in RAM first and storage can preserve a matching pair of damaged data and checksum. A flipped pointer might crash the program; an ordinary data value could just look plausible. Computing another checksum on disk afterward won't recover what was lost in RAM.

That puts RAM inside the integrity problem. The fault could be a radiation-induced upset or a persistent hardware problem; the rates depend on the hardware and environment. With common SEC-DED ECC, a single error within a protected word can be corrected and a double error detected. Larger patterns can exceed those guarantees and need stronger protection. I want ECC on a server holding data I care about, with monitoring for the errors it reports. PostgreSQL's reliability documentation assumes ECC RAM or better protection.

Before relying on repair in ZFS, Btrfs, Ceph, or GFS, I would ask where the checksum lives and how the system finds a good copy. All four check returned data and can seek a valid copy where their redundancy permits. Calling them all cryptographic Merkle trees skips the differences in their checksum layouts and repair paths.

Even a Merkle tree leaves me with one value to trust: its root. The leaves hash data blocks, each parent hashes its children, and the root commits to everything below it. Change a block and verification along that path fails against the trusted root. A noncryptographic checksum may be enough for accidental corruption. To check deliberate replacement, I also need a suitable hash and a root the attacker cannot replace along with the data.

Suppose two mirrored copies disagree. With just those copies, which one should a reader trust? An independently stored checksum lets it test both, return a copy that verifies, and use that copy to repair the other. If neither verifies, the mirror cannot recover the block. Scrubbing does this checking for blocks that ordinary reads haven't touched recently, while there may still be a good copy to repair from.

Some checks happen during the block transfer itself. T10 protection information, described in the Linux data-integrity documentation, has an 8-byte field with a guard tag, an application tag, and a reference tag. DIF carries protection information with the block; DIX allows a separate representation in host memory. Supporting components can check it along that path. Filesystem checksums cover another part of the journey.

I would follow any expected value back to where it was computed. A filesystem checksum can catch wrong bytes returned by a device, and a database can add page checks. TLS checks the transfer between its endpoints, leaving later disk storage outside that check. Checking end to end can expose faults missed by checks at individual hops, provided the expected value was right when the check began.

The four-state RAID calculation settles one small question: mixed versions can corrupt reconstruction of an untouched block. It leaves the storage promises to test separately. For durability, I would cut power and compare recovered records with the writes acknowledged as durable. For integrity, I would damage bytes on the read path and check that the system reports an error or recovers a copy that verifies. Starting up again would tell me too little about either one.