A NAS owner’s account of a firmware bug that dropped a healthy disk on every boot, and how I used Claude Code to reverse-engineer the fix out of the closed-source firmware.

A quick note on this post before we start: the NAS is mine, the problem was mine, and the tests (and the swearing) were mine. The deep reverse-engineering, disassembling the firmware’s own binaries and tracing data flow instruction by instruction, was done by Claude Code, which I drove and second-guessed throughout. So when you read “Claude found,” that is literally what happened. When you read “I rebooted and it was still broken,” that is me being the reality check. It was a genuine back-and-forth, and it is worth being honest about who did which part.

The symptom

A four-bay NAS. Four 10 TB drives in RAID 5. A little red light on bay 2 that would not go away.

That red light is where it started, and for over a week straight it refused to turn blue. Replace the drive: still red. Force the array back to full health by hand: still red. Reboot: red again, and now the array is missing a disk. Every single reboot, the same:

md1 : active raid5 [4/3] [U_UU]

The second disk, gone. Not failed: the drive is physically fine, SMART passes, the cable is clean. Just not in the array. Re-add it by hand and it resyncs in a couple of minutes, back to [4/4] and a happy-looking dashboard. Reboot: [4/3] again. And the light stays stubbornly red through all of it.

A degraded RAID 5 has zero redundancy. One more hiccup and the whole 27 TB volume is gone. So “just re-add it after every boot” is not a fix. It is a countdown.

This is the story of chasing that little red light all the way down to individual CPU instructions inside the NAS’s closed-source firmware, through a long string of dead ends and confidently-wrong theories, and finally landing a fix that a cold reboot actually proved.

How it started: the third dying disk, and a replacement that would not take

First, one fact about this NAS that matters enormously later: its root filesystem is a RAM disk, rebuilt from a firmware image on every boot. Only a couple of tiny flash partitions and the data array itself survive a power cycle. Everything else is reconstructed from scratch each time the machine starts. Hold onto that.

The drives in this box were old, about eight years, and they had started dying the way old drives do: one at a time. This was the third failure in about seven months. The first two replacements had been completely routine: pop out the dead drive, slot in a new one, and the firmware quietly rebuilt the array on its own while I got on with my day. By the third one it felt like muscle memory.

The third one did not cooperate.

It began, as always, with a red light on bay 2, the NAS’s way of announcing a failing disk. Routine. Out with the old drive, in with a brand-new one straight out of the box. Except the new drive did not seat on the first try. It took a couple of firm presses to get it home, and in the process the disk may well have connected and disconnected a few times, an “in, out, in, out” as it seated, though honestly that part is a guess. Eventually it clicked in, the bay door closed, and the light stayed red.

Worse, the dashboard would not report a rebuild ETA. Auto-rebuild was switched on, the drive showed as healthy in the interface, but nothing was actually happening. No progress bar, no estimate. Just a red light and an array quietly running one disk short.

When I pointed Claude at the logs, the story started to come together. During that fumbled insertion, the controller had briefly lost the drive:

sd 1:0:0:0: [sdb] Read Capacity(10) failed: hostbyte=DID_NO_CONNECT
sdb: unable to read partition table
sdb: detected capacity change from 10000831348736 to 0
sdb: partition table beyond EOD, enabling native capacity

For a moment the controller saw a zero-capacity, unreadable disk. That is the classic signature of a drive whose reported size collapses mid-insertion: the connect-and-disconnect churn from the awkward seating made the controller briefly misread the disk, and once the reported capacity flickered to zero, the kernel could no longer make sense of the drive’s partition layout (“beyond EOD” just means “the partition table points past what I currently think the end of the disk is”). The drive itself was brand new and perfectly healthy. This was a bad handshake, not a bad disk.

At the time, honestly, it looked like the NAS had simply run out of memory and killed whatever process was supposed to bring the new disk into the array. The job that should have joined and rebuilt it just seemed to vanish, leaving a red light and no ETA. On a box whose entire brain lives in RAM, “a background job quietly died” and “we ran out of memory” are almost impossible to tell apart from the couch, so that was my first theory.

It turned out to be wrong, and this is the actual mechanism Claude pinned down: the firmware’s rebuild is a one-shot. When it detects a new disk it fires a single mdadm --add command and then forgets about it. That single command fired at the exact instant the disk was mid-glitch, failed against the not-ready device, and was never retried. Nothing was killed, and nothing ran out of memory. The handler simply ran once, at the worst possible moment, and washed its hands. Re-seating the tray and rebooting did not re-arm it, because as far as the firmware was concerned, it had already done its one job.

That single failed one-shot is the seed of everything that follows.

Fixing the easy part: the manual rebuild

Getting the array back to four disks was, ironically, the easy part. The vendor firmware uses stock mdadm under the hood, so you can do by hand exactly what its rebuild would have done, and it will adopt the result without complaint. Here is what actually worked, with Claude walking me through each step and checking the result before moving to the next.

First, replicate a healthy disk’s partition layout onto the new one so the two match sector for sector. The only tools on the box were gdisk, mdadm, and blockdev, so this was done in place, on the NAS itself:

# back up a good disk's partition table first, as a safety net
printf 'b\n/tmp/sda-gpt.bak\nq\n' | gdisk /dev/sda

# then recreate the four partitions on the new disk at the identical
# start/end sectors and type codes, then write them out
gdisk /dev/sdb        # interactive: create the 4 partitions to match, then 'w'

Writing a fresh partition table this way also overwrote whatever the glitch had left on the disk, so no separate wipe step was needed. Then add the new partitions into the arrays, the small system mirror first as a low-risk canary, the big data array second:

mdadm --add /dev/md0 /dev/sdb1     # system RAID-1 mirror: resyncs in seconds
mdadm --add /dev/md1 /dev/sdb2     # RAID-5 data array: joins as spare, promotes to 'recover'
cat /proc/mdstat                   # watch the rebuild progress and ETA

The firmware happily displayed “Rebuilding… Time: 883 min” and adopted the hand-built member as its own. When it finished: md1 [4/4] [UUUU]. All four disks present, all in sync. The dashboard declared everything healthy.

The red light stayed on.

And the very first reboot dropped the disk right back out, [4/3] again. So the easy part had fixed the array’s contents but not the disease: the machine still believed, somewhere, that this disk did not belong. That belief survived the manual rebuild, survived the dashboard insisting all was well, and quietly re-degraded the array on every single power cycle, red light and all.

The real problem: it will not survive a reboot

Here is what made this genuinely hard: there were two independent bugs stacked on top of each other, and the first one was corrupting the evidence for the second.

Every reboot, the freshly-rebuilt disk was missing again. That raised an obvious question with a maddening answer. The question: was something failing the disk at shutdown, or was boot just not picking it up? The answer was both, and each half made the other harder to see.

Here are the dead ends in order, because the dead ends are the point.

Dead end #1: “the inventory file is stale”

The firmware keeps an XML inventory of the disks in the volume. After the botched insertion, that file listed three disks; the new one had never been inventoried, because the one-shot that would have done it failed.

Beautiful. Obvious. Claude edited the inventory to add the missing disk, I rebooted, and… still [4/3]. The disk was excluded anyway.

That should have been a louder alarm than it was. The lesson we should have taken there: editing a file that “logically must” drive the behavior means nothing until a test agrees. We filed it as “wrong field, keep looking” and moved on. (It was actually much more wrong than that, but we would not learn how wrong until many dead ends later.)

Dead end #2: “a hotplug script is failing the disk”

If boot was not the culprit, maybe shutdown was. There was a promising suspect: a hotplug script, triggered by device events, that runs mdadm --fail and --remove on disks. It even had a guard flag it was supposed to check, and Claude found that shutdown never set that flag. Clean theory: at power-down, a stray “device removed” event fires the script, it fails the disk out of the still-assembled array, and the array stops with three members.

Rather than just believe that, we tested it. I had Claude write a watcher, and this part I actually love: the root filesystem is a RAM disk that gets torn down during shutdown, so a normal log would vanish. The watcher instead logged to one of the tiny persistent flash partitions, polling disk state every 0.3 seconds straight through the shutdown sequence, so its record survived to be read after the next boot. (It also taught us that this NAS’s BusyBox shell was built without arithmetic expansion, so every counter had to be done the long way. Embedded Linux keeps you humble.)

I ran the watcher, then rebooted. The result: the hotplug flags never appeared. The script never ran. The theory was refuted, by our own instrument, which is the good way to be wrong.

But the watcher caught something better. It showed the array being stopped while all four disks were still present, and the kernel singling out the new disk for special treatment, exporting it separately, before batch-unbinding the other three. Something was deliberately kicking that one disk out, right before the stop. And it was not the hotplug script.

Breakthrough #1: disassembling the shutdown binary

The thing stopping the array at shutdown was a closed-source vendor binary. So Claude pulled it off the box and disassembled it.

Buried in the shutdown path was the smoking gun. For each disk, the binary decides between two code paths:

strstr(used_device_string, disk_name)   // is this disk in the volume record?
   found     ->  clean stop   (mdadm --stop)
   not found ->  FAIL-REMOVE  (mdadm -f /dev/sdX2 ; -r /dev/sdX2)

There it is. At shutdown, the binary checks each disk against the volume record’s used_device field. Disks that are listed get a clean stop. Any disk not listed gets explicitly failed and removed, its RAID superblock scribbled with “I failed,” just before the array comes down.

The new disk was not in used_device. So every shutdown: fail-remove. Which corrupts its superblock. Which is why boot then legitimately refused it (a superblock that says “failed” is not one mdadm will silently reactivate). Which is why we had been staring at a boot symptom that was actually a shutdown crime.

The fix for this half was to edit the live volume record so used_device lists all four disks. That one field is exactly what the shutdown binary checks:

# /var/www/xml/used_volume_info.xml
# before: the new disk (sdb2) is missing, so shutdown fail-removes it
<used_device>/dev/sda2 /dev/sdc2 /dev/sdd2 </used_device>
# after: all four listed, so shutdown stops it cleanly and leaves it intact
<used_device>/dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2 </used_device>

We did it carefully: staged the edited file, diffed it, and I copied it into place myself rather than letting anything overwrite. Then the watcher-plus-reboot test.

The watcher came back clean: no fail-remove, no “Disk failure” message, all four disks unbound equally with matching event counts. The shutdown eviction was dead. Verified, not hoped.

Victory?

Reboot: [4/3].

Still. The disk survived shutdown clean this time, and boot still left it out. There was a second, independent bug at boot, and only now, with the shutdown crime finally stopped, was it possible to see it in isolation.

Dead end #3: the boot fix that edited a field nothing reads

The boot-time assembler is another vendor binary. Claude disassembled its device-list builder and found what looked like the exact gate:

bt   mask, slot        ; is this slot's bit set?
jae  skip
cmp  device_name, 0    ; and does it have a name?
je   skip
... include /dev/sdX2 in the assemble command

And elsewhere in the same binary, code that read a scsi_mapping number from a persistent config file called hd_info.xml. The value there was 13, which in binary is 1101, exactly slots 0, 2, and 3: the three disks that kept assembling, with the new disk’s slot (bit 1) clear. It was a perfect match. Change 13 to 15 (all four bits), add the missing serial field, reboot.

Reboot: [4/3]. Again.

That failure finally forced the right question. Fix after fix had targeted config fields with airtight-sounding logic, and every one of them turned out to be inert. Why?

The pattern behind the wrong turns was always the same: Claude kept connecting two separately-observed facts with a causal link it never actually traced. “The builder tests a mask” plus “the binary reads scsi_mapping” became “therefore the mask is scsi_mapping.” But those were two different pieces of code touching two different pieces of memory, welded together because the story was tidy and the number 13 was such a satisfying match. A value that mirrors the symptom is not necessarily the value that causes it.

Time to stop guessing.

The reset: stop inferring, start proving

The failure mode was clear. On a stripped binary, following data flow across functions is slow and painstaking, so Claude had been substituting plausible inference for actual tracing, and rounding “plausible” up to “confirmed” in how confidently it acted. Every wrong theory had cost me a real reboot.

So I asked Claude to change method. Instead of one narrative traced by hand, I asked it to split the binary forensics into independent, self-contained questions and run them in parallel, with one ironclad rule: every claim had to come with the actual instruction-level evidence, and each one got re-verified against the raw bytes before we acted on it. Splitting the work up sped up the legwork; it did not outsource the verification.

Four questions went out at once:

  1. Where does the assemble mask actually come from? Trace it to its real source.
  2. Does the assembler even run mdadm as a subprocess, so that capturing its command would even be possible?
  3. Does it log that command anywhere we could read after a boot?
  4. Where does the per-slot device name really get populated?

And in parallel, the one thing that needed the live machine: a full side-by-side dump of all four disks’ RAID superblocks.

What came back

The superblock comparison was the first clean signal. All four members were structurally identical and healthy: same event count, same consistent role map, all clean. The only oddity about the new disk was that it carried the highest internal device number, a scar from being re-added so many times. Nothing about its metadata justified exclusion. That ruled out the entire “bad superblock” family of theories and pointed straight at the assembler’s own logic.

Then the disassembly findings landed, and they were brutal about the earlier work:

  • That scsi_mapping config field we edited? A dead store. The binary parses it, writes it to a memory slot, and never reads that slot again. One write, zero reads, confirmed by grepping the whole binary. The boot fix had changed a value the program literally never looks at. (Claude verified this one byte by byte, precisely because it overturned something we had been sure of.)
  • That inventory field we edited back in dead end #1? It flows into a buffer the builder never touches. Also inert. Those “obvious” config edits had been aimed at fields the code simply does not read.
  • The real gate is a live string match: the builder includes a slot only if the disk’s name, from a live hardware scan, matches an entry in a “volume info list” built from a different file entirely.

So which file, and which field? A second, focused trace into the vendor shared library that builds that list revealed the true membership decision. The assembler reads its volume list from a RAM-disk XML file, and branches on the volume’s status:

  • if the volume is flagged degraded, it gates on a scsi_mapping bitmask (a different scsi_mapping than the dead one, this one is live) plus a serial-number match;
  • otherwise it gates on a plain string match of the device list against the live-scanned disk names.

Two different scsi_mapping values in two different files: the dead one sat in the persistent hd_info.xml I had wasted a reboot editing, and the live one lived in used_volume_info.xml, the RAM file that gets regenerated from scratch on every boot. I had been editing the fossil.

The “aha”: it was a loop the whole time

Step back and the shape finally resolves. The RAM-disk file the boot assembler trusts is re-seeded at every boot from the persistent records, and those persistent records are written to reflect whatever the array currently looks like.

That closes a vicious circle:

  degraded array (3 disks)
        |
        v
  persistent records get written as "3 disks"
        |
        v
  next boot re-seeds the RAM file as "3 disks"
        |
        v
  assembler includes only 3 disks  ->  degraded array
        |
        +--------------  (repeat forever)  --------------+

  meanwhile: manual re-add fixes the ARRAY but not the RECORD,
  and the shutdown fail-remove keeps re-corrupting the disk,
  so the loop always reconverges on "3."

The new disk was never in the records, because the one-shot that would have added it failed on day one. Every layer downstream faithfully propagated “3 disks” forever. The config edits failed not just because they hit dead fields, but because even the right field would have been overwritten on the next boot; the RAM file that matters is regenerated, not edited.

And here is the beautiful part: the boot half never needed a fix at all.

The shutdown fail-remove was the pump keeping the loop alive. Stop the fail-remove, which the used_device edit already did, and the disk stays a clean, in-sync member across shutdown. A clean member means the persistent records regenerate as four disks. Four-disk records mean the next boot re-seeds a four-disk RAM file. Which means the assembler’s string-match finds all four. The loop does not just stop; it reconverges on “4” and stays there.

There is one more detail that makes this self-healing, and it is the thing that finally explained why the boot half needed no fix. The boot assembler has a switch. If the volume record says the array is degraded, it uses a strict path: the scsi_mapping bitmask plus a serial-number check, which is exactly the path that kept dropping the new disk. If the record says healthy, it uses a forgiving path: a plain match of the disk’s name. So the instant the array survived one shutdown intact and the record flipped to healthy, boot started taking the forgiving path, found the disk by name, and the strict bitmask that had been excluding it never ran again.

The entire fix was one edit (the shutdown-side used_device, already applied) plus one manual re-add to seed a healthy four-disk state for the records to capture. The clever boot-side binary patches were not just wrong; they were unnecessary.

The verified fix

The bar for “fixed” was non-negotiable: a cold reboot that comes up whole, with no manual re-add afterward.

Before rebooting, Claude snapshotted every relevant record to a persistent flash log, so that if it failed we would know exactly what fed the decision instead of guessing again. Every record, the RAM file, the persistent config, all four on-disk copies, now read four disks, consistently.

Then: reboot. No edits. No pre-emptive re-add.

$ grep -oE '\[[U_]+\]' /proc/mdstat | head -1
[UUUU]

$ dmesg | grep -c 'bind<sdb2>'
1

$ cat /proc/mdstat
md1 : active raid5 sda2[4] sdd2[3] sdc2[5] sdb2[6]
      29286720960 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

[UUUU]. All four. And dmesg shows the new disk bound at boot; it was in the assembler’s command from the very first moment, not re-added afterward. The loop was broken, and it stayed broken on its own.

Fixed. Verified, not hoped.

Root cause, in one paragraph

A brand-new drive glitched during a fumbled hot-insertion, so the firmware’s one-shot, no-retry rebuild handler failed to enter it into the volume’s bookkeeping. From then on, a self-perpetuating loop kept the array degraded: a shutdown routine explicitly failed-and-removed any disk absent from the volume record (corrupting its superblock), so boot legitimately excluded it, so the records regenerated as “three disks,” so the RAM file the boot assembler trusts was re-seeded as “three disks,” forever. Manual re-adds fixed the array but never the record. The cure was to stop the shutdown fail-remove (one edit to a used_device list) and seed a healthy four-disk state once; from there the records self-maintain and every subsequent boot comes up whole.

What I took away from having an AI reverse-engineer my NAS

A value that matches the symptom is not the same as the value that causes it. The scsi_mapping = 13 that mapped perfectly onto the three surviving disks was a mirror, not a lever; the code wrote it from the array state, so of course it matched. That coincidence cost a reboot.

On a stripped binary, “plausible data flow” is a hypothesis, not a finding. The confident wrong turns, and there were more of them than this write-up dwells on, spread across several sessions, were all the same mistake: two real observations, welded together with a causal link nobody had actually traced. The hop that got skipped was always the one that was false. The fix that held was the one where the whole chain got read end to end, with a live test to back it. Claude is fast and tireless at this, but left to run on inference alone it will hand you a beautiful, confident theory; the discipline that made it genuinely useful was forcing every claim back to the raw bytes, and back to a reboot that either agreed or did not.

Instrument, do not argue. The single best move in the whole saga was a dumb shell loop logging to persistent flash across a shutdown. It refuted our favorite theory and handed us the real one in the same run. When you can watch the machine do the thing, do that instead of reasoning about what it must do.

Layered bugs mask each other. As long as the shutdown fix was not in place, the boot behavior was a consequence, not an independent fact. The second bug only became visible, and, as it turned out, moot, once the first was truly gone. If you cannot reproduce a symptom cleanly, suspect that something upstream is still moving under you.

Sometimes the second half of a problem evaporates when you fix the first half correctly. We spent real effort dissecting a boot-time assembler that, it turned out, needed no patch at all.

And keep the safety net regardless. The data was fully backed up before any of this, and a health monitor plus an auto-recovery script stayed in place the entire time. A “verified” fix on a closed-source appliance you do not fully control still deserves a net under it.

The whole thing took a disassembler, a shell loop writing to flash, several parallel binary-forensics passes, a pile of wrong theories, and one very satisfying [UUUU]. The permanent fix was a single line in a config file; the hard part was the marathon of finding out which line.

If this happens to you (in plain English)

Not everyone who owns one of these boxes wants to read assembly language. So here is the whole thing without the jargon: what went wrong, how to tell if it is happening to you, and, importantly, what to actually do about it.

What was going on, in one breath: the NAS keeps a private little “guest list” of which drives belong to the storage pool. When I swapped the failed drive, the new one had a bumpy install and never got written onto that guest list. From then on, two things fed each other: at shutdown the NAS kicked any drive that was not on the list back out of the pool, and at startup it only let the drives on the list back in. So every reboot, the new drive was left out, even though the drive itself was perfectly fine, and even though the dashboard cheerfully said everything was OK.

Signs you have the same problem:

  • You replaced a failed drive, and the bay’s light stays red even though the dashboard shows the disk as healthy.
  • After the swap, the NAS never shows a rebuild progress bar or ETA, or the rebuild just never starts.
  • Or: you get the array healthy, but every reboot it goes back to “degraded” and drops the same drive.
  • The dropped drive is physically fine (it passes a health check). It just will not stay in the pool.

Do this first, before anything else: back up your data. A storage pool running one drive short has no safety margin left. It looks like it is working, but if any other drive so much as hiccups before this is fixed, you can lose everything. Copy your important files to an external USB drive or another computer now. This is the step people skip and regret.

Then, here is exactly what we did that actually fixed it, that you could do too. I want to be clear about what this is and is not. This is a software fix, done over SSH, with the array left running; we did not fix it by re-doing the physical install. And it is specific to this style of firmware: the file paths below will differ on other brands, but the idea carries over, which is to get the new disk back onto the pool’s internal membership record so it stops getting evicted.

  1. Get the pool back to full health. If the replacement disk is blank, whether fresh out of the box or scrambled by a bad insertion like ours, you first have to give it the same partition layout as the other disks, because the array is built from partitions (like sdb2), not whole disks. Our box had gdisk, mdadm, and blockdev, but no parted or sgdisk.

Start by reading the exact layout off a healthy disk so you can copy it precisely:

gdisk -l /dev/sda        # print a good disk's partition table

On ours that printed four partitions. The layout is not in disk order (partitions 1 and 4 sit before 2, and 3 is at the very end), so do not guess it, read it. These sector numbers are ours; use whatever your own healthy disk reports:

Number   Start (sector)    End (sector)   Size        Code   Role
   1             2048          4196351     2.0 GiB     8200   system mirror (md0)
   2          6293504      19530774527     9.1 TiB     0700   data array   (md1)
   3      19530774528      19532873694     1.0 GiB     0700   reserved
   4          4196352          6293503     1024 MiB    0700   firmware scratch

Back up that good disk’s partition table first, then recreate the same partitions on the new disk. gdisk is interactive, but you can drive it with a script: for each partition it wants a number, a start sector, an end sector, and a hex type code, and it defaults the type to 8300, so you must type the real code (8200 or 0700) every single time or it comes out wrong:

# 1. back up the healthy disk's GPT as insurance (reads sda, writes a file)
printf 'b\n/tmp/sda-gpt.bak\nq\n' | gdisk /dev/sda

# 2. recreate the four partitions on the new disk with the SAME sectors and codes.
#    each partition is: n, number, start, end, typecode; then w, Y to write it out.
printf 'n\n1\n2048\n4196351\n8200\nn\n2\n6293504\n19530774527\n0700\nn\n3\n19530774528\n19532873694\n0700\nn\n4\n4196352\n6293503\n0700\nw\nY\n' | gdisk /dev/sdb

Then confirm the new disk’s partitions came out identical to the healthy one, before you touch the array:

blockdev --rereadpt /dev/sdb                  # re-read the fresh partition table
grep -E ' sd[ab][1-4]$' /proc/partitions      # sdb* block counts must match sda*

Only once they match, add the partitions back into their arrays, the small system mirror first as a deliberate canary (if the firmware is going to balk at a manual add, you want to find out on the 2 GiB mirror, not the 27 TB data array), the data array second:

mdadm --add /dev/md0 /dev/sdb1     # system mirror: resyncs in seconds
mdadm --add /dev/md1 /dev/sdb2     # data array: rebuilds over several hours
cat /proc/mdstat                   # wait until both read [UUUU], fully in sync

(If the disk already has its partitions and simply keeps dropping out after each reboot, skip the gdisk step and just re-add. Many of these NASes also ship an auto-repair script that does the re-add for you.)

  1. Put the disk back on the pool’s “guest list” so shutdown stops evicting it. This is the actual cure. The firmware keeps a volume record listing which partitions belong to the pool; ours had the new disk missing, which is what made the shutdown routine kick it out on every power cycle. Add it back:
# /var/www/xml/used_volume_info.xml   (exact path varies by model)
# change the used_device line from three disks:
<used_device>/dev/sda2 /dev/sdc2 /dev/sdd2 </used_device>
# to all four, including the disk that kept vanishing:
<used_device>/dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2 </used_device>

Edit a copy, compare it against the original, then put it in place. Do not blind-overwrite anything. used_device is the one field that actually stops the eviction, but while you are in there it is worth making the rest of the record consistent (the device name list, the scsi_mapping bitmask, and the disk’s serial line), which is what the firmware would have written itself had the disk enrolled normally. The full record, field by field, is in the appendix.

  1. Reboot and confirm it comes up whole on its own (cat /proc/mdstat shows every member, with no manual re-add needed). Once it survives one clean cycle, the firmware keeps its own records in step from then on, and the loop stays broken. In our case it has come up complete on every reboot since, with nothing done by hand.

That is the whole tested fix. Everything else we tried along the way (editing other config files, chasing a hotplug script) did nothing, so it is not in this list.

One honest note on prevention: I would love to tell you how to stop this from ever happening, but we fixed it after the fact, in software, and never re-ran the physical install, so I cannot claim a tested prevention routine. My best guess, and it is only a guess, is that a firmer, unhurried insertion, ideally into a powered-off bay, would have let the firmware’s single automatic rebuild attempt succeed the way it did on the two previous swaps. Treat that as a hunch, not a proven step.

And whatever else you do, keep a backup. RAID protects you from a drive dying. It does not protect you from a firmware quirk, a fumble, a fat-fingered command, or two drives failing close together. A backup does. If you are staring at a red light, a “healthy” dashboard, and a drive that keeps vanishing, and the steps above have not cleared it, that is a completely reasonable moment to back up your files and hand the box to someone who does this for a living. There is no shame in it. The bug in this story took a small forensic campaign to pin down; you are not expected to reverse-engineer your appliance’s firmware to keep your photos safe.

Appendix: the volume record, field by field

For anyone who wants to understand, or safely edit, the record the fix touches, here is the full structure of used_volume_info.xml, with the real serials and array UUID swapped for placeholders. This is the RAM-disk file the shutdown routine reads to decide who to evict, and the boot assembler reads to decide who to include.

<volume_info>
  <item>
    <volume>1</volume>
    <raid_mode>raid5</raid_mode>
    <file_type>ext4</file_type>
    <raid_status>0</raid_status>
    <device>sdasdbsdcsdd</device>
    <mount>/dev/md1</mount>
    <used_device>/dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2 </used_device>
    <dev_num>4</dev_num>
    <scsi_mapping>15</scsi_mapping>
    <raid_uuid>UUID=xxxxxxxx:xxxxxxxx:xxxxxxxx:xxxxxxxx</raid_uuid>
    <scsi0_serial>sda:SERIAL-A</scsi0_serial>
    <scsi1_serial>sdb:SERIAL-B</scsi1_serial>
    <scsi2_serial>sdc:SERIAL-C</scsi2_serial>
    <scsi3_serial>sdd:SERIAL-D</scsi3_serial>
  </item>
</volume_info>

The fields that matter, and why:

  • used_device is the load-bearing one. It is a space-separated list of the array’s member partitions (the trailing space is real). At shutdown the firmware does a literal substring search of this string for each disk’s name; a disk that is not found here gets failed and removed. Missing sdb2 here is the entire bug.
  • device is the same membership expressed as concatenated disk names (sdasdbsdcsdd). On the healthy boot path the assembler substring-matches live disk names against this, so sdb has to appear here too.
  • scsi_mapping is a four-bit mask, one bit per bay, with the least-significant bit being the first bay. 15 is 1111, all four present. 13 is 1101, which is bays 1, 3, and 4 present with the second bay (the new disk) missing. That was the degraded value that kept dropping it. This mask is only consulted on the strict, degraded boot path.
  • raid_status is the switch between the two boot paths: 0 means healthy, and the assembler uses the forgiving name match; 2 means degraded, and it uses the strict scsi_mapping mask plus a serial check. Getting this to 0 while the array is genuinely whole is what routes future boots down the forgiving path for good.
  • scsi0_serial through scsi3_serial tag each bay with its disk’s serial, in a disk:serial form. On the strict path a bay with no serial line is skipped, so the new disk’s missing serial was the other half of why it was excluded while degraded.
  • raid_uuid is the array’s identity, the same UUID that mdadm --assemble --uuid uses to gather the right members.

The same membership is mirrored in a few other places the firmware maintains: a persistent copy in hd_info.xml on the internal flash, and a small copy tucked on each disk itself. Once the array survives a single clean shutdown as a healthy four, the firmware rewrites all of them to match, which is why the fix sticks without having to hand-edit every copy.