That Darn RAID Array

So I sat down to write a thoughtful "update" of my software RAID problem and it turns out I never wrote the original piece. Oops. So, first the update. Things are fine.

Now the backstory...

A few days ago I got a rather scary e-mail from mdadm saying that a Fail event had been detected on md device /dev/md1. That is the root partition in a software RAID array on a RHEL 3.0 box. Uh-oh. It didn't sound good and yet the box was running along just fine. This was because the RAID array did it's job. Yay.

Between then and now we have correspondence with the main IT people. But we'll just skip over that part. The solution, provided by a kind friend over irc, was pretty simple. Run raidhotremove and then raidhotadd. Then the magic will start to happen. Of course you have to check /proc/mdstat to actually see the magic happening.

md1 : active raid1 sdb2[2] sda2[0]
      34435200 blocks [2/1] [U_]
      [=========>...........]  recovery = 48.2% (16607936/34435200) finish=52.0min speed=5706K/sec