Thomas (boggyb) wrote,
Thomas
boggyb

  • Music:
Since I'm sure talismancer will post even more snark if I just leave it at the two previous posts for today, have a not-even-slightly-epic snippet of computer fail.

There's an unexpected downside to reducing the amount of servers you need by consolidating several under-used ones into a bunch of virtual machines on one real server: when (not if) the server fails you lose all the virtual machines.

In this case it was a disk failure that took out the server. Now, disk failures had been planned for by creating a RAID-5 array, which uses MATHS to ensure that you can lose a disk in the array without losing any data.

So Murphy ensured that on the server in question two disks failed a few hours apart, at which point the RAID controller threw its hands up in the air and gave up. IT then had to spend several hours rebuilding the various VMs from backups.

I wonder how common double disk failures actually are? I do recall reading something about this a few years ago - someone had a look at the specified uncorrectable error rate (how likely the drive is to be unable to read a sector). They worked out that with high-capacity hard disks (on the order of 1TB) there's a scarily high chance of a RAID-5 rebuild failing due to an uncorrectable error on one of the remaining disks.
Tags: computing, nablopomo
Subscribe
  • Post a new comment

    Error

    default userpic
    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.
  • 1 comment