Thomas (boggyb) wrote,

  • Music:
Since I'm sure talismancer will post even more snark if I just leave it at the two previous posts for today, have a not-even-slightly-epic snippet of computer fail.

There's an unexpected downside to reducing the amount of servers you need by consolidating several under-used ones into a bunch of virtual machines on one real server: when (not if) the server fails you lose all the virtual machines.

In this case it was a disk failure that took out the server. Now, disk failures had been planned for by creating a RAID-5 array, which uses MATHS to ensure that you can lose a disk in the array without losing any data.

So Murphy ensured that on the server in question two disks failed a few hours apart, at which point the RAID controller threw its hands up in the air and gave up. IT then had to spend several hours rebuilding the various VMs from backups.

I wonder how common double disk failures actually are? I do recall reading something about this a few years ago - someone had a look at the specified uncorrectable error rate (how likely the drive is to be unable to read a sector). They worked out that with high-capacity hard disks (on the order of 1TB) there's a scarily high chance of a RAID-5 rebuild failing due to an uncorrectable error on one of the remaining disks.
Tags: computing, nablopomo

  • Training swag!

    At work we're thinking of using C++ on an upcoming embedded project (historically our embedded stuff has been in C). As it happened while we were…

  • Work Secret Santa!

    Work's Secret Santa happened today. I got gifted with this, which raised a few eyebrows... Colleague: "That looks delightfully dangerous" Boss:…

  • Work Secret Santa?

    Oh dear, I seem to have become responsible for a Secret Santa at work. It all started innocently enough - there had been vague mentions, so I asked…

  • Post a new comment


    default userpic
    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.
  • 1 comment