The red light, glowing brightly among the usual blue LEDs, was not an expected sight yesterday morning.
Back in February 2010, we installed a LaCie NAS (network attached storage) appliance as the primary data repository for office. It’s been chugging along ever since, with five little blue LEDs flickering constantly as the five disks were accessed.
Blink, blink, blink. Documents, source code, graphics, media files, everything went onto the big silver appliance. The appliance itself was backed up for disaster recovery, but as you know, restoring from backup can be both labor-intensive and time consuming.
Then yesterday morning, I came into the office… and something was different on the NAS appliance on the equipment shelf. One of the LEDs was shining an ominous red. And there was a strange clicking sound.
Yes, one of the five Seagate Barracuda hard disks in the NAS appliance had failed.
Fortunately, there was no data loss, because the LaCie system included hardware-based RAID 5. RAID, or “Redundant Array of Inexpensive Disks,” spread every file across all five disks in such a way so that if any one disk filed, duplicate data could be found on the other four disks.
The downside of RAID 5 is that you get a little reduced capacity. This old NAS appliance has five 1TB hard disks, but after setting up RAID 5 striping, we only had 4TB capacity.
The reduced capacity was offset by two upsides. The first is that with the RAID 5 setup, we could suffer a single disk failure without any data loss or any interruption in the use of the device. (If two disks failed, we would be hosed, and would have had to restore from backup.) The second is that because data is striped across multiple disks, reading and writing are done in parallel, so we get superior file-access performance. The real benefit, though, is in the redundancy.
Because we purchased the NAS appliance with the explicit intent to protect data, we had taken the precaution to order a sixth identical Seagate Barracuda hard disk. It took fewer than 10 minutes to find the spare disk, unlock and remove the dead disk from the NAS appliance, and replace it—without even having to power anything down.
After the new disk was installed, the solid red light turned into a blinking blue-blue-blue-red pattern, as the firmware inside the NAS appliance began rebuilding the RAID 5 array using the new disk. About two hours later, all five lights were flickering blue. We were back to full performance, and full protection.
All we had to do is order a new Seagate Barracuda disk so that we’ll have a “cold spare” in case we suffer another disk failure. (The LaCie appliance, and many similar NAS appliances, will let you configure the system as “RAID 5 + 1,” which gives you a hot spare that’s brought online automatically. This reduces the available storage, however.)
Now, you might argue that this isn’t necessary: You can access files in the cloud, for example. In my experience, though, file performance over the Internet is too slow (and can be too unreliable) for, say, builds—unless, of course, you want to push your entire dev environment into the cloud, or to set up another system to avoid having a single point of failure. That’s a different conversation.
I have run into too many development teams that store their working files on their own desktops, or on a cheap local file server that stores everything on a single hard disk, often without a solid backup strategy. When something goes wrong, intellectual property is lost, real-life data is lost, time is lost, and money is wasted.
If you store your data locally, my advice is to make sure that your file server (either a standard server or NAS appliance) is set up using a multi-disk RAID scheme that includes redundancy: RAID 1, RAID 5, RAID 5+1, RAID 6, or some variation thereof. Make sure you keep a spare hard disk handy (and a spare power supply, if the server supports field-replaceable power supplies).
Of course, even with RAID, it’s essential to have a live, real-time backup scheme as well.
You don’t want red lights to mean data loss. Be prepared!
Alan Zeichick, founding editor of SD Times, is principal analyst of Camden Associates.