Deprecated: mysql_pconnect(): The mysql extension is deprecated and will be removed in the future: use mysqli or PDO instead in /home/darkflib/ on line 16
Ask Flib - Maintaining RAID

Maintaining RAID

By Flib


Category: Linux

The Problem

RAID is often used without much understanding of what it truly provides. People have no idea of how it can fail or what to do if it does actually fail.

Worse still, they have no idea of what maintenance RAID needs or how to see if it is working propoerly.

In this article I try to explain some of this without too much complexity.

The Solution

What is RAID?

RAID stands for 'Redundant Array of Inexpensive Disks'. 'Redundant' meaning that they are generally used in such a way so as to have spare capacity to cope with failure. 'Array' meaning many of them working together and 'Inexpensive Disks' coming from the time it was originally coined, when cheap potentially more failure prone disks were used in place of massively more expensive 'enterprise quality' disks. This is mostly unimportant, but gives a little background.

Put simply, RAID is a way of having a second copy of the data on one or more drives so that if a drive fails, you can keep working without data loss and recover from it.

It is not to be confused with backups, which are point in time copies; meaning that at a particular time they were accurate, however the older they get, the more out of sync they are likely to be from the data at the time of the backup.

RAID on the otherhand is constantly kept in-sync with the data. This has many advatanges but one drawback; if you delete a file, you have deleted it across the whole RAID. This is why backups and RAID should be used together.

Types of RAID

RAID comes in many varieties. These are generally called 'Levels'. The levels generally differ in how many copies of the data they keep and their speed advantages.

RAID LevelDescription
0RAID 0 has zero redundancy. It is used primarily to speed up disks which is its main advantage. Reads and writes essentially work at twice the speed of a single drive alone (for two drives). Its main disadvantage is that if any one drive fails, the whole RAID is essentially gone. This means its best used for fast, non critical stuff such as fast temporary working space which isn't critical if it gets deleted. Minimum 2 drives.
1RAID 1 has 100% redundancy. It keeps two (or more) drives synced, so that they both contain the exact same data. If any drive fails, the other drive, since it contains the exact same data, can service any requests that would have gone to the failed drive. Read speed is doubled when compared to two drives, however write speed is generally the same as a single drive alone since any writes need to go to both drives. RAID 1 is one of the best levels for normal use and probably should be considered before RAID 5 by most people. Minimum 2 drives.
3/4Seldom used and seldom supported, steer clear unless you know more about why you would want to use these. Minimum 3 drives.
5RAID 5 is what most people use as a default standard. What generally attracts them is the protection for minimal overhead. For example, 3 drives with RAID 5 would give similar capacity to 2 drives alone with the advantage of being able to cope with a single drive failure. RAID 5 is fairly fast on both reads and writes, since both reads and writes are spread over all drives. Minimum 3 drives.
6RAID 6 is a fairly new kid of the block and not yet supported everywhere. Like RAID 5, it also spreads writes and reads across all disks and as such very similar in performance. Its main advantage (at the cost of 1 more disk for recovery) is that is can cope with the failure of any two drives and keep working. Minimum 4 drives.
10RAID 10 is normally thought of as a hybrid RAID. It uses two RAID 1 arrays and then applies RAID 0 across them. This allows excellent read performance, and moderate write performance. It can also cope with up to 2 drive failures (if the two drives are on seperate RAID 1 arrays). Minimum of 4 drives.

Problems of RAID

People trust their data to RAID without understanding the things that can go wrong. Generally these problems fall into one or more groups:

  • Silent Corruption
  • Multiple Failures
  • Failed Rebuilds

Of these, silent corruption is probably the most serious. It silently corrupts your data and due to the way RAID works, can occur as an intermittent error.

Besides catastrophic failures like lightning strikes, floods or power supply failures that can all destroy the whole machine, most of these can be detected to some extent by some basic maintenance.

Maintenance of Linux software RAID

Checking State of the Arrays

From time to time you may need to check the state of the arrays on the server. This can be accomplished using:

[root@xen1 ~]# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [raid1] md0 : active raid1 sda1[0] sdb1[1] sdc1[2] sdd1[3] sde1[4] sdf1[5] 104320 blocks [6/6] [UUUUUU] md1 : active raid6 sdf2[5] sde2[4] sdd2[3] sdc2[2] sdb2[1] sda2[0] 1924014336 blocks level 6, 256k chunk, algorithm 2 [6/6] [UUUUUU] unused devices:

This was taken from a virtualisation server with 6x 500Gig drives. The [UUUUUU] allows you to see at a glance that all elements of the RAID array are up and working.

RAID scrubbing (resyncing)

RAID scrubbing is a process that goes through each sector of the RAID array and checks that all drives agree on the data that should be present.

It is recommended to run a scrub of the RAID weekly, or at least monthly. This allows the drives to detect any 'weak' sectors and if possible relocate them. Its better to detect this as part of the routine testing than when you are trying to recover, since a read error is very likely to break your rebuild.

In Linux is can be accomplished using:

echo check > /sys/block/md2/md/sync_action

If you need to set upper and lower speeds (in kilobytes/sec) for the resync process, then you can do so using:

echo 50000 > /sys/block/md1/md/sync_speed_max echo 5000 > /sys/block/md1/md/sync_speed_min

To check status of the resync check, you can do

cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [raid1] md0 : active raid1 sda1[0] sdb1[1] sdc1[2] sdd1[3] sde1[4] sdf1[5] 104320 blocks [6/6] [UUUUUU] [>....................] resync = 0.0% (0/104320) finish=17.3min speed=0K/sec . .

This gives a progress indication and estimated time to completion.

Checking drives with smartctl

Most modern drives have basic error and performance reporting in the form of SMART. SMART must be enabled in the BIOS on most systems, but can provide more data than most sysadmins will ever need about thier drives.

On most Linux distributions, the tools are provided in a package called 'smartmontools'. It consists of a daemon to check in the background, and a command line program to allow on demand interrogation and testing of the drives.

To view all the smart data of a drive, simply use the following command:

smartctl -a <drivename>

alternatively, to see what options are available

smartctl --help

If you don't like what you see, perhaps, the drive is starting to develop errors at a faster rate than in the past, you may choose to do one of the tests available:

#smartctl -t <offline|short|long|conveyance|select,M-N|pending,N|afterselect,[on|off]|scttempint,N[,p]> #eg. smartctl -t short /dev/sda

Not all tests will be available on all drives, for example, both my Maxlines and WD RE2 drives support both short, long, offline and selective, but only the WD RE2 drives support the Conveyance test.

smartctl -c <drivename>

will display the drive capavilities and the recommended polling (time after which the test is likely to be finished) time for each of the supported tests.

smartctl -l selftest <drivename>

will also display the test history and the results of the tests priviously executed on the drives.

It was recommended to myself, that I run short tests daily and long test weekly. I haven't seen anything that suggests not doing the tests this often besides a potential performance degradation while they are actually running, but since the short test is only about 2 minutes long, I don't see this as a real issue.


Powered by Sysdom Support Services