Nethence Newdoc Olddoc Lab Your IP BBDock  

Warning: those guides are mostly obsolete, please have a look at the new documentation.


Software RAID with MD devices on Linux
We have the main disk on which the system is currently running : sda. And we want to make an RAID1 array while adding the second disk : sdb. The both disks have identical sizes. But the RAID1 device (/dev/mdX) will be a little bit smaller than the originating device partitions (/dev/sdXn), because of RAID internals, even in mirror mode. Therefore we'll have to create a degrated array on the second disk and migrate the data to it. We'll then be able to reconstruct the array, erasing the original disk.
Note. this way, if ever you missed one step or did something wrong, you can always get back to the original system with the installation media typing "linux rescue" at boot prompt, to finish up or fix the RAID configuration,
chroot /mnt/sysimage
#service network start
#service sshd start
mdadm --assemble /dev/md0
mdadm --assemble /dev/md1
mdadm --assemble /dev/md2
swapon /dev/md1
mkdir -p /raid
mount /dev/md2 /raid
mkdir -p /raid/boot
mount /dev/md0 /raid/boot
We're assuming a simple partition layout (no LVM),
sda1  /boot
sda2  swap
sda3  /
Configuration (temporarly degrated mode)
Copy the partition layout on the second disk,
cd ~/
sfdisk -d /dev/sda > sdab.layout
sfdisk /dev/sdb < sdab.layout
#dd if=/dev/sda of=/dev/sdb bs=512 count=1
then change partition types to FD (Linux raid autodetect),
fdisk /dev/sdb
t 1 fd
t 2 fd
t 3 fd
fdisk -l /dev/sdb
Clean up the partitions just to make sure there's no filesystems from possible previous attempts,
dd if=/dev/zero of=/dev/sdb1 bs=1024K count=100
dd if=/dev/zero of=/dev/sdb2 bs=1024K count=100
dd if=/dev/zero of=/dev/sdb3 bs=1024K count=100
Create the empty arrays on the second disk,
mdadm --create /dev/md0 --level=1 --raid-devices=2 missing /dev/sdb1
mdadm --create /dev/md1 --level=1 --raid-devices=2 missing /dev/sdb2
mdadm --create /dev/md2 --level=1 --raid-devices=2 missing /dev/sdb3
Switch to init 1 to safely copy the data to the RAID devices,
telinit 1
Format the new filesystems and mount them somewhere,
mkfs.ext3 /dev/md0
mkfs.ext3 /dev/md2
mkdir -p /raid
mount /dev/md2 /raid
mkdir -p /raid/boot
mount /dev/md0 /raid/boot
Copy the data into it,
cd /
find . -xdev | cpio -pm /raid
cd /dev
find . | cpio -pm /raid/dev
cd /boot
find . -xdev | cpio -pm /raid/boot
Note. shouldn't we also exclude "lost+found" ?
Note. maybe we could optimize those commands, all at once including /dev
Replace the swap by a new one into the RAID1 device,
swapon -s
swapoff /dev/sda2
mkswap /dev/md1
swapon /dev/md1
Write the RAID configuration into the RAID volume,
cd /raid/etc
cat > mdadm.conf <<EOF9
ARRAY /dev/md0 level=raid1 num-devices=2 devices=missing,/dev/sdb1
ARRAY /dev/md1 level=raid1 num-devices=2 devices=missing,/dev/sdb2
ARRAY /dev/md2 level=raid1 num-devices=2 devices=missing,/dev/sdb3
Note. the DEVICE line isn't mandatory in that case
Note. carefull this step is absolutely mandatory so the system knows which RAID devices to start during bootup.
Note. otherwise,
#cat > mdadm.conf <<EOF9
#DEVICE partitions
#mdadm -Es | grep md0 >> mdadm.conf
Note. or even,
#echo "DEVICE /dev/sd[ab]1" > mdadm.conf
#mdadm --detail --scan >> mdadm.conf
Fix the partitions pathes in the RAID volume,
cd /raid/etc
vi fstab
/dev/md2 / ext3 defaults 1 1
/dev/md0 /boot ext3 defaults 1 2
/dev/md1 swap swap defaults 0 0
Update the initramfs,
mkinitrd --fstab=/raid/etc/fstab /raid/boot/initraid.img `uname -r`
Update the bootloader configuration on the original filesystem and on RAID volume,
vi /boot/grub/menu.lst
vi /raid/boot/grub/menu.lst
title CentOS (2.6.18-128.4.1.el5) RAID
root (hd1,0)
kernel /vmlinuz-2.6.18-128.4.1.el5 ro root=/dev/md2
initrd /initraid.img
Note. we changed title ... RAID
Note. we changed root (hd1,0)
Note. we changed kernel ... root=/dev/md2
Note. eventually define md1,
title CentOS (2.6.18-164.11.1.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-164.11.1.el5 ro root=/dev/md1 md=1,/dev/sda2,/dev/sdb2
initrd /initrd-2.6.18-164.11.1.el5.img
Make the both disks bootable,
grub --no-floppy
root (hd0,0)
setup (hd0)
root (hd1,0)
setup (hd1)
Note. GRUB is still able to look in the RAID device, donno how.
Rady to go, reboot,
Note. we're rebooting from single user mode, the "reboot" alias isn't activated but that's what we want since no services are running.
Note. since RHEL5.4 you may configure this too,
vi /etc/sysconfig/raid-check
CHECK_DEVS="md0 md1"
REPAIR_DEVS="md0 md1"
Adding sda to the array
Change partition types to FD (Linux raid autodetect),
fdisk /dev/sda
t 1 fd
t 2 fd
t 3 fd
fdisk -l /dev/sda
Add sda to the RAID1 array,
mdadm /dev/md0 -a /dev/sda1
mdadm /dev/md1 -a /dev/sda2
mdadm /dev/md2 -a /dev/sda3
cat /proc/mdstat
Make sure /etc/mdadm.conf is up to date.
Check the state of the array,
cat /proc/mdstat
#tail /var/log/messages
#cat /sys/block/md2/md/sync_action
mdadm --detail /dev/md0
#mdadm --examine /dev/md0
#mdadm --examine /dev/sda1
#mdadm --examine --brief --scan --config=partitions
#mdadm -Ebsc partitions
#cat /var/run/mdadm/map
Monitor the array,
mdadm --monitor --scan --daemonise > /var/run/mdadm
Mark a faulty device,
mdadm /dev/md0 --fail detached --remove detached
rebuild the array and start what can be started,
mdadm --incremental --rebuild --run --scan
Deployment issues
- Make sure /etc/mdadm.conf exists and is up to date
- Make sure the initrd is up to date and points to the correct root directory and swap
- Make sure the devices files (/dev) exist into the RAID volume
- Make sure the partitions types (sdb1, sdb2, sdb3, ...) are type FD (Linux raid autodetect)
Maintainance issues
Since RHEL5.4, the weekly crontab includes RAID checks (/etc/cron.weekly/99-raid-check) and you may receive some of those warnings by email,
WARNING: mismatch_cnt is not 0 on /dev/md0
WARNING: mismatch_cnt is not 0 on /dev/md1
Repair and then check the arrays (now on a RAID1 MD array, repair is different than just a check),
echo repair > /sys/block/md0/md/sync_action
echo repair > /sys/block/md1/md/sync_action
cat /proc/mdstat
cat /sys/block/md0/md/sync_action
cat /sys/block/md1/md/sync_action
echo check > /sys/block/md1/md/sync_action
echo check > /sys/block/md1/md/sync_action
cat /proc/mdstat
cat /sys/block/md0/md/sync_action
cat /sys/block/md1/md/sync_action
check again,
cat /sys/block/md0/md/mismatch_cnt
cat /sys/block/md1/md/mismatch_cnt
Note. you should now see "0" for both

(obsolete, see the new doc)