this is obsolete doc -- see instead

Software RAID with MD devices on Linux 




We have the main disk on which the system is currently running : sda. And we want to make an RAID1 array while adding the second disk : sdb. The both disks have identical sizes. But the RAID1 device (/dev/mdX) will be a little bit smaller than the originating device partitions (/dev/sdXn), because of RAID internals, even in mirror mode. Therefore we'll have to create a degrated array on the second disk and migrate the data to it. We'll then be able to reconstruct the array, erasing the original disk. 


Note. this way, if ever you missed one step or did something wrong, you can always get back to the original system with the installation media typing "linux rescue" at boot prompt, to finish up or fix the RAID configuration, 

chroot /mnt/sysimage
#service network start
#service sshd start
mdadm --assemble /dev/md0
mdadm --assemble /dev/md1
mdadm --assemble /dev/md2
swapon /dev/md1
mkdir -p /raid
mount /dev/md2 /raid
mkdir -p /raid/boot
mount /dev/md0 /raid/boot


We're assuming a simple partition layout (no LVM), 

sda1  /boot
sda2  swap
sda3  /



Configuration (temporarly degrated mode) 

Copy the partition layout on the second disk, 

cd ~/
sfdisk -d /dev/sda > sdab.layout
sfdisk /dev/sdb < sdab.layout
#dd if=/dev/sda of=/dev/sdb bs=512 count=1

then change partition types to FD (Linux raid autodetect), 

fdisk /dev/sdb
t 1 fd
t 2 fd
t 3 fd


fdisk -l /dev/sdb


Clean up the partitions just to make sure there's no filesystems from possible previous attempts, 

dd if=/dev/zero of=/dev/sdb1 bs=1024K count=100
dd if=/dev/zero of=/dev/sdb2 bs=1024K count=100
dd if=/dev/zero of=/dev/sdb3 bs=1024K count=100


Create the empty arrays on the second disk, 

mdadm --create /dev/md0 --level=1 --raid-devices=2 missing /dev/sdb1
mdadm --create /dev/md1 --level=1 --raid-devices=2 missing /dev/sdb2
mdadm --create /dev/md2 --level=1 --raid-devices=2 missing /dev/sdb3


Switch to init 1 to safely copy the data to the RAID devices, 

telinit 1


Format the new filesystems and mount them somewhere, 

mkfs.ext3 /dev/md0
mkfs.ext3 /dev/md2
mkdir -p /raid
mount /dev/md2 /raid
mkdir -p /raid/boot
mount /dev/md0 /raid/boot


Copy the data into it, 

cd /
find . -xdev | cpio -pm /raid
cd /dev
find . | cpio -pm /raid/dev
cd /boot
find . -xdev | cpio -pm /raid/boot

Note. shouldn't we also exclude "lost+found" ? 

Note. maybe we could optimize those commands, all at once including /dev 


Replace the swap by a new one into the RAID1 device, 

swapon -s
swapoff /dev/sda2
mkswap /dev/md1
swapon /dev/md1


Write the RAID configuration into the RAID volume, 

cd /raid/etc
cat > mdadm.conf <<EOF9
ARRAY /dev/md0 level=raid1 num-devices=2 devices=missing,/dev/sdb1
ARRAY /dev/md1 level=raid1 num-devices=2 devices=missing,/dev/sdb2
ARRAY /dev/md2 level=raid1 num-devices=2 devices=missing,/dev/sdb3

Note. the DEVICE line isn't mandatory in that case 

Note. carefull this step is absolutely mandatory so the system knows which RAID devices to start during bootup. 

Note. otherwise, 

#cat > mdadm.conf <<EOF9
#DEVICE partitions
#mdadm -Es | grep md0 >> mdadm.conf

Note. or even, 

#echo "DEVICE /dev/sd[ab]1" > mdadm.conf
#mdadm --detail --scan >> mdadm.conf


Fix the partitions pathes in the RAID volume, 

cd /raid/etc
vi fstab


/dev/md2                /                       ext3    defaults        1 1
/dev/md0                /boot                   ext3    defaults        1 2
/dev/md1                swap                    swap    defaults        0 0


Update the initramfs, 

mkinitrd --fstab=/raid/etc/fstab /raid/boot/initraid.img `uname -r`


Update the bootloader configuration on the original filesystem and on RAID volume, 

vi /boot/grub/menu.lst
vi /raid/boot/grub/menu.lst


title CentOS (2.6.18-128.4.1.el5) RAID
        root (hd1,0)
        kernel /vmlinuz-2.6.18-128.4.1.el5 ro root=/dev/md2
        initrd /initraid.img

Note. we changed title ... RAID 

Note. we changed root (hd1,0) 

Note. we changed kernel ... root=/dev/md2 

Note. eventually define md1, 

title CentOS (2.6.18-164.11.1.el5) 

        root (hd0,0)
        kernel /vmlinuz-2.6.18-164.11.1.el5 ro root=/dev/md1 md=1,/dev/sda2,/dev/sdb2
        initrd /initrd-2.6.18-164.11.1.el5.img


Make the both disks bootable, 

grub --no-floppy
root (hd0,0)
setup (hd0)
root (hd1,0)
setup (hd1)

Note. GRUB is still able to look in the RAID device, donno how. 


Rady to go, reboot, 


Note. we're rebooting from single user mode, the "reboot" alias isn't activated but that's what we want since no services are running. 


Note. since RHEL5.4 you may configure this too, 

vi /etc/sysconfig/raid-check


CHECK_DEVS="md0 md1"
REPAIR_DEVS="md0 md1"



Adding sda to the array 

Change partition types to FD (Linux raid autodetect), 

fdisk /dev/sda
t 1 fd
t 2 fd
t 3 fd


fdisk -l /dev/sda


Add sda to the RAID1 array, 

mdadm /dev/md0 -a /dev/sda1
mdadm /dev/md1 -a /dev/sda2
mdadm /dev/md2 -a /dev/sda3


cat /proc/mdstat


Make sure /etc/mdadm.conf is up to date. 




Check the state of the array, 

cat /proc/mdstat
#tail /var/log/messages
#cat /sys/block/md2/md/sync_action
mdadm --detail /dev/md0
#mdadm --examine /dev/md0
#mdadm --examine /dev/sda1
#mdadm --examine --brief --scan --config=partitions
#mdadm -Ebsc partitions
#cat /var/run/mdadm/map


Monitor the array, 

mdadm --monitor --scan --daemonise > /var/run/mdadm


Mark a faulty device, 

mdadm /dev/md0 --fail detached --remove detached

rebuild the array and start what can be started, 

mdadm --incremental --rebuild --run --scan




Deployment issues 

- Make sure /etc/mdadm.conf exists and is up to date 

- Make sure the initrd is up to date and points to the correct root directory and swap 

- Make sure the devices files (/dev) exist into the RAID volume 

- Make sure the partitions types (sdb1, sdb2, sdb3, ...) are type FD (Linux raid autodetect) 


Maintainance issues 

Since RHEL5.4, the weekly crontab includes RAID checks (/etc/cron.weekly/99-raid-check) and you may receive some of those warnings by email, 

WARNING: mismatch_cnt is not 0 on /dev/md0
WARNING: mismatch_cnt is not 0 on /dev/md1


Repair and then check the arrays (now on a RAID1 MD array, repair is different than just a check), 

echo repair > /sys/block/md0/md/sync_action
echo repair > /sys/block/md1/md/sync_action
cat /proc/mdstat
cat /sys/block/md0/md/sync_action
cat /sys/block/md1/md/sync_action
echo check > /sys/block/md1/md/sync_action
echo check > /sys/block/md1/md/sync_action
cat /proc/mdstat
cat /sys/block/md0/md/sync_action
cat /sys/block/md1/md/sync_action

check again, 

cat /sys/block/md0/md/mismatch_cnt
cat /sys/block/md1/md/mismatch_cnt

Note. you should now see "0" for both