this is obsolete doc -- see instead

Redhat GFS2 configuration 




GFS is fine for large files, it's not recommended for small files (Redhat supports it still...) 

Note. procedures there after need to be applied on all the nodes unless specified otherwise. 




Make sure the nodes resolve to each other, 

vi /etc/hosts


Make sure you've got NTP configured and running, 

rpm -q ntp
#vi /etc/ntp.conf (eventually change server)
service ntpd start
chkconfig ntpd on




Install the needed package groups, 

yum groupinstall Clustering
yum groupinstall "Cluster Storage"


Note the lvm configuration change, 

cd /etc/lvm
diff lvm.conf.lvmconfold lvm.conf


>     library_dir = "/usr/lib"


Also, you need to tweak lvm.conf a little, to adapt LVM2 tools for CLVM, 

cd /etc/lvm
cp lvm.conf lvm.conf.dist
sed '
' lvm.conf.dist > lvm.conf
vi lvm.conf


locking_type = 3
fallback_to_local_locking = 0

Note. locking_type type 3 for built-in clustered locking 

Note. fallback_to_local_locking 0 doesn't prevent LVM from working when another node is down 


Restart the clustered LVM daemon just to be sure the changes are immediately applied, 

service clvmd restart




Setup the cluster, 


then e.g., 

Cluster nodes > Add a cluster node > node1 / votes: 1
Cluster nodes > Add a cluster node > node2 / votes: 1
File > Save (OK)
File > Quit

Note. eventually use a quorum disk (recommended!) 


For a two node cluster w/o quorum disk and w/o fencing you get, 

cd /etc/cluster
cat cluster.conf


<?xml version="1.0" ?>
<cluster config_version="2" name="clustername">
        <fence_daemon post_fail_delay="0" post_join_delay="3"/>
                <clusternode name="node1" nodeid="1" votes="1">
                <clusternode name="node2" nodeid="2" votes="1">
        <cman expected_votes="1" two_node="1"/>

Note the '<cman expected_votes="1" two_node="1"/>' directive. You had to insert this kind of line manually in RHEL4 cluster suite for a two node cluster, otherwise it wouldn't run. 

Note. w/o fencing enabled you'll eventually get some of those in the logs, 

Jul 16 00:40:22 node2 fenced[1926]: fencing node "node1"
Jul 16 00:40:22 node2 fenced[1926]: fence "node1" failed


You can now start the daemons on ALL NODES : works best when you proceed with all the nodes at the same time for cluster startup, 

service cman start
service gfs2 start
service clvmd start
#service rgmanager start

don't forget to enable those at startup, 

chkconfig cman on
chkconfig gfs2 on
chkconfig clvmd on
#chkconfig rgmanager on



Make a GFS2 filesystem 

From only one node, create a CLVM logical volume, 

pvcreate /dev/dm-0
vgcreate vgcheck /dev/dm-0
lvcreate -L 10000 -n lvcheck vgcheck

note. lvm.conf has been adapted for CLVM see 'Installation' chapter 

note. FYI it's possible to prevent vgcreate to make the VG available to the cluster, hence visible only to one node, 

-c n

now look for the volume from the other nodes w/o rescaning, 

vgdisplay -v

note. no need to 'vgchange -ay' neither thanks to CLVM 


From only one node, make the filesystem, 

#man mkfs.gfs2
mkfs.gfs2 -p lock_dlm -t clustername:lvcheck -j 2 /dev/vgcheck/lvcheck

Note. use lock_dlm protocol for a shared disk storage configuration 

Note. clustername must match that in cluster.conf 

Note. lvcheck or any other name to distinguish this filesystem 


Mount it on all nodes and check, 

cd /mnt
mkdir -p gfs
mount.gfs2 /dev/vgcheck/lvcheck gfs
cd gfs
#echo ok > check
cat check

Note. mount.gfs2 doesn't tell if the filesystem is clean or not, hence the fsck before mounting when doing disater recovery, 

#fsck.gfs2 -y /dev/vgcheck/lvcheck


To mount the fs automaticly at boot, 

cd /etc
vi fstab

eventually add (here commented, see note below), 

#/dev/vgcheck/lvcheck /mnt/gfs gfs2 defaults 0 0

Note. but you might prefer to use a job scheduler or some cluster solution to enable/disable one node's services cleanly, w/o using fstab. Also, also mount.gfs2 doesn't tell if the filesystem is clean or not. 




View the running nodes, 

cman_tool nodes


If using fencing, to manually fence a node, 

fence_ack_manual -n node2




Note. to disable GFS2 completely, umount, clean up fstab and, 

service cman stop
service gfs2 stop
service clvmd stop
#service rgmanager stop


service cman status
service gfs2 status
service clvmd status
#service rgmanager status


chkconfig cman off
chkconfig gfs2 off
chkconfig clvmd off
#chkconfig rgmanager off


#rm -rf /etc/cluster


#cd /etc/lvm
#mv lvm.conf lvm.conf.cluster
#cp lvm.conf.lvmconfold lvm.conf




Note. if you get that error when trying to mount, 

mount.gfs2: can't connect to gfs_controld: Connection refused

start the cman service 


Note. if you get that error when trying to start cman, 

/usr/sbin/cman_tool: ccsd is not running