this is obsolete doc -- see http://doc.nethence.com/ instead

Veritas Storage Foundation Cluster File System v4.1 administration 

Monitoring 

Happy happy 

Check everything's fine, 

ps ax | grep vx

hastatus -summary

cfscluster status

vxdctl -c mode

Note. all systems should be RUNNING 

Note. all systems of the cvm group should be ONLINE 

Note. other possible commands, 

#vxparms

#hares -display | grep -i state

#hasys -display | grep -i sysstate

Initialize Disk Group 

Quoting /etc/vx/bin/vxdisksetup, 

# Determine if the device is a SCSI device or not; only SCSI devices are

# supported as CDS disks. Determine the starting disk address and size

# of the disk (for use as the WHOLEDISK slice on Solaris labeled disks

# or the extended partition on non-Solaris disks), including any CDS

# disk restrictions for non-Solaris disks.

Note. you need e.g. SAN or vmware shared raw disk. Software iSCSI may not work. 

Check you've got a shared disk available among the nodes, 

#find /sys/devices/platform/host* -name "block*"

fdisk -l

Check the shared disk is 'online' on all the nodes, 

vxdisk scandisks

vxdctl enable

vxdisk -o alldgs list

#vxdisk -eo alldgs list

On only one node, check disk geometry and make sure ncyl isn't equal to zero (which may happen with software iscsi), 

/usr/lib/vxvm/diag.d/vxscsi -g sdc

#hdparm  -i /dev/sdb | grep SCSI

On the master node, initialize a disk, 

vxdctl -c mode

vxdisksetup -i sdc

Note. format=cdsdisk is the default 

Note. '-f' forces a disk to be initialized even if it is a member of a deported disk group 

On the master node, create a disk group with it, 

vxdg -s init group0 sdc

#vxdg -g group0 set activation=sharedwrite

vxdg list

vxdisk -o alldgs list

#vxdisk -eo alldgs list

#vxdisk -g group0 list

#vxdisk -eg group0 list

Note. alternatively, 

#vxdg init group0 group0_disk1=sdb

Note. to add disks to the disk group, 

#vxdg -g ABCdg adddisk ABCdg02=c0t0d2

#vxdg -g ABCdg adddisk ABCdg09=c0t0d9

Note. we could also use vxdiskadd interactively for that, but heh, the hard way is always better, 

#vxdiskadd sdc

Note. vxdg list should return e.g. this for group0, 

group0       enabled,shared,cds   1279463494.34.vrts1.example.net

Note. vxdisk list should return e.g. this for group0, 

sdc          auto:cdsdisk    sdc          group0       online shared

Note. to destroy a DG, 

#vxdg destroy group0

Create a volume 

On only one node, create a new volume with that disk (50GB here), 

vxassist maxsize

vxassist make vol0 104820736

Note. other example, 

#vxassist make mirvol 500m layout=mirror,log mirror=ctlr \!ctlr:c1

Note. to remove a volume, 

#vxassist remove volume vol0

You should now be able to access the volume from any node in the cluster. 

Filesystem 

On the master node, 

ls -l /dev/vx/dsk/group0/vol0

mkfs.vxfs /dev/vx/dsk/group0/vol0

mkdir -p /mnt/vxfs

mount.vxfs -o log,rw,cluster,crw /dev/vx/dsk/group0/vol0 /mnt/vxfs

Note. -o largefiles (>1TB file capable) is the default, check with, 

fsadm /dev/vx/dsk/group0/vol0

On the other nodes, 

ls -l /dev/vx/dsk/group0/vol0

mkdir -p /mnt/vxfs

mount.vxfs -o log,rw,cluster,crw /dev/vx/dsk/group0/vol0 /mnt/vxfs

Ref. http://seer.entsupport.symantec.com/docs/300537.htm 

Usage 

Start/stop a volume, 

vxvol start vol0

vxvol stop vol0

Start/stop all volumes, 

vxvol startall

vxvol stopall

Make sure the disk group is imported, 

vxdg import group0

Make sure all volumes on the disk group are started, 

vxvol -g group0 startall

To remove a volume and recursively (-r) its plexes and subdisks, 

umount ...

vxvol stop vol0

vxedit -r rm vol0

To remove a disk group, 

vxdg list

vxdg destroy group0

Misc 

Note. to uninstall SFCFS, 

cd /opt/VRTS/install

./uninstallinfr

Troubbleshooting 

Watch the logs, 

cd /var/VRTSvcs/log

ls -ltr

tail -n 500 -F engine_A.log

VxVM vxdisksetup ERROR V-5-2-3461 sdb: Disk does not have enough cylinders for 'cdsdisk' format 

The 'vxdisksetup' scripts calls vxscsi, which returns ncyl=0 (cylinders), check yourself, 

/usr/lib/vxvm/diag.d/vxscsi -g sdc

You're proably using software iSCSI which doesn't seem to be enought. Try with a real SAN or with VMware shared raw disks. 

VxVM vxdg ERROR V-5-1-585 Disk group group0: cannot create: Cannot find disk on slave node. 

Here's the workaround. On the failing node (vxdisk list show an error on the disk), 

vxdisk offline sdc

blockdev --rereadpt /dev/vx/dmp/sdc

vxdisk online sdc

vxdisk scandisks

check again (should be online), 

vxdisk list

Ref. http://seer.entsupport.symantec.com/docs/282930.htm 

ERROR: V-3-24996 

Disassociate the volume, 

vxvol -g dg -f dis vol

Make the volume active by starting it, 

vxvol -g dg start vol

Run a consistency check on the existing filesystem, 

fsck -F vxfs /dev/vx/rdsk/dg/vol

Reassociate the volume to the root volume group, 

vxvol -g dg assoc rvg vol

Mount the filesystem within the Operating System, 

mount -F vxfs /dev/vx/dsk/dg/vol /mntpt 

Ref. http://seer.entsupport.symantec.com/docs/311556.htm 

Tuning 

Note. mount option for applications that O_SYNC but only require write data sync, 

-o convosync=dsync

Refs. 

http://www.mail-archive.com/veritas-vx@mailman.eng.auburn.edu/msg02378.html 

References 

http://sfdoccentral.symantec.com/ 

http://sfdoccentral.symantec.com/Storage_Foundation_HA_41_Linux.html 

Cheatsheets 

http://www.cuddletech.com/veritas/vxcheatsheet.txt 

http://www.blight.com/~rick/veritas/cheat.html 

Usage references 

http://uw714doc.sco.com/en/ODM_VMadmin/sag-6.html 

http://www.darklab.net/resources/veritas-cluster-server-debug.html 

http://cortiz.org/2008/08/13/veritas-storage-creating-dg-volumes-and-filesystem/ 

http://www.camelrichard.org/topics/VeritasVM/Volume_Manager_CLI_Examples 

http://adminschoice.com/veritas-volume-manager-vxassist 

http://www.cuddletech.com/veritas/vxassist/x41.html