this is obsolete doc -- see http://doc.nethence.com/ instead

XEN4 RHEL6 dom0 (pv_ops) 

 

 

Requirements 

Make sure the development tools package group and those additional packages are installed, 

yum groupinstall "Development tools"
yum install \
openssl-devel \
uuid-devel \
libX11-devel \
ncurses-devel \
dev86 \
iasl \
libuuid-devel \
python-devel \
texinfo

Note. among the development tools, you'll have gcc, make, zlib-devel 

 

 

XEN compilation and installation 

Fetch (http://xen.org/products/xen_source.html) and build the xen microkernel, xen tools and xen stubdom (hvm ioemu), 

cd ~/
#wget http://bits.xensource.com/oss-xen/release/4.1.1/xen-4.1.1.tar.gz
#tar xzf xen-4.1.1.tar.gz
#cd xen-4.1.1/
wget http://bits.xensource.com/oss-xen/release/4.1.2/xen-4.1.2.tar.gz
tar xzf xen-4.1.2.tar.gz
cd xen-4.1.2/
make xen && make tools && make stubdom

 

Put the XEN micro-kernel in place and install the userland tools, 

#cp xen/xen.gz /boot/xen-4.1.1.gz 

cp xen/xen.gz /boot/xen-4.1.2.gz 

make install-tools 

make install-stubdom 

 

 

Dom0 kernel compilation and installation 

Checkout Jeremy's repo (https://github.com/mirrors/linux-2.6 while kernel.org is down), 

cd ~/
#git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
git clone git://github.com/jsgf/linux-xen.git
cd linux-xen/
git branch
git pull

Note. we're on the next-2.6.32 branch by default. 

 

Configure your kernel, 

make mrproper
make defconfig

 

cat >> .config <<EOF9
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_BRIDGE=y
CONFIG_NFSD=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
EOF9
make silentoldconfig
# (press enter for the defaults)

 

cat >> .config <<EOF9
CONFIG_DUMMY=m
CONFIG_TUN=m
CONFIG_BONDING=m
EOF9
make silentoldconfig

 

  cat >> .config <<EOF9
CONFIG_PARAVIRT_GUEST=y
CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_BLKDEV_TAP=y
CONFIG_XEN_GNTDEV=y
CONFIG_XEN_PLATFORM_PCI=y
EOF9
make silentoldconfig
# (press enter for the defaults)

 

Add the needed drivers for your hardware. For example for a Dell XS11-VX8 mini-server, 

cat >> .config <<EOF9
CONFIG_PATA_VIA=y
CONFIG_I2C_VIAPRO=y
CONFIG_FB_VIA=y
CONFIG_E1000E=y
EOF9
make silentoldconfig

Note. disk, bus, video and network drivers. 

 

Compile your kernel, 

time nice make
ls -l /lib/modules/
make modules_install
cp arch/x86_64/boot/bzImage /boot/vmlinuz-2.6.32.46xen
mkdir -p /data/share/
cp arch/x86_64/boot/bzImage /data/share/vmlinuz-2.6.32.46xen

Note. the kernel you just compiled is both, dom0 and domU capable. 

 

Rebuild the initramfs, 

cd /boot/
dracut initramfs-2.6.32.46xen.img 2.6.32.46

 

Check the filenames, 

#ls -l xen-4.1.1.gz
ls -l xen-4.1.2.gz
ls -l vmlinuz-2.6.32.46xen 
  ls -l initramfs-2.6.32.46xen.img

make the new kernel bootable, 

vi grub/grub.conf

change/add, 

default=1
[...]
title XEN 4.1.2 with Linux 2.6.32.46xen
        root (hd0,0)
        kernel /xen-4.1.2.gz dom0_mem=512M
        module /vmlinuz-2.6.32.46xen ro root=/dev/sda2 md=
        module /initramfs-2.6.32.46xen.img

Note. assuming sda1 for /boot and sda2 for the root mount point. 

 

Eventually fix some useful kernel source symlinks, 

cd /usr/src/
ln -s ../../root/linux-xen
ln -s ../../root/linux-xen linux

 

Disable KSM and reboot, 

cd ~/
chkconfig ksm off
chkconfig ksmtuned off
shutdown -r now

once rebooted, check that you're actually running the new kernel on top of xen, 

uname -r
xm info

 

 

Dom0 configuration 

Mount XENFS, 

cat >> /etc/fstab <<EOF9
xenfs            /proc/xen      xenfs           defaults        0 0
EOF9
mount /proc/xen

check, 

mount | grep xen

 

Eventually edit your screen configuration, 

cd ~/
cat >> .screenrc <<EOF9
screen -t "xen" 1 tail -F /var/log/xen/xend.log
screen -t "xendebug" 1 tail -F /var/log/xen/xend-debug.log
EOF9

 

 

XEN network configuration 

Clean up the config file, 

cd /etc/xen/
mv xend-config.sxp xend-config.sxp.dist
sed '/^#/d; /^$/d' xend-config.sxp.dist > xend-config.sxp
vi xend-config.sxp

change/check, 

(network-script network-bridge)

Note. we're keeping the default (bridge for dom0 and guests). For guests, you can always force alternate bridges in their configs 

Note. if you're using a standalone xen dom0 without live migration, you may disable this to secure the 8002 port, 

#(xend-relocation-server yes)
#(xend-relocation-hosts-allow '^localhost$ ^localhost\\.localdomain$')

 

 

Rebuild bridge-utils 

On RHEL6 there's an issue with bridge-utils (IGMP snooping conflicts with the custom dom0 kernel). You need to rebuild it without it. 

 

Prepare your environment, 

cd ~/
yum install gcc make rpm-build
mkdir -p rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
ls -l .rpmmacros
echo '%_topdir %(echo $HOME)/rpmbuild' > .rpmmacros
groupadd mockbuild
useradd -g mockbuild mockbuild

Note. redhat-rpm-config isn't mandatory 

 

Install the src RPM and tweak the spec file, 

wget ftp://ftp.free.fr/mirrors/ftp.centos.org/6.0/os/SRPMS/Packages/bridge-utils-1.2-9.el6.src.rpm
rpm -ivh bridge-utils-1.2-9.el6.src.rpm
cd ~/rpmbuild/SPECS/
cp bridge-utils.spec bridge-utils.spec.dist
vi bridge-utils.spec

remove those two lines from it, 

Patch4: bridge-utils-1.2-Add_IGMP_snooping_support.patch
%patch4 -p1 -b .Add_IGMP_snooping_support

you can now build it with IGMP snooping disabled, 

yum install libsysfs-devel
rpmbuild -ba bridge-utils.spec

 

Install the thing, 

cd ~/rpmbuild/RPMS/x86_64/
rpm -e bridge-utils
rpm -ivh bridge-utils-1.2-9.el6.x86_64.rpm

 

Refs. 

http://wiki.centos.org/HowTos/RebuildSRPM 

http://wiki.centos.org/HowTos/SetupRpmBuildEnvironment 

 

 

Ready to go 

Start the xen daemons (xencommons now needs to be started before xend), 

chkconfig xencommons on
chkconfig xend on
service xencommons start
service xend start

 

 

Troubleshooting 

If you get this error when in the xen logs and when doing 'brctl show', 

/sys/class/net/virbr0/bridge: No such file or directory

==> make sure you rebuilt bridge-utils w/o IGMP snooping. 

 

If you get this error in the xen logs, 

xend start "cat: /sys/bus/scsi/devices/host0/vendor: No such file or directory"

==> make sure you started xencommons BEFORE xend. 

 

 

Update 

Eventually uninstall the previous version before you start from scratch again, 

cd ~/
cd xen-4.1.1/
make uninstall
cd ../
rm -rf xen-4.1.1/

 

 

References 

http://wiki.xen.org/xenwiki/RHEL6Xen4Tutorial 

http://wiki.xen.org/xenwiki/XenParavirtOps 

http://wiki.xen.org/xenwiki/XenSerialConsole 

http://wiki.xen.org/xenwiki/MigrationGuideToXen4.1%2B 

http://wiki.xen.org/xenwiki/XenCommonProblems 

http://wiki.gcu-squad.org/doku.php?id=linux:dediboxv3_debian6_xen4