Realm Linux install with VMXNET driver support
From NCSUTechstaffDocs
Within VMware ESX /ESXi 3.5 there are usually, but not always, two different NIC card choices available when you create or edit a virtual guest. You either get "E1000" and "Enhanced vmxnet" or "Flexible" and "Enhanced vmxnet". You will never see "Flexible" and "E1000." "Enhanced vmxnet" is really a VMXNET 2 adapter in ESX /ESXi 3.5. Whether you see "Flexible" or "E1000" or "Enhanced vmxnet" depends on what OS you have chosen in the creation wizard and also what version of VMware ESX you are running...
What do these choices mean? To understand this a little better we will discuss briefly the various network adapters available to our guest OS within ESX / ESXi. This is what VMware says about their various emulated NIC cards:
- Vlance - An emulated version of the AMD 79C970 PCnet32 LANCE NIC, an older 10 Mbps NIC with drivers available in most 32bit guest operating systems except Windows Vista and later. A virtual machine configured with this network adapter can use its network immediately.
- VMXNET - The VMXNET virtual network adapter has no physical counterpart. VMXNET is optimized for performance in a virtual machine. Because operating system vendors do not provide built-in drivers for this card, you must install VMware Tools to have a driver for the VMXNET network adapter available.
- Flexible - The Flexible network adapter identifies itself as a Vlance adapter when a virtual machine boots, but initializes itself and functions as either a Vlance or a VMXNET adapter, depending on which driver initializes it. With VMware Tools installed, the VMXNET driver changes the Vlance adapter to the higher performance VMXNET adapter.
- E1000 — An emulated version of the Intel 82545EM Gigabit Ethernet NIC, with drivers available in most newer guest operating systems, including Windows XP and later and Linux versions 2.4.19 and later.
- VMXNET 2 (Enhanced) - The VMXNET 2 adapter is based on the VMXNET adapter but provides some high-performance features commonly used on modern networks, such as jumbo frames and hardware offloads. This virtual network adapter is available only for some guest operating systems on ESX/ESXi 3.5 and later. VMXNET 2 is supported only for a limited set of guest operating systems. Operating system vendors, including Red Hat, do not provide built-in drivers for this card.
- VMXNET 3 - The VMXNET 3 adapter is the next generation of a paravirtualized NIC designed for performance, and is not related to VMXNET or VMXNET 2. It offers all the features available in VMXNET 2, and adds several new features like multiqueue support (also known as Receive Side Scaling in Windows), IPv6 offloads, and MSI/MSI-X interrupt delivery. VMXNET 3 is supported only for virtual machines version 7 and later, with a limited set of guest operating systems. Operating system vendors, including Red Hat, do not provide built-in drivers for this card.
ITECS is currently not using a new enough ESX cluster to use VMXNET 3 network cards. We plan to upgrade to ESX 4 at some future date. At that time we would like to have VMXNET 3 as an option. With our current ESX 3.5 cluster it would be possible to use VMXNET 2 NICs in our Realm Linux guests if we could install the OS...
There is a problem with doing Realm Linux installs using only a VMXNET 2/3 NIC since the network is required for the install process.
The vmxnet 2 "enhanced" virtual network adapter has no physical counterpart. One way to make Realm Linux install is to modify the various boot iso files so that they contain the vmxnet 2 NIC driver. Anaconda, which is the Red Hat Linux installer, must also be able to recognize the adapter as a NIC and to select it for the source of the install and the second stage. It turns out that it is possible to modify Realm Linux boot CDs such that they will work with this enhanced vmxnet 2/3 NIC.
Contents |
Prerequisites
A VM with the vmxnet 2 NIC installed and VMware tools configured. You will need this so that you can grab the kernel drivers. The version of the kernel is extremely important. For Red Hat 5.3 this is version 2.6.18-128.el5 of the Linux kernel.
Red Hat Enterprise Linux ISOs
First, we need to grab the Red Hat Enterprise Linux ISOs that go with the version we are trying to fix. They can be downloaded from: http://install.linux.ncsu.edu/pub/rhel/isos/
For example, the current "spin" of Realm Linux 5 is based on Red Hat Enterprise Linux 5.3 so we need the 5.3 iso files. Download these ISOs and copy them to your ESX or ESXi storage area for iso files. ITECS uses /vmfs/volumes/Local-XX/Linux/ for storing these iso files within our ESX cluster.
Install a virtual machine with a flavor of Red Hat. Lets start with server 5.3 i386. Install a virtual machine with this set of iso files:
- rhel-server-5.3-i386-disc1.iso
- rhel-server-5.3-i386-disc2.iso
- rhel-server-5.3-i386-disc3.iso
- rhel-server-5.3-i386-disc4.iso
- rhel-server-5.3-i386-disc5.iso
Once you have the OS installed, install VMware tools somehow. Make sure that you have the right NIC configured.("Enhanced vmxnet" for VMXNET 2 card) Make sure you do not register the machine in Red Hat network and do not upgrade the machine with yum. You must be running the same exact kernel as the 5.3 install time kernel which is 2.6.18-128.el5. You must have the virtual machine powered down to effect changes in the virtual network card. It seems it is not possible to change a NIC from "Flexible" type to "Enhanced vmxnet" type in ESX 3.5 so I power down the guest, remove the "Flexible" NIC, then add a "Enhanced vmxnet" NIC. Once the vmxnet 2 NIC is configured as the guest eth0 NIC and the system has powered on to runlevel 3 type this command:
/usr/bin/vmware-config-tools.pl -default
Verify that your network is working, waiting for DNS updates if required. Make sure that you got a lease for an IP address via DHCP and that you can connect / ping to NCSU IP addresses, etc...
Once you run "vmware-config-tools.pl" a driver module will be created in /lib/modules/2.6.18-128.el5/misc/ called vmxnet.ko. If you have a vmxnet 3 NIC installed you will also have a vmxnet3.ko file here.
This vmxnet.ko file is the driver that must be added to the Realm Linux install CD for server 5.3 i386. There are many steps to this process...
Realm Linux ISOs
We need to grab the Realm Linux ISOs that go with the version we are trying to fix. They can be downloaded from: http://web-kickstart.linux.ncsu.edu/
You should now have the Realm Linux iso:
- server-5.3-x86-webks.iso
Next we need to mount this iso file.
Type these commands:
mkdir /mnt/server53 mount -o loop /tmp/server-5.3-x86-webks.iso /mnt/server53
Next, we need to copy the contents of the iso to a temporary work area...
mkdir /tmp/EL5server53 cp -a /mnt/server53/* /tmp/EL5server53/
Since we are done with the original iso now we can unmount it...
umount /mnt/server53 rmdir /mnt/server53
Creating a temporary work area
At this point, lets also create a temporary work area...
mkdir /tmp/work cd /tmp/work
Next, we need to copy the init RAM disk to this work area...
cp /tmp/EL5server53/isolinux/initrd.img .
Next we need to unpack the init RAM disk. With a 2.6 kernel its a gzipped cpio archive....
mkdir initrd cd initrd zcat ../initrd.img | cpio -id
This will explode the contents of the initrd.img disk into the current working directory (initrd) so that we can modify things.
Gathering information
At this point you will need to gather some information for use later, so on your reference VM system perform the following:
lspci
Now that we know what PCI card we are looking for, we actually need some hex numbers, which you can get by typing:
lspci -n
Modifying modules.cgz
We now need to explode the modules cpio archive from within the initrd subdirectory
mkdir /tmp/work/modules cd /tmp/work/modules zcat ../initrd/modules/modules.cgz | cpio -id
Our initrd.img modules.cgz file contains i686 2.6.18-128.el5 modules.
We want to now copy in the modules from your reference VM system. On the virtual machine the module is located in
/lib/modules/2.6.18-128.el5/misc/vmxnet.ko
The vmxnet.ko module will only be in place if you have the VMware Tools rpm installed on your reference VM and you have run the vmware-config-tools.pl script on that same virtual machine.
cd /tmp/work/modules/2.6.18-128.el5/i686 cp /lib/modules/2.6.18-128.el5/misc/vmxnet*.ko . chmod 744 vmxnet*
Time to pack the modules back up again...
cd /tmp/work/modules find . | cpio -o -H crc | gzip -9 > /tmp/work/initrd/modules/modules.cgz
This will create a new cpio archive and replace the modules.cgz from the original initrd.img with our own modified modules.cgz file. Notice in this case the cpio format type is crc.
Making Anaconda find the driver
Now we need to modify a few more files to get this to work.
cd /tmp/work/initrd/modules
Using your favorite text editor, edit the pci.ids file.
You will have to search for VMware, and as you do you will notice there is a "15ad" number which came from the first part of the pair of hex codes you were supposed to remember.
It looks like this in the file:
15ad VMware Inc 0405 Abstract SVGA II Adapter 0710 Abstract SVGA Adapter
Now you can put the folowing line in:
0720 VMware Adapter
Where 0720 is replaced by whatever hex code you saw. So for our ESX cluster the file looks like this:
15ad VMware Inc
0405 Abstract SVGA II Adapter
0720 VMware Adapter
0710 Abstract SVGA Adapter
Next you will need to add the following to the file module-info:
vmxnet
eth
"VMware vmxnet ethernet driver"
The file should contain tab characters and not spaces before the second and third lines...
Next you will need to get the vmxnet entries for the modules.alias file. On the reference VM system, it was in the /lib/modules/2.6.18-128.el5/modules.alias file.
cd /lib/modules/2.6.18-128.el5/ grep vmxnet modules.alias >> /tmp/work/initrd/modules/modules.alias
The contents we were looking for:
alias pci:v000015ADd00000720sv*sd*bc*sc*i* vmxnet alias pci:v00001022d00002000sv*sd*bc*sc*i* vmxnet
Package our custom initrd.img
Ok, now that the hard work is over its time to package the initrd.img back up...
cd /tmp/work/initrd find . | cpio -o -H newc | gzip -9 > /tmp/work/initrd.img.vmxnet
Notice in this case the cpio format type is newc.
Now copy that new custom initrd into your temporary iso work environment blowing away the old file:
cp /tmp/work/initrd.img.vmxnet /tmp/EL5server53/isolinux/initrd.img
Since we've gone to this much trouble lets go ahead and make one more change at this point...
cd /tmp/EL5server53/isolinux/
Edit the isolinux.cfg file with your favorite text editor.
Change the line:
append ks=http://web-kickstart.linux.ncsu.edu/ks.py ksdevice=link noipv6 initrd=initrd.img
to
append ks=http://web-kickstart.linux.ncsu.edu/ks.py ksdevice=link noipv6 initrd=initrd.img text
This will force the installer to use text mode during the install. In this case we prefer text mode when installing into a VMware virtual guest because we don't have all of VMware tools running during the install process.
Create our custom iso file
Now as a last step we need to create the new iso file. We'll call it server-5.3-x86-eoswebks.iso and write it to /tmp. We can do that with this command:
mkisofs -R -T -J -V "server53x86" -b isolinux/isolinux.bin \ -c isolinux/boot.cat -o /tmp/server-5.3-x86-eoswebks.iso -no-emul-boot \ -boot-load-size 4 -boot-info-table /tmp/EL5server53
Now you need to copy the custom iso file you just created to your VMware ESX or ESXi storage area for install isos. At this point you should be able to install Realm Linux i386 server 5.3 virtual machines with a vmxnet 2 NIC. You need to repeat this process for each CD type for Realm Linux. So you need a x86_64 server and a x86 and x86_64 client CD before you are finished.
ITECS created ISOs in AFS
ITECS created a set of Realm Linux 5 install ISOs and they are available in an AFS locker if this would be useful to other groups on campus. The ISOs can be copied from:
/afs/eos.ncsu.edu/project/ESX-ISO/
The ISOs were built with the vendor provided VMware Tools version: VMwareTools-7303-158874. This is what we use on most Realm Linux virtual machines in our VMware clusters.
Realm Linux post install
If you use these ISOs and you set up a Realm Linux 5 box with a VMXNET 2 NIC you need to plan for what to do post install. You have two choices:
-
Use the open VMware tools provided by Campus Linux Services by adding the line:
package @ realmlinux-vmware
in the web kickstart config file. This is the simplest method.
- set up things so that post install you prepare for building the VMware Tools via the perl script VMware provides with the tools. This is usually how ITECS sets up Virtual machines.
Here is a listing of a %post script for a Realm Linux 5 server using the latter method that we currently use. This involves modifying the /etc/rc.d/rc.local script on the machine. It run the first time the machine boots up and after that on kernel uprade boots.
%post
chkconfig smartd off
rpm -ivh http://install.linux.ncsu.edu/pub/yum/itecs/ESX/VMwareTools-3.5.0-158874.i386.rpm
# We DO NOT want ntpd running. We use VMware tools to keep the machines in time sync.
# ntpd runs on the ESX server itself for this purpose.
cat << EOF >> /root/fix-ntpd
#!/bin/bash
sleep 2m
/sbin/chkconfig --level 123456 ntpd off
/sbin/service ntpd stop > /dev/null 2>&1
EOF
chmod 700 /root/fix-ntpd
at -f /root/fix-ntpd now > /dev/null 2>&1
cat << EOF >> /etc/rc.d/rc.local
if [ ! -e /lib/modules/\`uname -r\`/misc/.vmware_installed ]; then
echo -n "Building VMware tools kernel modules: "
/usr/bin/vmware-config-tools.pl --default > /dev/null 2>&1
vmwaretoolscompile=\$?
if [ \$vmwaretoolscompile = "0" ]; then
echo -en "[ "
echo -en "\\033[0;32m"
echo -en "OK"
echo -en "\\033[0;39m"
echo -e " ]"
sleep 5
touch /lib/modules/\`uname -r\`/misc/.vmware_installed
/etc/init.d/network stop
/sbin/rmmod vmxnet
/sbin/depmod -a
/sbin/modprobe vmxnet
/etc/init.d/network start
# AFS couldn't start earlier because there was no NIC. So start it now.
# comment out the next line if using flexible or E1000 NIC.
/etc/init.d/afs start
else
echo -en "["
echo -en "\\033[0;31m"
echo -en "FAILED"
echo -en "\\033[0;39m"
echo -e "]"
sleep 5
fi
fi
EOF
Here is what the actual rc.local script looks like post install afer applying the above kickstart scripts:
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
if [ ! -e /lib/modules/`uname -r`/misc/.vmware_installed ]; then
echo -n "Building VMware tools kernel modules: "
/usr/bin/vmware-config-tools.pl --default > /dev/null 2>&1
vmwaretoolscompile=$?
if [ $vmwaretoolscompile = "0" ]; then
echo -en "[ "
echo -en "\033[0;32m"
echo -en "OK"
echo -en "\033[0;39m"
echo -e " ]"
sleep 5
touch /lib/modules/`uname -r`/misc/.vmware_installed
/etc/init.d/network stop
/sbin/rmmod vmxnet
/sbin/depmod -a
/sbin/modprobe vmxnet
/etc/init.d/network start
# AFS couldn't start earlier because there was no NIC. So start it now.
# comment out the next line if using flexible or E1000 NIC.
/etc/init.d/afs start
else
echo -en "["
echo -en "\033[0;31m"
echo -en "FAILED"
echo -en "\033[0;39m"
echo -e "]"
sleep 5
fi
fi
ITECS will update this article whenever we have VMXNET 3 as an option in our cluster. The above scripts should work fine with either VMXNET 2 or 3 network adapters.
Categories: Linux | vm
