Installation

Overview

SL hosts are installed using kickstart

• SL3 online manual

• SL4 online manual

• SL5 online manual

The repository is mirrored from the ftp server at FNAL and is located on the installation server, pallas.ifh.de, in /nfs1/pallas/SL. The host profiles for the kickstart install are kept in /project/linux/SL/profiles, and some files needed during the postinstallation, before the AFS client is available, in /project/linux/SL/firstboot (accessible through the http server running on pallas). More files, and most utility scripts, are located in /project/linux/SL.

Installation takes the following steps:

• Configure the host in VAMOS
  This is important, because several variables must be set correctly since they are needed by the tools used in the following steps.

• Create a system profile
  Using CKS, information from VAMOS and possibly from the AMS directory or the live host, a kickstart file is generated that will steer the installation process.

• Activate private key distribution
  Only after this step, the host will be able to request its private keys and initial configuration cache from mentor.

• Prepare system boot into installation
  Current options include PXE, CD-ROM, and hard disk. Other possible methods like USB stick, multiple floppies, or a tftp grub floppy are not yet available.

• Boot the system into installation

  
  During boot, the system will load the kernel and initrd made available in step (d). Networking information comes from a DHCP server (possible with all methods) or is provided on the kernel command line (CD-ROM & hard disk methods only). The installation system locates the kickstart profile. Information is from a kernel command line provided by the tftp server (PXE method), manually (CD-ROM method), or the script preparing the hard disk boot.

  The kickstart profile contains all other information needed, including the repository location, partitioning & package selection, and a postinstall script that will do some very basic configuration and retrieve and install a one-time init script. After the first reboot, this init script (executing as the very last one) will retrieve the system's private keys and initial vamos configuration cache, and then bootstrap our site mechanisms for system maintenance.

System Configuration in VAMOS

Choose a default derived from sl3-def. Defaults starting with "sl3-" are 32bit, those starting with "sl3a-" are 64bit. These will mainly differ in the settings for OS_ARCH and AFS_SYSNAME (see the sl3a-mod modifier). 64bit capable systems can run the 32bit version as well. For SL4/5/..., obviously replace sl3 with the appropriate.

OS_ARCH is read by several tools in the following steps to determine what to install. The same is true for CF_SL_release: This variable determines which minor SL release the system will use. Both OS_ARCH and CF_SL_release affect the choice of installation kernel & initrd, installation repository, and yum repositories for updating and installing additional packages.

It should now be safe to do this step without disabling sue on the system, since sue.bootstrap will no longer permit OS_ARCH to change.

Run the Workflow whenever a system changes between major SL releases (say, from SL3 to SL4 or back), since some tools (scout) can only consult the netgroups to decide how things should be done. This is wrongwrongwrong, but ...

Creating System Profiles

This is done with the tool CKS.pl which reads "host.cks" files and creates "host.ks" files from them, using additional information from VAMOS, the AMS directory, or the live system still running SL, as well as pre/post script building blocks from /project/linux/SL/{pre|post}.

CKS.pl is located in /project/linux/SL/scripts, and is fully perldoc'd. There should be a link pointing to it in the profiles directory as well. A sample DEFAULT.cks with many comments is located in the same directory. The only SL4/5-specific option (SELINUX) is documented (and the default is almost certainly right).

To create a profile:

• You need
  • write access to /project/linux/SL/profiles
  • read access to VAMOS
  • ssh access to the system to install, if it is up and partitions to be kept
• Go into /project/linux/SL/profiles.
• Check whether a .cks file for your host exists.
  • If it does, and you find you have to modify the file, make sure it is not a link to some other file before you do so.
  • If it does not, create one by starting with a copy from a similar machine, or a copy of DEFAULT.cks.

  • NO host.cks IS NEEDED AT ALL if you just want to upgrade or reinstall a normal system without changing disk partitioning, since DEFAULT.cks is always read and should cover this case completely.

• Run CKS.pl, like this: ./CKS.pl host

  Always rerun CKS before installing a system, even if the existing .cks file looks fine. This is because the selection and the postinstall script are copied into the kickstart file, and the old file may no longer be correct.

• Watch the output. Make sure you understand what the profile is going to do to the machine! If in doubt, read and understand the

  <host>.ks file before actually installing.

  In particular, make sure you understand the partitioning, and any clearpart statements. Other than for DL5, these may wipe the disks even in "interactive" installs!

  Also make sure the SL release and architecture are what you want.

Activating Private Key Distribution

If you followed the instructions above (read the CKS output), you already know what to do:

ssh configsrv sudo prepare-ai <host>

This will activate the one-shot mechanism for giving the host (back) its private keys (root password, kerberos keyfile, vamos/ssh keys, ...). The init script retrieved during postinstall will start the script /products/ai/scripts/ai-web which will retrieve a tarball with these keys and a few other files from the install server. This works only once after each prepare-ai for a host. If after the installation the host has its credentials, it worked and no other system can possibly have them as well. If it hasn't, the keys are burned and have to be scrubbed. Hasn't happened yet, but who knows.

If ai-web fails, the system will retry after 5 minutes. Mails will be sent to linuxroot from both the AI server and the installing system, indicating that this happened. The reason is usually that the prepare-ai step was forgotten. Remember it has to be repeated before every reinstallation. The ai daemon writes a log /var/log/ai/ai_script.log.

Booting the system into installation

There are several options:

• Perl Script
  If the system is still running a working SL installation, this is the most convenient and reliable method: After logging on as root, run the script

  /project/linux/SL/scripts/SLU.pl yes please

  and either let the script reboot the system after the countdown, or interrupt the countdown with ^C and reboot (or have the user reboot) later. The script will create an additional, default, boot loader entry to start the installation system. By default, all needed information is appended to the kernel command line, including networking information. Hence not even DHCP is needed. The script comes with full perldoc documentation. Some additional options are available or may even be necessary for certain hosts (see 1.0).

• SL CD-ROM
  Images are http://pallas/SL/<release>/<arch>/images/SL/boot.iso The release and arch have to match exactly the planned installation, or the installation system will refuse to work.

  • If the system has a valid DHCP entry (inluding the MAC address):

    At the boot prompt enter "linux ks=http://pallas.ifh.de/SL/profiles/" If the system has more than one network interface, add "ksdevice=link". If the system has more than one network interface *connected*, instead add "ksdevice=eth0" or whatever is appropriate.

  • If the system has no valid DHCP entry yet: You have to add parameters like "ip=141.34.x.y netmask=255.255.255.0 gateway=141.34.x.1 dns=141.34.1.16" Or watch the log on the DHCP server, wait for the unknown MAC to appear, create a working entry for the host in /etc/dhcpd.conf, and restart the dhcp server. If you're quick enough, the client will receive an answer when it retries. Otherwise it has to be booted again. Don't forget to remove the CD before the first reboot, or installation will start all over.

• PXE
  This requires entries on both the DHCP and TFTP servers. The client will receive IP, netmask, gateway etc from the DHCP server, plus the information that it should fetch "pxelinux.0" from the TFTP server (actually, a) and run it. Then, pxelinux.0 will request the host configuration file (IP address in hex notation) from the TFTP server (a link in /tftpboot/pxelinux.cfg/). This will in turn tell pxelinux.0 which kernel & initrd to retrieve from the TFTP server, and what parameters the kernel should receive on the command line.

  • TFTP & DHCP (script)
    As root on pallas, run

    /project/linux/SL/scripts/pxe <host>

    This will add the right link for the system in /tftpboot/pxelinux.cfg and also attempt to update the DHCP configuration on the right server.

  • DHCP (manually)
    If the system has no valid DHCP entry yet, you have to use the last method given above in (b) to find out the MAC and create or complete the entry in the configuration file manually. In addition to IP and MAC, the following parameters have to be supplied:

    next-server 141.34.32.11;
    filename "pxelinux.0";

    If there was an incomplete entry for the system, these will already be present after running the "pxe" utility script.
  The changes on the DHCP server will be reverted the next time the dhcp feature runs, so no need for cleanup. To get rid of the link in /tftpboot/pxelinux.cfg, simply run (as root on pallas)

  /project/linux/SL/scripts/unpxe <host>

  If the client boots via PXE afterwards, it will pick up the default configuration which tells it to boot from its local disk. Anyway, using PXE is not recommended for systems which have no "one time boot menu" or a "request network boot" key stroke.

Package Handling & Automatic Updates

See the "aaru" feature for how all this (except kernels) is handled.

There are three distinct mechanisms for package handling on the client:

• aaru (package updates)
  Handled by the aaru feature, the scripts /sbin/aaru.yum.daily and /sbin/aaru.yum.boot run yum to update installed packages. Yum [2] is told to use specific repository descriptions for these tasks, which are created by /sbin/aaru.yum.create before, according to the values of VAMOS variables OS_ARCH, CF_SL_release, CF_YUM_extrarepos* and CF_DZPM_AGING.

• yumsel (addition and removal of packages)
  Handled by the aaru feature, the script /sbin/yumsel installs additional packages or removes installed ones. Configuration files for this task are read from /etc/yumsel.d/, which is populated by /sbin/yumsel.populate before, according to the values of VAMOS variables CF_yumsel_*.
  

  yumsel documentation, including the file format, is available with perldoc /sbin/yumsel

• KUSL(3) (everything related to kernels)<<BR> Handled by the kernel feature, this script deals with kernels and related packages (modules, source), according to the values of VAMOS variable Linux_kernel_version and a few others. On SL5, KUSL3 was replaced by KUSL, and this should happen eventually on SL3/4 as well. SL3/4 systems in update class "A" already use KUSL.pl.

SL Standard & Errata Packages

Errata are synced to pallas with /project/linux/SL/scripts/sync-pallas.pl (still manually). Packages to be installed additionally by /sbin/yumsel or updated by /sbin/aaru.yum.boot and /sbin/aaru.yum.daily are NOT taken from the errata mirror created like this, but instead from "staged errata" directories created (also, still manually) by the script /project/linux/SL/scripts/stage-errata. The sync/stage scripts send mail to linuxroot unless in dryrun mode. The stage_errata script is fully perldoc'ed, the others are too simple.

Addon Packages (Zeuthen)

These are generally found in /nfs1/pallas/SL/Z, with their (no)src rpms in /afs/ifh.de/packages/SRPMS/System and the source tarballs in /afs/ifh.de/packages/SOURCES (for .nosrc.rpms). Some come from external sources like the dag repository (http://dag.wieers.com/home-made/), freshrpms (http://freshrpms.net/) or the SuSE 8.2/9.0 distributions. These latter ones are typically not accompanied by a src rpm.

Available subdirectories under Z:

The distinction fo the INSTALL repositories is necessary because some of our add-ons do not work correctly when installed during initial system installation. Notice these repos may be populated by symlinks to packages or subdirectories, e.g. afs -> ../../x86_64/afs , but the metadata must be updated separately.

After adding a package, make it available to yum like this:

/project/linux/SL/scripts/UpdateRepo.pl -x /path/to/repo

Do not use createrepo manually.

Selectable Addon Packages (Zeuthen)

There's a way to provide packages in selectable repositories. For example, this was used to install an openafs-1.2.13 update on selected systems while the default for SL3 was still 1.2.11, and we didn't want to have 1.2.13 on every system.

These packages reside in directories Z/<major release>/extra/<arch>/<name> on the installation server. For example, the afs update packages for SL3/i386 would be in /nfs1/pallas/SL/Z/3/extra/i386/afs1213 . To have clients access this repository, set any vamos variable starting with CF_YUM_extrarepos (CF_YUM_extrarepos or CF_YUM_extrarepos_host or ...) to a space separated list of subdirectories in <arch>_extra.

For example, CF_YUM_extrarepos='afs1213' will make aaru.yum.create add this repository (accessible via nf or http) to the host's yum configuration.

To make available packages in such a repository, you must provide the full path, including the *sub*directory, to the repo update script:

/project/linux/SL/scripts/UpdateRepo.pl -x /nfs1/pallas/SL/Z/3/extra/i386/afs1213

Note that matching kernel modules must still reside in a directory searched by the update script (see below). This should generally not cause problems since these aren't updated by yum anyway.

Additional Modules for Kernel Updates

Handled by the kernel feature, the script /usr/sbin/KUSL3.pl reads its information about which kernels to install from VAMOS variables Linux_kernel_version and a few others, and carries out whatever needs to be done in order to install new kernels and remove old ones. The script is perldoc'ed.

Basically, set Linux_kernel_version in VAMOS, and on the host (after a sue.bootstrap) run KUSL3.pl. Make sure you like what it would do, then run KUSL[3].pl -x.

Kernels and additional packages are found in the repository mirror including the errata directory (CF_SL_release is used to find those), and in /afs/ifh.de/packages/RPMS/@sys/System (and some subdirectories).

If the variable Linux_kernel_modules is set to a (whitespace separated) list of module names, KUSL(3) will install (and require the availability of) the corresponding kernel-module rpm. For example, if Linux_kernel_version is 2.4.21-20.0.1.EL 2.4.21-27.0.2.EL, and Linux_kernel_modules is foo bar, the mandatory modules are:

• name	version	release
  kernel-module-foo-2.4.21-20.0.1.EL	latest	latest
  kernel-module-bar-2.4.21-20.0.1.EL	latest	latest
  kernel-module-foo-2.4.21-27.0.2.EL	latest	latest
  kernel-module-bar-2.4.21-27.0.2.EL	latest	latest

Generally speaking, kernel module packages must comply with the SL conventions. The new KUSL.pl will also handle packages complying with the kmod conventions introduced with RHEL5.

KUSL(3) will refuse to install a kernel if mandatory packages are not available. Non mandatory packages include kernel-source, sound modules, kernel-doc.

ALSA

This is not needed on SL4/5. For SL3 we no longer provide alsa modules as well.

ESD CAN Module (for PITZ Radiation Monitor)

This hasn't been done yet for SL4/5, and probably never will be.

This is similar to the ALSA modules, but;

• The srpms are in /packages/SRPMS/esdcan (different ACL from others due to proprietary license of source).
• There's no build script.
• Builds should be done manually, on pitzrap itself, and always against

  a fresh kernel-source package:

  1. remove the kernel-source package(s)
  2. install the right kernel-source package for the kernel you want to build the module for
  3. configure it:

     cd /usr/src/linux-2.4.....
     cp configs/kernel-2.4.21-i686.config .config
     sed -i 's/^\(EXTRAVERSION.*\)custom/\1/' Makefile
     make dep
     make clean

  4. install the srpm (it doesn't matter from which kernel build it is):

     rpm -ivh /packages/SRPMS/esdcan/kernel-module-esdcan-...3.3.3-1.src.rpm

  5. build:

     rpmbuild -ba [--define 'kernel 2.4.21....'] --target i686 kernel-module-esdcan.spec

  6. copy the .i686.rpms to /afs/.ifh.de/packages/i586_rhel30/System/esdcan, yum-arch and release

Nvidia

These are no longer provided for SL3/4.

Since nvidia no longer supports all cards in a single driver, "generations" had to be introduced to support all our PCs. There is a single SRPM, nvidia-driver-gx to build all binary packages from. It will only work on SL5.

The g1 "legacy driver" is required on the Dell Precision 350 with an NVS 280 SD (AGP) board. The new g2 driver should work an all newer systems. Notice the nvidia driver doesn't work on Xen kernels.

1. make sure the right kernel-devel RPMs are installed (there's a build requirement)
2. rebuild the SRPM, on 32bit:

   rpmbuild --rebuild --target i686 --define 'kernel 2.6.18-53.1.4.el5' --define 'nvgen 1' nvidia-driver-gx-...src.rpm
   rpmbuild --rebuild --target i686 --define 'kernel 2.6.18-53.1.4.el5' --define 'nvgen 2' nvidia-driver-gx-...src.rpm

   64bit:

   rpmbuild --rebuild --define 'build_module 1' --define 'kernel 2.6.18-53.1.4.el5' --define 'nvgen 2' nvidia-driver-gx-...src.rpm

For generation 1, we only need the 32bit packages because the Dell 350 doesn't support x86_64.

XFS

These modules only make sense on the x86_64 architecture since running xfs on 32bit EL is not considered a good idea.

XFS kernel modules are now included in the SL(4+) distribution, and there's no need to build them anymore - except when forgotten by the SL maintainers for some update kernel. When this happens, find the SRPM in the distribution mirror, and read the instructions in the spec. Or ask on the sl-users list.

Since SL5.4, no additional kernel-module package is needed any longer because the module is now part of the EL kernel. For the time being, a dummy module is still provided.

ARECA RAID (SL4)

As of kernel 2.6.9-67.0.1.EL (SL4) and 2.6.18-53.1.4.el5 (SL5), this driver is included in the distribution. The only system needing this driver (sbox1) hence no longer requires any special module package. And it's headed for the scrap pile.

Adding a new SL release

There are quarterly releases of SL, following Red Hat's updates to RHEL. Each new release must be made available for installation and updates. The procedure is the same for SL3 and SL4. Just substitute filenames and paths as appropriate:

Step 1: Mirror the new subdirectory

• Create a new logical volume on a:

  lvcreate -L 30G -n SL44 vg00

• Add an according line in /etc/fstab (mount with the acl option)

• Create the directory, mount the volume, and make sure permissions and security context are right:

  chgrp sysprog /nfs1/pallas/SL/44
  chmod g+w /nfs1/pallas/SL/44
  getfacl /nfs1/pallas/SL/43 | setfacl --set-file=- /nfs1/pallas/SL/44
  chcon system_u:object_r:httpd_sys_content_t /nfs1/pallas/SL/44

  The last command makes it possible to access the directory through apache. The chgrp and chmod are actually redundant if ACLs are used.
• Modify sync-pallas.pl to include the new release. Now sync-pallas.pl (do a dryrun first, and look wether additional subdirectories should be excluded).
• If you're using xrolling for testing, make a link like this:

  /nfs1/a/SL/44 -> 40rolling

• Check access through http

Step 2: Create staged errata directories

Modify /project/linux/SL/scripts/stage-errata.cf to include the new release. Note if you're trying 30rolling as a test for the release, you must configure 30rolling, not 304 (or whatever). The same for SL4. Now run stage-errata.

Step 3: Make the kernel/initrd available by TFTP for PXE boot

Run the script

/project/linux/SL/scripts/tftp_add_sl_release 44 i386

and accordingly for other releases and architectures. This will copy the kernel and the initrd, and create a pxelinux configuration file. You may still want/have to add a few lines in /tftpboot/pxelinux.cfg/default (for example, for Tier2 installs).

Step 4: Make the release available in VAMOS

Fire up the GUI, select "vars" as the top object, go to CF_SL_release, choose the "values_host" tab, and add the new value to the available choices. Set it on some test host.

Step 5: test

Make sure this works and sets the right link:

  /project/linux/SL/scripts/pxe <testhost>

Make sure this chooses the right directory:

  cd /project/linux/SL/profiles
  ./CKS.pl <testhost>

Make sure SLU works correctly:

  ssh <testhost>
  /project/linux/SL/scripts/SLU.pl yes please

Try an installation:

• activ-ai <testhost>

• then boot it

Try updating an existing installation:

• set CF_SL_release for the host in VAMOS

• sue.bootstrap

• sue.update aaru

• have a look into /var/log/yum.log, and check everything still works

Booting a rescue system

There are several ways to do this, including:

From CD1 of the distribution

Simply boot from CD1 of the distribution. At the boot prompt, type linux rescue.

Over the network using PXE

Make sure the system gets the "next-server" and "filename" responses from the dhcp server, but there's no link for the system in /tftpboot on the install server. At the boot prompt, then enter something like "sa53 rescue".

Building Kernel Packages

32-bit SL3

First install the kernel srpm (not kernel-source). Make your changes to the spec, add patches etc.

rpmbuild --sign -ba kernel-2.4.spec

This will build

• kernel.src.rpm
• kernel-source.i386.rpm
• kernel-doc.i386.rpm
• kernel-BOOT.i386.rpm

rpmbuild --sign -ba --target i686 kernel-2.4.spec

This will build

• kernel.i686.rpm
• kernel-smp.i686.rpm
• kernel-hugeme.i686.rpm
• kernel-unsupported.i686.rpm
• kernel-smp-unsupport.i686.rpm
• kernel-hugemem-unsupported.i686.rpm

Trying to turn off build of the hugemen kernel breaks the spec.

Additional modules for these are built as for any other SL kernel, with one exception of course:

Building the 1.2.x kernel-module-openafs (and openafs) packages

For ordinary SL kernels, this is done at FNAL, hence we needn't bother. But for our own kernels, or if we want to apply a change to the SRPM, we have to do this ourselves.

For each kernel version you want to build modules for, install the kernel-source, kernel, and kernel-smp RPMs on the build system, they're all needed. Then:

• To build the base packages on i686:

  PATH=/usr/kerberos/bin:$PATH rpmbuild --rebuild --sign --define 'kernel 2.4.21...' openafs-...src.rpm

• To build the kernel module packages (UP and SMP) on i686:

  PATH=/usr/kerberos/bin:$PATH rpmbuild --rebuild --sign --target i686 --define 'kernel 2.4.21...' openafs-...src.rpm

• To build the base packages plus the kernel modules (UP and SMP) on x86_64:

  PATH=/usr/kerberos/bin:$PATH rpmbuild --rebuild --sign --define 'kernel 2.4.21...' openafs-...src.rpm

• To build just the kernel modules (UP and SMP) on x86_64:

  PATH=/usr/kerberos/bin:$PATH rpmbuild --rebuild --sign --define 'kernel 2.4.21...' --define 'build_modules 1' openafs-...src.rpm

• To build just the kernel module for ia32e (intel CPUs) on x86_64:

  PATH=/usr/kerberos/bin:$PATH rpmbuild --rebuild --sign --target ia32e --define 'kernel 2.4.21...' openafs-...src.rpm

Building the NEW 1.4.x kernel-module-openafs packages

As of December 2006, there exists a unified SRPM for OpenAFS 1.4.2+, which doesn't have the build problems described above, and works in exactly the same way on SL3, SL4, and SL5. It's named openafs.SLx, but will create packages named openafs with SL3, SL4, SL5 in the release number. The SRPM can (and should) be rebuilt without being root. The steps are the same on every platform:

First install the right kernel-source (SL3) or matching kernel[-smp|-largesmp|-xen|...]-devel package for the target kernel.

Then run:

rpmbuild --rebuild --sign --target  i686 --define 'kernel 2.4.21...' --define 'build_modules 1' openafs.SLx-...src.rpm

There's always just one module built per invocation. Building on SMP systems is ok.

Supported targets include i686, athlon (SL3 only), ia32e (SL3 only, must build on 64bit system), x86_64 (must build on 64bit system), ia64 (untested).

Supported kernel flavours include smp (SL3/4), hugemem (SL3/4), largesmp (SL4), xen (SL5), xenU (SL4), PAE (SL5).

Notes on 10GbE

Cards

So far we have used two different cards:

• Intel 82598EB

• Broadcom NetXtreme II BCM57710

The intel card is used in zyklop41..43. It has no PXE ROM (but supposedly can be flashed), is cooled passively, and works out of the box. It can also do 1Gb, but not 10/100. Throughput is not very good, maybe because the cards won't fit into the PCIex8 Slot in the Dell R510.

The Broadcom card is used in dusk & dawn so far. It does have a PXE ROM (not tested yet though), has a cooling fan , and currently needs an updated driver because the bnx2x driver in the SL5.6 kernel (2.6.18-238.*) has a bug. The host will need a powercycle after the SL5.6 driver was used, a reset is not sufficient. The driver update is packaged in kernel-module-bnx2 rpms. These can currently only be built if the target kernel is running on the build host. The driver update should become unnecessary with SL5.7. The cards (purchased from Dell) can only do 10000baseT - not even Gigabit is supported, hence they must be connected to a 10GbE switch with an appropriate cable.

LRO/TPA and bridged networking

Large Receive Offload is a hardware feature significantly enhancing throughput. Alas, it is incompatible with bridged networking. To use such a device with Xen or KVM virtualization, LRO has to be turned off by specifying options bnx2x disable_tpa=1 in /etc/modprobe.conf (or modprobe.d/...). Otherwise, all kinds of weird things happen although the network basically works. Unfortunately, this reduces the throughput as measured with qperf (under the Xen kernel in DOM0) to just 30% of what can be achieved with the normal kernel and LRO enabled. NB "TPA" = "Transparent Packet Aggregation".

References

http://www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/sysadmin-guide/ch-kickstart2.html

http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/en-US/System_Administration_Guide_/Kickstart_Installations.html

http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Installation_Guide/ch-kickstart2.html

http://linux.duke.edu/projects/yum/