## page was renamed from AfsOsd/Instructions <> = Client setup = == Client protocol extension == === Install a suitable kernel module === * via rpm {{{ % rpm -q kernel-module-openafs-`uname -r` kernel-module-openafs-2.6.18-128.1.10.el5-1.4.7-68.2.SL5.i686 }}} * or manually 1. From the source directory '''copy''' {{{ src/libafs/MODLOAD-`uname -r`-*/libafs.ko }}} . to {{{ /lib/modules/`uname -r`/kernel/fs/openafs/libafs.ko }}} 1. {{{ depmod }}} === Restart of the AFS service === Upon startup of the ''afs'' service, the following should show up: {{{ # service afs restart ... # dmesg | grep patch libafs with 'rxosd support' 'vicep-access' patches }}} (''vicep-access'' is not necessary for RxOSD operation per se, but is required for making proper use of a Lustre or GPFS backend.) Because ''afs restart'' doesn't work often, is is better to '''reboot''' the client. == Proper afs commands == Install a suitable openafs package {{{ # rpm -q openafs openafs-1.4.10.osd.vpa.r691-77alma.sl5.x86_64 # fs protocol Enabled protocols are RXOSD (1 parallel streams on connections with rtt > 10 ms). # osd help osd: Commands are: ... }}} == Memory cache == When using a cluster filesystem backend, memcache has proven to be the fastest alternative (at least for Lustre via Infiniband). In ''/etc/sysconfig/afs'': {{{ OPTIONS="-memcache -stat 8192 -chunksize 20 -daemons 8 -volumes 64" }}} This selects memcache and boosts the chunk size to 1MB. To avoid a low overall number of cache chunks: {{{ CACHESIZE="262144" }}} 256 MB seems a reasonable size. == Accessing a cluster filesystem backend == In Zeuthen, the Lustre symlinks is handled by the ''SLZ_vicep_lustre_links'' RPM. Generally (e.g. Lustre): * Make sure the shared vice partition is accessible: {{{ # ln -s /vicepal /lustre/... }}} . or resp. {{{ # mount -t lustre 141.34.218.7@tcp:/zn_test /vicept }}} . (see "Using a Lustre storage backend" below) * make the client detect the vice partition {{{ # service afs-vicep-access restart ... # dmesg | tail ... Visible OSD 2 lun 37 cell 33 == ifh.de }}} = Fileserver setup = == Adding an rxosd service == It is sensible to have ''fileserver'' and ''rxosd'' processes share the same machines. This way, access to files even in a single volume can be spread to all fileservers (given that clients don't access the same (set of) file(s) all the time). * create a dedicated vice partition {{{ # mkdir /vicepc # mount /vicepc # touch /vicepc/OnlyRxosd }}} . The last command ensures that the fileserver process will not claim this partition. * create the service instance (with admin privileges or `-localauth`) {{{ # bos create rxosd simple /usr/afs/bin/rxosd }}} . (!) There is no harm in doing this even before creating a vice partition. /!\ Please note the discussion of '''Lustre''' storage backends below. * add the new OSD to the OSDDB {{{ # osd createosd -id -name -ip -lun 1m 64g }}} id:: must not have existed in the OSDDB before, check with `osd l -all` osd-name:: must also be unique lun:: numeric representation of the vice partition, ''/vicepa'' is 0, ''/vicepc'' is 2 and ''/vicepal'' is 37 size constraints:: 1m and 64g are arbitrary. /!\ Note that the minimum size directly influences what files get stored to OSD. (!) If this is done before creating vice partition and/or service instance, the OSD will appear as "down" in the `osd l` output. No harm done. == Using a Lustre storage backend == Basically, this is the same process as adding a regular ''rxosd'' service. The vice partition resides in a Lustre filesystem, however. The ''rxosd'' process must be launched with a special parameter: {{{ # bos create rxosd simple "/usr/afs/bin/rxosd -lustrehack" }}} (!) A larger maximum file size for your Lustre-OSD is not required, the fileserver will never get to see a size that surpasses any client's cache size. === Lustre mountpoint === E.g. {{{ # mount -t lustre 141.34.218.7@tcp:/zn_test /vicept # touch /vicept/OnlyRxosd }}} The rest is identical to the above. === Symlink === If Lustre is mounted anywhere else: {{{ # ln -s /lustre/vpxl /vicepal # touch /vicepal/OnlyRxosd # touch /vicepal/AlwaysAttach }}} The last command enables the use of a regular directory as a vice partition. The rest is identical to the above. == Verifying the correct setup of the OSD == Assuming both service instance and vice partition were available at the time of updating the database, `osd l` should contain something along the lines of {{{ 2 zyklop20-al 10694 gb 0.0 % up 64 64 zyklop20 37 (1mb-2tb) }}} The "up" signifies that the OSD is ready. A "down" entry does not have to be a problem. It can (among other things) mean * an unreachable host * a missing ''rxosd'' process on the host machine {{{ bos status }}} * a missing vice partition on the host machine * a wrong LUN {{{ bos getlog RxosdLog ... Thu Jun 4 13:20:00 2009 [0] FiveMinuteCheckLWP: statfs for /vicepam failed with 2 }}} . Here, the LUN had been one too large initially. = Database servers = == Packages == If appropriate ''openafs-server'' packages are in use, there is nothing to do. {{{ # rpm -q openafs-server openafs-server-1.4.10.osd.vpa.r657-74alma.2.sl5.x86_64 # rpm -qf /usr/afs/bin/osdserver openafs-server-1.4.10.osd.vpa.r657-74alma.2.sl5.x86_64 }}} For a minimum intrusion on existing DB servers, use the dedicated ''openafs-osdserver'' package: {{{ % rpm -q openafs-server openafs-osdserver openafs-server-1.4.7-68.2.SL5.x86_64 openafs-osdserver-1.4.10.osd.vpa.r689-76alma.sl5.x86_64 % rpm -qf /usr/afs/bin/osdserver openafs-osdserver-1.4.10.osd.vpa.r689-76alma.sl5.x86_64 }}} It has no dependencies and will install regardless of the rest of your local AFS installation. == Launching the OSDDB service == With admin privileges: {{{ bos create osddb simple /usr/afs/bin/osdserver }}} Once the osdservers are up and running, clients should be able to see this: {{{ % osd l id name(loc) ---total space--- flag prior. own. server lun size range 1 local_disk wr rd (0kb-1mb) }}} The ''local_disk'' entry is a default. = Enabling OSD for Volumes ... = {{{ # vos setfields osdtestvol2 -osd 1 }}} = Update Client and Server = * At First: * check if all packages are aviable and that no errors will be send because of a downtime or a not available service * Stop AFS process . On Client: . . {{{ % /etc/init.d/afs stop }}} . On Server: (maybe Client process also running on a server!) . {{{ % /etc/init.d/afs-server stop }}} * Update the packages . First Erase Kernel Package: . {{{ % rpm -e kernel-module-openafs-2.6.18-164.11.1.el5-1.4.11.osd.vpa.r901-80alma.sl5.x86_64 }}} . Update all the other packages, Kernel-module should be installed too. Add openafs-server on a server machine. . {{{ % yum update openafs-authlibs-devel openafs-authlibs openafs-client openafs openafs-krb5 openafs-debug openafs-compat openafs-devel }}} * Start AFS process . On Client: . {{{ % /etc/init.d/afs start }}} . On Server: . {{{ % /etc/init.d/afs-server start }}} * Start Vicep-Access on the Client . {{{ % /etc/init.d/afs-vicep-access status Enabled protocols are RXOSD (1 parallel streams on connections with rtt > 10 ms). % /etc/init.d/afs-vicep-access stop % /etc/init.d/afs-vicep-access start % /etc/init.d/afs-vicep-access status Enabled protocols are VICEP-ACCESS (with Lustre hack) RXOSD (1 parallel streams on connections with rtt > 10 ms). }}} . * Check Configuration . On Client: . {{{ % dmesg ... Visible OSD 2 lun 37 cell 33 == ifh.de }}} . {{{ % KUSL.pl }}} . dont want to install something . . On Server: . . {{{ % bos status localhost -localauth Instance fs, currently running normally. Auxiliary status is: file server running. Instance rxosd, currently running normally. }}} . Or other Instances.