- Client setup
- Fileserver setup
- Database servers
- Enabling OSD for Volumes ...
- Update Client and Server
Client protocol extension
Install a suitable kernel module
- via rpm
% rpm -q kernel-module-openafs-`uname -r` kernel-module-openafs-2.6.18-128.1.10.el5-1.4.7-68.2.SL5.i686
- or manually
From the source directory copy
Restart of the AFS service
Upon startup of the afs service, the following should show up:
# service afs restart ... # dmesg | grep patch libafs with 'rxosd support' 'vicep-access' patches
(vicep-access is not necessary for RxOSD operation per se, but is required for making proper use of a Lustre or GPFS backend.)
Because afs restart doesn't work often, is is better to reboot the client.
Proper afs commands
Install a suitable openafs package
# rpm -q openafs openafs-1.4.10.osd.vpa.r691-77alma.sl5.x86_64 # fs protocol Enabled protocols are RXOSD (1 parallel streams on connections with rtt > 10 ms). # osd help osd: Commands are: ...
When using a cluster filesystem backend, memcache has proven to be the fastest alternative (at least for Lustre via Infiniband).
OPTIONS="-memcache -stat 8192 -chunksize 20 -daemons 8 -volumes 64"
This selects memcache and boosts the chunk size to 1MB. To avoid a low overall number of cache chunks:
256 MB seems a reasonable size.
Accessing a cluster filesystem backend
In Zeuthen, the Lustre symlinks is handled by the SLZ_vicep_lustre_links RPM.
Generally (e.g. Lustre):
- Make sure the shared vice partition is accessible:
# ln -s /vicepal /lustre/...
- or resp.
# mount -t lustre 184.108.40.206@tcp:/zn_test /vicept
- (see "Using a Lustre storage backend" below)
- make the client detect the vice partition
# service afs-vicep-access restart ... # dmesg | tail ... Visible OSD 2 lun 37 cell 33 == ifh.de
Adding an rxosd service
It is sensible to have fileserver and rxosd processes share the same machines. This way, access to files even in a single volume can be spread to all fileservers (given that clients don't access the same (set of) file(s) all the time).
- create a dedicated vice partition
# mkdir /vicepc <edit /etc/fstab> # mount /vicepc # touch /vicepc/OnlyRxosd
- The last command ensures that the fileserver process will not claim this partition.
create the service instance (with admin privileges or -localauth)
# bos create <server-machine> rxosd simple /usr/afs/bin/rxosd
There is no harm in doing this even before creating a vice partition. Please note the discussion of Lustre storage backends below.
- add the new OSD to the OSDDB
# osd createosd -id <id> -name <osd-name> -ip <machine-ip> -lun <lun> 1m 64g
must not have existed in the OSDDB before, check with osd l -all
- must also be unique
numeric representation of the vice partition, /vicepa is 0, /vicepc is 2 and /vicepal is 37
- size constraints
- 1m and 64g are arbitrary.
Note that the minimum size directly influences what files get stored to OSD. If this is done before creating vice partition and/or service instance, the OSD will appear as "down" in the osd l output. No harm done.
Using a Lustre storage backend
Basically, this is the same process as adding a regular rxosd service. The vice partition resides in a Lustre filesystem, however. The rxosd process must be launched with a special parameter:
# bos create <server-machine> rxosd simple "/usr/afs/bin/rxosd -lustrehack"
A larger maximum file size for your Lustre-OSD is not required, the fileserver will never get to see a size that surpasses any client's cache size.
# mount -t lustre 220.127.116.11@tcp:/zn_test /vicept # touch /vicept/OnlyRxosd
The rest is identical to the above.
If Lustre is mounted anywhere else:
# ln -s /lustre/vpxl /vicepal # touch /vicepal/OnlyRxosd # touch /vicepal/AlwaysAttach
The last command enables the use of a regular directory as a vice partition. The rest is identical to the above.
Verifying the correct setup of the OSD
Assuming both service instance and vice partition were available at the time of updating the database, osd l should contain something along the lines of
2 zyklop20-al 10694 gb 0.0 % up 64 64 zyklop20 37 (1mb-2tb)
The "up" signifies that the OSD is ready.
A "down" entry does not have to be a problem. It can (among other things) mean
- an unreachable host
a missing rxosd process on the host machine
bos status <osd-machine>
- a missing vice partition on the host machine
- a wrong LUN
bos getlog <osd-machine> RxosdLog ... Thu Jun 4 13:20:00 2009  FiveMinuteCheckLWP: statfs for /vicepam failed with 2
- Here, the LUN had been one too large initially.
If appropriate openafs-server packages are in use, there is nothing to do.
# rpm -q openafs-server openafs-server-1.4.10.osd.vpa.r657-74alma.2.sl5.x86_64 # rpm -qf /usr/afs/bin/osdserver openafs-server-1.4.10.osd.vpa.r657-74alma.2.sl5.x86_64
For a minimum intrusion on existing DB servers, use the dedicated openafs-osdserver package:
% rpm -q openafs-server openafs-osdserver openafs-server-1.4.7-68.2.SL5.x86_64 openafs-osdserver-1.4.10.osd.vpa.r689-76alma.sl5.x86_64 % rpm -qf /usr/afs/bin/osdserver openafs-osdserver-1.4.10.osd.vpa.r689-76alma.sl5.x86_64
It has no dependencies and will install regardless of the rest of your local AFS installation.
Launching the OSDDB service
With admin privileges:
bos create <db-machine> osddb simple /usr/afs/bin/osdserver
Once the osdservers are up and running, clients should be able to see this:
% osd l id name(loc) ---total space--- flag prior. own. server lun size range 1 local_disk wr rd (0kb-1mb)
The local_disk entry is a default.
Enabling OSD for Volumes ...
# vos setfields osdtestvol2 -osd 1
Update Client and Server
- At First:
- check if all packages are aviable and that no errors will be send because of a downtime or a not available service
- Stop AFS process
- On Client: .
% /etc/init.d/afs stop
- On Server: (maybe Client process also running on a server!)
% /etc/init.d/afs-server stop
- Update the packages
- First Erase Kernel Package:
% rpm -e kernel-module-openafs-2.6.18-164.11.1.el5-1.4.11.osd.vpa.r901-80alma.sl5.x86_64
- Update all the other packages, Kernel-module should be installed too. Add openafs-server on a server machine. .
% yum update openafs-authlibs-devel openafs-authlibs openafs-client openafs openafs-krb5 openafs-debug openafs-compat openafs-devel
- Start AFS process
- On Client:
% /etc/init.d/afs start
- On Server: .
% /etc/init.d/afs-server start
- Start Vicep-Access on the Client
% /etc/init.d/afs-vicep-access status Enabled protocols are RXOSD (1 parallel streams on connections with rtt > 10 ms). % /etc/init.d/afs-vicep-access stop % /etc/init.d/afs-vicep-access start % /etc/init.d/afs-vicep-access status Enabled protocols are VICEP-ACCESS (with Lustre hack) RXOSD (1 parallel streams on connections with rtt > 10 ms).
- Check Configuration
- On Client: .
% dmesg ... Visible OSD 2 lun 37 cell 33 == ifh.de
- dont want to install something . . On Server: . .
% bos status localhost -localauth Instance fs, currently running normally. Auxiliary status is: file server running. Instance rxosd, currently running normally.
- Or other Instances.