6249
Comment:
|
8216
|
Deletions are marked like this. | Additions are marked like this. |
Line 7: | Line 7: |
* via rpm |
|
Line 9: | Line 11: |
kernel-module-openafs-2.6.18-128.1.10.el5-1.4.7-68.2.SL5.i686 }}} | kernel-module-openafs-2.6.18-128.1.10.el5-1.4.7-68.2.SL5.i686 }}} * or manually 1. From the source directory '''copy''' {{{ src/libafs/MODLOAD-`uname -r`-*/libafs.ko }}} . to {{{ /lib/modules/`uname -r`/kernel/fs/openafs/libafs.ko }}} 1. {{{ depmod }}} |
Line 11: | Line 28: |
Upon startup of the ''afs'' service, the following should show up: {{{ | Upon startup of the ''afs'' service, the following should show up: {{{ |
Line 14: | Line 33: |
# dmesg | grep patch libafs with 'rxosd support' 'vicep-access' patches }}} |
# dmesg | grep patch libafs with 'rxosd support' 'vicep-access' patches }}} |
Line 18: | Line 37: |
To have a clean AFS Installation, is is better to '''reboot''' the client. |
Because ''afs restart'' doesn't work often, is is better to '''reboot''' the client. |
Line 22: | Line 41: |
Install a suitable openafs package {{{ | Install a suitable openafs package {{{ |
Line 24: | Line 45: |
openafs-1.4.10.osd.vpa.r691-77alma.sl5.x86_64 | openafs-1.4.10.osd.vpa.r691-77alma.sl5.x86_64 |
Line 26: | Line 47: |
Enabled protocols are RXOSD (1 parallel streams on connections with rtt > 10 ms). | Enabled protocols are RXOSD (1 parallel streams on connections with rtt > 10 ms). |
Line 29: | Line 50: |
... }}} |
... }}} |
Line 34: | Line 55: |
In ''/etc/sysconfig/afs'': {{{ OPTIONS="-memcache -stat 8192 -chunksize 20 -daemons 8 -volumes 64" }}} This selects memcache and boosts the chunk size to 1MB. To avoid a low overall number of cache chunks: {{{ CACHESIZE="262144" }}} |
In ''/etc/sysconfig/afs'': {{{ OPTIONS="-memcache -stat 8192 -chunksize 20 -daemons 8 -volumes 64" }}} This selects memcache and boosts the chunk size to 1MB. To avoid a low overall number of cache chunks: {{{ CACHESIZE="262144" }}} |
Line 44: | Line 71: |
* Make sure the shared vice partition is accessible: {{{ # ln -s /vicepal /lustre/... }}} or resp. {{{ # mount -t lustre 141.34.218.7@tcp:/zn_test /vicept }}} (see "Using a Lustre storage backend" below) * make the client detect the vice partition {{{ # service afs-vicep-access restart |
* Make sure the shared vice partition is accessible: {{{ # ln -s /vicepal /lustre/... }}} . or resp. {{{ # mount -t lustre 141.34.218.7@tcp:/zn_test /vicept }}} . (see "Using a Lustre storage backend" below) * make the client detect the vice partition {{{ # service afs-vicep-access restart |
Line 54: | Line 88: |
Visible OSD 2 lun 37 cell 33 == ifh.de }}} |
Visible OSD 2 lun 37 cell 33 == ifh.de }}} |
Line 61: | Line 95: |
* create a dedicated vice partition {{{ | * create a dedicated vice partition {{{ |
Line 65: | Line 100: |
# touch /vicepc/OnlyRxosd }}} The last command ensures that the fileserver process will not claim this partition. * create the service instance (with admin privileges or `-localauth`) {{{ # bos create <server-machine> rxosd simple /usr/afs/bin/rxosd }}} (!) There is no harm in doing this even before creating a vice partition. /!\ Please note the discussion of '''Lustre''' storage backends below. * add the new OSD to the OSDDB {{{ # osd createosd -id <id> -name <osd-name> -ip <machine-ip> -lun <lun> 1m 64g }}} |
# touch /vicepc/OnlyRxosd }}} . The last command ensures that the fileserver process will not claim this partition. * create the service instance (with admin privileges or `-localauth`) {{{ # bos create <server-machine> rxosd simple /usr/afs/bin/rxosd }}} . (!) There is no harm in doing this even before creating a vice partition. /!\ Please note the discussion of '''Lustre''' storage backends below. * add the new OSD to the OSDDB {{{ # osd createosd -id <id> -name <osd-name> -ip <machine-ip> -lun <lun> 1m 64g }}} |
Line 77: | Line 117: |
size constraints:: 1m and 64g are arbitrary. /!\ Note that the minimum size directly influences what files get stored to OSD. (!) If this is done before creating vice partition and/or service instance, the OSD will appear as "down" in the `osd l` output. No harm done. |
size constraints:: 1m and 64g are arbitrary. /!\ Note that the minimum size directly influences what files get stored to OSD. (!) If this is done before creating vice partition and/or service instance, the OSD will appear as "down" in the `osd l` output. No harm done. |
Line 84: | Line 121: |
Basically, this is the same process as adding a regular ''rxosd'' service. The vice partition resides in a Lustre filesystem, however. The ''rxosd'' process must be launched with a special parameter: {{{ # bos create <server-machine> rxosd simple "/usr/afs/bin/rxosd -lustrehack" }}} |
Basically, this is the same process as adding a regular ''rxosd'' service. The vice partition resides in a Lustre filesystem, however. The ''rxosd'' process must be launched with a special parameter: {{{ # bos create <server-machine> rxosd simple "/usr/afs/bin/rxosd -lustrehack" }}} |
Line 90: | Line 129: |
E.g. {{{ # mount -t lustre 141.34.218.7@tcp:/zn_test /vicept # touch /vicept/OnlyRxosd }}} |
E.g. {{{ # mount -t lustre 141.34.218.7@tcp:/zn_test /vicept # touch /vicept/OnlyRxosd }}} |
Line 97: | Line 139: |
Line 100: | Line 143: |
# touch /vicepal/AlwaysAttach }}} | # touch /vicepal/AlwaysAttach }}} |
Line 104: | Line 148: |
Assuming both service instance and vice partition were available at the time of updating the database, `osd l` should contain something along the lines of {{{ 2 zyklop20-al 10694 gb 0.0 % up 64 64 zyklop20 37 (1mb-2tb) }}} |
Assuming both service instance and vice partition were available at the time of updating the database, `osd l` should contain something along the lines of {{{ 2 zyklop20-al 10694 gb 0.0 % up 64 64 zyklop20 37 (1mb-2tb) }}} |
Line 109: | Line 156: |
Line 110: | Line 158: |
* a missing ''rxosd'' process on the host machine {{{ bos status <osd-machine> }}} |
* a missing ''rxosd'' process on the host machine {{{ bos status <osd-machine> }}} |
Line 113: | Line 164: |
* a wrong LUN {{{ bos getlog <osd-machine> RxosdLog ... Thu Jun 4 13:20:00 2009 [0] FiveMinuteCheckLWP: statfs for /vicepam failed with 2 }}} Here, the LUN had been one too large initially. |
* a wrong LUN {{{ bos getlog <osd-machine> RxosdLog ... Thu Jun 4 13:20:00 2009 [0] FiveMinuteCheckLWP: statfs for /vicepam failed with 2 }}} . Here, the LUN had been one too large initially. |
Line 121: | Line 174: |
If appropriate ''openafs-server'' packages are in use, there is nothing to do. {{{ | If appropriate ''openafs-server'' packages are in use, there is nothing to do. {{{ |
Line 125: | Line 180: |
openafs-server-1.4.10.osd.vpa.r657-74alma.2.sl5.x86_64 }}} For a minimum intrusion on existing DB servers, use the dedicated ''openafs-osdserver'' package: {{{ |
openafs-server-1.4.10.osd.vpa.r657-74alma.2.sl5.x86_64 }}} For a minimum intrusion on existing DB servers, use the dedicated ''openafs-osdserver'' package: {{{ |
Line 130: | Line 188: |
% rpm -qf /usr/afs/bin/osdserver openafs-osdserver-1.4.10.osd.vpa.r689-76alma.sl5.x86_64 }}} |
% rpm -qf /usr/afs/bin/osdserver openafs-osdserver-1.4.10.osd.vpa.r689-76alma.sl5.x86_64 }}} |
Line 135: | Line 194: |
With admin privileges: {{{ bos create <db-machine> osddb simple /usr/afs/bin/osdserver }}} Once the osdservers are up and running, clients should be able to see this: {{{ |
With admin privileges: {{{ bos create <db-machine> osddb simple /usr/afs/bin/osdserver }}} Once the osdservers are up and running, clients should be able to see this: {{{ |
Line 140: | Line 204: |
1 local_disk wr rd (0kb-1mb) }}} | 1 local_disk wr rd (0kb-1mb) }}} |
Line 142: | Line 207: |
= Enabling OSD for Volumes ... = {{{ # vos setfields osdtestvol2 -osd 1 }}} = Update Client and Server = * Stop AFS process . On Client: . . {{{ % /etc/init.d/afs stop }}} . On Server: (maybe Client process also running on a server!) . {{{ % /etc/init.d/afs-server stop }}} * Update the packages . First Erase Kernel Package: . {{{ % rpm -e kernel-module-openafs-2.6.18-164.11.1.el5-1.4.11.osd.vpa.r901-80alma.sl5.x86_64 }}} . Update all the other packages, Kernel-module should be installed too. Add openafs-server on a server machine. . {{{ % yum update openafs-authlibs-devel openafs-authlibs openafs-client openafs openafs-krb5 openafs-debug openafs-compat openafs-devel }}} * Start AFS process . On Client: . {{{ % /etc/init.d/afs start }}} . On Server: . {{{ % /etc/init.d/afs-server start }}} * Start Vicep-Access on the Client . {{{ % /etc/init.d/afs-vicep-access status Enabled protocols are RXOSD (1 parallel streams on connections with rtt > 10 ms). % /etc/init.d/afs-vicep-access stop % /etc/init.d/afs-vicep-access start % /etc/init.d/afs-vicep-access status Enabled protocols are VICEP-ACCESS (with Lustre hack) RXOSD (1 parallel streams on connections with rtt > 10 ms). }}} . * Check Configuration . On Client: . {{{ % dmesg ... Visible OSD 2 lun 37 cell 33 == ifh.de }}} . {{{ % KUSL.pl }}} . dont want to install something . . On Server: . . {{{ % bos status localhost -localauth Instance fs, currently running normally. Auxiliary status is: file server running. Instance rxosd, currently running normally. }}} . Or other Instances. |
Contents
Client setup
Client protocol extension
Install a suitable kernel module
- via rpm
% rpm -q kernel-module-openafs-`uname -r` kernel-module-openafs-2.6.18-128.1.10.el5-1.4.7-68.2.SL5.i686
- or manually
From the source directory copy
src/libafs/MODLOAD-`uname -r`-*/libafs.ko
- to
/lib/modules/`uname -r`/kernel/fs/openafs/libafs.ko
depmod
Restart of the AFS service
Upon startup of the afs service, the following should show up:
# service afs restart ... # dmesg | grep patch libafs with 'rxosd support' 'vicep-access' patches
(vicep-access is not necessary for RxOSD operation per se, but is required for making proper use of a Lustre or GPFS backend.)
Because afs restart doesn't work often, is is better to reboot the client.
Proper afs commands
Install a suitable openafs package
# rpm -q openafs openafs-1.4.10.osd.vpa.r691-77alma.sl5.x86_64 # fs protocol Enabled protocols are RXOSD (1 parallel streams on connections with rtt > 10 ms). # osd help osd: Commands are: ...
Memory cache
When using a cluster filesystem backend, memcache has proven to be the fastest alternative (at least for Lustre via Infiniband).
In /etc/sysconfig/afs:
OPTIONS="-memcache -stat 8192 -chunksize 20 -daemons 8 -volumes 64"
This selects memcache and boosts the chunk size to 1MB. To avoid a low overall number of cache chunks:
CACHESIZE="262144"
256 MB seems a reasonable size.
Accessing a cluster filesystem backend
In Zeuthen, the Lustre symlinks is handled by the SLZ_vicep_lustre_links RPM.
Generally (e.g. Lustre):
- Make sure the shared vice partition is accessible:
# ln -s /vicepal /lustre/...
- or resp.
# mount -t lustre 141.34.218.7@tcp:/zn_test /vicept
- (see "Using a Lustre storage backend" below)
- make the client detect the vice partition
# service afs-vicep-access restart ... # dmesg | tail ... Visible OSD 2 lun 37 cell 33 == ifh.de
Fileserver setup
Adding an rxosd service
It is sensible to have fileserver and rxosd processes share the same machines. This way, access to files even in a single volume can be spread to all fileservers (given that clients don't access the same (set of) file(s) all the time).
- create a dedicated vice partition
# mkdir /vicepc <edit /etc/fstab> # mount /vicepc # touch /vicepc/OnlyRxosd
- The last command ensures that the fileserver process will not claim this partition.
create the service instance (with admin privileges or -localauth)
# bos create <server-machine> rxosd simple /usr/afs/bin/rxosd
There is no harm in doing this even before creating a vice partition. Please note the discussion of Lustre storage backends below.
- add the new OSD to the OSDDB
# osd createosd -id <id> -name <osd-name> -ip <machine-ip> -lun <lun> 1m 64g
- id
must not have existed in the OSDDB before, check with osd l -all
- osd-name
- must also be unique
- lun
numeric representation of the vice partition, /vicepa is 0, /vicepc is 2 and /vicepal is 37
- size constraints
- 1m and 64g are arbitrary.
Note that the minimum size directly influences what files get stored to OSD. If this is done before creating vice partition and/or service instance, the OSD will appear as "down" in the osd l output. No harm done.
Using a Lustre storage backend
Basically, this is the same process as adding a regular rxosd service. The vice partition resides in a Lustre filesystem, however. The rxosd process must be launched with a special parameter:
# bos create <server-machine> rxosd simple "/usr/afs/bin/rxosd -lustrehack"
A larger maximum file size for your Lustre-OSD is not required, the fileserver will never get to see a size that surpasses any client's cache size.
Lustre mountpoint
E.g.
# mount -t lustre 141.34.218.7@tcp:/zn_test /vicept # touch /vicept/OnlyRxosd
The rest is identical to the above.
Symlink
If Lustre is mounted anywhere else:
# ln -s /lustre/vpxl /vicepal # touch /vicepal/OnlyRxosd # touch /vicepal/AlwaysAttach
The last command enables the use of a regular directory as a vice partition. The rest is identical to the above.
Verifying the correct setup of the OSD
Assuming both service instance and vice partition were available at the time of updating the database, osd l should contain something along the lines of
2 zyklop20-al 10694 gb 0.0 % up 64 64 zyklop20 37 (1mb-2tb)
The "up" signifies that the OSD is ready.
A "down" entry does not have to be a problem. It can (among other things) mean
- an unreachable host
a missing rxosd process on the host machine
bos status <osd-machine>
- a missing vice partition on the host machine
- a wrong LUN
bos getlog <osd-machine> RxosdLog ... Thu Jun 4 13:20:00 2009 [0] FiveMinuteCheckLWP: statfs for /vicepam failed with 2
- Here, the LUN had been one too large initially.
Database servers
Packages
If appropriate openafs-server packages are in use, there is nothing to do.
# rpm -q openafs-server openafs-server-1.4.10.osd.vpa.r657-74alma.2.sl5.x86_64 # rpm -qf /usr/afs/bin/osdserver openafs-server-1.4.10.osd.vpa.r657-74alma.2.sl5.x86_64
For a minimum intrusion on existing DB servers, use the dedicated openafs-osdserver package:
% rpm -q openafs-server openafs-osdserver openafs-server-1.4.7-68.2.SL5.x86_64 openafs-osdserver-1.4.10.osd.vpa.r689-76alma.sl5.x86_64 % rpm -qf /usr/afs/bin/osdserver openafs-osdserver-1.4.10.osd.vpa.r689-76alma.sl5.x86_64
It has no dependencies and will install regardless of the rest of your local AFS installation.
Launching the OSDDB service
With admin privileges:
bos create <db-machine> osddb simple /usr/afs/bin/osdserver
Once the osdservers are up and running, clients should be able to see this:
% osd l id name(loc) ---total space--- flag prior. own. server lun size range 1 local_disk wr rd (0kb-1mb)
The local_disk entry is a default.
Enabling OSD for Volumes ...
# vos setfields osdtestvol2 -osd 1
Update Client and Server
- Stop AFS process
- On Client: .
% /etc/init.d/afs stop
- On Server: (maybe Client process also running on a server!)
% /etc/init.d/afs-server stop
- Update the packages
- First Erase Kernel Package:
% rpm -e kernel-module-openafs-2.6.18-164.11.1.el5-1.4.11.osd.vpa.r901-80alma.sl5.x86_64
Update all the other packages, Kernel-module should be installed too. Add openafs-server on a server machine. .
% yum update openafs-authlibs-devel openafs-authlibs openafs-client openafs openafs-krb5 openafs-debug openafs-compat openafs-devel
- Start AFS process
- On Client:
% /etc/init.d/afs start
On Server: .
% /etc/init.d/afs-server start
- Start Vicep-Access on the Client
% /etc/init.d/afs-vicep-access status Enabled protocols are RXOSD (1 parallel streams on connections with rtt > 10 ms). % /etc/init.d/afs-vicep-access stop % /etc/init.d/afs-vicep-access start % /etc/init.d/afs-vicep-access status Enabled protocols are VICEP-ACCESS (with Lustre hack) RXOSD (1 parallel streams on connections with rtt > 10 ms).
- Check Configuration
On Client: .
% dmesg ... Visible OSD 2 lun 37 cell 33 == ifh.de
% KUSL.pl
dont want to install something . . On Server: . .
% bos status localhost -localauth Instance fs, currently running normally. Auxiliary status is: file server running. Instance rxosd, currently running normally.
- Or other Instances.