Differences between revisions 9 and 10
⇤ ← Revision 9 as of 2009-08-17 16:11:13 →
Size: 6556
Editor: WaltrautNiepraschk
Comment: Enabling OSD
← Revision 10 as of 2010-03-03 11:42:19 → ⇥
Size: 7763
Editor: ReneStandke
Comment:
Deletions are marked like this. Additions are marked like this.
Line 8: Line 8:
Line 10: Line 11:
kernel-module-openafs-2.6.18-128.1.10.el5-1.4.7-68.2.SL5.i686 }}} kernel-module-openafs-2.6.18-128.1.10.el5-1.4.7-68.2.SL5.i686
}}}
Line 12: Line 14:
  1. From the source directory '''copy''' {{{
src/libafs/MODLOAD-`uname -r`-*/libafs.ko }}}
  to {{{
/lib/modules/`uname -r`/kernel/fs/openafs/libafs.ko }}}
  2. {{{ 
depmod }}}		   1. From the source directory '''copy'''
  {{{
src/libafs/MODLOAD-`uname -r`-*/libafs.ko
}}}
  to
  {{{
/lib/modules/`uname -r`/kernel/fs/openafs/libafs.ko
}}}
  1. {{{
depmod
}}}
Line 20: Line 27:
Upon startup of the ''afs'' service, the following should show up: {{{ Upon startup of the ''afs'' service, the following should show up:

{{{
Line 23: Line 32:
# dmesg | grep patch                           
libafs with 'rxosd support' 'vicep-access' patches }}}
 # dmesg | grep patch
libafs with 'rxosd support' 'vicep-access' patches
}}}
Line 27: Line 36:
 
Line 31: Line 40:
Install a suitable openafs package {{{ Install a suitable openafs package

{{{
Line 33: Line 44:
openafs-1.4.10.osd.vpa.r691-77alma.sl5.x86_64  openafs-1.4.10.osd.vpa.r691-77alma.sl5.x86_64
Line 35: Line 46:
Enabled protocols are RXOSD (1 parallel streams on connections with rtt > 10 ms).  Enabled protocols are RXOSD (1 parallel streams on connections with rtt > 10 ms).
Line 38: Line 49:
... }}}
 ...
}}}
Line 43: Line 54:
In ''/etc/sysconfig/afs'': {{{
OPTIONS="-memcache -stat 8192 -chunksize 20 -daemons 8 -volumes 64" }}}
This selects memcache and boosts the chunk size to 1MB. To avoid a low overall number of cache chunks: {{{
CACHESIZE="262144" }}}		 In ''/etc/sysconfig/afs'':

{{{
OPTIONS="-memcache -stat 8192 -chunksize 20 -daemons 8 -volumes 64"
}}}
This selects memcache and boosts the chunk size to 1MB. To avoid a low overall number of cache chunks:

{{{
CACHESIZE="262144"
}}}
Line 53: Line 70:
 * Make sure the shared vice partition is accessible: {{{
# ln -s /vicepal /lustre/... }}}
 or resp. {{{
# mount -t lustre 141.34.218.7@tcp:/zn_test /vicept }}}
* Make sure the shared vice partition is accessible:
 {{{
# ln -s /vicepal /lustre/...
}}}
 or resp.
 {{{
# mount -t lustre 141.34.218.7@tcp:/zn_test /vicept
}}}
Line 58: Line 80:
 * make the client detect the vice partition {{{
# service afs-vicep-access restart 		  * make the client detect the vice partition
 {{{
# service afs-vicep-access restart
Line 63: Line 86:
Visible OSD 2 lun 37 cell 33 == ifh.de }}}
 Visible OSD 2 lun 37 cell 33 == ifh.de
}}}
Line 70: Line 93:
 * create a dedicated vice partition {{{  * create a dedicated vice partition
 {{{
Line 74: Line 98:
# touch /vicepc/OnlyRxosd }}}
  The last command ensures that the fileserver process will not claim this partition.
 * create the service instance (with admin privileges or `-localauth`) {{{
# bos create <server-machine> rxosd simple /usr/afs/bin/rxosd }}}
  (!) There is no harm in doing this even before creating a vice partition.
 # touch /vicepc/OnlyRxosd
}}}
  . The last command ensures that the fileserver process will not claim this partition.
 * create the service instance (with admin privileges or `-localauth`)
 {{{
# bos create <server-machine> rxosd simple /usr/afs/bin/rxosd
}}}
  . (!) There is no harm in doing this even before creating a vice partition.
Line 81: Line 107:
 * add the new OSD to the OSDDB {{{
# osd createosd -id <id> -name <osd-name> -ip <machine-ip> -lun <lun> 1m 64g }}}		  * add the new OSD to the OSDDB
 {{{
# osd createosd -id <id> -name <osd-name> -ip <machine-ip> -lun <lun> 1m 64g
}}}
Line 86: Line 114:
  size constraints:: 1m and 64g are arbitrary.     size constraints:: 1m and 64g are arbitrary.
Line 89: Line 116:
  
Line 93: Line 119:
Basically, this is the same process as adding a regular ''rxosd'' service. The vice partition resides in a Lustre filesystem, however. The ''rxosd'' process must be launched with a special parameter: {{{
# bos create <server-machine> rxosd simple "/usr/afs/bin/rxosd -lustrehack" }}}
 Basically, this is the same process as adding a regular ''rxosd'' service. The vice partition resides in a Lustre filesystem, however. The ''rxosd'' process must be launched with a special parameter:

{{{
# bos create <server-machine> rxosd simple "/usr/afs/bin/rxosd -lustrehack"
}}}
Line 99: Line 127:
E.g. {{{
# mount -t lustre 141.34.218.7@tcp:/zn_test /vicept 
# touch /vicept/OnlyRxosd }}}		 E.g.

{{{
# mount -t lustre 141.34.218.7@tcp:/zn_test /vicept
# touch /vicept/OnlyRxosd
}}}
Line 106: Line 137:
Line 109: Line 141:
# touch /vicepal/AlwaysAttach }}} # touch /vicepal/AlwaysAttach
}}}
Line 113: Line 146:
Assuming both service instance and vice partition were available at the time of updating the database, `osd l` should contain something along the lines of {{{
  2 zyklop20-al 10694 gb 0.0 % up 64 64 zyklop20 37 (1mb-2tb) }}}		 Assuming both service instance and vice partition were available at the time of updating the database, `osd l` should contain something along the lines of

{{{
  2 zyklop20-al 10694 gb 0.0 % up 64 64 zyklop20 37 (1mb-2tb)
}}}
Line 118: Line 154:
Line 119: Line 156:
 * a missing ''rxosd'' process on the host machine {{{
bos status <osd-machine> }}}		  * a missing ''rxosd'' process on the host machine
 {{{
bos status <osd-machine>
}}}
Line 122: Line 161:
 * a wrong LUN {{{
bos getlog <osd-machine> RxosdLog 
...
Thu Jun 4 13:20:00 2009 [0] FiveMinuteCheckLWP: statfs for /vicepam failed with 2 }}}		  * a wrong LUN
 {{{
bos getlog <osd-machine> RxosdLog
...
Thu Jun 4 13:20:00 2009 [0] FiveMinuteCheckLWP: statfs for /vicepam failed with 2
}}}
Line 130: Line 171:
If appropriate ''openafs-server'' packages are in use, there is nothing to do. {{{ If appropriate ''openafs-server'' packages are in use, there is nothing to do.

{{{
Line 134: Line 177:
openafs-server-1.4.10.osd.vpa.r657-74alma.2.sl5.x86_64 }}}
For a minimum intrusion on existing DB servers, use the dedicated ''openafs-osdserver'' package: {{{		 openafs-server-1.4.10.osd.vpa.r657-74alma.2.sl5.x86_64
}}}
For a minimum intrusion on existing DB servers, use the dedicated ''openafs-osdserver'' package:

{{{
Line 139: Line 185:
% rpm -qf /usr/afs/bin/osdserver 
openafs-osdserver-1.4.10.osd.vpa.r689-76alma.sl5.x86_64 }}}		 % rpm -qf /usr/afs/bin/osdserver
openafs-osdserver-1.4.10.osd.vpa.r689-76alma.sl5.x86_64
}}}
Line 144: Line 191:
With admin privileges: {{{
bos create <db-machine> osddb simple /usr/afs/bin/osdserver }}}
Once the osdservers are up and running, clients should be able to see this: {{{		 With admin privileges:

{{{
bos create <db-machine> osddb simple /usr/afs/bin/osdserver
}}}
Once the osdservers are up and running, clients should be able to see this:

{{{
Line 149: Line 201:
  1 local_disk wr rd (0kb-1mb) }}}   1 local_disk wr rd (0kb-1mb)
}}}
Line 156: Line 209:
= Update Client and Server =
 * Stop AFS process

On Client:

{{{
 % /etc/init.d/afs stop
}}}
On Server: (maybe Client process also running on a server!)

{{{
 % /etc/init.d/afs-server stop
}}}
 * Update the packages

First Erase Kernel Package:

{{{
 % rpm -e kernel-module-openafs-2.6.18-164.11.1.el5-1.4.11.osd.vpa.r901-80alma.sl5.x86_64
}}}
Update all the other packages, Kernel-module should be installed too. Add openafs-server on a server machine.

{{{
% yum update openafs-authlibs-devel openafs-authlibs openafs-client openafs openafs-krb5 openafs-debug openafs-compat openafs-devel
}}}
 * Start AFS process

On Client:

{{{
% /etc/init.d/afs start
}}}
On Server:

{{{
% /etc/init.d/afs-server start
}}}
 * Start Vicep-Access on the Client

{{{
% /etc/init.d/afs-vicep-access status
        Enabled protocols are RXOSD (1 parallel streams on connections with rtt > 10 ms).
% /etc/init.d/afs-vicep-access stop
% /etc/init.d/afs-vicep-access start
% /etc/init.d/afs-vicep-access status
        Enabled protocols are VICEP-ACCESS (with Lustre hack) RXOSD (1 parallel streams on connections with rtt > 10 ms).
}}}

Contents

  1. Client setup
    1. Client protocol extension
      1. Install a suitable kernel module
      2. Restart of the AFS service
    2. Proper afs commands
    3. Memory cache
    4. Accessing a cluster filesystem backend
  2. Fileserver setup
    1. Adding an rxosd service
    2. Using a Lustre storage backend
      1. Lustre mountpoint
      2. Symlink
    3. Verifying the correct setup of the OSD
  3. Database servers
    1. Packages
    2. Launching the OSDDB service
  4. Enabling OSD for Volumes ...
  5. Update Client and Server

Client setup

Client protocol extension

Install a suitable kernel module

  • via rpm 

% rpm -q kernel-module-openafs-`uname -r`
kernel-module-openafs-2.6.18-128.1.10.el5-1.4.7-68.2.SL5.i686
  • or manually

    1. From the source directory copy 

      src/libafs/MODLOAD-`uname -r`-*/libafs.ko
      to 
      /lib/modules/`uname -r`/kernel/fs/openafs/libafs.ko
    2. depmod

Restart of the AFS service

Upon startup of the afs service, the following should show up: 

# service afs restart
...
# dmesg | grep patch
libafs with   'rxosd support' 'vicep-access' patches

(vicep-access is not necessary for RxOSD operation per se, but is required for making proper use of a Lustre or GPFS backend.)

Because afs restart doesn't work often, is is better to reboot the client.

Proper afs commands

Install a suitable openafs package 

# rpm -q openafs
openafs-1.4.10.osd.vpa.r691-77alma.sl5.x86_64
# fs protocol
Enabled protocols are  RXOSD (1 parallel streams on connections with rtt > 10 ms).
# osd help
osd: Commands are:
...

Memory cache

When using a cluster filesystem backend, memcache has proven to be the fastest alternative (at least for Lustre via Infiniband).

In /etc/sysconfig/afs: 

OPTIONS="-memcache -stat 8192 -chunksize 20 -daemons 8 -volumes 64"

This selects memcache and boosts the chunk size to 1MB. To avoid a low overall number of cache chunks: 

CACHESIZE="262144"

256 MB seems a reasonable size.

Accessing a cluster filesystem backend

<!> In Zeuthen, the Lustre symlinks is handled by the SLZ_vicep_lustre_links RPM.

Generally (e.g. Lustre):

  • Make sure the shared vice partition is accessible: 
    # ln -s /vicepal /lustre/...
    or resp. 
    # mount -t lustre 141.34.218.7@tcp:/zn_test /vicept
    (see "Using a Lustre storage backend" below)
  • make the client detect the vice partition 
    # service afs-vicep-access restart
...
# dmesg | tail
...
Visible OSD 2 lun 37 cell 33 == ifh.de

Fileserver setup

Adding an rxosd service

It is sensible to have fileserver and rxosd processes share the same machines. This way, access to files even in a single volume can be spread to all fileservers (given that clients don't access the same (set of) file(s) all the time).

  • create a dedicated vice partition 
    # mkdir /vicepc
<edit /etc/fstab>
# mount /vicepc
# touch /vicepc/OnlyRxosd
    • The last command ensures that the fileserver process will not claim this partition.

  • create the service instance (with admin privileges or -localauth) 

    # bos create <server-machine> rxosd simple /usr/afs/bin/rxosd

    • (!) There is no harm in doing this even before creating a vice partition. /!\ Please note the discussion of Lustre storage backends below.

  • add the new OSD to the OSDDB 
    # osd createosd -id <id> -name <osd-name> -ip <machine-ip> -lun <lun> 1m 64g
    id

    must not have existed in the OSDDB before, check with osd l -all

    osd-name
    must also be unique
    lun

    numeric representation of the vice partition, /vicepa is 0, /vicepc is 2 and /vicepal is 37

    size constraints
    1m and 64g are arbitrary.

    /!\ Note that the minimum size directly influences what files get stored to OSD. (!) If this is done before creating vice partition and/or service instance, the OSD will appear as "down" in the osd l output. No harm done.

Using a Lustre storage backend

Basically, this is the same process as adding a regular rxosd service. The vice partition resides in a Lustre filesystem, however. The rxosd process must be launched with a special parameter: 

# bos create <server-machine> rxosd simple "/usr/afs/bin/rxosd -lustrehack"

(!) A larger maximum file size for your Lustre-OSD is not required, the fileserver will never get to see a size that surpasses any client's cache size.

Lustre mountpoint

E.g. 

# mount -t lustre 141.34.218.7@tcp:/zn_test /vicept
# touch /vicept/OnlyRxosd

The rest is identical to the above.

If Lustre is mounted anywhere else: 

# ln -s /lustre/vpxl /vicepal
# touch /vicepal/OnlyRxosd
# touch /vicepal/AlwaysAttach

The last command enables the use of a regular directory as a vice partition. The rest is identical to the above.

Verifying the correct setup of the OSD

Assuming both service instance and vice partition were available at the time of updating the database, osd l should contain something along the lines of 

  2 zyklop20-al    10694 gb   0.0 % up         64  64     zyklop20 37 (1mb-2tb)

The "up" signifies that the OSD is ready.

A "down" entry does not have to be a problem. It can (among other things) mean

  • an unreachable host

  • a missing rxosd process on the host machine 

    bos status <osd-machine>
  • a missing vice partition on the host machine
  • a wrong LUN 
    bos getlog <osd-machine> RxosdLog
...
Thu Jun  4 13:20:00 2009 [0] FiveMinuteCheckLWP: statfs for /vicepam failed with 2
    Here, the LUN had been one too large initially.

Database servers

Packages

If appropriate openafs-server packages are in use, there is nothing to do. 

# rpm -q openafs-server
openafs-server-1.4.10.osd.vpa.r657-74alma.2.sl5.x86_64
# rpm -qf /usr/afs/bin/osdserver
openafs-server-1.4.10.osd.vpa.r657-74alma.2.sl5.x86_64

For a minimum intrusion on existing DB servers, use the dedicated openafs-osdserver package: 

% rpm -q openafs-server openafs-osdserver
openafs-server-1.4.7-68.2.SL5.x86_64
openafs-osdserver-1.4.10.osd.vpa.r689-76alma.sl5.x86_64
% rpm -qf /usr/afs/bin/osdserver
openafs-osdserver-1.4.10.osd.vpa.r689-76alma.sl5.x86_64

It has no dependencies and will install regardless of the rest of your local AFS installation.

Launching the OSDDB service

With admin privileges: 

bos create <db-machine> osddb simple /usr/afs/bin/osdserver

Once the osdservers are up and running, clients should be able to see this: 

% osd l
 id name(loc)     ---total space---      flag  prior. own. server lun size range
  1 local_disk                                 wr  rd                 (0kb-1mb)

The local_disk entry is a default.

Enabling OSD for Volumes ...

 # vos setfields osdtestvol2 -osd 1

Update Client and Server

  • Stop AFS process

On Client: 

 % /etc/init.d/afs stop

On Server: (maybe Client process also running on a server!) 

 % /etc/init.d/afs-server stop
  • Update the packages

First Erase Kernel Package: 

 % rpm -e kernel-module-openafs-2.6.18-164.11.1.el5-1.4.11.osd.vpa.r901-80alma.sl5.x86_64

Update all the other packages, Kernel-module should be installed too. Add openafs-server on a server machine. 

% yum update openafs-authlibs-devel openafs-authlibs openafs-client openafs openafs-krb5 openafs-debug openafs-compat openafs-devel
  • Start AFS process

On Client: 

% /etc/init.d/afs start

On Server: 

% /etc/init.d/afs-server start
  • Start Vicep-Access on the Client 

% /etc/init.d/afs-vicep-access status
        Enabled protocols are  RXOSD (1 parallel streams on connections with rtt > 10 ms).
% /etc/init.d/afs-vicep-access stop
% /etc/init.d/afs-vicep-access start
% /etc/init.d/afs-vicep-access status
        Enabled protocols are  VICEP-ACCESS (with Lustre hack) RXOSD (1 parallel streams on connections with rtt > 10 ms).

AfsOsd/SetupInstructions (last edited 2010-03-08 10:11:58 by ReneStandke)