Differences between revisions 10 and 11

Contents

Client setup
Fileserver setup
Database servers
1. Packages
2. Launching the OSDDB service
Enabling OSD for Volumes ...
Update Client and Server

Client setup

Client protocol extension

Install a suitable kernel module

via rpm

% rpm -q kernel-module-openafs-`uname -r`
kernel-module-openafs-2.6.18-128.1.10.el5-1.4.7-68.2.SL5.i686

or manually

From the source directory copy

src/libafs/MODLOAD-`uname -r`-*/libafs.ko

/lib/modules/`uname -r`/kernel/fs/openafs/libafs.ko

```
depmod
```

Restart of the AFS service

Upon startup of the afs service, the following should show up:

# service afs restart
...
# dmesg | grep patch
libafs with   'rxosd support' 'vicep-access' patches

(vicep-access is not necessary for RxOSD operation per se, but is required for making proper use of a Lustre or GPFS backend.)

Because afs restart doesn't work often, is is better to reboot the client.

Proper afs commands

Install a suitable openafs package

# rpm -q openafs
openafs-1.4.10.osd.vpa.r691-77alma.sl5.x86_64
# fs protocol
Enabled protocols are  RXOSD (1 parallel streams on connections with rtt > 10 ms).
# osd help
osd: Commands are:
...

Memory cache

When using a cluster filesystem backend, memcache has proven to be the fastest alternative (at least for Lustre via Infiniband).

In /etc/sysconfig/afs:

OPTIONS="-memcache -stat 8192 -chunksize 20 -daemons 8 -volumes 64"

This selects memcache and boosts the chunk size to 1MB. To avoid a low overall number of cache chunks:

CACHESIZE="262144"

256 MB seems a reasonable size.

Accessing a cluster filesystem backend

In Zeuthen, the Lustre symlinks is handled by the SLZ_vicep_lustre_links RPM.

Generally (e.g. Lustre):

Make sure the shared vice partition is accessible:
```
# ln -s /vicepal /lustre/...
```
- or resp.
```
# mount -t lustre 141.34.218.7@tcp:/zn_test /vicept
```
- (see "Using a Lustre storage backend" below)

make the client detect the vice partition

# service afs-vicep-access restart
...
# dmesg | tail
...
Visible OSD 2 lun 37 cell 33 == ifh.de

Fileserver setup

Adding an rxosd service

It is sensible to have fileserver and rxosd processes share the same machines. This way, access to files even in a single volume can be spread to all fileservers (given that clients don't access the same (set of) file(s) all the time).

create a dedicated vice partition
```
# mkdir /vicepc
<edit /etc/fstab>
# mount /vicepc
# touch /vicepc/OnlyRxosd
```
- The last command ensures that the fileserver process will not claim this partition.
create the service instance (with admin privileges or -localauth)
```
# bos create <server-machine> rxosd simple /usr/afs/bin/rxosd
```
- There is no harm in doing this even before creating a vice partition. $/!\$ Please note the discussion of Lustre storage backends below.
add the new OSD to the OSDDB
```
# osd createosd -id <id> -name <osd-name> -ip <machine-ip> -lun <lun> 1m 64g
```
id
must not have existed in the OSDDB before, check with osd l -all
osd-name
must also be unique
lun
numeric representation of the vice partition, /vicepa is 0, /vicepc is 2 and /vicepal is 37
size constraints
1m and 64g are arbitrary.
$/!\$ Note that the minimum size directly influences what files get stored to OSD. If this is done before creating vice partition and/or service instance, the OSD will appear as "down" in the osd l output. No harm done.

Using a Lustre storage backend

Basically, this is the same process as adding a regular rxosd service. The vice partition resides in a Lustre filesystem, however. The rxosd process must be launched with a special parameter:

# bos create <server-machine> rxosd simple "/usr/afs/bin/rxosd -lustrehack"

A larger maximum file size for your Lustre-OSD is not required, the fileserver will never get to see a size that surpasses any client's cache size.

Lustre mountpoint

E.g.

# mount -t lustre 141.34.218.7@tcp:/zn_test /vicept
# touch /vicept/OnlyRxosd

The rest is identical to the above.

Symlink

If Lustre is mounted anywhere else:

# ln -s /lustre/vpxl /vicepal
# touch /vicepal/OnlyRxosd
# touch /vicepal/AlwaysAttach

The last command enables the use of a regular directory as a vice partition. The rest is identical to the above.

Verifying the correct setup of the OSD

Assuming both service instance and vice partition were available at the time of updating the database, osd l should contain something along the lines of

  2 zyklop20-al    10694 gb   0.0 % up         64  64     zyklop20 37 (1mb-2tb)

The "up" signifies that the OSD is ready.

A "down" entry does not have to be a problem. It can (among other things) mean

an unreachable host
a missing rxosd process on the host machine
```
bos status <osd-machine>
```
a missing vice partition on the host machine

a wrong LUN

bos getlog <osd-machine> RxosdLog
...
Thu Jun  4 13:20:00 2009 [0] FiveMinuteCheckLWP: statfs for /vicepam failed with 2

Here, the LUN had been one too large initially.

Database servers

Packages

If appropriate openafs-server packages are in use, there is nothing to do.

# rpm -q openafs-server
openafs-server-1.4.10.osd.vpa.r657-74alma.2.sl5.x86_64
# rpm -qf /usr/afs/bin/osdserver
openafs-server-1.4.10.osd.vpa.r657-74alma.2.sl5.x86_64

For a minimum intrusion on existing DB servers, use the dedicated openafs-osdserver package:

% rpm -q openafs-server openafs-osdserver
openafs-server-1.4.7-68.2.SL5.x86_64
openafs-osdserver-1.4.10.osd.vpa.r689-76alma.sl5.x86_64
% rpm -qf /usr/afs/bin/osdserver
openafs-osdserver-1.4.10.osd.vpa.r689-76alma.sl5.x86_64

It has no dependencies and will install regardless of the rest of your local AFS installation.

Launching the OSDDB service

With admin privileges:

bos create <db-machine> osddb simple /usr/afs/bin/osdserver

Once the osdservers are up and running, clients should be able to see this:

% osd l
 id name(loc)     ---total space---      flag  prior. own. server lun size range
  1 local_disk                                 wr  rd                 (0kb-1mb)

The local_disk entry is a default.

Enabling OSD for Volumes ...

 # vos setfields osdtestvol2 -osd 1

Update Client and Server

Stop AFS process
On Client: .
```
% /etc/init.d/afs stop
```
On Server: (maybe Client process also running on a server!)
```
 % /etc/init.d/afs-server stop
```
Update the packages
First Erase Kernel Package:

 % rpm -e kernel-module-openafs-2.6.18-164.11.1.el5-1.4.11.osd.vpa.r901-80alma.sl5.x86_64

Update all the other packages, Kernel-module should be installed too. Add openafs-server on a server machine.

% yum update openafs-authlibs-devel openafs-authlibs openafs-client openafs openafs-krb5 openafs-debug openafs-compat openafs-devel

Start AFS process
On Client:

% /etc/init.d/afs start

On Server:

% /etc/init.d/afs-server start

Start Vicep-Access on the Client

% /etc/init.d/afs-vicep-access status
        Enabled protocols are  RXOSD (1 parallel streams on connections with rtt > 10 ms).
% /etc/init.d/afs-vicep-access stop
% /etc/init.d/afs-vicep-access start
% /etc/init.d/afs-vicep-access status
        Enabled protocols are  VICEP-ACCESS (with Lustre hack) RXOSD (1 parallel streams on connections with rtt > 10 ms).

Check Configuration

On Client:

% dmesg
...
Visible OSD 2 lun 37 cell 33 == ifh.de

-  ⇤ ← Revision 10 as of 2010-03-03 11:42:19 → 
  Size: 7763
  Editor: ReneStandke
  Comment:
+   ← Revision 11 as of 2010-03-03 12:02:37 → ⇥
  Size: 8017
  Editor: ReneStandke
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 14:
-. From the source directory '''copy'''
+.
  From the source directory '''copy'''
-Line 18:
+Line 20:
-  to
+   . to
-Line 22:
+Line 26:
-. {{{
+.
  {{{
-Line 25:
+Line 32:
-Line 75:
+Line 84:
- or resp.
+  . or resp.
-Line 79:
+Line 90:
- (see "Using a Lustre storage backend" below)
+  . (see "Using a Lustre storage backend" below)
-Line 88:
+Line 102:
-Line 100:
+Line 115:
-Line 101:
+Line 117:
- * create the service instance (with admin privileges or `-localauth`)
+ *
 create the service instance (with admin privileges or `-localauth`)
-Line 105:
+Line 125:
-  . (!) There is no harm in doing this even before creating a vice partition.
  /!\ Please note the discussion of '''Lustre''' storage backends below.
+  .
  (!) There is no harm in doing this even before creating a vice partition. /!\ Please note the discussion of '''Lustre''' storage backends below.
-Line 111:
+Line 135:
-  id:: must not have existed in the OSDDB before, check with `osd l -all`
+  id::
  must not have existed in the OSDDB before, check with `osd l -all`
-Line 113:
+Line 140:
-  lun:: numeric representation of the vice partition, ''/vicepa'' is 0, ''/vicepc'' is 2 and ''/vicepal'' is 37
+  lun::
  numeric representation of the vice partition, ''/vicepa'' is 0, ''/vicepc'' is 2 and ''/vicepal'' is 37
-Line 115:
+Line 144:
-  /!\ Note that the minimum size directly influences what files get stored to OSD.
  (!) If this is done before creating vice partition and/or service instance, the OSD will appear as "down" in the `osd l` output. No harm done.
+  /!\ Note that the minimum size directly influences what files get stored to OSD. (!) If this is done before creating vice partition and/or service instance, the OSD will appear as "down" in the `osd l` output. No harm done.
-Line 156:
+Line 186:
- * a missing ''rxosd'' process on the host machine
+ *
 a missing ''rxosd'' process on the host machine
-Line 160:
+Line 192:
-Line 167:
+Line 201:
- Here, the LUN had been one too large initially.
+  . Here, the LUN had been one too large initially.
-Line 212:
+Line 248:
-On Client:

{{{
 % /etc/init.d/afs stop
}}}
On Server: (maybe Client process also running on a server!)

{{{
+ . On Client: .

 .
 {{{
% /etc/init.d/afs stop
}}}


 . On Server: (maybe Client process also running on a server!)

 .
 {{{
-Line 222:
+Line 262:
-Line 224:
+Line 266:
-First Erase Kernel Package:
+ . First Erase Kernel Package:
 .
-Line 229:
+Line 272:
-Update all the other packages, Kernel-module should be installed too. Add openafs-server on a server machine.
+ . Update all the other packages, Kernel-module should be installed too. Add openafs-server on a server machine.
 .
-Line 236:
+Line 280:
-On Client:
+ . On Client:
 .
-Line 241:
+Line 286:
-On Server:
+ . On Server:
 .
-Line 247:
+Line 293:
+ .
-Line 256:
+Line 303:
+ * Check Configuration

On Client:

 .

{{{
% dmesg
...
Visible OSD 2 lun 37 cell 33 == ifh.de
}}}

Wiki

Page