Contents

Important notes
Status
Building OpenAFS-OSD
1. Quick RPM creation
Enabeling the client
Enabling OSD
Issues
1. Missing token is not handled well
Archiving

Sub pages:

Important notes

On iokaste, /root/.zshenv now contains

PATH=/usr/heimdal/bin:/usr/afs/bin:$PATH

Status

Overview of installs ( $/!\$ this is how it's currently supposed to be / under way):

Machine	service	version
iokaste	fileserver	manually from svn
	ubik servers	manually from svn
	afsd	manually from svn
phobos	fileserver	manually from svn, for debugging only
	afsd	manually from svn
ssu12/13, zyklop19/20	fileserver	OpenAFS+OSD RPM from svn (not zyklop20)
	rxosd	OpenAFS+OSD RPM from svn
	afsd	OpenAFS+OSD RPM from svn 1.4.7 or SL5 RPM
deimos	ubik servers	OpenAFS+OSD RPM from svn

Building OpenAFS-OSD

To build openafs, kernel headers are recommended to be installed (so configure can detect the local sysname). On SL5, as root do a

yum install kernel-devel

and you're set (for the time being). If there's still no /usr/src/linux:

ln -s /usr/src/kernels/2.6.18-92.1.10.el5-i686 /usr/src/linux

...or whatever directory the kernel source was installed to.

In the openafs-version directory:

mkdir `sys`
cd `sys`
../configure --prefix=/usr --libdir=/usr/lib --bindir=/usr/bin --sbindir=/usr/sbin --with-krb5-conf --enable-bitmap-later --enable-bos-restricted-mode --enable-fast-restart --enable-bos-new-config --enable-supergroups --enable-transarc-paths --enable-object-storage --enable-osd-policies

Quick RPM creation

Package building is described in TestCellSetup/IncludingFileservers.

Currently, this works only with the openafs.SLx.spec file, revision 70. Install any openafs-1.4.x.osd.*.src.rpm package to retrieve it.

On the build machine:

cd /usr/src/packages/SOURCES
svn export https://svnsrv.desy.de/desy/openafs-osd/trunk/openafs openafs-1.4.x.osd.rXXX
tar cjf openafs-1.4.x.osd.rXXX-src.tar.bz2
rm -r openafs-1.4.x.osd.rXXX
vim ../SPECS/openafs.SLx.spec
rpmbuild -ba ../SPECS/openafs.SLx.spec

Change 1.4.x to the AFS version being built and rXXX to the revision number of the HEAD in the subversion repository (obtain using svn info https://...).
To build packages with policy support, substitute trunk in the URL by branches/policies and the version to 1.4.x.osd.pol.rXXX.

Enabeling the client

From the source directory copy

src/libafs/MODLOAD-2.6.18-8.1.15.el5-MP/libafs.ko

(where 2.6.18-8.1.15.el5-MP is the kernel version)

/lib/modules/2.6.18-8.1.10.el5/kernel/fs/openafs/libafs.ko

```
depmod 
```
reboot
launch afsd

Enabling OSD

[iokaste] ~ # vos setfields osdtestvol2 -osd 1
[iokaste] ~ # fs protocol -enable RXOSD

Issues

Missing token is not handled well

$ osd volumes io -lun 1
RX xdr error
Cannot list volumes
Request aborted.
    
$ osd createv io 1 536870924
Createvolume failed: RXOSD_create_part ended with 13
Request aborted.

In general, missing privileges are an issue but there is no "friendly output" describing the cause of the problem. Some examples of unprivileged access attempts:

fs osd	Error 13 reading length of metadata	works with r privilege granted to system:anyuser
osd read	RX xdr error \\ Cannot read the object \\ rx_EndCall returns: 13 \\ Request aborted.	requires token
osd volumes	RX xdr error \\ Cannot list volumes \\ Request aborted.	requires token
osd examine	RXOSD_examine failed with code 13 \\ Request aborted.	requires token

Database queries osd listosds and osd servers seem to work unauthenticated.

Archiving

Manually creating archival copies

The most simple way is to tell the fileserver to create a file copy in an archival osd:

# fs archive ascii-file
ascii-file done

An alternative to this is the fs fidarchive subcommand.

It is also possible to specify the desired archive:

# osd l
 id name(loc)     ---total space---      flag  prior. own. server lun size range
  1 local_disk                                 wr  rd                 (0kb-1mb)
 13 ssu12           1781 gb   0.0 % up         55  64     ssu12.if  2 (1mb-64gb)
 14 ssu13           1781 gb   0.0 % up         55  64     ssu13.if  2 (1mb-64gb)
 15 z19mini           91 gb   2.3 % up   hsm   64  64     zyklop19  0 (1mb-1gb)
 16 z19maxi          458 gb   0.0 % up         55  64     zyklop19  2 (1mb-64gb)
 17 arch1            458 gb   0.0 % up   arch  55  64     zyklop20  0 (0kb-2tb)
 18 arch2            458 gb   0.0 % up   arch  55  64     zyklop20  1 (0kb-2tb)
 19 z20online        391 gb   0.0 % up         55  64     zyklop20  2 (1mb-64gb)
# fs archive ascii-file -osd 18
ascii-file done
# fs osd archive-file
ascii-file has 284 bytes of osd metadata, v=3
On-line, 1 segm, flags=0x0
    segment:
        lng=0, offs=0, stripes=1, strsize=4096, cop=1, 1 objects
        object:
            obj=536871001.6.207.0, osd=15, stripe=0
Archive, dv=3, 2009-01-28 09:55:34, 1 segm, flags=0x0
    segment:
        lng=44537893, offs=0, stripes=1, strsize=0, cop=1, 1 objects
        object:
            obj=536871001.6.207.0, osd=18, stripe=0
    metadata:
        md5=a64535612df9ff989b4bd6d6a133fb5d  as from  2009-01-28 09:56:36

The archival copy is on osd 18 (zyklop20, vicepb = lun 1).

$/!\$ There is no warning about existent copies. Repeat the command:

[iokaste] /afs/desytest/field_test # fs arch ascii-file -osd 18
ascii-file done
[iokaste] /afs/desytest/field_test # fs osd ascii-file         
ascii-file has 284 bytes of osd metadata, v=3
On-line, 1 segm, flags=0x0
    segment:
        lng=0, offs=0, stripes=1, strsize=4096, cop=1, 1 objects
        object:
            obj=536871001.6.207.0, osd=15, stripe=0
Archive, dv=3, 2009-01-28 09:55:34, 1 segm, flags=0x0
    segment:
        lng=44537893, offs=0, stripes=1, strsize=0, cop=1, 1 objects
        object:
            obj=536871001.6.207.0, osd=18, stripe=0
    metadata:
        md5=a64535612df9ff989b4bd6d6a133fb5d  as from  2009-01-28 09:56:36

The call is silently ignored.

Change the data:

[iokaste] /afs/desytest/field_test # echo More data >> ascii-file 
[iokaste] /afs/desytest/field_test # fs arch ascii-file -osd 18
ascii-file done
[iokaste] /afs/desytest/field_test # fs osd ascii-file 
ascii-file has 284 bytes of osd metadata, v=3
On-line, 1 segm, flags=0x0
    segment:
        lng=0, offs=0, stripes=1, strsize=4096, cop=1, 1 objects
        object:
            obj=536871001.6.207.0, osd=15, stripe=0
Archive, dv=5, 2009-01-28 09:59:27, 1 segm, flags=0x0
    segment:
        lng=44537903, offs=0, stripes=1, strsize=0, cop=1, 1 objects
        object:
            obj=536871001.6.207.1, osd=18, stripe=0
    metadata:
        md5=1891d17de5eb67c0bd30d54952ea0325  as from  2009-01-28 09:59:43

The copy is updated.

Another update:

[iokaste] /afs/desytest/field_test # echo More data yet >>| ascii-file
[iokaste] /afs/desytest/field_test # fs arch ascii-file           
ascii-file done
[iokaste] /afs/desytest/field_test # fs osd ascii-file 
ascii-file has 284 bytes of osd metadata, v=3
On-line, 1 segm, flags=0x0
    segment:
        lng=0, offs=0, stripes=1, strsize=4096, cop=1, 1 objects
        object:
            obj=536871001.6.207.0, osd=15, stripe=0
Archive, dv=7, 2009-01-28 10:00:46, 1 segm, flags=0x0
    segment:
        lng=44537917, offs=0, stripes=1, strsize=0, cop=1, 1 objects
        object:
            obj=536871001.6.207.0, osd=18, stripe=0
    metadata:
        md5=eec1410b6ae326e6c9173802122372c0  as from  2009-01-28 10:00:51

Even without specifying the target osd, the system correctly assumes that the archive should be refreshed.

Without updates:

[iokaste] /afs/desytest/field_test # fs arch ascii-file
ascii-file done
[iokaste] /afs/desytest/field_test # fs osd ascii-file 
ascii-file has 436 bytes of osd metadata, v=3
On-line, 1 segm, flags=0x0
    segment:
        lng=0, offs=0, stripes=1, strsize=4096, cop=1, 1 objects
        object:
            obj=536871001.6.207.0, osd=15, stripe=0
Archive, dv=7, 2009-01-28 10:00:46, 1 segm, flags=0x0
    segment:
        lng=44537917, offs=0, stripes=1, strsize=0, cop=1, 1 objects
        object:
            obj=536871001.6.207.0, osd=18, stripe=0
    metadata:
        md5=eec1410b6ae326e6c9173802122372c0  as from  2009-01-28 10:00:51
Archive, dv=7, 2009-01-28 10:00:51, 1 segm, flags=0x0
    segment:
        lng=44537917, offs=0, stripes=1, strsize=0, cop=1, 1 objects
        object:
            obj=536871001.6.207.0, osd=17, stripe=0
    metadata:
        md5=eec1410b6ae326e6c9173802122372c0  as from  2009-01-28 10:02:07

Another archived copy was created, same data (as per md5data).

A last update:

[iokaste] /afs/desytest/field_test # echo And even more data >> ascii-file 
[iokaste] /afs/desytest/field_test # fs arch ascii-file 
ascii-file done
[iokaste] /afs/desytest/field_test # fs osd ascii-file 
ascii-file has 284 bytes of osd metadata, v=3
On-line, 1 segm, flags=0x0
    segment:
        lng=0, offs=0, stripes=1, strsize=4096, cop=1, 1 objects
        object:
            obj=536871001.6.207.0, osd=15, stripe=0
Archive, dv=9, 2009-01-28 10:02:54, 1 segm, flags=0x0
    segment:
        lng=44537936, offs=0, stripes=1, strsize=0, cop=1, 1 objects
        object:
            obj=536871001.6.207.1, osd=17, stripe=0
    metadata:
        md5=150103c2d7bcfc367b707df35801cdfa  as from  2009-01-28 10:02:59

The copy in osd 17 is updated, the "old" archive in osd 18 is discarded.

Wiping files

If the data is archived, it can be wiped from its actual OSD:

[iokaste] /afs/desytest/field_test # fs wipe ascii-file 
done
[iokaste] /afs/desytest/field_test # fs osd ascii-file 
ascii-file has 160 bytes of osd metadata, v=3
Archive, dv=9, 2009-01-28 10:02:54, 1 segm, flags=0x2
    segment:
        lng=44537936, offs=0, stripes=1, strsize=0, cop=1, 1 objects
        object:
            obj=536871001.6.207.1, osd=17, stripe=0
    metadata:
        md5=150103c2d7bcfc367b707df35801cdfa  as from  2009-01-28 10:02:59

On an access, it is silently restored to an "normal" osd:

[iokaste] /afs/desytest/field_test # head ascii-file 
[1177711200] LOG ROTATION: DAILY
[1177711200] LOG VERSION: 2.0
[1177711200] CURRENT HOST STATE: a;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.13 ms
[1177711200] CURRENT HOST STATE: amafs;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.14 ms
[1177711200] CURRENT HOST STATE: aphrodite;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.54 ms
[1177711200] CURRENT HOST STATE: aquila;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.25 ms
[1177711200] CURRENT HOST STATE: ares;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.59 ms
[1177711200] CURRENT HOST STATE: argus;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.16 ms
[1177711200] CURRENT HOST STATE: artax;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 1.08 ms
[1177711200] CURRENT HOST STATE: arwen;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.18 ms
[iokaste] /afs/desytest/field_test # fs osd ascii-file 
ascii-file has 284 bytes of osd metadata, v=3
On-line, 1 segm, flags=0x0
    segment:
        lng=0, offs=0, stripes=1, strsize=0, cop=1, 1 objects
        object:
            obj=536871001.6.207.0, osd=15, stripe=0
Archive, dv=9, 2009-01-28 10:02:54, 1 fetches, last: 2009-01-28, 1 segm, flags=0x2
    segment:
        lng=44537936, offs=0, stripes=1, strsize=0, cop=1, 1 objects
        object:
            obj=536871001.6.207.1, osd=17, stripe=0
    metadata:
        md5=150103c2d7bcfc367b707df35801cdfa  as from  2009-01-28 10:02:59

The osd is selected the same way as always - depending on priorities, usage percentage, owner etc.

Automatic archiving

The archiver script is included in the OpenAFS+OSD sources. It runs infinitely and searches unarchived files in short intervals. It runs fs fidarchive on them.

The wiper script will traverse OSDs that have the "hsm" flag set using

# osd setosd <id> -wipeable

On those, files that have not been accessed for a while are wiped, so that only their latest archival copy remains in storage.

The scripts have yet to be ported to our specific requirements (as far as they differ from Garching's).

Wiki

Page