Differences between revisions 30 and 47 (spanning 17 versions)
⇤ ← Revision 30 as of 2008-08-25 10:53:45 →
Size: 14779
Editor: FelixFrank
Comment: removed obsolete "mirroring from garching". update of build instruction.
← Revision 47 as of 2009-07-30 08:55:10 → ⇥
Size: 2053
Editor: WaltrautNiepraschk
Comment: - issue
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
[[TableOfContents]] <<TableOfContents>>
Line 3: Line 3:
Sub pages: [[Navigation(children)]] Sub pages: <<Navigation(children)>>
Line 6: Line 6:
On ''iokaste'', /root/.zshenv now contains {{{ On ''phobos'', /root/.zshenv now contains {{{
Line 8: Line 8:

= Status =
Overview of installs:
|| Machine || service || version ||
|| io || kdc || - ||
|| || rxosd || 1.4.7+OSD ||
|| || afsd || 1.4.6-58 (SL5 rpm) ||
|| iokaste || fileserver || 1.4.7+OSD ||
|| || vldb,pts || 1.4.7(+OSD) ||
|| || osddb || 1.4.7+OSD ||
|| || afsd || 1.4.7+OSD ||
|| phobos || fileserver || 1.4.7, '''currently being used for testing various sub-patches of the Garching patch''' ||
|| || afsd || 1.4.6-58 (SL5 rpm) ||
Line 24: Line 11:
To build openafs, kernel headers are recommended to be installed (so configure can detect the local sysname). On SL5, as root do a {{{yum install kernel-devel To build openafs, kernel headers are recommended to be installed (so configure can detect the local sysname). On SL5, as root do a {{{
yum install kernel-devel
Line 26: Line 14:
If there's still no ''/usr/src/linux'': {{{ ln -s /usr/src/kernels/2.6.18-92.1.10.el5-i686 /usr/src/linux If there's still no ''/usr/src/linux'': {{{
ln -s /usr/src/kernels/2.6.18-92.1.10.el5-i686 /usr/src/linux
Line 29: Line 18:
In the ''openafs-1.4.6-osd'' directory: {{{
./configure --enable-bitmap-later --enable-bos-restricted-mode --enable-bos-new-config --enable-fast-restart --enable-supergroups --enable-largefile-fileserver --enable-transarc-paths --with-krb5-conf --enable-object-storage		 In the openafs-''version'' directory: {{{
mkdir `sys`
cd `sys`
../configure --prefix=/usr --libdir=/usr/lib --bindir=/usr/bin --sbindir=/usr/sbin --with-krb5-conf --enable-bitmap-later --enable-bos-restricted-mode --enable-fast-restart --enable-bos-new-config --enable-supergroups --enable-transarc-paths --enable-object-storage --enable-osd-policies
Line 33: Line 24:
== Quick RPM creation ==
See TestCellSetup/IncludingFileservers.
Line 34: Line 28:
 * From the source directory '''copy''' {{{
src/libafs/MODLOAD-2.6.18-8.1.15.el5-MP/libafs.ko }}}
  (where ''2.6.18-8.1.15.el5-MP'' is the kernel version)
 to {{{
/lib/modules/2.6.18-8.1.10.el5/kernel/fs/openafs/libafs.ko }}}
 * {{{ 		 * Manually:
  * From the source directory '''copy''' {{{
src/libafs/MODLOAD-`uname -r`-*/libafs.ko }}}
  to {{{
/lib/modules/`uname -r`/kernel/fs/openafs/libafs.ko }}}
  * {{{
Line 41: Line 35:
 * or
  * install appropriate kernel module RPM
Line 47: Line 43:
[iokaste] ~ # fs protocol -enable RXOSD
Line 50: Line 45:
= Issues =
Line 52: Line 46:
== Missing token is not handled well ==
{{{ $ osd volumes io -lun 1
RX xdr error
Cannot list volumes
Request aborted.
    
$ osd createv io 1 536870924
Createvolume failed: RXOSD_create_part ended with 13
Request aborted.}}}		 = Performance =
Line 62: Line 48:
In general, missing privileges are an issue but there is no "friendly output" describing the cause of the problem. Some examples of unprivileged access attempts:
|| '''fs osd''' || Error 13 reading length of metadata || works with '''r''' privilege granted to system:anyuser ||
|| '''osd read''' || RX xdr error \\ Cannot read the object \\ rx_End``Call returns: 13 \\ Request aborted. || requires token ||
|| '''osd volumes''' || RX xdr error \\ Cannot list volumes \\ Request aborted. || requires token ||
|| '''osd examine''' || RXOSD_examine failed with code 13 \\ Request aborted. || requires token ||

Database queries '''osd listosds''' and '''osd servers''' seem to work unauthenticated.

= Subcommands =

== fs createstripedfile ==

=== creating mirrored files ===

{{{ # fs creates /afs/desytest/osdtest/mirrored-file 1 14 -copies 2
}}} where 1 is the ''number of stripes'' and 14 is ''log_2(stripesize in bytes)''.
 Note:: Allowed exponents are 12..19, allowing stripe sizes 4kB..512kB.

The above command created an unstriped but mirrored file.

=== creating striped files ===

This is how we created the first striped file: {{{
# fs creates /afs/desytest/osdtest/striped_file_1 2 14
done }}}
We stored a kernel image inside: {{{
# cp /boot/vmlinuz-2.6.18-8.1.10.el5 striped_file_1
# mv striped_file_1 vmlinuz-2.6.18-8.1.10.el5
# ll
total 7615
-rw-r--r-- 1 bin root 417032 Jan 14 14:54 libafs.ko
-rw-r--r-- 1 bin root 5612872 Jan 28 15:31 mirrored._usr_afs.2007-12-17.tar.bz2
-rw-r--r-- 1 bin root 1765652 Feb 14 13:37 vmlinuz-2.6.18-8.1.10.el5
# fs osd vmlinuz-2.6.18-8.1.10.el5
vmlinuz-2.6.18-8.1.10.el5 has 172 bytes of osd metadata, v=3
On-line, 1 segm, flags=0x0
    segment:
        lng=0, offs=0, stripes=2, strsize=16384, cop=1, 2 objects
        object:
            pid=536870924, oid=720578573194231814, osd=12, stripe=0
            obj=536870924.6.167772773.0
        object:
            pid=536870924, oid=3026421582407925766, osd=11, stripe=1
            obj=536870924.6.704643685.0 }}}
As the stripes were put in OSDs 11 and 12 respectively: {{{
# osd listo
 id name(loc) ---total space--- flag prior. own. server lun size range
  1 local_disk wr rd (0kb-1mb)
  8 io_a 279 gb 0.0 % up hsm 50 0 io.ifh.d 0 (1mb-64gb)
  9 io_b 279 gb 0.0 % up hsm 50 0 io.ifh.d 1 (1mb-64gb)
 10 io_c 279 gb 0.0 % up hsm 50 0 io.ifh.d 2 (1mb-64gb)
 11 io_d 279 gb 0.0 % up hsm 50 0 io.ifh.d 3 (1mb-64gb)
 12 io_e 278 gb 0.0 % up hsm 50 0 io.ifh.d 4 (1mb-64gb)
# osd vol io
1 volumes found:
        536870924
# osd obj io 536870924 3
536870924.4.33555043.0 fid 536870924.4.611 tag 0 not-striped lng 4209654 lc 1
536870924.6.704643685.0 fid 536870924.6.613 tag 0 1/2/16384 lng 880916 lc 1
536870924.4.704643683.1 fid 536870924.4.611 tag 1 1/2/16384 lng 0 lc 1
3 object(s) with totally 5090570 bytes for volume 536870924 found
# osd obj io 536870924 4
536870924.4.167772771.0 fid 536870924.4.611 tag 0 0/2/16384 lng 0 lc 1
536870924.6.167772773.0 fid 536870924.6.613 tag 0 0/2/16384 lng 884736 lc 1
2 object(s) with totally 884736 bytes for volume 536870924 found }}}

The first file on ''io/3'' (vicepd) belongs to the mirrored file created above, not the striped file.

=== adding segments to an existing OSD file ===
{{{
# fs creates /afs/desytest/osdtest/mirrored._usr_afs.2007-12-17.tar.bz2 2 14
File /afs/desytest/osdtest/mirrored._usr_afs.2007-12-17.tar.bz2 already exists. Create new segment? (Y|N)
y
done
# fs osd mirrored._usr_afs.2007-12-17.tar.bz2
mirrored._usr_afs.2007-12-17.tar.bz2 has 436 bytes of osd metadata, v=3
On-line, 3 segm, flags=0x0
    segment:
        lng=5612872, offs=0, stripes=1, strsize=16384, cop=2, 2 objects
        object:
            pid=536870924, oid=144117812300873732, osd=10, stripe=0
            obj=536870924.4.33555043.0
        object:
            pid=536870924, oid=144117812300873732, osd=11, stripe=0
            obj=536870924.4.33555043.0
    segment:
        lng=5612872, offs=5612872, stripes=2, strsize=16384, cop=1, 2 objects
        object:
            pid=536870924, oid=720578564604297220, osd=12, stripe=0
            obj=536870924.4.167772771.0
        object:
            pid=536870924, oid=3026421573885100036, osd=11, stripe=1
            obj=536870924.4.704643683.1
    segment:
        lng=0, offs=5612872, stripes=2, strsize=16384, cop=1, 2 objects
        object:
            pid=536870924, oid=720578564671406084, osd=12, stripe=0
            obj=536870924.4.167772771.1
        object:
            pid=536870924, oid=3026421573952208900, osd=11, stripe=1
            obj=536870924.4.704643683.2 }}}


== fs osd ==
{{{ # fs osd /afs/desytest/osdtest/mirrored-file
/afs/desytest/osdtest/mirrored-file has 172 bytes of osd metadata, v=3
On-line, 1 segm, flags=0x0
    segment:
        lng=0, offs=0, stripes=1, strsize=16384, cop=2, 2 objects
        object:
            pid=536870924, oid=144117812300873732, osd=10, stripe=0
            obj=536870924.4.33555043.0
        object:
            pid=536870924, oid=144117812300873732, osd=11, stripe=0
            obj=536870924.4.33555043.0 }}}
where ''osd=10'' and ''osd=11'' identify the storing OSD of each file copy. IDs are looked up using `osd listo`.

== osd wipecandidates ==
{{{
# osd wipe io
# osd wipe iokaste
            [ some waiting time ]
RXOSD_wipe_candidates failed with code -1
Request aborted.
# osd wipe /afs/desytest/osdtest/mirrored._usr_afs.2007-12-17.tar.bz2
(no error output)
# osd wipe io 0
# osd wipe io 1
# osd wipe io 2
0 Jan 24 16:03 4209654 536870924.4.33555043.0 536870924.4.33555043
# osd wipe io 3
0 Jan 28 15:31 4209654 536870924.4.33555043.0 536870924.4.33555043
# osd wipe io 4
RXOSD_wipe_candidates failed with code 5
Request aborted. }}}
 Note:: The last request propably failed because there was no /vicepe/AFSIDat/ directory on io at the time.
 Note:: Using a filename as the only argument to `osd wipe` should have at least yielded an error similar to when using ''iokaste'' (which is not an OSD server).
 Note:: The entries for io/2 and io/3 are the two copies of the file created above.

== fs wipe ==
{{{
# fs wipe mirrored._usr_afs.2007-12-17.tar.bz2
Could not wipe mirrored._usr_afs.2007-12-17.tar.bz2, error code was 22 }}}
 Note:: This appears to be the default answer to a tried wipe of a file that has not yet been archived using `fs archive`.

== osd createvolume ==
{{{
[iokaste] ~ # vos create iokaste a osdtestvol2
Volume 536870927 created on partition /vicepa of iokaste
[iokaste] ~ # osd createvol io 536870927 0
Createvolume: volume 536870927 successfully created.
[iokaste] ~ # osd volu io 0
2 volumes found:
        536870924
        536870927
[iokaste] ~ # osd volu io 1
1 volumes found:
        536870924
}}}

== osd volumes ==
{{{
# osd volumes io
1 volumes found:
        536870924
}}}
That's the same VolID as in AFS:
{{{
# vos listvl
VLDB entries for all servers

osdtestvol
    RWrite: 536870924
    number of sites -> 1
       server iokaste.ifh.de partition /vicepa RW Site

root.cell
    RWrite: 536870921
    number of sites -> 1
       server iokaste.ifh.de partition /vicepa RW Site

Total entries: 2
}}}

 Note:: This information seems to be persistent, even after removing all of a volume's files from Object Storage: {{{
[iokaste] /afs/desytest/osdtest # osd volu io 0
1 volumes found:
        536870924
[iokaste] /afs/desytest/osdtest # osd volu io 1
1 volumes found:
        536870924
[iokaste] /afs/desytest/osdtest # osd volu io 2
1 volumes found:
        536870924
[iokaste] /afs/desytest/osdtest # osd volu io 3
1 volumes found:
        536870924
[iokaste] /afs/desytest/osdtest # osd volu io 4
1 volumes found:
        536870924
[iokaste] /afs/desytest/osdtest # osd volu io 5
0 volumes found:
[iokaste] /afs/desytest/osdtest # osd obj io 536870924 0
0 object(s) with totally 0 bytes for volume 536870924 found
[iokaste] /afs/desytest/osdtest # osd obj io 536870924 1
0 object(s) with totally 0 bytes for volume 536870924 found
[iokaste] /afs/desytest/osdtest # osd obj io 536870924 2
0 object(s) with totally 0 bytes for volume 536870924 found
[iokaste] /afs/desytest/osdtest # osd obj io 536870924
0 object(s) with totally 0 bytes for volume 536870924 found
[iokaste] /afs/desytest/osdtest # osd obj io 536870924 3
0 object(s) with totally 0 bytes for volume 536870924 found
[iokaste] /afs/desytest/osdtest # osd obj io 536870924 4
0 object(s) with totally 0 bytes for volume 536870924 found }}}


== osd objects ==
Both volume and partition are needed:
{{{
# osd obj io 536870924
0 object(s) with totally 0 bytes for volume 536870924 found

# osd obj io 536870924 2
536870924.4.33555043.0 fid 536870924.4.611 tag 0 not-striped lng 4209654 lc 1
1 object(s) with totally 4209654 bytes for volume 536870924 found
}}}
 Note:: Without specifying a lun, the default of 0 (vicepa) is used.

== osd examine ==
This command also has issues with missing LUNs: {{{
[iokaste] /afs/desytest/osdtest # osd exa io 536870924.6.621.0
RXOSD_examine failed with code 5
Request aborted.
[iokaste] /afs/desytest/osdtest # osd exa io 536870924.6.621.0 -lun 4
536870924.6.621.0 fid 536870924.6.621 tag 0 not-striped lng 44537893 lc 1 Feb 21 13:53 }}}

== osd read ==
This works for unstriped files, but suffers from the "missing LUN syndrome" as well: {{{
[iokaste] /afs/desytest/osdtest # osd read io 536870924.6.621
RX xdr error
Cannot read the object
rx_EndCall returns: 5
Request aborted.
[iokaste] /afs/desytest/osdtest # osd read io 536870924.6.621 0 512 -lun 4
sscanf failed at offset (0x0, 0x0) [1177711200] LOG ROTATION: DAILY
[1177711200] LOG VERSION: 2.0
[1177711200] CURRENT HOST STATE: a;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.13 ms
[1177711200] CURRENT HOST STATE: amafs;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.14 ms
[1177711200] CURRENT HOST STATE: aphrodite;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.54 ms
[1177711200] CURRENT HOST STATE: aquila;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.25 ms
[1177711200] CURRENT HOST STATE: ares;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.
reading of 512 bytes took 0.003 sec.
Total data rate = 180 Kbytes/sec. for read. }}}
 Note:: '''0''' and '''512''' in example 2 are ''offset'' and ''bytes to read'', resp.

== fs replaceosd ==
First tries were very discouraging:
{{{
[iokaste] ~ # fs osd /afs/desytest/osdtest/nagios-04-29-2007-00.log
/afs/desytest/osdtest/nagios-04-29-2007-00.log has 132 bytes of osd metadata, v=3
On-line, 1 segm, flags=0x0
    segment:
        lng=0, offs=0, stripes=1, strsize=0, cop=1, 1 objects
        object:
            pid=536870924, oid=2667174690822, osd=12, stripe=0
            obj=536870924.6.621.0
[iokaste] ~ # fs rep /afs/desytest/osdtest/nagios-04-29-2007-00.log 12 8
failed to replace osd 12 for /afs/desytest/osdtest/nagios-04-29-2007-00.log, error code is 5
[iokaste] ~ # fs rep /afs/desytest/osdtest/nagios-04-29-2007-00.log 12
}}}

The last command hung and could not be TERMed or KILLed. A subsequent {{{
[iokaste] ~ # fs osd /afs/desytest/osdtest/nagios-04-29-2007-00.log
}}} got stuck much the same.

= Case study: Creating a long striped file =
{{{
[iokaste] /afs/desytest/osdtest # fs creates striped-ascii-44m 2 12
done
[iokaste] /afs/desytest/osdtest # cp /root/nagios-04-29-2007-00.log striped-ascii-44m
  [ took about 3 seconds ]
[iokaste] /afs/desytest/osdtest # fs osd striped-ascii-44m
striped-ascii-44m has 172 bytes of osd metadata, v=3
On-line, 1 segm, flags=0x0
    segment:
        lng=0, offs=0, stripes=2, strsize=4096, cop=1, 2 objects
        object:
            pid=536870924, oid=576463393708310532, osd=12, stripe=0
            obj=536870924.4.134218343.0
        object:
            pid=536870924, oid=2882306402922004484, osd=11, stripe=1
            obj=536870924.4.671089255.0
[iokaste] /afs/desytest/osdtest # osd obj io 536870924 3
536870924.4.671089255.0 fid 536870924.4.615 tag 0 1/2/4096 lng 22267941 lc 1
1 object(s) with totally 22267941 bytes for volume 536870924 found
[iokaste] /afs/desytest/osdtest # osd obj io 536870924 4
536870924.4.134218343.0 fid 536870924.4.615 tag 0 0/2/4096 lng 22269952 lc 1
1 object(s) with totally 22269952 bytes for volume 536870924 found
}}}
 Note:: The lengths ('''lng''') of both objects amount to the size of the stored file: `22267941 + 22269952 = 44537893`.		 || '''Server version''' || '''Clients version''' || process || max. throughput [MB/s] ||
|| 1.4.7.osd.r302.sec || 1.4.7.osd.r302.sec || fileserver || 160 ||
|| || || rxosd || 120 ||
|| 1.4.7 || || fileserver || 160 ||
|| 1.4.8 || || fileserver || 125 ||
|| 1.4.10 || || fileserver || 125 ||
|| 1.4.8.osd.r573x || || fileserver || 125 ||
|| || || rxosd || 100 ||
|| || 1.4.10.osd.nosvn || fileserver || 125 ||
|| || || rxosd || 72 ||
|| 1.4.7 || || fileserver || 160 ||
|| 1.4.8 || || fileserver || 125 ||
|| 1.4.10 || || fileserver || 125 ||
|| || 1.4.10 || fileserver || 125 ||
|| 1.4.7 || || fileserver || 160 ||

Contents

  1. Important notes
  2. Building OpenAFS-OSD
    1. Quick RPM creation
  3. Enabeling the client
  4. Enabling OSD
  5. Performance

Sub pages:

/Archivierung   /Build   /Debugging   /FAQ   /Issues   /Logs   /LustreTestsZeuthen   /Notes   /Patches   /Performance   /Performance/Kernel   /Performance/System   /Performance/Szenario   /Production   /Protocol-2008-09-03   /SetupInstructions   /Subversion   /UsageInstructions   /UsageInstructions/FsSubcommands   /UsageInstructions/OsdSubcommands   /UsageInstructions/VosSubcommands   /policies  

1. Important notes

On phobos, /root/.zshenv now contains 

PATH=/usr/heimdal/bin:/usr/afs/bin:$PATH

2. Building OpenAFS-OSD

To build openafs, kernel headers are recommended to be installed (so configure can detect the local sysname). On SL5, as root do a 

yum install kernel-devel

and you're set (for the time being). If there's still no /usr/src/linux: 

ln -s /usr/src/kernels/2.6.18-92.1.10.el5-i686 /usr/src/linux

...or whatever directory the kernel source was installed to.

In the openafs-version directory: 

mkdir `sys`
cd `sys`
../configure --prefix=/usr --libdir=/usr/lib --bindir=/usr/bin --sbindir=/usr/sbin --with-krb5-conf --enable-bitmap-later --enable-bos-restricted-mode --enable-fast-restart --enable-bos-new-config --enable-supergroups --enable-transarc-paths --enable-object-storage --enable-osd-policies

2.1. Quick RPM creation

See TestCellSetup/IncludingFileservers.

3. Enabeling the client

  • Manually:

    • From the source directory copy 

      src/libafs/MODLOAD-`uname -r`-*/libafs.ko

      to 

      /lib/modules/`uname -r`/kernel/fs/openafs/libafs.ko 
    • depmod
  • or
    • install appropriate kernel module RPM
  • reboot

  • launch afsd

4. Enabling OSD

[iokaste] ~ # vos setfields osdtestvol2 -osd 1

5. Performance

Server version

Clients version

process

max. throughput [MB/s]

1.4.7.osd.r302.sec

1.4.7.osd.r302.sec

fileserver

160

rxosd

120

1.4.7

fileserver

160

1.4.8

fileserver

125

1.4.10

fileserver

125

1.4.8.osd.r573x

fileserver

125

rxosd

100

1.4.10.osd.nosvn

fileserver

125

rxosd

72

1.4.7

fileserver

160

1.4.8

fileserver

125

1.4.10

fileserver

125

1.4.10

fileserver

125

1.4.7

fileserver

160

AfsOsd (last edited 2009-08-17 16:11:48 by WaltrautNiepraschk)