Differences between revisions 7 and 8
⇤ ← Revision 7 as of 2008-02-14 14:45:17 →
Size: 6553
Editor: FelixFrank
Comment: another fs creates example
← Revision 8 as of 2008-02-14 15:02:31 → ⇥
Size: 8065
Editor: FelixFrank
Comment: new segment creation
Deletions are marked like this. Additions are marked like this.
Line 45: Line 45:

=== creating mirrored files ===
Line 50: Line 53:

=== creating striped files ===
Line 97: Line 102:
=== adding segments to an existing OSD file ===
{{{
# fs creates /afs/desytest/osdtest/mirrored._usr_afs.2007-12-17.tar.bz2 2 14
File /afs/desytest/osdtest/mirrored._usr_afs.2007-12-17.tar.bz2 already exists. Create new segment? (Y|N)
y
done
# fs osd mirrored._usr_afs.2007-12-17.tar.bz2
mirrored._usr_afs.2007-12-17.tar.bz2 has 436 bytes of osd metadata, v=3
On-line, 3 segm, flags=0x0
    segment:
        lng=5612872, offs=0, stripes=1, strsize=16384, cop=2, 2 objects
        object:
            pid=536870924, oid=144117812300873732, osd=10, stripe=0
            obj=536870924.4.33555043.0
        object:
            pid=536870924, oid=144117812300873732, osd=11, stripe=0
            obj=536870924.4.33555043.0
    segment:
        lng=5612872, offs=5612872, stripes=2, strsize=16384, cop=1, 2 objects
        object:
            pid=536870924, oid=720578564604297220, osd=12, stripe=0
            obj=536870924.4.167772771.0
        object:
            pid=536870924, oid=3026421573885100036, osd=11, stripe=1
            obj=536870924.4.704643683.1
    segment:
        lng=0, offs=5612872, stripes=2, strsize=16384, cop=1, 2 objects
        object:
            pid=536870924, oid=720578564671406084, osd=12, stripe=0
            obj=536870924.4.167772771.1
        object:
            pid=536870924, oid=3026421573952208900, osd=11, stripe=1
            obj=536870924.4.704643683.2 }}}

TableOfContents

Important notes

On iokaste, /root/.zshenv now contains 

PATH=/usr/afs/bin:$PATH

Status

Currently io runs an rxosd and a Heimdal KDC and the SL5 openafs client.

iokaste runs fileserver, vldbserver, osdserver and ptserver, all OSD-aware, as is the client.

Both run openafs-1.4.6, the servers being compiled with OSD patch.

Building OpenAFS-OSD

In the openafs-1.4.5-osd directory: 

./configure --enable-bitmap-later --enable-bos-restricted-mode --enable-bos-new-config --enable-fast-restart --enable-supergroups --enable-largefile-fileserver --enable-transarc-paths --with-krb5-conf

Enabeling the client

  • From the source directory copy 

    src/libafs/MODLOAD-2.6.18-8.1.15.el5-MP/libafs.ko

    • (where 2.6.18-8.1.15.el5-MP is the kernel version)

    to 

    /lib/modules/2.6.18-8.1.10.el5/kernel/fs/openafs/libafs.ko 
  • depmod
  • reboot

  • launch afsd

Issues

Missing token is not handled well

{{{ $ osd volumes io -lun 1 RX xdr error Cannot list volumes Request aborted.

$ osd createv io 1 536870924 Createvolume failed: RXOSD_create_part ended with 13 Request aborted.}}}

Subcommands

fs createstripedfile

creating mirrored files

{{{ # fs creates /afs/desytest/osdtest/mirrored-file 1 14 -copies 2 }}} where 1 is the number of stripes and 14 is log_2(stripesize in bytes).

Note
Allowed exponents are 12..19, allowing stripe sizes 4kB..512kB.

The above command created an unstriped but mirrored file.

creating striped files

This is how we created the first striped file: 

# fs creates /afs/desytest/osdtest/striped_file_1 2 14
done

We stored a kernel image inside: 

# cp /boot/vmlinuz-2.6.18-8.1.10.el5 striped_file_1
# mv striped_file_1 vmlinuz-2.6.18-8.1.10.el5
# ll
total 7615
-rw-r--r-- 1 bin root  417032 Jan 14 14:54 libafs.ko
-rw-r--r-- 1 bin root 5612872 Jan 28 15:31 mirrored._usr_afs.2007-12-17.tar.bz2
-rw-r--r-- 1 bin root 1765652 Feb 14 13:37 vmlinuz-2.6.18-8.1.10.el5
# fs osd vmlinuz-2.6.18-8.1.10.el5 
vmlinuz-2.6.18-8.1.10.el5 has 172 bytes of osd metadata, v=3
On-line, 1 segm, flags=0x0
    segment:
        lng=0, offs=0, stripes=2, strsize=16384, cop=1, 2 objects
        object:
            pid=536870924, oid=720578573194231814, osd=12, stripe=0 
            obj=536870924.6.167772773.0
        object:
            pid=536870924, oid=3026421582407925766, osd=11, stripe=1 
            obj=536870924.6.704643685.0

As the stripes were put in OSDs 11 and 12 respectively: 

# osd listo
 id name(loc)     ---total space---      flag  prior. own.  server lun size range
  1 local_disk                                 wr  rd                 (0kb-1mb)
  8 io_a             279 gb   0.0 % up   hsm   50   0     io.ifh.d  0 (1mb-64gb)
  9 io_b             279 gb   0.0 % up   hsm   50   0     io.ifh.d  1 (1mb-64gb)
 10 io_c             279 gb   0.0 % up   hsm   50   0     io.ifh.d  2 (1mb-64gb)
 11 io_d             279 gb   0.0 % up   hsm   50   0     io.ifh.d  3 (1mb-64gb)
 12 io_e             278 gb   0.0 % up   hsm   50   0     io.ifh.d  4 (1mb-64gb)
# osd vol io 
1 volumes found:
        536870924
# osd obj io 536870924 3
536870924.4.33555043.0 fid 536870924.4.611 tag 0 not-striped lng 4209654 lc 1
536870924.6.704643685.0 fid 536870924.6.613 tag 0 1/2/16384 lng 880916 lc 1
536870924.4.704643683.1 fid 536870924.4.611 tag 1 1/2/16384 lng 0 lc 1
3 object(s) with totally 5090570 bytes for volume 536870924 found
# osd obj io 536870924 4
536870924.4.167772771.0 fid 536870924.4.611 tag 0 0/2/16384 lng 0 lc 1
536870924.6.167772773.0 fid 536870924.6.613 tag 0 0/2/16384 lng 884736 lc 1
2 object(s) with totally 884736 bytes for volume 536870924 found

The first file on io/3 (vicepd) belongs to the mirrored file created above, not the striped file.

adding segments to an existing OSD file

# fs creates /afs/desytest/osdtest/mirrored._usr_afs.2007-12-17.tar.bz2 2 14
File /afs/desytest/osdtest/mirrored._usr_afs.2007-12-17.tar.bz2 already exists. Create new segment? (Y|N)
y
done 
# fs osd mirrored._usr_afs.2007-12-17.tar.bz2 
mirrored._usr_afs.2007-12-17.tar.bz2 has 436 bytes of osd metadata, v=3
On-line, 3 segm, flags=0x0
    segment:
        lng=5612872, offs=0, stripes=1, strsize=16384, cop=2, 2 objects
        object:
            pid=536870924, oid=144117812300873732, osd=10, stripe=0 
            obj=536870924.4.33555043.0
        object:
            pid=536870924, oid=144117812300873732, osd=11, stripe=0 
            obj=536870924.4.33555043.0
    segment:
        lng=5612872, offs=5612872, stripes=2, strsize=16384, cop=1, 2 objects
        object:
            pid=536870924, oid=720578564604297220, osd=12, stripe=0 
            obj=536870924.4.167772771.0
        object:
            pid=536870924, oid=3026421573885100036, osd=11, stripe=1 
            obj=536870924.4.704643683.1
    segment:
        lng=0, offs=5612872, stripes=2, strsize=16384, cop=1, 2 objects
        object:
            pid=536870924, oid=720578564671406084, osd=12, stripe=0 
            obj=536870924.4.167772771.1
        object:
            pid=536870924, oid=3026421573952208900, osd=11, stripe=1 
            obj=536870924.4.704643683.2

fs osd

{{{ # fs osd /afs/desytest/osdtest/mirrored-file /afs/desytest/osdtest/mirrored-file has 172 bytes of osd metadata, v=3 On-line, 1 segm, flags=0x0

  • segment:
    • lng=0, offs=0, stripes=1, strsize=16384, cop=2, 2 objects object:
      • pid=536870924, oid=144117812300873732, osd=10, stripe=0 obj=536870924.4.33555043.0
      object:
      • pid=536870924, oid=144117812300873732, osd=11, stripe=0 obj=536870924.4.33555043.0 }}}

where osd=10 and osd=11 identify the storing OSD of each file copy. IDs are looked up using osd listo -ip.

osd wipecandidates

# osd wipe io
# osd wipe iokaste
            [ some waiting time ]
RXOSD_wipe_candidates failed with code -1
Request aborted.
# osd wipe /afs/desytest/osdtest/mirrored._usr_afs.2007-12-17.tar.bz2
(no error output)
# osd wipe io 0
# osd wipe io 1
# osd wipe io 2
0  Jan 24 16:03      4209654 536870924.4.33555043.0 536870924.4.33555043
# osd wipe io 3
0  Jan 28 15:31      4209654 536870924.4.33555043.0 536870924.4.33555043
# osd wipe io 4
RXOSD_wipe_candidates failed with code 5
Request aborted.
Note
The last request propably failed because there was no /vicepe/AFSIDat/ directory on io at the time.
Note

Using a filename as the only argument to osd wipe should have at least yielded an error similar to when using iokaste (which is not an OSD server).

Note
The entries for io/2 and io/3 are the two copies of the file created above.

fs wipe

# fs wipe mirrored._usr_afs.2007-12-17.tar.bz2
Could not wipe mirrored._usr_afs.2007-12-17.tar.bz2, error code was 22
Note

This appears to be the default answer to a tried wipe of a file that has not yet been archived using fs archive.

osd volumes

# osd volumes io
1 volumes found:
        536870924

That's the same VolID as in AFS: 

# vos listvl
VLDB entries for all servers 

osdtestvol 
    RWrite: 536870924 
    number of sites -> 1
       server iokaste.ifh.de partition /vicepa RW Site 

root.cell 
    RWrite: 536870921 
    number of sites -> 1
       server iokaste.ifh.de partition /vicepa RW Site 

Total entries: 2

osd objects

Both volume and partition are needed: 

# osd obj io 536870924
0 object(s) with totally 0 bytes for volume 536870924 found

# osd obj io 536870924 2
536870924.4.33555043.0 fid 536870924.4.611 tag 0 not-striped lng 4209654 lc 1
1 object(s) with totally 4209654 bytes for volume 536870924 found
Note
Without specifying a lun, the default of 0 (vicepa) is used.

AfsOsd (last edited 2009-08-17 16:11:48 by WaltrautNiepraschk)