|
Size: 15063
Comment: mirroring of remote code dir
|
← Revision 50 as of 2009-08-17 16:11:48 ⇥
Size: 60
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| [[TableOfContents]] | <<TableOfContents>> |
| Line 3: | Line 3: |
| = Important notes = On ''iokaste'', /root/.zshenv now contains {{{ PATH=/usr/heimdal/bin:/usr/afs/bin:$PATH }}} = Status = Currently '''io''' runs an ''rxosd'' and a Heimdal ''KDC'' and the SL5 openafs client. '''iokaste''' runs ''fileserver'', ''vldbserver'', ''osdserver'' and ''ptserver'', all OSD-aware, as is the client. Both run '''openafs-1.4.6''', the servers being compiled with OSD patch. = Mirroring of the code from Garching = There are two directories with AFS+OSD code on the servers: {{{ /usr/local/src/openafs-1.4.6-osd /usr/local/src/openafs-1.4.6-osd.orig }}} The latter is used to diff against Garching and is usually my primary update site: {{{ rsync -r /afs/ipp-garching.mpg.de/.cs/openafs/openafs-1.4.6-osd.readonly/src openafs-1.4.6-osd.orig }}} This saves me from excluding all the platform-specific directories, but files outside the ''src/'' subdirectory '''must''' be copied manually. For building the code, the ''openafs-1.5.6-osd'' directory (sans ''.orig'' suffix) is used. I update it using another rsync: {{{ rsync openafs-1.4.6-osd.orig/ openafs-1.4.6-osd }}} /!\ When specifying the source for `rsync`, the presence or absence of the trailing slash is vital for the effect of the command. See `rsync (1)`. = Building OpenAFS-OSD = In the ''openafs-1.4.6-osd'' directory: {{{ ./configure --enable-bitmap-later --enable-bos-restricted-mode --enable-bos-new-config --enable-fast-restart --enable-supergroups --enable-largefile-fileserver --enable-transarc-paths --with-krb5-conf --enable-object-storage }}} = Enabeling the client = * From the source directory '''copy''' {{{ src/libafs/MODLOAD-2.6.18-8.1.15.el5-MP/libafs.ko }}} (where ''2.6.18-8.1.15.el5-MP'' is the kernel version) to {{{ /lib/modules/2.6.18-8.1.10.el5/kernel/fs/openafs/libafs.ko }}} * {{{ depmod }}} * reboot * launch ''afsd'' = Enabling OSD = {{{ [iokaste] ~ # vos setfields osdtestvol2 -osd 1 [iokaste] ~ # fs protocol -enable RXOSD }}} = Issues = == Missing token is not handled well == {{{ $ osd volumes io -lun 1 RX xdr error Cannot list volumes Request aborted. $ osd createv io 1 536870924 Createvolume failed: RXOSD_create_part ended with 13 Request aborted.}}} In general, missing privileges are an issue but there is no "friendly output" describing the cause of the problem. Some examples of unprivileged access attempts: || '''fs osd''' || Error 13 reading length of metadata || works with '''r''' privilege granted to system:anyuser || || '''osd read''' || RX xdr error \\ Cannot read the object \\ rx_End``Call returns: 13 \\ Request aborted. || requires token || || '''osd volumes''' || RX xdr error \\ Cannot list volumes \\ Request aborted. || requires token || || '''osd examine''' || RXOSD_examine failed with code 13 \\ Request aborted. || requires token || Database queries '''osd listosds''' and '''osd servers''' seem to work unauthenticated. = Subcommands = == fs createstripedfile == === creating mirrored files === {{{ # fs creates /afs/desytest/osdtest/mirrored-file 1 14 -copies 2 }}} where 1 is the ''number of stripes'' and 14 is ''log_2(stripesize in bytes)''. Note:: Allowed exponents are 12..19, allowing stripe sizes 4kB..512kB. The above command created an unstriped but mirrored file. === creating striped files === This is how we created the first striped file: {{{ # fs creates /afs/desytest/osdtest/striped_file_1 2 14 done }}} We stored a kernel image inside: {{{ # cp /boot/vmlinuz-2.6.18-8.1.10.el5 striped_file_1 # mv striped_file_1 vmlinuz-2.6.18-8.1.10.el5 # ll total 7615 -rw-r--r-- 1 bin root 417032 Jan 14 14:54 libafs.ko -rw-r--r-- 1 bin root 5612872 Jan 28 15:31 mirrored._usr_afs.2007-12-17.tar.bz2 -rw-r--r-- 1 bin root 1765652 Feb 14 13:37 vmlinuz-2.6.18-8.1.10.el5 # fs osd vmlinuz-2.6.18-8.1.10.el5 vmlinuz-2.6.18-8.1.10.el5 has 172 bytes of osd metadata, v=3 On-line, 1 segm, flags=0x0 segment: lng=0, offs=0, stripes=2, strsize=16384, cop=1, 2 objects object: pid=536870924, oid=720578573194231814, osd=12, stripe=0 obj=536870924.6.167772773.0 object: pid=536870924, oid=3026421582407925766, osd=11, stripe=1 obj=536870924.6.704643685.0 }}} As the stripes were put in OSDs 11 and 12 respectively: {{{ # osd listo id name(loc) ---total space--- flag prior. own. server lun size range 1 local_disk wr rd (0kb-1mb) 8 io_a 279 gb 0.0 % up hsm 50 0 io.ifh.d 0 (1mb-64gb) 9 io_b 279 gb 0.0 % up hsm 50 0 io.ifh.d 1 (1mb-64gb) 10 io_c 279 gb 0.0 % up hsm 50 0 io.ifh.d 2 (1mb-64gb) 11 io_d 279 gb 0.0 % up hsm 50 0 io.ifh.d 3 (1mb-64gb) 12 io_e 278 gb 0.0 % up hsm 50 0 io.ifh.d 4 (1mb-64gb) # osd vol io 1 volumes found: 536870924 # osd obj io 536870924 3 536870924.4.33555043.0 fid 536870924.4.611 tag 0 not-striped lng 4209654 lc 1 536870924.6.704643685.0 fid 536870924.6.613 tag 0 1/2/16384 lng 880916 lc 1 536870924.4.704643683.1 fid 536870924.4.611 tag 1 1/2/16384 lng 0 lc 1 3 object(s) with totally 5090570 bytes for volume 536870924 found # osd obj io 536870924 4 536870924.4.167772771.0 fid 536870924.4.611 tag 0 0/2/16384 lng 0 lc 1 536870924.6.167772773.0 fid 536870924.6.613 tag 0 0/2/16384 lng 884736 lc 1 2 object(s) with totally 884736 bytes for volume 536870924 found }}} The first file on ''io/3'' (vicepd) belongs to the mirrored file created above, not the striped file. === adding segments to an existing OSD file === {{{ # fs creates /afs/desytest/osdtest/mirrored._usr_afs.2007-12-17.tar.bz2 2 14 File /afs/desytest/osdtest/mirrored._usr_afs.2007-12-17.tar.bz2 already exists. Create new segment? (Y|N) y done # fs osd mirrored._usr_afs.2007-12-17.tar.bz2 mirrored._usr_afs.2007-12-17.tar.bz2 has 436 bytes of osd metadata, v=3 On-line, 3 segm, flags=0x0 segment: lng=5612872, offs=0, stripes=1, strsize=16384, cop=2, 2 objects object: pid=536870924, oid=144117812300873732, osd=10, stripe=0 obj=536870924.4.33555043.0 object: pid=536870924, oid=144117812300873732, osd=11, stripe=0 obj=536870924.4.33555043.0 segment: lng=5612872, offs=5612872, stripes=2, strsize=16384, cop=1, 2 objects object: pid=536870924, oid=720578564604297220, osd=12, stripe=0 obj=536870924.4.167772771.0 object: pid=536870924, oid=3026421573885100036, osd=11, stripe=1 obj=536870924.4.704643683.1 segment: lng=0, offs=5612872, stripes=2, strsize=16384, cop=1, 2 objects object: pid=536870924, oid=720578564671406084, osd=12, stripe=0 obj=536870924.4.167772771.1 object: pid=536870924, oid=3026421573952208900, osd=11, stripe=1 obj=536870924.4.704643683.2 }}} == fs osd == {{{ # fs osd /afs/desytest/osdtest/mirrored-file /afs/desytest/osdtest/mirrored-file has 172 bytes of osd metadata, v=3 On-line, 1 segm, flags=0x0 segment: lng=0, offs=0, stripes=1, strsize=16384, cop=2, 2 objects object: pid=536870924, oid=144117812300873732, osd=10, stripe=0 obj=536870924.4.33555043.0 object: pid=536870924, oid=144117812300873732, osd=11, stripe=0 obj=536870924.4.33555043.0 }}} where ''osd=10'' and ''osd=11'' identify the storing OSD of each file copy. IDs are looked up using `osd listo`. == osd wipecandidates == {{{ # osd wipe io # osd wipe iokaste [ some waiting time ] RXOSD_wipe_candidates failed with code -1 Request aborted. # osd wipe /afs/desytest/osdtest/mirrored._usr_afs.2007-12-17.tar.bz2 (no error output) # osd wipe io 0 # osd wipe io 1 # osd wipe io 2 0 Jan 24 16:03 4209654 536870924.4.33555043.0 536870924.4.33555043 # osd wipe io 3 0 Jan 28 15:31 4209654 536870924.4.33555043.0 536870924.4.33555043 # osd wipe io 4 RXOSD_wipe_candidates failed with code 5 Request aborted. }}} Note:: The last request propably failed because there was no /vicepe/AFSIDat/ directory on io at the time. Note:: Using a filename as the only argument to `osd wipe` should have at least yielded an error similar to when using ''iokaste'' (which is not an OSD server). Note:: The entries for io/2 and io/3 are the two copies of the file created above. == fs wipe == {{{ # fs wipe mirrored._usr_afs.2007-12-17.tar.bz2 Could not wipe mirrored._usr_afs.2007-12-17.tar.bz2, error code was 22 }}} Note:: This appears to be the default answer to a tried wipe of a file that has not yet been archived using `fs archive`. == osd createvolume == {{{ [iokaste] ~ # vos create iokaste a osdtestvol2 Volume 536870927 created on partition /vicepa of iokaste [iokaste] ~ # osd createvol io 536870927 0 Createvolume: volume 536870927 successfully created. [iokaste] ~ # osd volu io 0 2 volumes found: 536870924 536870927 [iokaste] ~ # osd volu io 1 1 volumes found: 536870924 }}} == osd volumes == {{{ # osd volumes io 1 volumes found: 536870924 }}} That's the same VolID as in AFS: {{{ # vos listvl VLDB entries for all servers osdtestvol RWrite: 536870924 number of sites -> 1 server iokaste.ifh.de partition /vicepa RW Site root.cell RWrite: 536870921 number of sites -> 1 server iokaste.ifh.de partition /vicepa RW Site Total entries: 2 }}} Note:: This information seems to be persistent, even after removing all of a volume's files from Object Storage: {{{ [iokaste] /afs/desytest/osdtest # osd volu io 0 1 volumes found: 536870924 [iokaste] /afs/desytest/osdtest # osd volu io 1 1 volumes found: 536870924 [iokaste] /afs/desytest/osdtest # osd volu io 2 1 volumes found: 536870924 [iokaste] /afs/desytest/osdtest # osd volu io 3 1 volumes found: 536870924 [iokaste] /afs/desytest/osdtest # osd volu io 4 1 volumes found: 536870924 [iokaste] /afs/desytest/osdtest # osd volu io 5 0 volumes found: [iokaste] /afs/desytest/osdtest # osd obj io 536870924 0 0 object(s) with totally 0 bytes for volume 536870924 found [iokaste] /afs/desytest/osdtest # osd obj io 536870924 1 0 object(s) with totally 0 bytes for volume 536870924 found [iokaste] /afs/desytest/osdtest # osd obj io 536870924 2 0 object(s) with totally 0 bytes for volume 536870924 found [iokaste] /afs/desytest/osdtest # osd obj io 536870924 0 object(s) with totally 0 bytes for volume 536870924 found [iokaste] /afs/desytest/osdtest # osd obj io 536870924 3 0 object(s) with totally 0 bytes for volume 536870924 found [iokaste] /afs/desytest/osdtest # osd obj io 536870924 4 0 object(s) with totally 0 bytes for volume 536870924 found }}} == osd objects == Both volume and partition are needed: {{{ # osd obj io 536870924 0 object(s) with totally 0 bytes for volume 536870924 found # osd obj io 536870924 2 536870924.4.33555043.0 fid 536870924.4.611 tag 0 not-striped lng 4209654 lc 1 1 object(s) with totally 4209654 bytes for volume 536870924 found }}} Note:: Without specifying a lun, the default of 0 (vicepa) is used. == osd examine == This command also has issues with missing LUNs: {{{ [iokaste] /afs/desytest/osdtest # osd exa io 536870924.6.621.0 RXOSD_examine failed with code 5 Request aborted. [iokaste] /afs/desytest/osdtest # osd exa io 536870924.6.621.0 -lun 4 536870924.6.621.0 fid 536870924.6.621 tag 0 not-striped lng 44537893 lc 1 Feb 21 13:53 }}} == osd read == This works for unstriped files, but suffers from the "missing LUN syndrome" as well: {{{ [iokaste] /afs/desytest/osdtest # osd read io 536870924.6.621 RX xdr error Cannot read the object rx_EndCall returns: 5 Request aborted. [iokaste] /afs/desytest/osdtest # osd read io 536870924.6.621 0 512 -lun 4 sscanf failed at offset (0x0, 0x0) [1177711200] LOG ROTATION: DAILY [1177711200] LOG VERSION: 2.0 [1177711200] CURRENT HOST STATE: a;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.13 ms [1177711200] CURRENT HOST STATE: amafs;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.14 ms [1177711200] CURRENT HOST STATE: aphrodite;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.54 ms [1177711200] CURRENT HOST STATE: aquila;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.25 ms [1177711200] CURRENT HOST STATE: ares;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0. reading of 512 bytes took 0.003 sec. Total data rate = 180 Kbytes/sec. for read. }}} Note:: '''0''' and '''512''' in example 2 are ''offset'' and ''bytes to read'', resp. == fs replaceosd == First tries were very discouraging: {{{ [iokaste] ~ # fs osd /afs/desytest/osdtest/nagios-04-29-2007-00.log /afs/desytest/osdtest/nagios-04-29-2007-00.log has 132 bytes of osd metadata, v=3 On-line, 1 segm, flags=0x0 segment: lng=0, offs=0, stripes=1, strsize=0, cop=1, 1 objects object: pid=536870924, oid=2667174690822, osd=12, stripe=0 obj=536870924.6.621.0 [iokaste] ~ # fs rep /afs/desytest/osdtest/nagios-04-29-2007-00.log 12 8 failed to replace osd 12 for /afs/desytest/osdtest/nagios-04-29-2007-00.log, error code is 5 [iokaste] ~ # fs rep /afs/desytest/osdtest/nagios-04-29-2007-00.log 12 }}} The last command hung and could not be TERMed or KILLed. A subsequent {{{ [iokaste] ~ # fs osd /afs/desytest/osdtest/nagios-04-29-2007-00.log }}} got stuck much the same. = Case study: Creating a long striped file = {{{ [iokaste] /afs/desytest/osdtest # fs creates striped-ascii-44m 2 12 done [iokaste] /afs/desytest/osdtest # cp /root/nagios-04-29-2007-00.log striped-ascii-44m [ took about 3 seconds ] [iokaste] /afs/desytest/osdtest # fs osd striped-ascii-44m striped-ascii-44m has 172 bytes of osd metadata, v=3 On-line, 1 segm, flags=0x0 segment: lng=0, offs=0, stripes=2, strsize=4096, cop=1, 2 objects object: pid=536870924, oid=576463393708310532, osd=12, stripe=0 obj=536870924.4.134218343.0 object: pid=536870924, oid=2882306402922004484, osd=11, stripe=1 obj=536870924.4.671089255.0 [iokaste] /afs/desytest/osdtest # osd obj io 536870924 3 536870924.4.671089255.0 fid 536870924.4.615 tag 0 1/2/4096 lng 22267941 lc 1 1 object(s) with totally 22267941 bytes for volume 536870924 found [iokaste] /afs/desytest/osdtest # osd obj io 536870924 4 536870924.4.134218343.0 fid 536870924.4.615 tag 0 0/2/4096 lng 22269952 lc 1 1 object(s) with totally 22269952 bytes for volume 536870924 found }}} Note:: The lengths ('''lng''') of both objects amount to the size of the stored file: `22267941 + 22269952 = 44537893`. |
Sub pages: <<Navigation(children)>> |
Contents
Sub pages: