|
Size: 11681
Comment: first try of osd read
|
← Revision 50 as of 2009-08-17 16:11:48 ⇥
Size: 60
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| [[TableOfContents]] | <<TableOfContents>> |
| Line 3: | Line 3: |
| = Important notes = On ''iokaste'', /root/.zshenv now contains {{{ PATH=/usr/afs/bin:$PATH }}} = Status = Currently '''io''' runs an ''rxosd'' and a Heimdal ''KDC'' and the SL5 openafs client. '''iokaste''' runs ''fileserver'', ''vldbserver'', ''osdserver'' and ''ptserver'', all OSD-aware, as is the client. Both run '''openafs-1.4.6''', the servers being compiled with OSD patch. = Building OpenAFS-OSD = In the ''openafs-1.4.5-osd'' directory: {{{ ./configure --enable-bitmap-later --enable-bos-restricted-mode --enable-bos-new-config --enable-fast-restart --enable-supergroups --enable-largefile-fileserver --enable-transarc-paths --with-krb5-conf }}} = Enabeling the client = * From the source directory '''copy''' {{{ src/libafs/MODLOAD-2.6.18-8.1.15.el5-MP/libafs.ko }}} (where ''2.6.18-8.1.15.el5-MP'' is the kernel version) to {{{ /lib/modules/2.6.18-8.1.10.el5/kernel/fs/openafs/libafs.ko }}} * {{{ depmod }}} * reboot * launch ''afsd'' = Issues = == Missing token is not handled well == {{{ $ osd volumes io -lun 1 RX xdr error Cannot list volumes Request aborted. $ osd createv io 1 536870924 Createvolume failed: RXOSD_create_part ended with 13 Request aborted.}}} = Subcommands = == fs createstripedfile == === creating mirrored files === {{{ # fs creates /afs/desytest/osdtest/mirrored-file 1 14 -copies 2 }}} where 1 is the ''number of stripes'' and 14 is ''log_2(stripesize in bytes)''. Note:: Allowed exponents are 12..19, allowing stripe sizes 4kB..512kB. The above command created an unstriped but mirrored file. === creating striped files === This is how we created the first striped file: {{{ # fs creates /afs/desytest/osdtest/striped_file_1 2 14 done }}} We stored a kernel image inside: {{{ # cp /boot/vmlinuz-2.6.18-8.1.10.el5 striped_file_1 # mv striped_file_1 vmlinuz-2.6.18-8.1.10.el5 # ll total 7615 -rw-r--r-- 1 bin root 417032 Jan 14 14:54 libafs.ko -rw-r--r-- 1 bin root 5612872 Jan 28 15:31 mirrored._usr_afs.2007-12-17.tar.bz2 -rw-r--r-- 1 bin root 1765652 Feb 14 13:37 vmlinuz-2.6.18-8.1.10.el5 # fs osd vmlinuz-2.6.18-8.1.10.el5 vmlinuz-2.6.18-8.1.10.el5 has 172 bytes of osd metadata, v=3 On-line, 1 segm, flags=0x0 segment: lng=0, offs=0, stripes=2, strsize=16384, cop=1, 2 objects object: pid=536870924, oid=720578573194231814, osd=12, stripe=0 obj=536870924.6.167772773.0 object: pid=536870924, oid=3026421582407925766, osd=11, stripe=1 obj=536870924.6.704643685.0 }}} As the stripes were put in OSDs 11 and 12 respectively: {{{ # osd listo id name(loc) ---total space--- flag prior. own. server lun size range 1 local_disk wr rd (0kb-1mb) 8 io_a 279 gb 0.0 % up hsm 50 0 io.ifh.d 0 (1mb-64gb) 9 io_b 279 gb 0.0 % up hsm 50 0 io.ifh.d 1 (1mb-64gb) 10 io_c 279 gb 0.0 % up hsm 50 0 io.ifh.d 2 (1mb-64gb) 11 io_d 279 gb 0.0 % up hsm 50 0 io.ifh.d 3 (1mb-64gb) 12 io_e 278 gb 0.0 % up hsm 50 0 io.ifh.d 4 (1mb-64gb) # osd vol io 1 volumes found: 536870924 # osd obj io 536870924 3 536870924.4.33555043.0 fid 536870924.4.611 tag 0 not-striped lng 4209654 lc 1 536870924.6.704643685.0 fid 536870924.6.613 tag 0 1/2/16384 lng 880916 lc 1 536870924.4.704643683.1 fid 536870924.4.611 tag 1 1/2/16384 lng 0 lc 1 3 object(s) with totally 5090570 bytes for volume 536870924 found # osd obj io 536870924 4 536870924.4.167772771.0 fid 536870924.4.611 tag 0 0/2/16384 lng 0 lc 1 536870924.6.167772773.0 fid 536870924.6.613 tag 0 0/2/16384 lng 884736 lc 1 2 object(s) with totally 884736 bytes for volume 536870924 found }}} The first file on ''io/3'' (vicepd) belongs to the mirrored file created above, not the striped file. === adding segments to an existing OSD file === {{{ # fs creates /afs/desytest/osdtest/mirrored._usr_afs.2007-12-17.tar.bz2 2 14 File /afs/desytest/osdtest/mirrored._usr_afs.2007-12-17.tar.bz2 already exists. Create new segment? (Y|N) y done # fs osd mirrored._usr_afs.2007-12-17.tar.bz2 mirrored._usr_afs.2007-12-17.tar.bz2 has 436 bytes of osd metadata, v=3 On-line, 3 segm, flags=0x0 segment: lng=5612872, offs=0, stripes=1, strsize=16384, cop=2, 2 objects object: pid=536870924, oid=144117812300873732, osd=10, stripe=0 obj=536870924.4.33555043.0 object: pid=536870924, oid=144117812300873732, osd=11, stripe=0 obj=536870924.4.33555043.0 segment: lng=5612872, offs=5612872, stripes=2, strsize=16384, cop=1, 2 objects object: pid=536870924, oid=720578564604297220, osd=12, stripe=0 obj=536870924.4.167772771.0 object: pid=536870924, oid=3026421573885100036, osd=11, stripe=1 obj=536870924.4.704643683.1 segment: lng=0, offs=5612872, stripes=2, strsize=16384, cop=1, 2 objects object: pid=536870924, oid=720578564671406084, osd=12, stripe=0 obj=536870924.4.167772771.1 object: pid=536870924, oid=3026421573952208900, osd=11, stripe=1 obj=536870924.4.704643683.2 }}} == fs osd == {{{ # fs osd /afs/desytest/osdtest/mirrored-file /afs/desytest/osdtest/mirrored-file has 172 bytes of osd metadata, v=3 On-line, 1 segm, flags=0x0 segment: lng=0, offs=0, stripes=1, strsize=16384, cop=2, 2 objects object: pid=536870924, oid=144117812300873732, osd=10, stripe=0 obj=536870924.4.33555043.0 object: pid=536870924, oid=144117812300873732, osd=11, stripe=0 obj=536870924.4.33555043.0 }}} where ''osd=10'' and ''osd=11'' identify the storing OSD of each file copy. IDs are looked up using `osd listo -ip`. == osd wipecandidates == {{{ # osd wipe io # osd wipe iokaste [ some waiting time ] RXOSD_wipe_candidates failed with code -1 Request aborted. # osd wipe /afs/desytest/osdtest/mirrored._usr_afs.2007-12-17.tar.bz2 (no error output) # osd wipe io 0 # osd wipe io 1 # osd wipe io 2 0 Jan 24 16:03 4209654 536870924.4.33555043.0 536870924.4.33555043 # osd wipe io 3 0 Jan 28 15:31 4209654 536870924.4.33555043.0 536870924.4.33555043 # osd wipe io 4 RXOSD_wipe_candidates failed with code 5 Request aborted. }}} Note:: The last request propably failed because there was no /vicepe/AFSIDat/ directory on io at the time. Note:: Using a filename as the only argument to `osd wipe` should have at least yielded an error similar to when using ''iokaste'' (which is not an OSD server). Note:: The entries for io/2 and io/3 are the two copies of the file created above. == fs wipe == {{{ # fs wipe mirrored._usr_afs.2007-12-17.tar.bz2 Could not wipe mirrored._usr_afs.2007-12-17.tar.bz2, error code was 22 }}} Note:: This appears to be the default answer to a tried wipe of a file that has not yet been archived using `fs archive`. == osd volumes == {{{ # osd volumes io 1 volumes found: 536870924 }}} That's the same VolID as in AFS: {{{ # vos listvl VLDB entries for all servers osdtestvol RWrite: 536870924 number of sites -> 1 server iokaste.ifh.de partition /vicepa RW Site root.cell RWrite: 536870921 number of sites -> 1 server iokaste.ifh.de partition /vicepa RW Site Total entries: 2 }}} Note:: This information seems to be persistent, even after removing all of a volume's files from Object Storage: {{{ [iokaste] /afs/desytest/osdtest # osd volu io 0 1 volumes found: 536870924 [iokaste] /afs/desytest/osdtest # osd volu io 1 1 volumes found: 536870924 [iokaste] /afs/desytest/osdtest # osd volu io 2 1 volumes found: 536870924 [iokaste] /afs/desytest/osdtest # osd volu io 3 1 volumes found: 536870924 [iokaste] /afs/desytest/osdtest # osd volu io 4 1 volumes found: 536870924 [iokaste] /afs/desytest/osdtest # osd volu io 5 0 volumes found: [iokaste] /afs/desytest/osdtest # osd obj io 536870924 0 0 object(s) with totally 0 bytes for volume 536870924 found [iokaste] /afs/desytest/osdtest # osd obj io 536870924 1 0 object(s) with totally 0 bytes for volume 536870924 found [iokaste] /afs/desytest/osdtest # osd obj io 536870924 2 0 object(s) with totally 0 bytes for volume 536870924 found [iokaste] /afs/desytest/osdtest # osd obj io 536870924 0 object(s) with totally 0 bytes for volume 536870924 found [iokaste] /afs/desytest/osdtest # osd obj io 536870924 3 0 object(s) with totally 0 bytes for volume 536870924 found [iokaste] /afs/desytest/osdtest # osd obj io 536870924 4 0 object(s) with totally 0 bytes for volume 536870924 found }}} == osd objects == Both volume and partition are needed: {{{ # osd obj io 536870924 0 object(s) with totally 0 bytes for volume 536870924 found # osd obj io 536870924 2 536870924.4.33555043.0 fid 536870924.4.611 tag 0 not-striped lng 4209654 lc 1 1 object(s) with totally 4209654 bytes for volume 536870924 found }}} Note:: Without specifying a lun, the default of 0 (vicepa) is used. = Case study: Creating a big, striped file = {{{ [iokaste] /afs/desytest/osdtest # fs creates striped-ascii-44m 2 12 done [iokaste] /afs/desytest/osdtest # cp /root/nagios-04-29-2007-00.log striped-ascii-44m [ took about 3 seconds ] [iokaste] /afs/desytest/osdtest # fs osd striped-ascii-44m striped-ascii-44m has 172 bytes of osd metadata, v=3 On-line, 1 segm, flags=0x0 segment: lng=0, offs=0, stripes=2, strsize=4096, cop=1, 2 objects object: pid=536870924, oid=576463393708310532, osd=12, stripe=0 obj=536870924.4.134218343.0 object: pid=536870924, oid=2882306402922004484, osd=11, stripe=1 obj=536870924.4.671089255.0 [iokaste] /afs/desytest/osdtest # osd obj io 536870924 3 536870924.4.671089255.0 fid 536870924.4.615 tag 0 1/2/4096 lng 22267941 lc 1 1 object(s) with totally 22267941 bytes for volume 536870924 found [iokaste] /afs/desytest/osdtest # osd obj io 536870924 4 536870924.4.134218343.0 fid 536870924.4.615 tag 0 0/2/4096 lng 22269952 lc 1 1 object(s) with totally 22269952 bytes for volume 536870924 found }}} Note:: The lengths ('''lng''') of both objects amount to the size of the stored file: `22267941 + 22269952 = 44537893`. First attempt at reading from an object directly: {{{ [iokaste] /afs/desytest/osdtest # osd read -server io -fid 536870924.4.134218343.0 -offset 0 -length 1024 -lun 3 -rxdebug debug.out RX xdr error Cannot read the object rx_EndCall returns: 5 Request aborted. [iokaste] /afs/desytest/osdtest # cat debug.out 1203598296.452:rx_NewConnection(host 6416228d, port 25371, service 900, securityObject 9f72710, serviceSecurityIndex 2) 1203598296.453:rx_MakeCall(conn 9f73948) 1203598296.453:Alloc 9f84220, class 1 1203598296.453:S 1 data: 6416228d.25371.1.3065751495.3809172764.1.1 flags 5, packet 9f84220 resend 1203598298.453 len 836 1203598296.453:R 1 challenge: 6416228d.25371.900.-1229215801.-485794532.0.0 flags 0, packet 9f661b0 1203598296.453:S 2 response: 6416228d.25371.2.3065751495.3809172764.0.0 flags 1, packet 9f661b0 resend 0.000 len 307 1203598296.454:R 2 abort: 6416228d.25371.900.-1229215801.-485794532.1.0 flags 0, packet 9f661b0 1203598296.454:rx_EndCall(call 9f84898) }}} |
Sub pages: <<Navigation(children)>> |
Contents
Sub pages: