⇤ ← Revision 1 as of 2008-05-19 17:32:23
4722
Comment: first buch of info from garching and aftermath
|
5595
some more notes
|
Deletions are marked like this. | Additions are marked like this. |
Line 4: | Line 4: |
== Goodies == * the `threads` subcommand lets you know what each thread is currently doing {{{osd threads io }}} |
|
Line 17: | Line 21: |
This is how a fileserver and OSD server can share one machine: * touch the file ''/vicepx/OnlyRXOSD'' to forbid the fileserver to use this partition |
|
Line 34: | Line 41: |
* the fill percentage plays a role if above 50% or 95% respectively | |
Line 46: | Line 54: |
== How to upgrade a cell to AFS+OSD == 1. set up OSDDB on the database servers 2. set up pristine AFS+OSD fileservers + OSDs 3. move volumes to the AFS+OSD fileservers * volserver is supposed to be armed with a `-convertvolumes` switch for that purpose * otherwise, set the osdflag by hand {{{vos setfields <volume> -osd 1 }}} |
|
Line 126: | Line 140: |
* many osd subcommands require the user to onter IP+LUN, where OSD ID would be better * this has to be changed, and care must be taken as these commands are in production at Garching |
General concepts
Goodies
the threads subcommand lets you know what each thread is currently doing {{{osd threads io
}}}
Working with an AFS+OSD cell
The following techniques are only useful when using OSD as a frontend for HSM:
- {{{vos listobj
}}} is useful for finding all objects stored on a given OSD
- {{{fs prefetch
}}} allows the user to schedule tape restore operations
it is useful to have bosserver run a script to wipe objects off the OSDs according to OSD fill rate etc.
These are general techniques:
- {{{osd addserver
}}} makes an AFS fileserver known to the OSDDB
this is required in order to make use of ownerships and locations
This is how a fileserver and OSD server can share one machine:
touch the file /vicepx/OnlyRXOSD to forbid the fileserver to use this partition
How to migrate data from an OSD
- set a low write priority to stop fileservers from storing data on the OSD in question {{{osd setosd -wrprior 0
}}}
- use {{{vos listobj
}}} to identify the files (by fid) that have data on the OSD
- use {{{fs replaceosd
}}} to move each file's data to another OSD
Priorities and choice of storing OSD
- OSDs are used in a Round Robin fashion
- priorities add weight to each OSD
- static priorities can be set by the administrator {{{osd setosd -wrprior ... -rdprior ...
}}}
- priorities are dynamic
- ownerships apply a modifier of +/- 10 to the static priority
- locations apply a modifier of +/- 20
- the fill percentage plays a role if above 50% or 95% respectively
Data held in volumes, DBs etc.
- all metadata belonging to files in a volume are stored in a designated volume special file
metadata references OSDs by id
OSD id is an index into the OSDDB
- these IDs are permanent and can never be reused after deletion of an OSDDB entry
- view the OSDDB using {{{osd l
}}}
a file's metadata is found using the metadataindex stored in the file's vnode
view the metadata using {{{fs osd <file>
}}}
How to upgrade a cell to AFS+OSD
- set up OSDDB on the database servers
- set up pristine AFS+OSD fileservers + OSDs
- move volumes to the AFS+OSD fileservers
volserver is supposed to be armed with a -convertvolumes switch for that purpose
otherwise, set the osdflag by hand {{{vos setfields <volume> -osd 1
}}}
Policies
- policies are going to decide 4 basic things:
- whether object storage is to be used for a given file
- whether the objects are to be mirrored and how often
- how many stripes the file will comprise
- the size of the stripes
- two pieces of information will be used to make decisions:
- file size
- file name (i.e. prefix and suffix)
Open questions
- will inheritance be supported for policy definitions?
how many levels of inheritance are permitted (1 or n)?
- in what way can policies be represented/stored?
Technical aspects
Performance
- client connections to OSDs are cheap because they are unauthorized
- clients do connect to each single OSD they have business with
osd psread acts as though reading stripes from multiple OSDs
- i.e. it opens several connections, in this case all targets are the same however
- each stripe has an altered offset (e.g., normally first n stripes start at offset 0 on each OSD, here it is 0+(i*stripesize) etc.)
- this is not impossible for access to AFS fileservers, but making connections is more costly
Notes on the code (changes)
- original linktable was insufficient, further fields were required
- link table v2 uses 30 bits instead of 15
- hashing is used for management of DB contents
choice of OSDs resides in osddbuser.c
Debugging techniques
- add trace* calls to the code
- CM_TRACE_WASHERE especially handy
use