cabling
This is the rear view of the SCSI cabling:
attachment:iole12-setup.png [attachment:iole12-setup.fig xfig source]
RAID array setup and mapping
RAID |
drives |
LD |
Partition |
SCSI Channel |
ID |
Host |
Adapter |
device |
raid-iole |
1-4 |
0 |
0 |
0 |
1 |
iole1 |
0 |
sda |
|
|
|
1 |
|
2 |
|
|
sdb |
|
5-8 |
1 |
- |
|
3 |
|
|
sdc |
|
9-12 |
2 |
0 |
1 |
1 |
iole2 |
0 |
sda |
|
|
|
1 |
|
2 |
|
|
sdb |
|
13-16 |
3 |
- |
|
3 |
|
|
sdc |
raid-iolaos |
1-4 |
0 |
0 |
0 |
1 |
iole1 |
1 |
sdd |
|
|
|
1 |
|
2 |
|
|
sde |
|
5-8 |
1 |
- |
|
3 |
|
|
sdf |
|
9-12 |
2 |
0 |
1 |
1 |
iole2 |
1 |
sdd |
|
|
|
1 |
|
2 |
|
|
sde |
|
13-16 |
3 |
- |
|
3 |
|
|
sdf |
kickstart partitioning
clearpart --drives=sda,sdd --initlabel part raid.01 --size 256 --ondisk sda part raid.03 --size 1024 --ondisk sda part raid.05 --size 2048 --ondisk sda part raid.07 --size 10240 --ondisk sda part raid.09 --size 1 --ondisk sda --grow part raid.02 --size 256 --ondisk sdd part raid.04 --size 1024 --ondisk sdd part raid.06 --size 2048 --ondisk sdd part raid.08 --size 10240 --ondisk sdd part raid.10 --size 1 --ondisk sdd --grow raid /boot --level=1 --device=md0 --fstype ext2 raid.01 raid.02 raid /afs_cache --level=1 --device=md1 --fstype ext3 raid.03 raid.04 raid swap --level=1 --device=md2 --fstype swap raid.05 raid.06 raid / --level=1 --device=md3 --fstype ext3 raid.07 raid.08 raid /usr1 --level=1 --device=md4 --fstype ext3 raid.09 raid.10
This should be safe to reuse if anything ever has to be reinstalled, since there are no data partitions on any of these block devices. To play safe, add the following to the cks3 files:
$cfg{PREINSTALL_ADD} = "sleep 86400";and rerun CKS3. Then check /proc/partitions on virtual console #2: You should see 6 disks, and sda and sdd should be the small ones. It is then safe to "killall -TERM sleep" to continue the installation.
adding the MD devices for the vice partitions
First, created primary partitions spanning the whole device on sdb, sdc, sde, sdf. This was done with fdisk. Note the type must be fd (Linux RAID Autodetect).
Then created the devices:
mdadm -create /dev/md5 -l 1 --raid-devices=2 /dev/sdb1 /dev/sde1 mdadm -create /dev/md6 -l 1 --raid-devices=2 /dev/sdc1 /dev/sdf1
Initialization takes very long! When all four devices were initialized at the same time, it took > 24 hours, even though the max bandwidth was set to 50000 in /proc/sys/dev/raid/speed_limit_max. The limiting factor seemed to be the writes ot raid-iolaos.
installing GRUB on sdd
While kickstart will happily put /boot onto /dev/md0, it will install GRUB in the master boot record of /dev/sda only. Hence when raid-iole is not operational, neither iole1 nor iole2 could boot.
To remedy this, GRUB was installed into the master boot record of sdd on both systems manually. After starting grub (as root):
grub > device (hd0) /dev/sdd grub > root (hd0,0) grub > setup (hd0) grub > quit
This of course assumes that /boot is a separate partition. The device command accounts for the fact that /dev/hdd will be the first BIOS drive if raid-iole is unavailable.
Some more info [http://www.linuxquestions.org/questions/archive/8/2005/03/1/297043 here].
TODO: mdadm.conf
maintenance
everything ok ?
This is what /proc/mdstat looks like if all is fine:
Personalities : [raid1]
read_ahead 1024 sectors
Event: 7
md0 : active raid1 sdd1[1] sda1[0]
264960 blocks [2/2] [UU]
md3 : active raid1 sdd2[1] sda2[0]
10482304 blocks [2/2] [UU]
md2 : active raid1 sdd3[1] sda3[0]
2096384 blocks [2/2] [UU]
md1 : active raid1 sdd5[1] sda5[0]
1052160 blocks [2/2] [UU]
md4 : active raid1 sdd6[1] sda6[0]
88501952 blocks [2/2] [UU]
md5 : active raid1 sde1[1] sdb1[0]
1068924800 blocks [2/2] [UU]
md6 : active raid1 sdf1[1] sdc1[0]
1171331136 blocks [2/2] [UU]
unused devices: <none>If an MD array is degraded, it looks like this:
md5 : active raid1 sdb1[0]
1068924800 blocks [2/1] [U_]
TODO: mdmonitor
Logical Drive or Partition removed from MD array
This happened upon the very first reboot of iole2 after the vice filesystems had been created.
Although all devices showed up:
SCSI subsystem driver Revision: 1.00
scsi0 : Adaptec AIC79XX PCI-X SCSI HBA DRIVER, Rev 1.3.10-RH1
<Adaptec AIC7902 Ultra320 SCSI adapter>
aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs
scsi1 : Adaptec AIC79XX PCI-X SCSI HBA DRIVER, Rev 1.3.10-RH1
<Adaptec AIC7902 Ultra320 SCSI adapter>
aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs
blk: queue c36b0618, I/O limit 4095Mb (mask 0xffffffff)
(scsi0:A:1): 160.000MB/s transfers (80.000MHz DT, 16bit)
(scsi1:A:1): 160.000MB/s transfers (80.000MHz DT, 16bit)
(scsi1:A:2): 160.000MB/s transfers (80.000MHz DT, 16bit)
(scsi0:A:2): 160.000MB/s transfers (80.000MHz DT, 16bit)
(scsi1:A:3): 160.000MB/s transfers (80.000MHz DT, 16bit)
(scsi0:A:3): 160.000MB/s transfers (80.000MHz DT, 16bit)
Vendor: IFT Model: A16U-G1A3 Rev: 342J
Type: Direct-Access ANSI SCSI revision: 03
blk: queue c36b0418, I/O limit 4095Mb (mask 0xffffffff)
Vendor: IFT Model: A16U-G1A3 Rev: 342J
Type: Direct-Access ANSI SCSI revision: 03
blk: queue c36b0018, I/O limit 4095Mb (mask 0xffffffff)
Vendor: IFT Model: A16U-G1A3 Rev: 342J
Type: Direct-Access ANSI SCSI revision: 03
blk: queue c36b0218, I/O limit 4095Mb (mask 0xffffffff)
scsi0:A:1:0: Tagged Queuing enabled. Depth 32
scsi0:A:2:0: Tagged Queuing enabled. Depth 32
scsi0:A:3:0: Tagged Queuing enabled. Depth 32
Attached scsi disk sda at scsi0, channel 0, id 1, lun 0
Attached scsi disk sdb at scsi0, channel 0, id 2, lun 0
Attached scsi disk sdc at scsi0, channel 0, id 3, lun 0
SCSI device sda: 204800000 512-byte hdwr sectors (104858 MB)
Partition check:
sda: sda1 sda2 sda3 sda4 < sda5 sda6 >
SCSI device sdb: 2137864192 512-byte hdwr sectors (1094586 MB)
sdb: sdb1
SCSI device sdc: 2342664192 512-byte hdwr sectors (1200418 MB)
sdc: sdc1
blk: queue f779b018, I/O limit 4095Mb (mask 0xffffffff)
Vendor: IFT Model: A16U-G1A3 Rev: 342J
Type: Direct-Access ANSI SCSI revision: 03
blk: queue f7794e18, I/O limit 4095Mb (mask 0xffffffff)
Vendor: IFT Model: A16U-G1A3 Rev: 342J
Type: Direct-Access ANSI SCSI revision: 03
blk: queue f7794c18, I/O limit 4095Mb (mask 0xffffffff)
Vendor: IFT Model: A16U-G1A3 Rev: 342J
Type: Direct-Access ANSI SCSI revision: 03
blk: queue f7794a18, I/O limit 4095Mb (mask 0xffffffff)
scsi1:A:1:0: Tagged Queuing enabled. Depth 32
scsi1:A:2:0: Tagged Queuing enabled. Depth 32
scsi1:A:3:0: Tagged Queuing enabled. Depth 32
Attached scsi disk sdd at scsi1, channel 0, id 1, lun 0
Attached scsi disk sde at scsi1, channel 0, id 2, lun 0
Attached scsi disk sdf at scsi1, channel 0, id 3, lun 0
SCSI device sdd: 204800000 512-byte hdwr sectors (104858 MB)
sdd: sdd1 sdd2 sdd3 sdd4 < sdd5 sdd6 >
SCSI device sde: 2137864192 512-byte hdwr sectors (1094586 MB)
SCSI device sdf: 2342664192 512-byte hdwr sectors (1200418 MB)
Nov 1 15:12:03 iole2 sysctl: kernel.core_uses_pid = 1
sdf: sdf1this is what happened a moment later:
md: considering sdd1 ... md: adding sdd1 ... md: adding sda1 ... md: created md0 md: bind<sda1,1> md: bind<sdd1,2> md: running: <sdd1><sda1> md: sdd1's event counter: 00000006 md: sda1's event counter: 00000006 md: RAID level 1 does not need chunksize! Continuing anyway. md0: max total readahead window set to 124k md0: 1 data-disks, max readahead per data-disk: 124k raid1: device sdd1 operational as mirror 1 raid1: device sda1 operational as mirror 0 raid1: raid set md0 active with 2 out of 2 mirrors md: considering sdb1 ... md: adding sdb1 ... md: created md5 md: bind<sdb1,1> md: running: <sdb1> md: sdb1's event counter: 00000002 md5: former device sde1 is unavailable, removing from array! md5: max total readahead window set to 124k md5: 1 data-disks, max readahead per data-disk: 124k raid1: device sdb1 operational as mirror 0 raid1: md5, not all disks are operational -- trying to recover array raid1: raid set md5 active with 1 out of 2 mirrors md5: no spare disk to reconstruct array! -- continuing in degraded mode
Since sde1 was accessible (again - still no clue what happened during boot; there were no further messages), it could be added to the array:
# mdadm -a /dev/md5 /dev/sde1
Rebuild progress will be displayed in /proc/mdstat.