Experience with online capacity expansion
How to expand filesystem using Broadcom RAID online capacity expansion (OCE)
I added one drive, then 2 more drives after getting more cabling. This shows one having been added and attached, but not fully initialized so it can be added to the array.
[root@bfraid ~]# bcadm -M -qa
bcadm -M -qa
<VERSIONS>
bcadm: RC-140-2005128.2
driver: RC-140-2005128.2
bios: RC-140-2005079.3
<CONTROLLER LIST>
Serial License
Number Type Number Key
—— ——– ———— ———————–
0 BC4852 400000000422 32211-1111U-Z1HYK-VKJ81
<DISK LIST> Largest
Free Free G Ctrl Model Firmware
Disk State Type Size Space Space S Ca Chan Number Version
—- ——- ——- ——- ——- ——- - — —- ————— ——–
6 Online New 245.8GB 245.7GB 245.7GB - NC 0:5 WDC WD2500JD-00 02.05D02
5 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:4 WDC WD2500JD-75 08.02D08
4 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:3 WDC WD2500JD-75 08.02D08
3 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:2 WDC WD2500JD-75 08.02D08
2 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:1 WDC WD2500JD-75 08.02D08
1 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:0 WDC WD2500JD-75 08.02D08
<ARRAY LIST>
A Type OS Name State Size Hide Id Task %
CA Name
- ——- ———- ——— ——- —- —————— ———— —-
- — ——————————-
1 RAID5 /dev/sda NORMAL 982.9GB NO 0×0000f715ed248225 NOT_ACTIVE ….
. RW <none>To add, you need to initialize the drive
[root@bfraid ~]# bcadm -M -id -d 6
bcadm -M -id -d 6
initializing disk 6This shows max size of new array if you were to add drive 6.
[root@bfraid ~]# bcadm -T –raid5 –disk 1 2 3 4 5 6 –max-size -a 1
bcadm -T –raid5 –disk 1 2 3 4 5 6 –max-size -a 1
Max Possible Size: 1171725
After being initialized, the drive status shows as “Disk” instead of “New”
[root@bfraid ~]# bcadm -M -qa
bcadm -M -qa
<VERSIONS>
bcadm: RC-140-2005128.2
driver: RC-140-2005128.2
bios: RC-140-2005079.3
<CONTROLLER LIST>
Serial License
Number Type Number Key
—— ——– ———— ———————–
0 BC4852 400000000422 32211-1111U-Z1HYK-VKJ81
<DISK LIST> Largest
Free Free G Ctrl Model Firmware
Disk State Type Size Space Space S Ca Chan Number Version
—- ——- ——- ——- ——- ——- - — —- ————— ——–
6 Online Disk 245.8GB 245.7GB 245.7GB - RW 0:5 WDC WD2500JD-00 02.05D02
5 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:4 WDC WD2500JD-75 08.02D08
4 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:3 WDC WD2500JD-75 08.02D08
3 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:2 WDC WD2500JD-75 08.02D08
2 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:1 WDC WD2500JD-75 08.02D08
1 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:0 WDC WD2500JD-75 08.02D08
<ARRAY LIST>
A Type OS Name State Size Hide Id Task %
CA Name
- ——- ———- ——— ——- —- —————— ———— —-
- — ——————————-
1 RAID5 /dev/sda NORMAL 982.9GB NO 0×0000f715ed248225 NOT_ACTIVE ….
. RW <none>
I add new drive to array, but I forgot to specify size, so we didn’t actually enlarge the array. We just spread the existing storage over a larger number of disks.
[root@bfraid ~]# bcadm -T –raid5 –disk 1 2 3 4 5 6 -a 1
bcadm -T –raid5 –disk 1 2 3 4 5 6 -a 1
transform started successfullyThe entire transformation takes quite a few hours.
(started at 12/30/05 12:45pm)
[root@bfraid ~]# bcadm -M -qa
bcadm -M -qa
<VERSIONS>
bcadm: RC-140-2005128.2
driver: RC-140-2005128.2
bios: RC-140-2005079.3
<CONTROLLER LIST>
Serial License
Number Type Number Key
—— ——– ———— ———————–
0 BC4852 400000000422 32211-1111U-Z1HYK-VKJ81
<DISK LIST> Largest
Free Free G Ctrl Model Firmware
Disk State Type Size Space Space S Ca Chan Number Version
—- ——- ——- ——- ——- ——- - — —- ————— ——–
6 Online Disk 245.8GB 49.1GB 49.1GB - RW 0:5 WDC WD2500JD-00 02.05D02
5 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:4 WDC WD2500JD-75 08.02D08
4 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:3 WDC WD2500JD-75 08.02D08
3 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:2 WDC WD2500JD-75 08.02D08
2 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:1 WDC WD2500JD-75 08.02D08
1 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:0 WDC WD2500JD-75 08.02D08
<ARRAY LIST>
A Type OS Name State Size Hide Id Task % CA Name
- ——- ———- ——— ——- —- —————— ———— —– — ——————————-
1 RAID5 /dev/sda NORMAL 982.9GB NO 0×0000f715ed248225 TRANSFORM 00.01 RW <none>
[root@bfraid ~]# sfdisk -l /dev/sda
Disk /dev/sda: 119494 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sda1 0+ 119493 119494- 959835523+ 83 Linux
/dev/sda2 0 - 0 0 0 Empty
/dev/sda3 0 - 0 0 0 Empty
/dev/sda4 0 - 0 0 0 Empty
1:59PM, 10.11% done.
2:55PM, 17.86% done.
3:53PM, 25.75% done.
9:08PM, 67.82% done.
When it’s done, you can see the previous storage is spread across all the drives. There’s still free space.
[root@bfraid ~]# date; bcadm -M -qa
Sun Jan 1 14:33:57 PST 2006
bcadm -M -qa
<VERSIONS>
bcadm: RC-140-2005128.2
driver: RC-140-2005128.2
bios: RC-140-2005079.3
<CONTROLLER LIST>
Serial License
Number Type Number Key
—— ——– ———— ———————–
0 BC4852 400000000422 32211-1111U-Z1HYK-VKJ81
<DISK LIST> Largest
Free Free G Ctrl Model Firmware
Disk State Type Size Space Space S Ca Chan Number Version
—- ——- ——- ——- ——- ——- - — —- ————— ——–
6 Online Disk 245.8GB 49.1GB 49.1GB - RW 0:5 WDC WD2500JD-00 02.05D02
5 Online Disk 245.8GB 49.1GB 49.1GB - RW 0:4 WDC WD2500JD-75 08.02D08
4 Online Disk 245.8GB 49.1GB 49.1GB - RW 0:3 WDC WD2500JD-75 08.02D08
3 Online Disk 245.8GB 49.1GB 49.1GB - RW 0:2 WDC WD2500JD-75 08.02D08
2 Online Disk 245.8GB 49.1GB 49.1GB - RW 0:1 WDC WD2500JD-75 08.02D08
1 Online Disk 245.8GB 49.1GB 49.1GB - RW 0:0 WDC WD2500JD-75 08.02D08
<ARRAY LIST>
A Type OS Name State Size Hide Id Task % CA Name
- ——- ———- ——— ——- —- —————— ———— —– — ——————————-
1 RAID5 /dev/sda NORMAL 982.9GB NO 0×0000f715ed248225 NOT_ACTIVE ….. RW <none>
Now we actually enlarge the array. We could have done this in one shot.
[root@bfraid ~]# bcadm -T –raid5 –disk 1 2 3 4 5 6 –size 1171725 -a 1
bcadm -T –raid5 –disk 1 2 3 4 5 6 –size 1171725 -a 1
transform started successfullyThis shows the entire space is allocated instead of each drive having free space. The entire transformation takes another number of hours to complete.
[root@bfraid ~]# date; bcadm -M -qa
Sun Jan 1 14:37:05 PST 2006
bcadm -M -qa
<VERSIONS>
bcadm: RC-140-2005128.2
driver: RC-140-2005128.2
bios: RC-140-2005079.3
<CONTROLLER LIST>
Serial License
Number Type Number Key
—— ——– ———— ———————–
0 BC4852 400000000422 32211-1111U-Z1HYK-VKJ81
<DISK LIST> Largest
Free Free G Ctrl Model Firmware
Disk State Type Size Space Space S Ca Chan Number Version
—- ——- ——- ——- ——- ——- - — —- ————— ——–
6 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:5 WDC WD2500JD-00 02.05D02
5 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:4 WDC WD2500JD-75 08.02D08
4 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:3 WDC WD2500JD-75 08.02D08
3 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:2 WDC WD2500JD-75 08.02D08
2 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:1 WDC WD2500JD-75 08.02D08
1 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:0 WDC WD2500JD-75 08.02D08
<ARRAY LIST>
A Type OS Name State Size Hide Id Task % CA Name
- ——- ———- ——— ——- —- —————— ———— —– — ——————————-
1 RAID5 /dev/sda NORMAL 982.9GB NO 0×0000f715ed248225 TRANSFORM 03.20 RW <none>Now after adding two more drives and initializing them, we reorganize and expand the array to all 8 drives. I really should make one of them a hot spare, but I think at this point, I can take the risk of 2nd drive failure while waiting 3 or 4 days for a new drive. Well, that isn’t quite true as a subsequent article will detail.
[root@bfraid ~]# bcadm -T –raid5 –disk 1 2 3 4 5 6 7 8 –size 1640415 -a 1
bcadm -T –raid5 –disk 1 2 3 4 5 6 7 8 –size 1640415 -a 1
transform started successfully
[root@bfraid ~]# date; bcadm -M -qa
Sun Jan 1 20:14:35 PST 2006
bcadm -M -qa
<VERSIONS>
bcadm: RC-140-2005128.2
driver: RC-140-2005128.2
bios: RC-140-2005079.3
<CONTROLLER LIST>
Serial License
Number Type Number Key
—— ——– ———— ———————–
0 BC4852 400000000422 32211-1111U-Z1HYK-VKJ81
<DISK LIST> Largest
Free Free G Ctrl Model Firmware
Disk State Type Size Space Space S Ca Chan Number Version
—- ——- ——- ——- ——- ——- - — —- ————— ——–
8 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:7 WDC WD2500JD-00 02.05D02
7 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:6 WDC WD2500JD-00 02.05D02
6 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:5 WDC WD2500JD-00 02.05D02
5 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:4 WDC WD2500JD-75 08.02D08
4 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:3 WDC WD2500JD-75 08.02D08
3 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:2 WDC WD2500JD-75 08.02D08
2 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:1 WDC WD2500JD-75 08.02D08
1 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:0 WDC WD2500JD-75 08.02D08
<ARRAY LIST>
A Type OS Name State Size Hide Id Task % CA Name
- ——- ———- ——— ——- —- —————— ———— —– — ——————————-
1 RAID5 /dev/sda NORMAL 1.2TB NO 0×0000f715ed248225 TRANSFORM 00.00 RW <none>
0% at Sun Jan 1 20:14:35 PST 2006
2.86% Sun Jan 1 20:48:16 PST 2006 (35m,41s)
est done 20h, 48m, 5:02PM Monday.
Actually completed at 3:00pm.Yup, the transform took almost 21 hours. That’s just for a small 1.7TB array. The professionals probably use much more sophisticated techniques.
[root@bfraid ~]# date; bcadm -M -qa
Mon Jan 2 15:28:23 PST 2006
bcadm -M -qa
<VERSIONS>
bcadm: RC-140-2005128.2
driver: RC-140-2005128.2
bios: RC-140-2005079.3
<CONTROLLER LIST>
Serial License
Number Type Number Key
—— ——– ———— ———————–
0 BC4852 400000000422 32211-1111U-Z1HYK-VKJ81
<DISK LIST> Largest
Free Free G Ctrl Model Firmware
Disk State Type Size Space Space S Ca Chan Number Version
—- ——- ——- ——- ——- ——- - — —- ————— ——–
8 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:7 WDC WD2500JD-00 02.05D02
7 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:6 WDC WD2500JD-00 02.05D02
6 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:5 WDC WD2500JD-00 02.05D02
5 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:4 WDC WD2500JD-75 08.02D08
4 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:3 WDC WD2500JD-75 08.02D08
3 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:2 WDC WD2500JD-75 08.02D08
2 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:1 WDC WD2500JD-75 08.02D08
1 Online Disk 245.8GB 0.0MB 0.0MB - RW 0:0 WDC WD2500JD-75 08.02D08
<ARRAY LIST>
A Type OS Name State Size Hide Id Task % CA Name
- ——- ———- ——— ——- —- —————— ———— —– — ——————————-
1 RAID5 /dev/sda NORMAL 1.7TB NO 0×0000f715ed248225 NOT_ACTIVE ….. RW <none>We have to reboot at this point because Linux thinks the in-core label still has the old geometry.
[root@bfraid ~]# rebootThis shows our partition only consumes 119493 of the now available 209123 cylinders. We need to create a new partition representing the newly available space.
[root@bfraid ~]# sfdisk -l /dev/sda
Disk /dev/sda: 209123 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sda1 0+ 119493 119494- 959835523+ 83 Linux
/dev/sda2 0 - 0 0 0 Empty
/dev/sda3 0 - 0 0 0 Empty
/dev/sda4 0 - 0 0 0 EmptyWe create a second partition – this implies we can resize at most 2 more times before we have to resort to another technique.
[root@bfraid ~]# fdisk /dev/sda
The number of cylinders for this disk is set to 209123.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): p
Disk /dev/sda: 1720.0 GB, 1720099799040 bytes
255 heads, 63 sectors/track, 209123 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 1 119494 959835523+ 83 Linux
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (119495-209123, default 119495):
Using default value 119495
Last cylinder or +size or +sizeM or +sizeK (119495-209123, default 209123):
Using default value 209123
Command (m for help): p
Disk /dev/sda: 1720.0 GB, 1720099799040 bytes
255 heads, 63 sectors/track, 209123 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 1 119494 959835523+ 83 Linux
/dev/sda2 119495 209123 719944942+ 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.Since kernel can’t update in-memory label, we have to reboot again.
[root@bfraid ~]# rebootNow we initailize the new partition and add it to the volume group.
[root@bfraid ~]# pvcreate /dev/sda2
Physical volume "/dev/sda2" successfully created
[root@bfraid ~]# vgextend vga /dev/sda2
Volume group "vga" successfully extendedThis shows free space now available on /dev/sda2.
[root@bfraid ~]# vgdisplay -v
Finding all volume groups
Finding volume group "vga"
— Volume group —
VG Name vga
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 2
Act PV 2
VG Size 1.56 TB
PE Size 4.00 MB
Total PE 410100
Alloc PE / Size 234333 / 915.36 GB
Free PE / Size 175767 / 686.59 GB
VG UUID EpZ6hz-bkgB-uCB2-C8M9-SSBj-HAx6-1AWsDY
— Logical volume —
LV Name /dev/vga/va
VG Name vga
LV UUID H28oHS-0edX-buds-I6so-rEDk-HE2I-KE1xkR
LV Write Access read/write
LV Status available
# open 1
LV Size 915.36 GB
Current LE 234333
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:0
— Physical volumes —
PV Name /dev/sda1
PV UUID xdDBzU-B2bt-E12f-tSDS-54Wx-bUaa-JmoUSF
PV Status allocatable
Total PE / Free PE 234333 / 0
PV Name /dev/sda2
PV UUID 6z6Q82-oVCW-tzCo-F3k7-rMES-Nfz7-SBHsgN
PV Status allocatable
Total PE / Free PE 175767 / 175767Now we just extend the logical volume to consume newly available space.
[root@bfraid ~]# lvextend -l+175767 /dev/vga/va
Extending logical volume va to 1.56 TB
Logical volume va successfully resizedFree space not available anymore.
[root@bfraid ~]# vgdisplay -v
Finding all volume groups
Finding volume group "vga"
— Volume group —
VG Name vga
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 4
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 2
Act PV 2
VG Size 1.56 TB
PE Size 4.00 MB
Total PE 410100
Alloc PE / Size 410100 / 1.56 TB
Free PE / Size 0 / 0
VG UUID EpZ6hz-bkgB-uCB2-C8M9-SSBj-HAx6-1AWsDY
— Logical volume —
LV Name /dev/vga/va
VG Name vga
LV UUID H28oHS-0edX-buds-I6so-rEDk-HE2I-KE1xkR
LV Write Access read/write
LV Status available
# open 1
LV Size 1.56 TB
Current LE 410100
Segments 2
Allocation inherit
Read ahead sectors 0
Block device 253:0
— Physical volumes —
PV Name /dev/sda1
PV UUID xdDBzU-B2bt-E12f-tSDS-54Wx-bUaa-JmoUSF
PV Status allocatable
Total PE / Free PE 234333 / 0
PV Name /dev/sda2
PV UUID 6z6Q82-oVCW-tzCo-F3k7-rMES-Nfz7-SBHsgN
PV Status allocatable
Total PE / Free PE 175767 / 0We cannot run resize immediately. We are told to fsck first, which we do.
[root@bfraid ~]# resize2fs /dev/mapper/vga-va
resize2fs 1.35 (28-Feb-2004)
Please run ‘e2fsck -f /dev/mapper/vga-va’ first.
#### started around 18:10 PST jan 2 2006
[root@bfraid ~]# e2fsck -f /dev/mapper/vga-va
e2fsck 1.35 (28-Feb-2004)
Pass 1: Checking inodes, blocks, and sizes
#### done at Mon Jan 2 18:43:35 PST 2006Now we resize the filesystem while it is unmounted and enjoy the newly available space.
[root@bfraid ~]# resize2fs /dev/mapper/vga-va
resize2fs 1.35 (28-Feb-2004)
Resizing the filesystem on /dev/mapper/vga-va to 419942400 (4k) blocks.
The filesystem on /dev/mapper/vga-va is now 419942400 blocks long.
[root@bfraid ~]# mount /d/40
[root@bfraid ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sdb3 75G 3.4G 68G 5% /
/dev/sdb1 99M 12M 82M 13% /boot
none 1014M 0 1014M 0% /dev/shm
heatsink:/d/1 84G 53G 28G 66% /d/1
/dev/mapper/vga-va 1.6T 693G 820G 46% /d/40So far, so good. Stay tuned for the next installment where we are forced to test the replacement and rebuild functionality due to an unexpected drive failure of one of the “refurbished” drives after less than 6 months service.
Experience with online capacity expansion, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License.
