Experience with online capacity expansion


How to expand filesystem using Broadcom RAID online capacity expansion (OCE)

I added one drive, then 2 more drives after getting more cabling.
This shows one having been added and attached, but not fully
initialized so it can be added to the array.

[root@bfraid ~]# bcadm -M -qa


bcadm -M -qa



bcadm:  RC-140-2005128.2
driver: RC-140-2005128.2
bios:   RC-140-2005079.3


                   Serial           License
Number   Type      Number             Key
------ -------- ------------ -----------------------
  0    BC4852   400000000422 32211-1111U-Z1HYK-VKJ81

                          Largest
                              Free   Free    G    Ctrl     Model       Firmware
Disk  State   Type    Size    Space  Space   S Ca Chan     Number      Version
---- ------- ------- ------- ------- ------- - -- ---- --------------- --------
6    Online  New     245.8GB 245.7GB 245.7GB - NC  0:5 WDC WD2500JD-00 02.05D02
5    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:4 WDC WD2500JD-75 08.02D08
4    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:3 WDC WD2500JD-75 08.02D08
3    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:2 WDC WD2500JD-75 08.02D08
2    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:1 WDC WD2500JD-75 08.02D08
1    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:0 WDC WD2500JD-75 08.02D08



A  Type    OS Name     State    Size   Hide        Id               Task      % 
  CA              Name
- ------- ---------- --------- ------- ---- ------------------ ------------ ----
- -- -------------------------------
1   RAID5   /dev/sda NORMAL    982.9GB NO   0x0000f715ed248225 NOT_ACTIVE   ....
. RW             

To add, you need to initialize the drive

[root@bfraid ~]# bcadm  -M -id -d 6


bcadm -M -id -d 6

initializing disk 6

This shows max size of new array if you were to add drive 6.

[root@bfraid ~]# bcadm -T --raid5 --disk 1 2 3 4 5 6 --max-size -a 1


bcadm -T --raid5 --disk 1 2 3 4 5 6 --max-size -a 1

Max Possible Size: 1171725

After being initialized, the drive status shows as “Disk” instead of “New”

[root@bfraid ~]# bcadm -M -qa


bcadm -M -qa



bcadm:  RC-140-2005128.2
driver: RC-140-2005128.2
bios:   RC-140-2005079.3


                   Serial           License
Number   Type      Number             Key
------ -------- ------------ -----------------------
  0    BC4852   400000000422 32211-1111U-Z1HYK-VKJ81

                          Largest
                              Free   Free    G    Ctrl     Model       Firmware
Disk  State   Type    Size    Space  Space   S Ca Chan     Number      Version
---- ------- ------- ------- ------- ------- - -- ---- --------------- --------
6    Online  Disk    245.8GB 245.7GB 245.7GB - RW  0:5 WDC WD2500JD-00 02.05D02
5    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:4 WDC WD2500JD-75 08.02D08
4    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:3 WDC WD2500JD-75 08.02D08
3    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:2 WDC WD2500JD-75 08.02D08
2    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:1 WDC WD2500JD-75 08.02D08
1    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:0 WDC WD2500JD-75 08.02D08



A  Type    OS Name     State    Size   Hide        Id               Task      % 
  CA              Name
- ------- ---------- --------- ------- ---- ------------------ ------------ ----
- -- -------------------------------
1   RAID5   /dev/sda NORMAL    982.9GB NO   0x0000f715ed248225 NOT_ACTIVE   ....
. RW             

I add new drive to array, but I forgot to specify size, so we didn’t
actually enlarge the array. We just spread the existing storage over a larger number of disks.

[root@bfraid ~]# bcadm -T --raid5 --disk 1 2 3 4 5 6  -a 1


bcadm -T --raid5 --disk 1 2 3 4 5 6 -a 1

transform started successfully

The entire transformation takes quite a few hours.

(started at 12/30/05 12:45pm)
[root@bfraid ~]# bcadm -M -qa


bcadm -M -qa



bcadm:  RC-140-2005128.2
driver: RC-140-2005128.2
bios:   RC-140-2005079.3


                   Serial           License
Number   Type      Number             Key
------ -------- ------------ -----------------------
  0    BC4852   400000000422 32211-1111U-Z1HYK-VKJ81

                          Largest
                              Free   Free    G    Ctrl     Model       Firmware
Disk  State   Type    Size    Space  Space   S Ca Chan     Number      Version
---- ------- ------- ------- ------- ------- - -- ---- --------------- --------
6    Online  Disk    245.8GB  49.1GB  49.1GB - RW  0:5 WDC WD2500JD-00 02.05D02
5    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:4 WDC WD2500JD-75 08.02D08
4    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:3 WDC WD2500JD-75 08.02D08
3    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:2 WDC WD2500JD-75 08.02D08
2    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:1 WDC WD2500JD-75 08.02D08
1    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:0 WDC WD2500JD-75 08.02D08



A  Type    OS Name     State    Size   Hide        Id               Task      %   CA              Name
- ------- ---------- --------- ------- ---- ------------------ ------------ ----- -- -------------------------------
1   RAID5   /dev/sda NORMAL    982.9GB NO   0x0000f715ed248225 TRANSFORM    00.01 RW             
[root@bfraid ~]# sfdisk -l /dev/sda

Disk /dev/sda: 119494 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sda1          0+ 119493  119494- 959835523+  83  Linux
/dev/sda2          0       -       0          0    0  Empty
/dev/sda3          0       -       0          0    0  Empty
/dev/sda4          0       -       0          0    0  Empty

1:59PM, 10.11% done.
2:55PM, 17.86% done.
3:53PM, 25.75% done.
9:08PM, 67.82% done.

When it’s done, you can see the previous storage is spread across all the drives. There’s still free space.

[root@bfraid ~]# date; bcadm -M -qa
Sun Jan  1 14:33:57 PST 2006


bcadm -M -qa



bcadm:  RC-140-2005128.2
driver: RC-140-2005128.2
bios:   RC-140-2005079.3


                   Serial           License
Number   Type      Number             Key
------ -------- ------------ -----------------------
  0    BC4852   400000000422 32211-1111U-Z1HYK-VKJ81

                          Largest
                              Free   Free    G    Ctrl     Model       Firmware
Disk  State   Type    Size    Space  Space   S Ca Chan     Number      Version
---- ------- ------- ------- ------- ------- - -- ---- --------------- --------
6    Online  Disk    245.8GB  49.1GB  49.1GB - RW  0:5 WDC WD2500JD-00 02.05D02
5    Online  Disk    245.8GB  49.1GB  49.1GB - RW  0:4 WDC WD2500JD-75 08.02D08
4    Online  Disk    245.8GB  49.1GB  49.1GB - RW  0:3 WDC WD2500JD-75 08.02D08
3    Online  Disk    245.8GB  49.1GB  49.1GB - RW  0:2 WDC WD2500JD-75 08.02D08
2    Online  Disk    245.8GB  49.1GB  49.1GB - RW  0:1 WDC WD2500JD-75 08.02D08
1    Online  Disk    245.8GB  49.1GB  49.1GB - RW  0:0 WDC WD2500JD-75 08.02D08



A  Type    OS Name     State    Size   Hide        Id               Task      %   CA              Name
- ------- ---------- --------- ------- ---- ------------------ ------------ ----- -- -------------------------------
1   RAID5   /dev/sda NORMAL    982.9GB NO   0x0000f715ed248225 NOT_ACTIVE   ..... RW             

Now we actually enlarge the array. We could have done this in one shot.

[root@bfraid ~]# bcadm -T --raid5 --disk 1 2 3 4 5 6 --size 1171725 -a 1


bcadm -T --raid5 --disk 1 2 3 4 5 6 --size 1171725 -a 1

transform started successfully

This shows the entire space is allocated instead of each drive having
free space. The entire transformation takes another number of hours to complete.

[root@bfraid ~]# date; bcadm -M -qa
Sun Jan  1 14:37:05 PST 2006


bcadm -M -qa



bcadm:  RC-140-2005128.2
driver: RC-140-2005128.2
bios:   RC-140-2005079.3


                   Serial           License
Number   Type      Number             Key
------ -------- ------------ -----------------------
  0    BC4852   400000000422 32211-1111U-Z1HYK-VKJ81

                          Largest
                              Free   Free    G    Ctrl     Model       Firmware
Disk  State   Type    Size    Space  Space   S Ca Chan     Number      Version
---- ------- ------- ------- ------- ------- - -- ---- --------------- --------
6    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:5 WDC WD2500JD-00 02.05D02
5    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:4 WDC WD2500JD-75 08.02D08
4    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:3 WDC WD2500JD-75 08.02D08
3    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:2 WDC WD2500JD-75 08.02D08
2    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:1 WDC WD2500JD-75 08.02D08
1    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:0 WDC WD2500JD-75 08.02D08



A  Type    OS Name     State    Size   Hide        Id               Task      %   CA              Name
- ------- ---------- --------- ------- ---- ------------------ ------------ ----- -- -------------------------------
1   RAID5   /dev/sda NORMAL    982.9GB NO   0x0000f715ed248225 TRANSFORM    03.20 RW             

Now after adding two more drives and initializing them, we reorganize
and expand the array to all 8 drives. I really should make one of
them a hot spare, but I think at this point, I can take the risk of
2nd drive failure while waiting 3 or 4 days for a new drive. Well, that isn’t quite true as a subsequent article will detail.

[root@bfraid ~]# bcadm -T --raid5 --disk 1 2 3 4 5 6 7 8 --size 1640415 -a 1


bcadm -T --raid5 --disk 1 2 3 4 5 6 7 8 --size 1640415 -a 1

transform started successfully
[root@bfraid ~]# date; bcadm -M -qa
Sun Jan  1 20:14:35 PST 2006


bcadm -M -qa



bcadm:  RC-140-2005128.2
driver: RC-140-2005128.2
bios:   RC-140-2005079.3


                   Serial           License
Number   Type      Number             Key
------ -------- ------------ -----------------------
  0    BC4852   400000000422 32211-1111U-Z1HYK-VKJ81

                          Largest
                              Free   Free    G    Ctrl     Model       Firmware
Disk  State   Type    Size    Space  Space   S Ca Chan     Number      Version
---- ------- ------- ------- ------- ------- - -- ---- --------------- --------
8    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:7 WDC WD2500JD-00 02.05D02
7    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:6 WDC WD2500JD-00 02.05D02
6    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:5 WDC WD2500JD-00 02.05D02
5    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:4 WDC WD2500JD-75 08.02D08
4    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:3 WDC WD2500JD-75 08.02D08
3    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:2 WDC WD2500JD-75 08.02D08
2    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:1 WDC WD2500JD-75 08.02D08
1    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:0 WDC WD2500JD-75 08.02D08



A  Type    OS Name     State    Size   Hide        Id               Task      %   CA              Name
- ------- ---------- --------- ------- ---- ------------------ ------------ ----- -- -------------------------------
1   RAID5   /dev/sda NORMAL      1.2TB NO   0x0000f715ed248225 TRANSFORM    00.00 RW             
0% at Sun Jan  1 20:14:35 PST 2006
2.86%  Sun Jan  1 20:48:16 PST 2006  (35m,41s)

est done 20h, 48m, 5:02PM Monday.

Actually completed at 3:00pm.

Yup, the transform took almost 21 hours. That’s just for a small 1.7TB array. The professionals probably use much more sophisticated techniques.

[root@bfraid ~]# date; bcadm -M -qa
Mon Jan  2 15:28:23 PST 2006


bcadm -M -qa



bcadm:  RC-140-2005128.2
driver: RC-140-2005128.2
bios:   RC-140-2005079.3


                   Serial           License
Number   Type      Number             Key
------ -------- ------------ -----------------------
  0    BC4852   400000000422 32211-1111U-Z1HYK-VKJ81

                          Largest
                              Free   Free    G    Ctrl     Model       Firmware
Disk  State   Type    Size    Space  Space   S Ca Chan     Number      Version
---- ------- ------- ------- ------- ------- - -- ---- --------------- --------
8    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:7 WDC WD2500JD-00 02.05D02
7    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:6 WDC WD2500JD-00 02.05D02
6    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:5 WDC WD2500JD-00 02.05D02
5    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:4 WDC WD2500JD-75 08.02D08
4    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:3 WDC WD2500JD-75 08.02D08
3    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:2 WDC WD2500JD-75 08.02D08
2    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:1 WDC WD2500JD-75 08.02D08
1    Online  Disk    245.8GB   0.0MB   0.0MB - RW  0:0 WDC WD2500JD-75 08.02D08



A  Type    OS Name     State    Size   Hide        Id               Task      %   CA              Name
- ------- ---------- --------- ------- ---- ------------------ ------------ ----- -- -------------------------------
1   RAID5   /dev/sda NORMAL      1.7TB NO   0x0000f715ed248225 NOT_ACTIVE   ..... RW             

We have to reboot at this point because Linux thinks the in-core label still has the old geometry.

[root@bfraid ~]# reboot

This shows our partition only consumes 119493 of the now available
209123 cylinders. We need to create a new partition representing the
newly available space.

[root@bfraid ~]# sfdisk -l /dev/sda

Disk /dev/sda: 209123 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sda1          0+ 119493  119494- 959835523+  83  Linux
/dev/sda2          0       -       0          0    0  Empty
/dev/sda3          0       -       0          0    0  Empty
/dev/sda4          0       -       0          0    0  Empty

We create a second partition — this implies we can resize at most 2
more times before we have to resort to another technique.

[root@bfraid ~]# fdisk /dev/sda

The number of cylinders for this disk is set to 209123.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sda: 1720.0 GB, 1720099799040 bytes
255 heads, 63 sectors/track, 209123 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1      119494   959835523+  83  Linux

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (119495-209123, default 119495): 
Using default value 119495
Last cylinder or +size or +sizeM or +sizeK (119495-209123, default 209123): 
Using default value 209123

Command (m for help): p

Disk /dev/sda: 1720.0 GB, 1720099799040 bytes
255 heads, 63 sectors/track, 209123 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1      119494   959835523+  83  Linux
/dev/sda2          119495      209123   719944942+  83  Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

Since kernel can’t update in-memory label, we have to reboot again.

[root@bfraid ~]# reboot

Now we initailize the new partition and add it to the volume group.

[root@bfraid ~]# pvcreate /dev/sda2
  Physical volume "/dev/sda2" successfully created
[root@bfraid ~]# vgextend vga /dev/sda2
  Volume group "vga" successfully extended

This shows free space now available on /dev/sda2.

[root@bfraid ~]# vgdisplay -v
    Finding all volume groups
    Finding volume group "vga"
  --- Volume group ---
  VG Name               vga
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               1.56 TB
  PE Size               4.00 MB
  Total PE              410100
  Alloc PE / Size       234333 / 915.36 GB
  Free  PE / Size       175767 / 686.59 GB
  VG UUID               EpZ6hz-bkgB-uCB2-C8M9-SSBj-HAx6-1AWsDY

  --- Logical volume ---
  LV Name                /dev/vga/va
  VG Name                vga
  LV UUID                H28oHS-0edX-buds-I6so-rEDk-HE2I-KE1xkR
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                915.36 GB
  Current LE             234333
  Segments               1
  Allocation             inherit
  Read ahead sectors     0
  Block device           253:0

  --- Physical volumes ---
  PV Name               /dev/sda1     
  PV UUID               xdDBzU-B2bt-E12f-tSDS-54Wx-bUaa-JmoUSF
  PV Status             allocatable
  Total PE / Free PE    234333 / 0

  PV Name               /dev/sda2     
  PV UUID               6z6Q82-oVCW-tzCo-F3k7-rMES-Nfz7-SBHsgN
  PV Status             allocatable
  Total PE / Free PE    175767 / 175767

Now we just extend the logical volume to consume newly available space.

[root@bfraid ~]# lvextend -l+175767 /dev/vga/va
  Extending logical volume va to 1.56 TB
  Logical volume va successfully resized

Free space not available anymore.

[root@bfraid ~]# vgdisplay -v
    Finding all volume groups
    Finding volume group "vga"
  --- Volume group ---
  VG Name               vga
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               1.56 TB
  PE Size               4.00 MB
  Total PE              410100
  Alloc PE / Size       410100 / 1.56 TB
  Free  PE / Size       0 / 0   
  VG UUID               EpZ6hz-bkgB-uCB2-C8M9-SSBj-HAx6-1AWsDY

  --- Logical volume ---
  LV Name                /dev/vga/va
  VG Name                vga
  LV UUID                H28oHS-0edX-buds-I6so-rEDk-HE2I-KE1xkR
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.56 TB
  Current LE             410100
  Segments               2
  Allocation             inherit
  Read ahead sectors     0
  Block device           253:0

  --- Physical volumes ---
  PV Name               /dev/sda1     
  PV UUID               xdDBzU-B2bt-E12f-tSDS-54Wx-bUaa-JmoUSF
  PV Status             allocatable
  Total PE / Free PE    234333 / 0

  PV Name               /dev/sda2     
  PV UUID               6z6Q82-oVCW-tzCo-F3k7-rMES-Nfz7-SBHsgN
  PV Status             allocatable
  Total PE / Free PE    175767 / 0

We cannot run resize immediately. We are told to fsck first, which we do.

[root@bfraid ~]# resize2fs /dev/mapper/vga-va 
resize2fs 1.35 (28-Feb-2004)
Please run 'e2fsck -f /dev/mapper/vga-va' first.
#### started around 18:10 PST jan 2 2006
[root@bfraid ~]# e2fsck -f /dev/mapper/vga-va 
e2fsck 1.35 (28-Feb-2004)
Pass 1: Checking inodes, blocks, and sizes
#### done at Mon Jan  2 18:43:35 PST 2006

Now we resize the filesystem while it is unmounted and enjoy the newly
available space.

[root@bfraid ~]# resize2fs /dev/mapper/vga-va 
resize2fs 1.35 (28-Feb-2004)
Resizing the filesystem on /dev/mapper/vga-va to 419942400 (4k) blocks.
The filesystem on /dev/mapper/vga-va is now 419942400 blocks long.
[root@bfraid ~]# mount /d/40
[root@bfraid ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb3              75G  3.4G   68G   5% /
/dev/sdb1              99M   12M   82M  13% /boot
none                 1014M     0 1014M   0% /dev/shm
heatsink:/d/1          84G   53G   28G  66% /d/1
/dev/mapper/vga-va    1.6T  693G  820G  46% /d/40

So far, so good. Stay tuned for the next installment where we are forced to test the replacement and rebuild functionality due to an unexpected drive failure of one of the “refurbished” drives after less than 6 months service.

TwitterFacebookGoogle GmailGoogle+Share/Save