Personal Terabyte Store

Disks are cheap enough now that we can have an affordable(*) personal terabyte sized store.

Other parts arrive separately, but these 250GB drives from western digital were a good deal at $96 each. However, I am now hearing that there’s a long standing WD firmware problem causing random writes to time out, and these cheap drives are “recertified”, i.e. returns, so I’m hoping these don’t have the bad firmware…

(*) as affordable as a 40MB hard disk was a few years back…

$500 or 1TB worth of disk

I’m configuring a rather high-end (for home use) NAS box with hardware RAID-5 and gigabit LAN to safely backup my baby photos – it’s a little overkill but given the choice between:

  1. burning N CDs that are almost sure to go bad in a few years unless they are re-read and re-burned periodically – $50 + continuing effort + risk that CD will die before you reburn it.
  2. printing photos to archival paper using archival ink and storing in a dark, cool place – $100.
  3. throwing together a hardware RAID with a periodic online reverify to catch early disk failure – $1500 + power.

I chose #3. I value my time more than to do #1, and I’ll probably do #2 anyway as a second backup.

It uses the following:

  • $120 - Antec P160 tower - room for 7 3.5 inch drives and more Silent PC review
  • $170 - Phantom 500W power supply Tom’s hardware found this to work well.
  • $111 - Gigabyte GA8i915P duo pro motherboard Hardware zone survey of i915P motherboards My criteria was onboard gigabit network using native PCI express, and PCI-X for the RAID card.
  • $270 - Pentium 4 650 3.6GHz CPU – cheapest $ per DIVX at this point.
  • 4GB of PC3200 DDR SDRAM that was lying around
  • $500 - 5 Western Digital 7200 RPM 250GB SATA drives – cheapest $ per GB at this point.
  • $52 - 1 Hitachi 80GB SATA drive for system drive
  • $358 - Broadcom Raidcore BC4852 8-port SATA RAID controller
  • Linux 2.6 with nothing special – maybe ext3 and LVM.
  • samba, NFS, etc.

$1581 total plus heatsink, fan, video card, shipping & handling, sales tax.

Supposedly, the broadcom supports OCE (online capacity expansion) so when the 500GB drives drop in cost, i can swap out the 250GB drives w/o losing data on the array and then expand the array device and LVM partition. By that time, drives may have started to fail, so they will need to be replaced anyway.

The choice of SATA over PATA was easy – cabling is cleaner and there is little price difference between the drives.

The choice of hardware raid over software was another easy one – this eval shows the performance of the various cards, the conclusion and I wanted something that had been on the market for a while. I wasn’t aware of, and currently don’t want to test the Linux implementation of OCE right now…

I’m thinking the hardware raid will have been tested more thoroughly than the linux software solution. After all hardware is used widely from linux to windows to freebsd, while linux is used by… linux enthusiasts. $350 worth peace of mind to me? Yes.

Can I always go back to just-a-bunch-of-disks with the CLI nightmare that is dm/md? Yes.

Why pentium 4 and not athlon64? In my experience, the athlon is less forgiving of OEM DRAM. Other P4 systems have run memtest86 fine with OEM DRAM while the athlon systems have failed with memory errors. This is like 4 or 5 machines that have been built personally or by vendors in the last 18 months with memory test failures, and all were athlons.

Update 2005-Sep-09:

lvm2 is very simply configured:

  • one physical volume group ‘vga’ which is /dev/sda1, the entire RAID array
  • one logical volume ‘va’ which is sized to be entire physical volume ‘vga’
  • ext3 filesystem with dir_index option on the ‘vga’ logical volume
  • no concatenations or stripes or mirrors, etc.

We didn’t quite get 1TB, but close to 1TiB (trillion, not tera) due to fact that 250GB drives are really 250GiB drives.

Looking into how lvm2 handles the physical device expansion (due toexpanding the RAID with OCE), there is a way to expand the physical volume group, but it’s not as clean as I would have hoped.

The supported way to expand the PV group is actually to create a new partition on the expanded device, and then add that partition to the PV group. Then you can expand the logical volume with lvresize, and finally expand the filesystem with resize2fs.

After bringing system up, we see reasonable performance, as would be expected for a non PCIX attached device. 150MB/sec Bursts of write activity every 5 seconds due to buffer flush activity. 300MBit/sec writes via NFS.

I am now consolidating storage so I can sell off my PATA drives (and buy more SATA for the RAID(!!))

Possibly to be added:

  • temperature monitor for fan control - maybe $100
  • 3×5.25 to 5×3.5 hot swap internal enclosure - maybe $120.

Other folks doing the same thing:

Creative Commons License
Personal Terabyte Store, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License.