[Novalug] best hard disk setup for home file server?

[Brandon Saxe]

First let me say I'm using consumer level hard drives and a SATA sil3124 controller card. This is hardly enterprise stuff. This is my stuff at home and I'm cheap.....

My personal experience with md RAID5 was terrible. This was only a 3 drive setup, but the write performance was so awful it was virtually unusable and aligning filesystems on stripe boundaries was a pain. Add the XFS filesystem and LVM on top of that and it was a major flop.

First, I got rid of LVM and put XFS right onto the md partitions. Then after that I threw out the md partitions together and used the drives as raw md devices. This seems to help quite a bit and no more alignment issues. Even with all this I decided to try RAID 10 with 3 drives (Linux software RAID supports this even though true RAID 10 needs 4 minimum). With the software md RAID 10 my setup was noticably better and definitely usable (I've got vmware server on Ubuntu 8.04 running virtual machines from this array).

I recommend going RAID 10 if you're using software md. I'm using it with my 3 drives  and it slaughters the performance of the RAID 5.

See http://www.baarf.com/ for reasons why RAID 5 is bad. After doing a lot of searching on the web I now believe that nested RAID 10 setups are much better than RAID5 or RAID6.

With Linux Software RAID 10, you have flexible options for odd combinations of drives. RAID5 rebuilds and alignment is a pain in the butt. Also, forget about creating partitions (/dev/sda1, /dev/sda2, etc.) for md partitions. Just use the whole disks for md (/dev/sda). There are articles on the web discussing this, I just don't have access to my references right now. 

Don't use LVM with software raid, especially RAID5 or RAID6 as it knows nothing about alignment and doesn't honor barriers either (I think there's experimental support in latest kernels, but nothing stable that I know of). If you really must use LVM, make sure you align your RAID5/6 LVM extents and the filesystem that goes on the logical volumes as well. You can google how to align LVM and filesystems to a raid array. Each filesystem has different mkfs options for this and some are downright confusing.

If you really like LVM, then use it with RAID10 volumes where alignment is not as big a deal as with RAID5/6.

Good luck.



--- On Wed, 10/14/09, Richard Ertel <richard.ertel at gmail.com> wrote:

> From: Richard Ertel <richard.ertel at gmail.com>
> Subject: Re: [Novalug] best hard disk setup for home file server?
> To: "Bryan J. Smith" <b.j.smith at ieee.org>
> Cc: novalug-bounces at calypso2.tux.org, "Novalug" <novalug at calypso2.tux.org>
> Date: Wednesday, October 14, 2009, 5:16 PM
> didn't you just say that you advocate
> "on the cheap" software RAID-6
> for non-system drives?
> 
> in my server, i care about my storage data infinitely more
> than my
> system drive. if my system drive dies, i can just put a new
> one in,
> reinstall the os, and re-create user accounts, scripts
> (from backup),
> settings, etc. my server being up isn't critical. i just
> like for it
> to be up :)
> 
> my struggle right now is trying to figure out how to have a
> good sized
> MD-RAID-5 array without the probability of it rebuilding
> all the time
> due to this error recovery business.
> 
> On Wed, Oct 14, 2009 at 17:07, Bryan J. Smith <b.j.smith at ieee.org>
> wrote:
> > In general, I avoid MD, especially for anything
> booting.  But that's just me.
> >
> > To make matters worse, most of the time most MD issues
> are caused by
> > people not understanding MD.
> >
> > E.g., people used to blame 3Ware for lack of hot-swap
> because they
> > were using 3Ware cards in JBOD mode with MD.  3Ware
> only supports
> > hot-swap when it's managing the RAID array (for
> various reasons, including
> > the fact that the kernel did not support a "missing
> device" for a long time
> > which happens when you use JBOD ;).
> >
> > The common argument that "I don't want to rely on a
> hardware vendor
> > for my data" didn't hold much water with 3Ware, given
> how open they
> > were with their RAID volume organization so it could
> be read without
> > their cards (e.g., DeviceMapper).  Again, people not
> knowing what they
> > are doing.
> >
> > Although even 3Ware "inits" volumes after a hang/power
> failure.  By
> > "init" they mean check the entire volume for
> consistency, not necessarily
> > rebuild it (it's still redundant).  It's just good
> practice.
> >
> >
> > ----- Original Message ----
> > From: Richard Ertel <richard.ertel at gmail.com>
> >
> > the issue i have read about is that without TLER, a
> drive takes it's
> > sweet time recovering from an error which causes MD to
> drop it from
> > the array for being non-responsive. then when it
> responds again, MD
> > picks it back up, but then starts the rebuilding
> process.
> >
> > this sounds like an issue i want to avoid.
> >
> _______________________________________________
> Novalug mailing list
> Novalug at calypso.tux.org
> http://calypso.tux.org/mailman/listinfo/novalug
>