[Novalug] LVM vs Partitions

[Bryan J. Smith]

First off, I never use LVM atop of MD.  Too many years of race conditions.
I general, I don't trust MD, although the newer MD meta and mdadm
tools are much better than years ago.  Some still chance it, but I honestly
try to avoid putting LVM atop of MD, and use true (I/O off-load) hardware
RAID.  At most I'll use Device Mapper RAID (dmraid) for RAID-1 mirroring
only.

Two, LVM does _not_ slow down your disk access at all.  LVM works with
DeviceMapper to define the sector ranges.  From there, the kernel is
accessing the devices _raw_.  LVM is not part of the mix at all once the
devices are mapped (/dev/mapper/* are actually where the device are).
Same deal for LVM striping, it's just more DeviceMapper assignments and
ranging.  The kernel takes it from there.  In the case of dmraid, it incurs the
same performance hits as RAID-1 mirroring (two copies) and XORs in
RAID-3/4/5/6 (XOR calculation through the LOAD-XOR-STOR cycle of
the system interconnect) as any other software RAID.  Which brings me to ...

Third, MD _does_ very much slow down your disk access.  In fact, I'm
totally chuckling when you say this as you use MD -- with RAID-5 no less!
Yes, modern CPUs are fast at XOR operations, but they are _not_ I/O
processors (the whole LOAD-XOR-STOR cycle through the system
interconnect), or at least the common PC design is not.  That's how a sub-1GHz
I/O microcontroller with in-line XOR ASICs can beat a 3GHz PC system at
RAID-5.  It's fine to do software RAID-5 if your writes are infrequently (like on
a web server), but if your services are I/O bound, it's much more efficient to
do the XORs in-line with a dedicated, in-line XOR ASIC sporting microcontroller.

In other words, when you do Software RAID that requires XORs, you turn
your I/O into Programmed I/O (PIO) instead of block writes without CPU
with Direct Memory Access (DMA).  That's why software RAID-5 writes
typically doesn't break 20MiBps on a modern PC (when you remove
any buffering).  The only time I've seen it faster is when people are comparing
performance to a '90s i960-based or early StrongARM I/O processor.
But any modern X-Scale of 500MHz-1GHz with storage ASICs will best it.



----- Original Message ----
From: Gopher <gopher at 3wa.org>

I've got an mdadm raid 5 setup with LVM on top of it.  So far it meets 
my needs if I want to reallocate space between LVM partitions.

Other than this, I'm not sure why I would ever use LVM. As I understand it, 
LVM is an extra layer of code that the kernel has to slog through to get 
to the disk and to be able to retrieve data.