[Novalug] benchmarks and iSCSI

[Miguel Gonzalez Castaños]

John Franklin escribió:
> On Wed, Feb 06, 2008 at 02:31:56PM -0500, Miguel Gonzalez Castaños wrote:
>   
>> John Franklin escribió:
>>     
>>> 1900MB/s?  How are you connecting the iSCSI drive?  100Base-T tops out 
>>> at 12MB/s, GigE could do about 100-120MB/s (reality: less). 10GigE 
>>> will get you (best, utopian case) 1000-1200MB/s.  How are you getting 
>>> 1900MB/s?  It sounds like you've got some significant caching on the 
>>> client side polluting your results.
>>>
>>> 34MB/s reads sounds like what you should be seeing over GigE, 
>>> untuned.  You should be able to boost it some with Jumbo frames, 
>>> sysctl net.* tuning and some NIC interrupt tuning.  Also, change the 
>>> block size to 1MB.  IIRC, VMFS3 uses 1MB blocks.
>>>       
>> I'm using Gigabit and of course there should be some sort of caching.
>>
>> I know about jumbo frames but can you point me to any documentation of 
>> how to tune sysctl settings and some NIC interrupts?
>>
>> When you are talking about the block size to 1 MB, what you are talking 
>> about? The dd test? the filesystem?
>>
>> Finally, I'm using VMware. Some people say that the best setting is 
>> using iSCSI under Windows and save the virtual hard drives in the NTFS 
>> filesystem. I'm trying to compare this configuration with real iSCSI to 
>> see what is better. (I have Vmware with Windows as a host OS but I'm 
>> using Debian as a guest).
>>     
>
> Let's backup for a minute and define the environment.  You've got a
> VMware host system (let's call it vmhost1), and a storage server (let's
> call it vmnas1), and one or more VMs (let's call them vmguest1,
> vmguest2, ... ).
>
> vmhost1 is a Windows machine with a fair bit of storage connected to it.
>
> vmnas1 is a ??? box exporting a block device (the storage) as an iSCSI
> device which vmhost1 would like to mount and format NTFS.  vmguest1 and
> friends will then create their virtual disks as files on the iSCSI NTFS
> file system.
>
> For any given configuration, an iSCSI disk will never be faster than
> local attached storage.  The GigE network will limit transfers to about
> 100MB/s max.  Compared to SCSI (320MB/s) or SATA II (300MB/s), it just
> can't compete.
>
>   
Interesting, I didn't know all this :)
> This doesn't mean iSCSI is worthless, especially in VM environments.
> Xen and VMWare ESX both support live migrations of VMs from one server
> to another.  That is vmguest1 running on vmhost1 can be moved to vmhost2
> without shutting down vmguest1.  vmhost1 and vmhost2 manage this by
> keeping the virtual drives and a copy of the memory on storage they've
> both mounted.  The entire hand-off process can take several seconds, but
> vmguest1 keeps running on vmhost1 during most of it until the moment of
> transition, so the process to a user is a fraction of a second and
> honestly looks like a network hiccup.
>   
That sounds very cool.
> Typically, this shared mount is an NFS share, but it can be an iSCSI
> device mounted by both vmhost1 and vmhost2 formatted with a filesystem
> that supports multiple simultaneous mounts.  VMWare's VMFS3 is one such
> filesystem.  Because VMFS3 is built for VM storage, and the virtual
> disks will always be very large, VMFS3 uses a 1MB block size, hence the
> suggestion of 1MB blocks.
>
> Regarding the tuning HOWTOs, I can't help at all with Windows NIC
> tuning.  What kind of box is vmnas1?  If it is an appliance like NetApp
> or EMC would sell, contact the vendor. If it's a Linux box, do a Google
> search for "linux net sysctl." [1] is one of the better hits I get from
> that search.
>   
It's a HP StorageWorks All-in-One (Windows Storage).

I found this performance wiki site that looks very nice:

http://www.performancewiki.com/

Miguel