[Novalug] best hard disk setup for home file server?

[Richard Ertel]

also, in case anyone was on the edge of their seats regarding what i
am doing with my server disk setup, i've decided to keep the 4 disks i
have (two 1TB, two 1.5TB), and RAID-10 (or whatever it's called,
mirror+stripe) the two 1 TB drives and RAID-0 (stripe) the 1.5 TB
drives. this will give me a 3.0 TB array of non-fault-tolerant space
and 1.0 TB of single-drive-fault-tolerant space. in the 3TB space will
only be stored data that also exists on the backup. data that isn't
backed up will go on the mirrored array.

i came to a realization that i don't need fault tolerance on data that
is already backed up. why have THREE copies of it, essentially? any
drive in the server dies, all my data is covered.

now whether or not i use MD or LVM to do the mirroring and striping,
that i haven't decided yet.

On Tue, Oct 20, 2009 at 17:15, Richard Ertel <richard.ertel at gmail.com> wrote:
> this is actually the kind of setup i have started doing for my music
> collection. as MP3 ID3 tags don't support multiple genres, i find that
> navigating directories of symlinks based on both artist (including
> multiple artists) and genre is working pretty good.
>
> for my movies, though, it COULD work out ok, but nothing that i really
> need, as i use XBMC for watching, and that builds it's own database.
> still, automating it with metadata files like you suggest is an idea i
> hadn't thought of.
>
> On Tue, Oct 20, 2009 at 16:57, James Ewing Cottrell 3rd
> <JECottrell3 at comcast.net> wrote:
>> Of course they are viable. Consider two movies, The_Hustler and
>> Cool_Hand_Luke. Now I can't get to IMDB, so I am going to make a wild guess
>> that that the former came out in 1962 and the latter in 1967. Furthermore,
>> I'm making the rediculous assertion that TH was directed by Stanley Kubrick
>> and the latter by George Lucas.
>>
>> We have 10 partitions, /data[0-9]. Consider the following:
>>
>> /data0/The_Hustler
>> /data1/Cool_Hand_Luke
>> /data2/The_Shining
>> /data3/Star_Wars
>> /data4/2001
>>
>> /movies/year/1962/The_Hustler -> /data0/The_Hustler
>> /movies/year/1967/Cool_Hand_Luke -> /data1/Cool_Hand_Luke
>> /movies/year/1969/2001 -> /data4/2001
>> /movies/year/1977/Star_Wars -> /data3/Star_Wars
>>
>> /movies/director/George Lucas/Cool_Hand_Luke -> /data1/Cool_Hand_Luke
>> /movies/director/George Lucas/Star_Wars -> /data3/Star_Wars
>> /movies/director/Stanley_Kubrick/2001 -> /data4/2001
>> /movies/director/Stanley_Kubrick/The_Hustler -> /data0/The_Hustler
>>
>> /movies/actor/George_C_Scott/The_Hustler -> /data0/The_Hustler
>> /movies/actor/Paul_Newman/The_Hustler -> /data0/The_Hustler
>> /movies/actor/Paul_Newman/Cool_Hand_Luke -> /data1/Cool_Hand_Luke
>> /movies/actor/Darth_H_Gates_III/Star_Wars -> /data3/Star_Wars
>>
>> For each movie, you could build a MetaFile you can generate the symbolic
>> links from. The_Hustler.txt might have
>>
>> Name: The_Hustler
>> Year: 1962
>> Director: Stanley_Kubrick
>> Actor: Paul Newman
>> Actor: George_C_Scott
>>
>> I probably should have thrown Dr. Strangelove in there too. But you get the
>> idea.
>>
>> You could build a Poor Man's Dedup this way too.
>>
>> JIM
>>
>>
>> Bryan J. Smith wrote:
>>>
>>> Well, I don't know if I'd argue that symlinks are viable.
>>> I was talking more mounts than symlinks.
>>>
>>> My standard filesystem[1] for static data is ...
>>>  /home/mobile/Static
>>>
>>> And my Photos go under ...
>>>  /home/mobile/Static/Photos
>>>
>>> Over the last five (5) years, I've shot a good TiB of photos.
>>> That's not something I want on a single filesystem, much less
>>> mixed in with a lot of other files, even if they are all static.
>>>
>>> I always organized my photos by type -- Travel for general
>>> travel, UCF for photos taken at my Alma Mater's football
>>> games, etc...  Underneath those are years and places.
>>>
>>> It wasn't even two (2) years into shooting that half of my photos
>>> were a result of 2-3,000 shots at every football game (I travel to
>>> a good ten (10) games every year -- all home and most away
>>> games), that it was 200-300GB/season.
>>> So I created a new filesystem[1] and mounted it accordingly
>>> when I started running into over 256GiB of photos:
>>>  /home/mobile/Static/Photos/UCF
>>>
>>> Just this year, because the sizes are 250GB+/season, I decided
>>> to create a new filesystem[1] for:    /home/mobile/Static/Photos/UCF/2009
>>>
>>> The years of 2005, 2006, 2007 and 2008 might have their own
>>> directories, but they are on the original Static/Photos/UCF filesystem.
>>> The 2009 directory is in its own, new, UCF/2009 filesystem.  The
>>> user doesn't see any different at all, be it over HTTP, NFS, SFTP or
>>> SMB.
>>>
>>> I can grow filesystems as necessary, create new ones when I decide
>>> I want to localize more storage, etc...  These filesystems can even be
>>> on different VGs, but mounted in the same area.  I can use LVM
>>> commands to move things around too.
>>>
>>> -- Bryan
>>>
>>> [1] To be really accurate, they don't even need to be on the same
>>> server.  Why would I want such?  Well, I have two "home" networks
>>> separated by 1,000 miles.  ;)
>>>
>>> To start, the actual, real, original filesystem is ...
>>>  /export/dilbert51/Static
>>> Which automounts (NFS, Bind, etc...) to:    /home/mobile/Static
>>>
>>> My own, personal/professional, longstanding convention for data
>>> has been:    /export/(server)/(type)
>>> With the automount:    /home/(domain)/(type)
>>>
>>> There are countless reasons for this approach, but it breaks down
>>> these reasons:
>>> - The "source" filesystem has a path with the servername in it, which
>>> allows
>>> autofs to differentiate if it's local (and uses a bind mount) or remote
>>> (and
>>> uses NFS or another mount)
>>>
>>> - The "clients" use a path with the domain name in it, because automounter
>>> maps are usually defined per domain (be it NIS, NIS+, LDAP, Winbindd,
>>> custom schema in AD-LDAP, etc...).
>>>
>>> In my case, I have two domains, "Oviedo" and "Mobile."  Oviedo is Oviedo,
>>> FL, where my permanent residence is.  Mobile is my domain for traveling,
>>> including my current Maryland residence.
>>>
>>> And yes, this allows me to do all sorts of things.  Just because I hit
>>> /home/oviedo/Static/Photos/UCF doesn't mean I hit an Oviedo server.
>>> It just means I'm just using the automounter map for Oviedo.  I have less
>>> storage in Oviedo than my Mobile, so I could define that even Oviedo
>>> servers hit the Mobile server.
>>>
>>> Say, for example, I only define ./UCF/2009 as hitting the Oviedo server
>>> because I have more storage on the Mobile server and my wife doesn't
>>> hit my photos from years ago as much as she does current stuff.  So I'm
>>> only rsync'ing photos from this year back to Oviedo, and not 2008 or
>>> earlier.
>>>
>>> So my Oviedo automounter map has 2009 defined to dilbert.oviedo while
>>> all other UCF/200x mounts hit to dilber51.mobile (across the VPN).
>>>
>>>
>>>  -- Bryan J  Smith           Professional, Technical Annoyance Linked
>>> Profile:         http://www.linkedin.com/in/bjsmith
>>> ---------------------------------------------------------- Red Hat:  That
>>> 'other' American software company built on
>>> open customer selection of options and value, instead of
>>> controlled distribution channels of forced bundle and lock
>>>
>>>
>>>
>>> ----- Original Message ----
>>> From: James Ewing Cottrell 3rd <JECottrell3 at comcast.net>
>>> To: Richard Ertel <richard.ertel at gmail.com>
>>> Cc: Bryan J. Smith <b.j.smith at ieee.org>; novalug-bounces at calypso2.tux.org;
>>> Novalug <novalug at calypso2.tux.org>
>>> Sent: Tue, October 20, 2009 2:50:59 PM
>>> Subject: Re: [Novalug] best hard disk setup for home file server?
>>>
>>> i was arguing that you really don't care where the actual movies go. you
>>> are going to build an index to them elsewhere and symlink to the actual
>>> movie/tv show. heck, build several indices, a poor man's database.
>>>
>>> as for space, if you had a 1TB space carved up into ten 100G partitions,
>>> and each movie was an average of 2G, you'd end up wasting 10 x 2G / 2 or 10G
>>> on the average. ok, that's five movies, but it's just 1% of your space, and
>>> only makes a difference if you want to use it all.
>>>
>>> as they say, don't put all your eggs in one basket.
>>>
>>> jim
>>>
>>> Richard Ertel wrote:
>>>>
>>>> i have a movies directory, then inside that, each movie is in it's own
>>>> directory. ditto with tv shows, each series has it's own directory
>>>> inside the "tv shows" directory.
>>>>
>>>> my main interest in keeping one big filesystem is effecient use of
>>>> space. if i have separate filesystems for each kind of media, then
>>>> what happens when i load up proportionately more movies than tv shows?
>>>> i might run out of space on the movies filesystem, but still have
>>>> plenty of room in tv shows. feels more complicated from an end-user
>>>> standpoint, but i see the argument for it being technically more
>>>> sound.
>>>>
>>>> On Fri, Oct 16, 2009 at 12:39, James Ewing Cottrell 3rd
>>>> <JECottrell3 at comcast.net> wrote:
>>>>>
>>>>> Richard Ertel wrote:
>>>>>>
>>>>>> i prefer one volume group because that's the most efficient way to
>>>>>> store a bunch of files. if i have 2 TB of movies, for example, but two
>>>>>> 1.5 TB volumes, then i have to split the group somehow. seems messy to
>>>>>> me.
>>>>>>
>>>>>> are there better solutions?
>>>>>
>>>>> What, you're gonna put them all in one directory? If you put them in
>>>>> separate directories, like scifi (oops, syfy?), action, western, comedy,
>>>>> drama, pron, etc .....
>>>>>
>>>>> then you can just use smaller partitions and build your tree via mounts.
>>>>>
>>>>> heck, with stuff that big, just stuff them in a next available partition
>>>>> and
>>>>> build a tree of symlinks to them...that way you can index them by year,
>>>>> director, actor, as many ways as you like.
>>>>>
>>>>> JIM
>>>>>
>>>>> P.S. Yes, I think you should ditch those 1.5 TBs ... I'd be glad to
>>>>> recycle
>>>>> them for you.
>>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>>
>>>> No virus found in this incoming message.
>>>> Checked by AVG - www.avg.com Version: 8.5.421 / Virus Database:
>>>> 270.14.20/2440 - Release Date: 10/16/09 06:32:00
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>>
>>>> No virus found in this incoming message.
>>>> Checked by AVG - www.avg.com Version: 8.5.422 / Virus Database:
>>>> 270.14.23/2448 - Release Date: 10/20/09 10:43:00
>>>>
>>
>>
>