Need Super Geek Computer Advice....

[Arcane]

you really think you're geek enough? ok...here we go.....

I'm basically trying to figure the most ideal way to set up my hard drives in my media server...here's what i'm dealing with..

System Specs:
Processor: AMD Athlon 64 4000+
RAM: 2GB DDR PC3200
Enclosure: Standard Full-Size Tower

One 60GB HDD - OS
One 250GB HDD - Temp Storage
Four 500GB HDD - Primary Storage

Use: Stream HD and SD video to other network hosts. Mainly 1 host at a time....possibly 2-3 simultaneously. Occassionally rip DVD's to HDD's.

So, the 60GB is just for OS + updates and required apps for file streaming/sharing. The 250GB is where I'll rip DVDs or store movies that I'm "evaluating". My predicament is how to set up the other 2TB. My concerns are performance, heat, and availability sorta.....while maintaining a good amount of storage space.

Option 1: Right now, I have the 4 HDD's as individual volumes. I did this for 2 reasons. The first is preventing entire data loss should one HDD fail. If I keep them separate, I can essentially lose any HDD and not affect the other drives. Also, if I'm streaming one video...only 1 drive is spinning, thus keeping heat down. However, I'm guessing I'm sacrificing a bit of performance since I'm not using RAID 0. Also, cataloging the movies can be an issue since I'd have to search through 4 drives to find 1 video.

Option 2: The other option I was thinking of was setting up an array (RAID 0). The benefit is performance vs. 4 separate HDDs. Also, I essentially only have to search 1 HDD to find a video. The disadvantage is fault tolerance. If I lose a HDD, the whole array is lost. This actually isn't too bad, since I still have the actual DVDs to reburn. It'll just be a pain to re-rip 400+ DVDs. The other issue is HEAT. I would essentially have all 6 discs spinning (or at minimum 5) at the same time. This will generate quite a bit of heat and possibly reduce HDD life. Not to mention it's in a standard server tower....cooling isn't the greatest.

I'd like to avoid RAID 1 since I'll be losing 1TB of storage space. RAID 4 seems like a good option, but again...I have to consider heat. What about RAID 1+0 or 0+1? Seems a bit too complicated for a plain ol' media server. So, I guess Option 1 or 2 are my only choices (unless I totally missed something).

So, yeah...basically, my question is...what would be the best configuration if my priorities are pretty much focused on performance and cataloging. Reliability and availability are just bonuses.

ummm....halp?

[Arcane]

Oh, and if anyone is familiar with Media Player or other applications that contain media "libraries". Is the library just a group of pointers to media files? Or does the library make a copy of your media and store it into it's on directory?

I'd imagine it's just pointers since it'd be pretty inconvenient to make multiple copies of 1 file.

[matias]

[quote name='Arcane' date='09 June 2010 - 11:48 AM' timestamp='1276076896' post='471196']
Oh, and if anyone is familiar with Media Player or other applications that contain media "libraries". Is the library just a group of pointers to media files? Or does the library make a copy of your media and store it into it's on directory?

I'd imagine it's just pointers since it'd be pretty inconvenient to make multiple copies of 1 file.
[/quote]

I am answering this one first, as it is much quicker. They use pointers. I assume you are using windows so I would advise VLC Media Player before any other.

http://www.videolan.org/

It has a simple interface, all the necessary features, it's free, you can use it to stream video or audio over LAN or WAN. It supports libraries and playlists. It's fast, stable and very customisable. It natively supports most of the codecs and formats you will ever need. MUCH better than Windows Media Player. And if you are not using Windows, it's cross-platform anyway!

[mushrat]

pm me your facebook info. Ill put you in touch with just the person you need.

[matias]

Given the size of your disk array, and the uses you list, I would forget about raid right now. OS on 1 HDD and 2 data drives with one partition each is the best in your case (that would be solution 1.)

A correctly ventilated tower (just run of the mill cpu fan, box fan and power source fan) is all you need regarding heat. Keep HDs as separate as possible so as to maximize airflow and that's it. Streaming or watching a movie doesn't require much cpu so you only have to deal with HDD heat and that is nothing to worry about in the scenario you mention. Well, if you have reliable HDDs. If you don't, heat will probably be the least of your worries anyway. If you want to minimize current consumption, I heartily advise the Western Digital Green Caviar line. They have less access speed but much lower power needs.

If you are going to stream to different hosts it would make sense to balance the load on different hard drives, but in your scenario it would probably be overkill, given the reduced number of concurrent streams.

And now, regarding your main concern, cataloguing and searching, there are several solutions that I think completely cover your requirements, that is search from only one place:

windows:
[list][*]The simplest, which is possible because you are using a new drive you can partition at will. It works well with single or multiple partionned disks. Mount your partitions as folders. You would mount for instance your primary storage partition under E:\, in which you would create a folder called Temporary Storage. Then, using [b]Control Panel[/b] > (if you are not using the classic view: [b]Performance and Maintenance[/b] >) [b]Computer Management[/b] > [b]Storage[/b] > [b]Disk Management. [/b]Your different HDDs will be shown there. Existing partitions too or you can create them, delete them, format them there if needed (right-clicking on a volume). I won't go into details right now, unless you need me to, but here you can map your temporary storage partition to a folder, in this case E:\Temporary Storage. So everything will be under E:\[*]Use symbolic links. If you are under Windows 7 you can use natively mklink in the console. It works with Vista too, but not very well. You had better use this software: WimbolicLink. It works with XP too. It does the same as what I have described above but you can nest directories in any other directory, on any HDD. Asd you can work directly with partitions, it would be preferable for you to use the first option.[/list]*NIX, FreeBSD, Linux: use symbolic links ( ln -s destination origin . try man ln for more explanations)

Let me know if you need further clarifications

PS: even if you are storing your software on the OS partition, try to store the configuration files on your data partitions, so you don't lose them if you have to re-install or lose your OS

[Cole.]

[quote name='matias' date='09 June 2010 - 09:20 AM' timestamp='1276089602' post='471211']
[quote name='Arcane' date='09 June 2010 - 11:48 AM' timestamp='1276076896' post='471196']
Oh, and if anyone is familiar with Media Player or other applications that contain media "libraries". Is the library just a group of pointers to media files? Or does the library make a copy of your media and store it into it's on directory?

I'd imagine it's just pointers since it'd be pretty inconvenient to make multiple copies of 1 file.
[/quote]

I am answering this one first, as it is much quicker. They use pointers. I assume you are using windows so I would advise VLC Media Player before any other.

[url="http://www.videolan.org/"]http://www.videolan.org/[/url]

It has a simple interface, all the necessary features, it's free, you can use it to stream video or audio over LAN or WAN. It supports libraries and playlists. It's fast, stable and very customisable. It natively supports most of the codecs and formats you will ever need. MUCH better than Windows Media Player. And if you are not using Windows, it's cross-platform anyway!
[/quote]

Can't help much, but I second the use of VLC. I love love love it. It plays just about everything.

[Arcane]

thanks matias!!!

i do have a few questions though....

1. for VLC to stream, it seems as though you need to have VLC installed on all the client computers. Also, it doesn't seem to be DLNA compliant. Would the service/server be able to be automatically recognized by a PS3 or other "non-PC" type devices? It would be a hassle to consistently type the URL for the stream on a PS3 controller. At the moment, I'm using TVersity 1.6 Pro. It's DLNA compliant and offers several spiffy features like Hulu, YouTube, G4tv, and other various streams. TVersity is also a transcoding media server. Does VLC do its own transcoding or does it rely on the host? Are there any benefits to using VLC over TVersity?

2. when you say it's best to use 1 HDD for the OS and 2 HDD's for the data drives....are you suggesting that I convert the 5 HDD's I have into two logical volumes? Would you suggest setting up two RAID 0's then? Or should I just spanned 1 volume across a couple HDDs? Is there a downside to using 1 HDD for the OS and having 5 separate volumes? Also, I'm not sure of the need for soft links...aren't those more suited for Unix platforms or cross-platforming? [i]I'm guess I'm totally lost on how you're recommending I set up the HDDs[/i]. Also, I'm just using old hardware I have and don't really want to buy more stuff. The HDDs were brand new in the box, but purchased about 3-4 years ago. So, I'm just trying to make use of what I have.

3. I'm not sure I understand this statement: "even if you are storing your software on the OS partition, try to store the configuration files on your data partitions". What configuration files are you referring to? All I really have set up is Win7 and TVersity. Not sure of any other configs.


thanks in advance, matias.....and other geeks who want to chime in.

[matias]

Sorry, I didn't read correctly. Where you wrote four, I read one. How about you forget what I said. I'll start over... how do you store your dvds? DVD-Shrink style or one compressed file? Edited June 9, 2010 by matias

[Arcane]

all my movies are still on the DVDs...haven't ripped them yet. i was just planning to rip them to an avi file...then put that file in a folder with the title of the movie....basically, a folder named Avatar and in the folder is Avatar.avi

any opinions on VLC vs. TVersity?


i feel all geeky....yay!

[Dr. B]

Keep us posted, I'm in a similar predicament except that I've modded a tower case for cooling.

VLC FTW. I use VLC and Foobar for everything.

I've decided to store my DVDs as .iso files with a lightweight virtual drive to mount them to when i want to use VLC to watch them. The tradeoff for quality is size, but I prefer having a flawless copy that I can downgrade later. I guess that's why I have so much .FLAC music too... Edited June 10, 2010 by Dr. B

[Arcane]

well, after streaming a few episodes from hulu and a few youtube vids....i think TVersity is great. saves hdd space by not having to rip some tv seasons....

before i had my ps3, i did the exact same thing Des....created dvd images and used daemon tools to mount them on a client....works great. but, like you said...takes up quite a bit of space. now that i use my ps3 to stream a lot of stuff on my tv, i think that eliminates the possibility of using a virtual optical drive....

if you haven't tried it already, check out tversity...it's pretty nifty..


anxiously waiting for matias' reply on the HDD setup though....oooooooh!!

[matias]

Raid 0
[list][*]Performance: Raid 0 gives you nothing because you are going to be accessing large files. Raid 0 gives you a performance boost when you are accessing files that are smaller than the stripe size.[*]redundancy is a bitch: you lose 1 disk, you lose data on all 4[/list]Raid 5
[list][*]Min. 3 similar HDDs so you're good. Parity and data are both spanned over the 4 drives. Parity takes up a storage space equivalent to one of the hard drives. It would mean you would 'lose' 500GB storage. But IMO that's a good tradeoff: 500 GB for 1.5TB of data insured against the loss of any one of the 4 HDDs. And it is redundant over time. If you lose one HDD you can rebuild parity and keep your collection long after your DVDs are scratched to death.[*]Performance: go Raid 5 only if you have it at hardware level. Software-based RAID 5 is usually slow. Writes are not very performant but reads are, regardless of the size of your files. But reads are boosted, regardless of the size of the accessed file. IMO, that fits your case as you do not really care if there is some overhead at write time[/list]
VLC transcodes.

I don't know TVersity, but it sounds good. VLC seems to support DLNA with some tweaking (my own media server is down for maintenance until I get the money to put it back up so I can't test it) but the tweaking part means headaches, so it's better to stick to a solution you know works.

I like dvd shrink to store my DVDs. You can strip off all the fat and keep the meat at any compression level suits you. Major win.

conf files: what I meant is keep all the configuration data you can away from the OS partition. For example, I do not know how TVersity stores it's catalogue, but it is good if you can store that or those files in a partition that won't suffer from reinstalls and has redundancy. Another example is keep your "My Documents folder" away from the OS and store it on another partition. Same with email archives etc... It makes reinstalls much less of a headache and means the work you put into cataloguing is pretty much going to last long term. In Win world, reinstalls are just something that will happen, sooner or later. So are HDD fails. It is not a question of whether there will be a failure, but of "when" and "are you prepared?".

[Arcane]

ah....that's a great tip about moving the TVersity to another drive....i think i'll put it on the 250GB now...

i was reading up on RAID 5...if one of the disks crashes, would i have to replace the disk with the exact same HDD? or can i just plop in any 500GB drive?

lastly, expansion...if i wanted to expand my storage from 4x500GB to 4X1TB discs, would I have to wipe and rebuild the array? Edited June 10, 2010 by Arcane

[matias]

[quote name='Arcane' date='10 June 2010 - 11:13 AM' timestamp='1276161191' post='471328']
ah....that's a great tip about moving the TVersity to another drive....i think i'll put it on the 250GB now...

i was reading up on RAID 5...if one of the disks crashes, would i have to replace the disk with the exact same HDD? or can i just plop in any 500GB drive?

lastly, expansion...if i wanted to expand my storage from 4x500GB to 4X1TB discs, would I have to wipe and rebuild the array?
[/quote]

It has to be of the same size and type: any 500GB with the same characteristics would be ok. A larger disk would work too but you would not be able to use disk space over 500GB.

The array can be expanded dynamically if your controller allows it. If you are going to replace the whole 4 drives you can change them one at a time, rebuild the array data on the new disk, 4 times. In the end you get a copy of the array on the 4x1TB and just need to expand the partition to use the whole 4TB. Or you can mount the array on another computer, mount the new array on your server and transfer data onto it.

[porksandwich9113]

Raid 5 is the way to go for media servers. It gives you redundancy and the best price:redundancy ratio you will get.

It's an (n-1) config, which requires at least 3 drives to function.

I personally run raid 5 in my desktop that I store data on.

Basically n-1 means you have 4 500GB disks. 4-1 (500) = 1500 GB or 1.5TB.

Raid-0 is for the boot disk - it doesn't matter if those crash because nothing important is stored on them. All you do is replace and re-install your OS and apps and your back in business.

Raid-5 is for your media - one disk fails, the array still functions as if nothing happened. You just drop in another 500GB 7200RPM disk and your in business. The raid array with rebuild itself with parity without you even having to do anything.

Raid-1 will accomplish the same thing - however you can have one disk fail and not have to replace it until a second fails. It comes at a cost though. It's an n/2 config. Meaning 4(500) / 2 = 1TB. You lose half of that storage at the cost of 2nd redundancy.

If you want my 2 cents, go for a raid 5 config.