[NCLUG] A *nix DFS alternative?

[DJ Eshelman]

DFS does a similar thing, where it will (up to a specified limit) in the 
case of file conflicts store a copy and run a somewhat intelligent 
algorithm to determine which file is current, then only change those 
bits.  My problem with rsync the last time I tried it was exactly what 
you mention- if a file was open it wouldn't read and would try to 
overwrite with the one that was open (granted, that was on two Windows 
servers over a VPN, that issue may not exist on Linux servers).  Your 
thought about compression over rsync got me to thinking and there was a 
project going that was using 7-zip compression end-to-end but it's gone 
stale

I may end up trying rsync initially for just our files (kind of an 
active backup) and see how that goes.  My fear is really just loosing 
data more than keeping a realtime service going, I suppose- I just want 
to build it right from the start instead of having to constantly tinker 
with it while it's a production system.  At first we may only have a few 
files each week that change, for which rsync would be great.

Eventually I want to be able to offer clients an alternative to 
RDB/IronMountain, etc but it's really hard to find user-friendly 
automatic backups, especially with bit-level deltas and easy recovery.  
BackupPC is what I'm currently considering for that end of things, but 
I'm also targeting a heavy Mac audience (grumble grumble), so whatever I 
choose will have to be as good on the front end as on the back.  I can 
see why the existing solutions charge so much, but I'm thinking of 
selling it as a value add instead of a standalone solution.

keep the thoughts coming though, I really appreciate it!

On 2/16/2010 3:12 PM, grant at amadensor.com wrote:
>> No problem- this is sort of hard to explain.
>>
>> Basically when you're dealing with several hundred (eventually
>> thousands) of files that are upwards of 16 MB each, the most workable
>> solution is to have a local server that is syncing on the backend any
>> changes made- that way if I'm at one office making changes, and my wife
>> is at home making changes, she's not having to download each file to her
>> PC, save it and re-upload it.  Rsync doesn't seem a reliable enough
>> solution for this because the traffic it would generate would be
>> immense, too much to run during the day.  The benefit of Rsync if it
>> could be invoked in a smart way, is that I could have a Linux server on
>> one end and a Windows server on the other if I really wanted to.  I just
>> question if running Rsync on a cron job would be efficient once you get
>> up to a few terabytes of data- you'd be running a glorifed backup; the
>> way to do this efficiently is to sync only when changes occur and I
>> haven't yet found a way to do that with Rsync.
>>
>>      
>
> rsync is actually pretty good about efficiently handling big loads.   It
> works with block level compares, and so it only sends the data it needs
> to.  if you are sending across SSH, which is easy with rsync, you will get
> file compression as well, although lossless compression and photos really
> don't play well, so you won't get much compression.  The other good thing
> is that if it is in a crontab, it will pick up any new files, or any
> partial transfers where it left off and finish the job.
>
> rsync -auv /home/www/photostore/ bob at otheroffice:/home/www/photostore/
>
> You will need to be careful if syncing both ways not to hit the same file
> from both ends.   Test it with dummy small text files to make sure you
> have your settings correct.  Likely possible problems:  Every sync makes a
> new sub directory, which gets copied the other way one level deeper,
> leading to a never ending depth of path until you run out of disk or the
> pathname is too long.  Overwriting a good file with a bad one, for
> instance, I put the -u on there to make sure that it takes the newer file,
> no matter what, so that if office2 updates a file, the old one from
> office1 does not end up replacing it.
>
>
> _______________________________________________
> NCLUG mailing list       NCLUG at lists.nclug.org
>
> To unsubscribe, subscribe, or modify
> your settings, go to:
> http://lists.nclug.org/mailman/listinfo/nclug
>