[NCLUG] A *nix DFS alternative?

grant at amadensor.com grant at amadensor.com
Tue Feb 16 15:12:34 MST 2010


> No problem- this is sort of hard to explain.
>
> Basically when you're dealing with several hundred (eventually
> thousands) of files that are upwards of 16 MB each, the most workable
> solution is to have a local server that is syncing on the backend any
> changes made- that way if I'm at one office making changes, and my wife
> is at home making changes, she's not having to download each file to her
> PC, save it and re-upload it.  Rsync doesn't seem a reliable enough
> solution for this because the traffic it would generate would be
> immense, too much to run during the day.  The benefit of Rsync if it
> could be invoked in a smart way, is that I could have a Linux server on
> one end and a Windows server on the other if I really wanted to.  I just
> question if running Rsync on a cron job would be efficient once you get
> up to a few terabytes of data- you'd be running a glorifed backup; the
> way to do this efficiently is to sync only when changes occur and I
> haven't yet found a way to do that with Rsync.
>


rsync is actually pretty good about efficiently handling big loads.   It
works with block level compares, and so it only sends the data it needs
to.  if you are sending across SSH, which is easy with rsync, you will get
file compression as well, although lossless compression and photos really
don't play well, so you won't get much compression.  The other good thing
is that if it is in a crontab, it will pick up any new files, or any
partial transfers where it left off and finish the job.

rsync -auv /home/www/photostore/ bob at otheroffice:/home/www/photostore/

You will need to be careful if syncing both ways not to hit the same file
from both ends.   Test it with dummy small text files to make sure you
have your settings correct.  Likely possible problems:  Every sync makes a
new sub directory, which gets copied the other way one level deeper,
leading to a never ending depth of path until you run out of disk or the
pathname is too long.  Overwriting a good file with a bad one, for
instance, I put the -u on there to make sure that it takes the newer file,
no matter what, so that if office2 updates a file, the old one from
office1 does not end up replacing it.





More information about the NCLUG mailing list