[NCLUG] A *nix DFS alternative?

[Bob Proulx]

DJ Eshelman wrote:
> Basically when you're dealing with several hundred (eventually 
> thousands) of files that are upwards of 16 MB each, the most workable 
> solution is to have a local server that is syncing on the backend any 
> changes made-

As Chris mentioned, I also thought of DRBD when you were talking about
this.  But I haven't tried it.  It seems to have a non-trivial
activation energy and so it has been on my list of cool toys to play
with forever but never had enough energy to actually try it.  But it
seems very much like something you could use here.

> that way if I'm at one office making changes, and my wife is at home
> making changes, she's not having to download each file to her PC,
> save it and re-upload it.

Eww...  No, I wouldn't even suggest anything that manual.  You would
never be able to keep them in sync manually.  You would eventually get
drift between in that case.

> Rsync doesn't seem a reliable enough solution for this because the
> traffic it would generate would be immense, too much to run during
> the day.

As Grant mentioned, rsync is extremely reliable.  It is also very
efficient.  I highly recommend rsync.  Now as to whether rsync is the
best tool for the task here that is a different question.  It might or
might not be part of the total solution.  But it is a very good tool
for doing what it does.

Let's assume for the sake of discussion that you never modify an image
in place.  In that case there won't be anything to delta against on
the target.  In that case the entirety of the file needs to be copied
from site A to site B.  It won't matter if you are using the rsync
algorithm or not.  Whatever you use will need to copy the entire file.
Rsync will simply be copying the entire file.  The advantage to using
rsync is when you run it the second time and it avoids copying the
file since it is already in sync.  Since image data is already
compressed you won't be able to get much more compression out of the
copy operation even if you use a compressed data link.  Considering
that you must already tolerate the bandwidth of one image once to
upload it to the first site then you are only doubling your bandwidth
by mirroring it to the second site.  I think by that argument that you
must be able to tolerate syncing the mirror all of the time.  You
could certainly limit the bandwidth spent on mirroring and let it run
behind a little bit however.  But 2x is really not so terribly bad.

> The benefit of Rsync if it could be invoked in a smart way, is that
> I could have a Linux server on one end and a Windows server on the
> other if I really wanted to.

I think you are much better off keeping to only GNU/Linux machines on
the server side.  Then you have compatible tools everywhere.  It just
gets messy once you try to incorporate a really very different
heterogeneous set of machines.  And I won't even give you one guess as
to which type I would choose to standardize upon on the server side.

> I just question if running Rsync on a cron job would be efficient
> once you get up to a few terabytes of data-

The limitation on using rsync is the number of files (and number of
directories) in your directory hierarchy.  It needs to walk those
directories every time it runs.  If you have enough ram in your server
machine then this will cache in the filesystem buffer cache.  But
using that for other things will push it out of cache too.  The size
of the files or the total size of all files really isn't a problem.  I
have used rsync to keep about 1.5T of data in sync between here and
300ms of Internet away to another site.  It worked very well for me.

> you'd be running a glorifed backup;

Backups have different needs.  In particular this isn't a backup in
the sense that if you corrupt a file on site A then the file will be
corrupted on the mirror too.  Mirroring never replaces backup.

> the way to do this efficiently is to sync only when changes occur
> and I haven't yet found a way to do that with Rsync.

The inotify functionality mentioned by Kasey can make this more
efficient.  But I am skeptical because the limits seem to be
problematic.  In your situation I think you would hit the limits and
then be stuck about how to get past that point.

I think you would be better off keeping track of what directories have
had changes lately.  That is, when your web interface accepts an
upload then add that directory to the list to be sync'd from site to
site.  Since you already know what files have changed on the side that
has changed then you already know what directories to sync over.  That
doesn't have any limits.  To me that seems the most attractive path.

> I'm not really sure subversion et al would really work for this because 
> this is really just file storage, not version builds.

I wouldn't use version control for this.  Large blobs of binary data
really don't version very well.

> It's been 11 years since I've touched any kind of CVS program so I
> really wouldn't know any more :)

I would *definitely* version control all of your code that surrounds
everything.  Don't put it off.  Get it going early in your project.  I
use and recommend Git but any version control is better than none.

> I appreciate the feedback-

For using rsync the /best/ case is unidirectional directories.  It is
always best to know that you are syncing site A to B here and B to A
there.  Grant suggested using 'rsync -u' to avoid updating newer files
for bidirectional syncing.  It is a good suggestion.  I have done
that.  But it isn't without issues.  For example if you ever want to
delete a file the you need to delete the file from all mirrors all at
the same time.  Otherwise it will replicate back!  You can play
wack-a-mole trying to remove a file sometimes.  But if the sync
direction is unidirectional then you can use the 'rsync --delete'
option to remove files that were removed on the source side.  That
will keep the mirrors in perfect synchronization.  If you can't design
in unidirectional directories then you will need to handle the delete
case somehow.

Obviously from this you can tell that I like rsync.  But that doesn't
mean it is the best tool for this task.  It is just my hammer and so
all problems look like nails to me.  I would definitely investigate
DRBD.  I have used AFS before and it has replication.  You might
decide to try it or one of the decedents such as CODA.  I know there
are other distributed filesystems and I would probably push hard
investigating a distributed filesystem solution before looking too far
into a userland application solution.  You might consider Lustre with
ZFS.  You might consider Amazon's S3.

Just my 2 cents...

Bob