[NCLUG] A *nix DFS alternative?

[Chad Perrin]

On Tue, Feb 16, 2010 at 11:38:15AM -0700, DJ Eshelman wrote:
> 
> So I need:
> 1)  Bit-Level sync (delta- changes only, not the whole file every single 
> time)
> 2)  Automatic sync (the files arrive or change and immediately being to 
> synchronize)
> 3)  Very low overhead AND scalable (we're talking about storage that 
> could grow to several terabytes in a matter of a few years if we're 
> successful in this)

It's possible I misunderstand your needs somehow, of course, but what you
describe sounds to me like it might be a decent fit for a version control
system (plus a touch of glue code).  Have you considered Mercurial,
Subversion, or Git?  Their command line interfaces tend to be very easy
to script, their incremental rollback capabilities are incredibly smooth
and fine-grained, and they're generally pretty good at minimizing storage
size for a repository.  I imagine there's not really any need for much
"merge" capability for what you need, and if there is no need for local
independent commits I don't see any reason that raw Subversion wouldn't
work for you.  If you *do* need them for some reason, Mercurial and Git
have got your back, and SVK offers similar functionality as a front-end
to Subversion.

Regarding local commits, I wonder if that's exactly what you need when
you talk about not wanting to send backups across the network at the
wrong time of day.  With local independent commits, you can commit
changes to the local repo then, at your leisure, send the updates to the
repo across the network to another computer with everything necessary to
roll back to any point set during the period when you weren't sending
anything across the network included.  Cron scripts or bandwidth
monitoring to trigger pushing or committing at the best times can
automate the process of sending outside of peak usage.

If a version control system suits your needs here, it will probably
actually provide more, and more flexible, functionality in a smaller
package (in terms of storage, memory, and CPU resources) than from an MS
Windows based solution.  Note that I'm speaking out of my fourth point of
contact here and could easily be mistaken, but that's my guess anyway.

Does this come anywhere near helping with your problem, or am I way off
base?

-- 
Chad Perrin [ original content licensed OWL: http://owl.apotheon.org ]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL: <http://lists.nclug.org/pipermail/nclug/attachments/20100216/1d8f29c3/attachment.pgp>