[NCLUG] Tidbit: /var/log logrotation by date

[Sean Reifschneider]

On 11/10/2012 12:34 PM, Bob Proulx wrote:
> That is certainly a good and safe strategy.  Unfortunately I

Yeah.  I have more backup space than I have time to manage storage, so I
tend to go with what is easy.

> For example I always backup from '/' down the file system but I
> use -x, --one-file-system to exclude other mount points.  Which means

I don't use -x, though that sometimes bites me...  Like when a CD gets
automounted, or I mount my USB disc for something or other, etc...  But, I
just prefer to back up more rather than less, so I will add excludes for
/dev, /proc, /sys, /home/*/.gvfs, and other specific things I want to
avoid.  -x is great, but I've been bitten too many times by systems having
additional file-systems mounted and them not getting backed up...

> This sounds interesting.  Please say more!  Are you pruning the
> previous backup at that time?  So that you are just keeping one
> current backup snapshot?  Is this a recipe that you use globally or
> just one implemented specially for those large files?

So here's the story...  Our backup servers are running ZFS via zfs-fuse.
We have some scripts that manage the backups, the core of that set of
scripts is almost a decade old, and is pretty solid, is at:

   https://github.com/linsomniac/rdiffharness

That version uses rdiff-backup instead of the ZFS tools that we use.
rdiff-backup can keep deltas and expire older data, but it can't do
deletion of interim backups.

With ZFS we take a backup every day, and the first backup of the month we
call a "monthly", the first of the year we call a "yearly", the first of a
week we call a "weekly".  Then we just delete snapshots if they're called a
daily and are older than 14 days, weekly if it's older than 6 weeks, etc...

You can't really do that with rdiff-backup, unless you keep multiple copies
of the backup, one for daily, one for weekly, one for monthly, etc...  This
is where ZFS shines.

All this is wrapped up into a backup system that we use extensively
internally.  If you'd like to see it sometime, let me know and we can do
lunch or something.  It works super, it cut down number of backup servers
we needed from when we were running BackupPC, I think we originally went
down from around 8 machines (though 4 or 5 of them were virtual machines)
down to just one, with better performance.  Though we did fairly quickly
add a second one, because we were at 80+% utilization at that point.

Sean