[NCLUG] Backup idea

[Bob Proulx]

grant at amadensor.com wrote:
> I have a separate mirrored array for backups.   It uses a daily cp to  
> copy the files to a dated copy using hard links, then uses rsync to  
> overlay only the changes.   It also daily deletes the oldest backup.

Hmm...  This sounds very much like 'pdumpfs'.  Is that what you are
using?  I have been using it for a while and it works pretty well.

  http://0xcc.net/pdumpfs/index.html.en

It is a little slow the first time through a large system (as compared
to the speed of rsync, rsync is faster) but the advantage is that it
just-works and I have never needed to mess with it after that point.

> This is good, except for one little thing, it is hard to manage when  
> the disk gets close to full.  Sometimes I need to manually delete the  
> oldest backup an extra time to leave enough slack for the next day's  
> rsync delta.

Perhaps you should delta every other day instead of every day?

> Here is my idea:
> Rebuild the backup partition using LVM.   Do a daily rsync to it, then  
> do a snapshot.

That should work.  For one snapshot.

> Will this succeed in removing the oldest one any time I run out of  
> space?

Hmm...  One snapshot is one snapshot.  It is not a rotating pipeline
of snapshots.  AFAIK.  You create a snapshot with lvcreate.

  lvcreate --snapshot --name backup /dev/vg0/data

That will create one snapshot.  It can be mounted.

  mount /dev/vg0/backup /mnt/backup

And so forth.  If you want to keep a rotating pipeline of lvm
snapshots then you would need enough free volume group space to do
this and you would need to script up the rotation plan yourself.

  TODAY=$(date +%F)
  lvcreate --snapshot --name backup-$TODAY /dev/vg0/data
  mkdir -p /mnt/backup/$TODAY
  mount -o ro /dev/vg0/backup /mnt/backup/$TODAY

And then back them out later.  This is untested but the attempt is to
be careful of skipping days due to invocation time.  That is why the
date invocation is complicated.

  THEN=$(date -d "$(date -d '12:00 today' +%F) -30 days" +%F)
  umount /mnt/backup/$THEN
  lvremove /dev/vg0/backup-$THEN

Hmm...  If the cleanup skipped a day that would leave orphans behind.
I guess it needs a little more tinkering to only keep 30 days or some
such.  Again untested but something like this seems possible.

  list=$(ls -1r /mnt/backup | sed 1,${NUM_BACKUPS}d)
  for d in $list; do
    umount $d
    lv=$(df -P $d | awk '/^\/dev/{print$1}')
    lvremove $lv
  done

I have no idea if this is more or less efficient than using hard links
to share a backup copy.  I haven't actually tried this at all and so
all of what I have said here might be completely bad information.  It
would need to be tested and verified.

The primary advantage is that the snapshot happens at a single point
in time and the files then do not change after that point.  This is an
advantage over a long running backup process that may grab some files
from 4am, some from 5am, some from 6am, etc.

> I know that it will invalidate the snapshot, but will it drop  
> it and free up the space?

You remove snapshots with lvremove.

  lvremove /dev/vg0/backup

> What happens if that snapshot is mounted when it gets full,

Getting full implies to me that you are creating a writable lvm
snapshot.  In which case you need to specify how much space in the lvm
snapshot you want to allocate to hold changes.  Say 500MB of changes
or something.

  lvcreate --snapshot --name backup --size 500M /dev/vg0/data

This will allow you to overlay an incrementally different filesystem
image on top of an existing image without needing to reuse the common
disk space.  This is similar to unionfs.  I believe when the changes
hit the size limit the filesystem becomes full but previously I have
heard rumors of bugs there where the filesystem image failed for some
reason, YMMV, its only a vague memory of mine.

Do you need a writable snapshot?  Don't you simply need to park the
data in a read-only image for the purpose of backup?

> does it get unmounted and removed, or does the whole volume group go
> into a state of stuck?

I think it acts like a full filesystem.  It would be useful to test
this assumption before relying upon it for backup.  I have a vague
memory that this has not always worked correctly.  I assume that it
has been fixed by now.

> I wish I had extra hardware to play with this on, but right now I  
> really don't, so I am hoping someone else has done this.

Not recently.  But I have some work on lvm happening soon.  I will try
to remember to test this out then.

Bob