[NCLUG] Tidbit: /var/log logrotation by date

[Bob Proulx]

According to the FHS the /var directory hierarchy is for variable data
files.  This includes spool files and other cache files.  It also
includes log files.  Most of those used to live in /usr/lib but were
moved out of /usr and over to /var as part of general refactoring.

In theory you could recover a system from backup without recovering
/var but in practice it often isn't so simple.  There is often a lot
of useful stuff in /var that isn't so ephemeral.  I find it easier to
back up /var.  On most simple systems it isn't very large.  On one of
my machines I just looked at it is 265M in total.  Of course it
depends upon what is installed and if you have heavy applications then
you might have a large amount of data there.

But then after having decided to back up /var I know that /var/log has
a lot of data that *appears* to change every day but doesn't really.
The log files in /var/log are usually rotated by the logrotate
program.

By default logrotate renames log files to have a numbered extension.
The syslog file is renamed syslog.1.  But just before that any
syslog.1 file is renamed and gzip'd to syslog.2.gz.  But just before
that any syslog.2.gz file is renamed syslog.3.gz.  And so forth down
the log history trail.  Depending upon what is installed there may be
a lot of log files and all of them will get renamed every day.  There
well may be over a hundred files with a lot of data and all of them
will appear to change every day because of the file renaming.  A
backup program backing up that directory will need to back up all of
those files every day.  That is a lot of unneeded thrash.  At 265M in
total it is perhaps small.  But it is unneeded.

Of course the contents of the files are not actually changing.  After
the log files have been written they are never changed again.  This
should be perfect data for a stable archive.  But the renaming gets in
the way.  If only there were a way to cause the filenames to be set
once and then not changed again.  And there is!

The logrotate program includes a "dateext" option.  This was
originally added to the SuSE distribution as a downstream patch
specifically to address this problem but I has now migrated upstream
and is now available everywhere.  But it is a change and as with most
new-ish features it isn't the default to avoid upsetting people
expecting the previous behavior.  If you don't ask for it then nothing
changes.  But if you know about it then you can ask for it and then
you can get the new feature.

I now always add this to my /etc/logrotate.conf file:

  # Archive old versions of log files adding a daily extension like
  # YYYYMMDD instead of simply adding a number.
  dateext

With that small addition to the logrotate configuration the log files
are created with an immutable datestamp extension to the filename.
Instead of being renamed everyday they are renamed once to a datestamp
name and then never renamed again.  This is much nicer for anything
that keeps track of filesystem changes and particularly for doing
backup that include the /var/log directory.

Here is a listing of 'ls -ltr /var/log/syslog*' from my machine.  The
result is similar for every other log file that logrotate handles.

  -rw-r----- 1 root adm  227412 Oct 24 06:47 syslog-20121024.gz
  -rw-r----- 1 root adm  208553 Oct 25 06:42 syslog-20121025.gz
  -rw-r----- 1 root adm  230041 Oct 26 06:45 syslog-20121026.gz
  -rw-r----- 1 root adm  178838 Oct 27 06:52 syslog-20121027.gz
  -rw-r----- 1 root adm  135139 Oct 28 06:43 syslog-20121028.gz
  -rw-r----- 1 root adm  174881 Oct 29 06:50 syslog-20121029.gz
  -rw-r----- 1 root adm 2270457 Oct 30 06:40 syslog-20121030
  -rw-r----- 1 root adm  582674 Oct 30 11:04 syslog

That is much nicer!

Bob