Data recovery and XFS (was: Re: [NCLUG] XFS, "Can't read superblock")

[Sean Reifschneider]

On Mon, Dec 04, 2006 at 11:00:15AM -0700, Chad Perrin wrote:
>My father uses a Linux box as a fileserver.  The partition on which he
>stores all the important data uses an XFS filesystem.  This weekend,

Just FYI, I've had quite a few issues with XFS.  I've found that XFS can
take massive amounts of RAM to run the fsck, for one thing.  I've currently
got a system that has periodic XFS errors, even though xfs fsck has been
run repeatedly on it.  That's the last XFS system I have right now, I've
switched entirely back to ext3.

I know some people love XFS, but I don't like it.  Having a TB worth of
file-system data that requires heavy swapping on a system with 2GB of disc
to run the fsck isn't really an option for most production systems.  I let
this one run 15 hours before I went out and did an upgrade (luckily, I had
spare RAM) which allowed it to complete in less than an hour.

>We're hoping this won't turn into a need to go to a data recovery
>specialist, because he doesn't have the several thousand dollars that
>would cost right now.

If you aren't in a rush, ISTR that recovery places will charge you far less
than if you are.  The worst rates are when you need it back right away,
during a holiday or weekend.

I've only had to deal with a data recovery place once.  We had a client
with a system that, for some reason I've now forgotten, ran into data
corruption or loss on it's RAID array.  The backups, we found, were writing
the full backup to the DDS tape and then writing the incrementals over the
front of the tape, instead of appending.  This was due to some upgrade
which happened since we installed and tested the backup system, causing a
rewind after it seeked to the end of the tape to append.  Ugh.

The client was, obviously, not happy.  They wanted us to pay the thousands
of dollars to get the most expensive recovery (a team of recovery people
working in shifts, top priority).  We had been practically begging them to
approve our doing a backup audit on the system for something like the
previous 6 months, so I didn't feel that it was entirely our fault...

So, the client ended up paying for the recovery.  Of course, when it was
their dime, they decided the data wasn't quite as critical as they
initially thought, and instead of the "They've got us working in shifts"
recovery they selected the "2 to 4 weeks" recovery option.  I want to say
the price difference was something like $500 instead of $4,000.  Hence my
comments above.

We ended up covering the cost of some hardware upgrades, fixing the tape
backup system, an audit, as well as a huge amount of time spent writing
custom recovery because the backup software stored data in a format not
supported by any of the data recovery houses, but we did get all the user
data off.  Luckily, incrementals had been small, and nothing important was
at the front of the tape.

The backup media was DDS tape, which was why it was so challenging.  DDS
actually stores an idea of the length of the written segment in the header
of the tape.  It's not just an "EOT" marker that is written that you can
skip by.  The header says "This tape is X bytes in length".  When you write
1 byte to the tape and eject it, it writes a header saying "1 byte on the
tape".

Some people who had similar problems and asked for help got suggestions
that they try seeking to EOT, then start writing data and then pull the
power on the drive.  The drive won't write an EOT and won't write the
header, so you can just read past the EOT.  That may work on non-DDS, but I
ran a test on a DDS mock-up tape I created to see if it was possible, and
after that the tape would still believe it's length was the last length it
had been (because of the header saying the old length).  Updating the
header requires a tape drive with a special firmware, and special SCSI
tools to do.

The recovery house had the tools to do that.

I wrote up instructions saying that I just needed them to stream the data
after the EOT, or optionally all the data, out to another tape or write to
a hard disc, and I'd take care of it from there.  The tape used a
non-standard format that was kind of like a custom tar header, with the
file data being compressed (unless it was already compressed).  I
documented all this in a paper I sent with the tape, overnight to the
recovery vendor.

The vendor threw away the instructions, loaded up the tape, said "Oh, this
is compressed, so the data after the EOT mark is going to be unreadable
without the data before the EOT", and then sent the tape back via ground
service.  They then call us back, with the news that the data is
unrecoverable.

I tell them that, as I said in the instructions, it's compressed on a file
by file basis, so only the file that the EOT is written in the middle of
will be lost (and anything before it, obviously), and we just need them to
stream that data off to another tape.  "We already sent the tape back to
you, you'll have to send it back to us."  As I said, they sent it back
ground, so we had to wait the better part of another week to get the tape,
then overnight it back to them AGAIN.

At that point, it was a fairly straightforward job to look for the next
occurrence of the file header marks, find the ones that really matched the
data, and then extract it.  I want to say it was a few Python programs,
each less than a hundred or two hundred lines.

So, in general I wasn't very impressed with the service, because of
shipping the tape back ground, but they did get the data we needed off the
tape.  DDS is a wonderful media, but not if you need data after the EOT.

Sean
-- 
 I like to be different, so I built a lowercase a-frame house.
                 -- Sean Reifschneider, 2000
Sean Reifschneider, Member of Technical Staff <jafo at tummy.com>
tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability