[NCLUG] sw raid, recovery after install

[Bob Proulx]

Stephen Warren wrote:
> Bob Proulx wrote:
> > It is available but you have to enable it.  Something like:
> >   mdadm /dev/md1 --grow --bitmap=internal
> > Then you will see the additional bitmap line displayed in /proc/mdstat.
> >   md2 : active raid1 sda6[0] sdb6[1]
> >         312496256 blocks [2/2] [UU]
> >         bitmap: 0/3 pages [0KB], 65536KB chunk
> 
> I'm pretty sure that's something else; I just did an install of Ubuntu
> 12.10 a little while back and saw the incremental RAID resync, but
> didn't ever use the --bitmap option, not do I see the bitmap line in
> /proc/mdstat. I do have "super 1.2", so perhaps this is a facet of
> version 1.2 of the superblock structure.

There certainly may be a new kernel feature.  Ubuntu 12.10 took a
kernel jump to 3.6.9 and may have a new feature well beyond what was
there before.  I don't know.  But I think it is independent of the 1.2
format.  Here is a different machine with the 1.2 format.

  root at havoc:~# cat /proc/mdstat
  Personalities : [raid1] 
  md3 : active raid1 sda7[0] sdb7[1]
        119379896 blocks super 1.2 [2/2] [UU]
        bitmap: 0/1 pages [0KB], 65536KB chunk

  md2 : active raid1 sda6[0] sdb6[1]
        244136824 blocks super 1.2 [2/2] [UU]
        bitmap: 0/2 pages [0KB], 65536KB chunk

  md1 : active raid1 sda5[2] sdb5[3]
        77648824 blocks super 1.2 [2/2] [UU]
        bitmap: 1/1 pages [4KB], 65536KB chunk

  md0 : active raid1 sda1[2] sdb1[3]
        498676 blocks super 1.2 [2/2] [UU]
        
  unused devices: <none>

The sizes above reflect that I have increased the size of the disks on
that machine twice now.  Being raid meant that I could replace one
drive at a time to a larger size and sync.  There is very little
downtime that way.  Then when both drives are upgraded to larger
versions I can create a new partition and grow the LVM with the new
physical extents.  Works well.

> > How often are you seeing "some kind of weird RAID initialization issue
> > on reboot" that causes an array to hard fail degrade to one disk?
> > That actually sounds pretty scary.  I'm not seeing those.
> 
> I've never seen it.

Oh good.  I was worried for your system! :-)

> However, RAID is all about covering your bases, about introducing
> safety measures to protect you when something goes wrong. Making the
> safety measures as simple as possible, and as unlikely as possible
> to fail in painful ways, sure seems like a good idea.

Yes.  And it was on that basis that shared that I split the physical
volumes up so that I have smaller chunks to sync.  I am the first to
say that it is my personal preference.  It certainly isn't wrong to do
it other ways.  But in my opinion having smaller 250-300G chunks is
safer than having one large 2T chunk.

> W.r.t. drive failures, a slightly-flakey-during-boot drive (which is
> what might cause such a failure scenario) sure seems like it could
> easily be a failure mode.

A single flakey drive could produce a failure causing it to kick out
of the raid.  That is probably fairly likely.  But in order to have a
criss-cross mismatched set of disks with part of the remainder on one
and part on the other then it would take *two* flakey drives during
boot and each would need to flake out on different partitions.
Possible.  But unlikely.  (Remember that I try to make sure I don't
have two identical drives.)

If there was a frequent kicking of one specific drive out then I would
take that as an indication of failure or flakiness of that one drive
and would queue it for replacement.

I could see the newer "green" drives causing these problems.  They are
designed to power save.  They do power save.  They go to sleep.  And
then when they get woken up with a command sometimes it takes a while
for them to wake up and respond.  If they are a single disk in the
system then the kernel has no other choice than to retry until
something responds.  Especially because they aren't producing errors
but just taking a long time.  So they are okay as a single disk in a
system.

But if green drives are in a raid array and have gone to sleep and are
taking a long time to respond then the raid controller may think they
have timed out and will kick a green drive out of the array.  I have
avoided them for raid but I have read many accounts of why green
drives are bad for raid.  The one green drive I have used as a single
was so very slow that the raid timeout problem seemed very plausible.

> For reference, I've only had a drive in a RAID array (or anywhere)
> fail once, perhaps twice, out of the last 9+8+5 years (3 arrays) of
> running a SW RAID array either, so 0 and 1-or-2 are fairly
> comparable:-) One of the failure modes was definitely along the
> lines of sometimes the drive would work OK sometimes not.

Sure.  A flakey drive can definitely cause that disk to be kicked out
of the array.  Either once or frequently.  I would probably sync it
back into the raid if it happened once or twice.  But if it happened
too often I would tag that as a problem with that drive and replace
it.  Or a problem with the cable.  Or SATA port.  Cables are sometimes
flakey at a higher rate than makes sense.

But I still think it would take *both* drives being flakey to get them
into a mode where the array was mismatched criss-crossed across the
two disks.

Bob