Machine IO slowing over time

[Zak Smith]

Hi,

I had a situation, actually on a Unifi appliance, where the raid
performance was dropping enough to cause it to be unfit for purpose,
but not enough to cause system errors, or even log anything in dmesg.
I was able to identify the single disk that was causing problems
because "smartctl -a" would hang for 5-10 seconds when run on *that*
drive, even though the output itself did not (at that time) show any
indicators of failure.



On Thu, Feb 05, 2026 at 08:01:39AM -0700, Grant Johnson wrote:
> Based on that idea, since it is LVM across 2 RAID 1 arrays, I moved one
> of my large partitions that seems to be the most trouble from the array
> that was getting very high waits according to iostat to the other one
> that is having less io wait.   We will see tonight if the daily
> snapshots run any smoother not that I moved the bkup volume group to
> the array with less trouble.
>
> Maybe those disks are faster, maybe the freshening of the data will
> help.   Not sure, but good ideas from all of you sent me at least in a
> new direction that I have not tried before.
>
>
> On Tue, 2026-02-03 at 15:43 -0700, Bob Proulx wrote:
> > Grant Johnson wrote:
> > > What else should I be checking?
> >
> > Sometimes reading data from drives will start to take longer when the
> > data on the drives is very static over years.  In those cases the
> > data
> > might degrade where more error correction from the drive is needed to
> > read the data.  And the drive might have to retry reading those disk
> > sectors more times to get a good read.
> >
> > This can be "fluffed" by reading and writing the entire drive.  I
> > have
> > seen some drives have dramatic improvement.  I have seen other drives
> > have no change at all.
> >
> > The most direct way is to boot a live boot image so that nothing is
> > touching the disk and then read and write the entire disk in place.
> >
> >     time dd if=/dev/sdX of=/dev/sdX bs=1M iflag=fullblock
> > oflag=sync,direct
> >
> > Of course doing so can take a long time on a big drive.  I like to
> > see
> > progress.  The utility I like the best for this is "pv" pipe-viewer.
> > If you have a new enough utility then it will have the --direct-io
> > option.  But if not then use dd.
> >
> >     time pv /dev/sdX | dd of=/dev/sdX bs=1M iflag=fullblock
> > oflag=sync,direct
> >
> > The pv utility can even be used as the reader and writer itself when
> > using the --direct-io option.  But no control of block size so dd is
> > still useful.
> >
> > Since the data being written is the data being read this is safe but
> > only if nothing else is touching the drive at the same time.
> >
> > Bob
> >

--
Zak Smith
307-543-7820 office
Please do not send private or confidential information via email.