Question on ZFS

[Bob Proulx]

Phil Marsh wrote:
> Hi Bob, All,
> I was wondering. Do you recommend using an SSD cache for ZFS, i.e. an L2ARC
> cache?

I was rather hoping Zak would have jumped in with a response on this
one as I know Zak is running several large high performance arrays.
But not having heard anything I will try to muddle through. :-)

> I have 512GB on my main ZFS server and I use some nvme SSD drives for my
> virtual machines and chroots. I'm thinking that because ZFS now supports
> persistent L2ARC that it makes more sense to convert the SSDs to L2ARC and
> I could then eliminate the need to mirror these SSDs.

Maybe.  I don't see anything wrong with the idea.  The theory goes
that NVMe is very fast and the L2ARC would be a persistent read cache.
Adding a separate L2ARC that is faster than the zfs array disk
collection should give faster performance than the array.  And it
should be a very safe addition. ZFS knows when data is changed and
ensures that the read cache is synchronized.  It's internal to ZFS.

Probably the best answer is to try it and see.  Try a few benchmarks
before and after and see what the performance difference is between
the two configurations.  I don't see any reason not to do it.

> I'd transfer all my chroots and virtual machines to datasets in my main
> pool and use the freed-up SSDs for L2ARC drives. No mirroring = extra
> space. The only downside I see is that an initial use might result in the
> L2ARC needing to load in the VMs if they haven't been used recently.

I don't think that is any different than not having it.  When things
start out the data will need to be read from storage regardless.  A
cold start won't be any worse with an l2arc than without an l2arc.

> I think my L2ARC would start out at 2TB between 2 nvme drives and I
> could likely go up.

Give it a shot.  Give us a report after you have tried it.

> I'm already running a SLOG between two Intel Optane 59GB nvmes.
> What do you think?

For the SLOG separate write intent log that's a little more risky
because the system is more fragile during the time that data is only
in the SLOG ZIL write intent log and in flight to the data array.  I
would only do that with a good UPS and/or a battery backed controller.
If it loses power while the intent log is not yet consistent then it
can cause corruption that would need to be scrubbed out later.

But as far as I know adding an L2ARC as you propose does not have any
risky downside.  Give it a try.

Bob