Well, two steps forwards, one step back. The scrub I ran yesterday at least showed some errors, but I’m having trouble identifying exactly what is the actual problem. I think I’ll sleep on it and form a new plan in the morning.

Controller failure? RAM failure? Dmesg shows absolutely nothing, no panics no anything so I’m not thinking it’s ram. Hmmmm… maybe I’ll run mtest after I get some sleep.

3-2-1-backup@BackupServer:~$ sudo zpool status -vx
pool: data_pool3
state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
 see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
 scan: scrub repaired 40K in 07:07:07 with 4 errors on Tue Nov 28 22:39:33 2023
config:

    NAME                        STATE     READ WRITE CKSUM
    data_pool3                  ONLINE       0     0     0
      raidz2-0                  ONLINE       0     0     0
        wwn-0x5000ccax1  ONLINE       0     0     8
        wwn-0x5000ccax2 ONLINE       0     0    10
        wwn-0x5000ccax3 ONLINE       0     0     8
        wwn-0x5000ccax4 ONLINE       0     0     8
        wwn-0x5000ccax5 ONLINE       0     0     8
        wwn-0x5000ccax6 ONLINE       0     0     8
        wwn-0x5000ccax7 ONLINE       0     0     8
        wwn-0x5000ccax8 ONLINE       0     0     8

errors: Permanent errors have been detected in the following files:

    data_pool3/(redacted)/downloads@backup_script-2023-11-28-0901:/(redacted).mkv
    data_pool3/(redacted)@backup_script-2023-11-28-2001:/ISOs/Ubuntu/23.10/ubuntu-23.10.1-desktop-amd64.iso
    data_pool3/(redacted)@backup_script-2023-11-07-0901:/(redacted).mkv

Hey wow, even though my problem is getting worse (maybe), an actual honest-to-god ISO showed up in the problem file list!

This is my backup server, so no. Primary does.

zfs --version also does the trick.