weird md error or undetected bad sector

[DraconPern]

Just got a very interesting error that I couldn't recover from.  It looks like md device was unable to heal, but bad sector count also didn't show anything.  Clean smart tests.  DSM 7.1-42661 Update 4 running on a Thecus NAS with WD Red in a raid 5 config.  Volume fall back to readonly on reboot.

 

2023-12-05T04:13:19-06:00 thecus kernel: [  105.617036] BTRFS critical (device dm-0): corrupt leaf: block=17324287705088 slot=68 extent bytenr=17301599223808 len=16384 unknown inline ref type: 144
2023-12-05T04:13:19-06:00 thecus kernel: [  105.630685] md2: [Self Heal] Retry sector [1099922016] round [1/2] start: sh-sector [137490272], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null]
2023-12-05T04:13:19-06:00 thecus kernel: [  105.643940] md2: [Self Heal] Retry sector [1099922024] round [1/2] start: sh-sector [137490280], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null]
2023-12-05T04:13:19-06:00 thecus kernel: [  105.657181] md2: [Self Heal] Retry sector [1099922032] round [1/2] start: sh-sector [137490288], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null]
2023-12-05T04:13:19-06:00 thecus kernel: [  105.670415] md2: [Self Heal] Retry sector [1099922040] round [1/2] start: sh-sector [137490296], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null]
2023-12-05T04:13:19-06:00 thecus kernel: [  105.683658] md2: [Self Heal] Retry sector [1099922016] round [1/2] choose d-disk
2023-12-05T04:13:19-06:00 thecus kernel: [  105.691054] md2: [Self Heal] Retry sector [1099922016] round [1/2] finished: return result to upper layer
2023-12-05T04:13:19-06:00 thecus kernel: [  105.700608] md2: [Self Heal] Retry sector [1099922024] round [1/2] choose d-disk
2023-12-05T04:13:19-06:00 thecus kernel: [  105.707992] md2: [Self Heal] Retry sector [1099922024] round [1/2] finished: return result to upper layer
2023-12-05T04:13:19-06:00 thecus kernel: [  105.717543] md2: [Self Heal] Retry sector [1099922032] round [1/2] choose d-disk
2023-12-05T04:13:19-06:00 thecus kernel: [  105.724926] md2: [Self Heal] Retry sector [1099922032] round [1/2] finished: return result to upper layer
2023-12-05T04:13:19-06:00 thecus kernel: [  105.734473] md2: [Self Heal] Retry sector [1099922040] round [1/2] choose d-disk
2023-12-05T04:13:19-06:00 thecus kernel: [  105.741857] md2: [Self Heal] Retry sector [1099922040] round [1/2] finished: return result to upper layer
2023-12-05T04:13:19-06:00 thecus kernel: [  105.751428] BTRFS critical (device dm-0): corrupt leaf: block=17324287705088 slot=68 extent bytenr=17301599223808 len=16384 unknown inline ref type: 144
2023-12-05T04:13:19-06:00 thecus kernel: [  105.765106] md2: [Self Heal] Retry sector [1099922016] round [1/2] start: sh-sector [137490272], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null]
2023-12-05T04:13:19-06:00 thecus kernel: [  105.778394] md2: [Self Heal] Retry sector [1099922024] round [1/2] start: sh-sector [137490280], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null]
2023-12-05T04:13:19-06:00 thecus kernel: [  105.791642] md2: [Self Heal] Retry sector [1099922032] round [1/2] start: sh-sector [137490288], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null]
2023-12-05T04:13:19-06:00 thecus kernel: [  105.804912] md2: [Self Heal] Retry sector [1099922040] round [1/2] start: sh-sector [137490296], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null]
2023-12-05T04:13:19-06:00 thecus kernel: [  105.848248] md2: [Self Heal] Retry sector [1099922016] round [1/2] choose d-disk
2023-12-05T04:13:19-06:00 thecus kernel: [  105.855666] md2: [Self Heal] Retry sector [1099922016] round [1/2] finished: get same result, retry next round
2023-12-05T04:13:19-06:00 thecus kernel: [  105.865663] md2: [Self Heal] Retry sector [1099922024] round [1/2] choose d-disk
2023-12-05T04:13:19-06:00 thecus kernel: [  105.873059] md2: [Self Heal] Retry sector [1099922024] round [1/2] finished: get same result, retry next round
2023-12-05T04:13:19-06:00 thecus kernel: [  105.883048] md2: [Self Heal] Retry sector [1099922032] round [1/2] choose d-disk
2023-12-05T04:13:19-06:00 thecus kernel: [  105.890448] md2: [Self Heal] Retry sector [1099922032] round [1/2] finished: get same result, retry next round
2023-12-05T04:13:19-06:00 thecus kernel: [  105.900442] md2: [Self Heal] Retry sector [1099922040] round [1/2] choose d-disk
2023-12-05T04:13:19-06:00 thecus kernel: [  105.907832] md2: [Self Heal] Retry sector [1099922040] round [1/2] finished: get same result, retry next round
2023-12-05T04:13:19-06:00 thecus kernel: [  105.917828] md2: [Self Heal] Retry sector [1099922040] round [2/2] start: sh-sector [137490296], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null]
2023-12-05T04:13:19-06:00 thecus kernel: [  105.931036] md2: [Self Heal] Retry sector [1099922032] round [2/2] start: sh-sector [137490288], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null]
2023-12-05T04:13:19-06:00 thecus kernel: [  105.944240] md2: [Self Heal] Retry sector [1099922024] round [2/2] start: sh-sector [137490280], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null]
2023-12-05T04:13:19-06:00 thecus kernel: [  105.957466] md2: [Self Heal] Retry sector [1099922016] round [2/2] start: sh-sector [137490272], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null]
2023-12-05T04:13:20-06:00 thecus kernel: [  105.984666] md2: [Self Heal] Retry sector [1099922040] round [2/2] choose p-disk
2023-12-05T04:13:20-06:00 thecus kernel: [  105.992065] md2: [Self Heal] Retry sector [1099922040] round [2/2] finished: get same result, give up
2023-12-05T04:13:20-06:00 thecus kernel: [  106.001269] md2: [Self Heal] Retry sector [1099922032] round [2/2] choose p-disk
2023-12-05T04:13:20-06:00 thecus kernel: [  106.008669] md2: [Self Heal] Retry sector [1099922032] round [2/2] finished: get same result, give up
2023-12-05T04:13:20-06:00 thecus kernel: [  106.017894] md2: [Self Heal] Retry sector [1099922024] round [2/2] choose p-disk
2023-12-05T04:13:20-06:00 thecus kernel: [  106.025279] md2: [Self Heal] Retry sector [1099922024] round [2/2] finished: get same result, give up
2023-12-05T04:13:20-06:00 thecus kernel: [  106.034491] md2: [Self Heal] Retry sector [1099922016] round [2/2] choose p-disk
2023-12-05T04:13:20-06:00 thecus kernel: [  106.041890] md2: [Self Heal] Retry sector [1099922016] round [2/2] finished: get same result, give up
2023-12-05T04:13:20-06:00 thecus kernel: [  106.051146] BTRFS critical (device dm-0): corrupt leaf: block=17324287705088 slot=68 extent bytenr=17301599223808 len=16384 unknown inline ref type: 144
2023-12-05T04:13:20-06:00 thecus kernel: [  106.064802] ------------[ cut here ]------------
2023-12-05T04:13:20-06:00 thecus kernel: [  106.069428] WARNING: CPU: 0 PID: 17296 at fs/btrfs/disk-io.c:916 btree_io_failed_hook+0x12d/0x220 [btrfs]()
2023-12-05T04:13:20-06:00 thecus kernel: [  106.079167] Modules linked in: fuse 8021q ocs_fc_lio(PO) tcm_qla2xxx(O) qla2xxx(O) libfc scsi_transport_fc vhost_scsi(O) vhost(O) tcm_loop(O) iscsi_target_mod(O) target_core_user(O) target_core_ep(O) target_core_multi_file(O) target_core_file(O) target_core_iblock(O) target_core_mod(O) syno_extent_pool(PO) rodsp_ep(O) vfat fat udf isofs synoacl_vfs(PO) btrfs ecryptfs zstd_decompress zstd_compress xxhash raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq adt7475 zram aesni_intel glue_helper lrw gf128mul ablk_helper broadwellnk_synobios(PO) hid_generic usbhid hid usblp uhci_hcd etxhci_hcd bnxt_en(O) bnx2x(O) mdio qede(O) qed(O) mlx5_core(O) mlx4_en(O) mlx4_core(O) mlx_compat(O) atlantic_v2(O) atlantic(O) r8168(O) tn40xx(O) i40e(O) ixgbe(O) be2net(O) i2c_algo_bit igb(O)
2023-12-05T04:13:20-06:00 thecus kernel: [  106.150446]  dca sg dm_snapshot dm_bufio crc_itu_t crc_ccitt psnap p8022 llc hfsplus md4 hmac sit tunnel4 ipv6 flashcache_syno(O) flashcache(O) syno_flashcache_control(O) dm_mod arc4 crc32c_intel cryptd ecb aes_x86_64 authenc des_generic ansi_cprng cts md5 cbc cpufreq_powersave cpufreq_performance processor cpufreq_stats vxlan ip6_udp_tunnel udp_tunnel ip_tunnel loop sha256_generic synorbd(O) synofsbd(O) e1000e(OE) usb_storage xhci_pci xhci_hcd ehci_pci ehci_hcd usbcore usb_common mv14xx(O) [last unloaded: adt7475]
2023-12-05T04:13:20-06:00 thecus kernel: [  106.195440] CPU: 0 PID: 17296 Comm: kworker/u16:13 Tainted: P           OE   4.4.180+ #42661
2023-12-05T04:13:20-06:00 thecus kernel: [  106.203857] Hardware name: Intel Corporation SugarBay Platform/SCPT_CRB, BIOS TBML_T06 03/12/2012
2023-12-05T04:13:20-06:00 thecus kernel: [  106.212722] Workqueue: btrfs-endio-meta btrfs_endio_meta_helper [btrfs]
2023-12-05T04:13:20-06:00 thecus kernel: [  106.219335]  0000000000000000 ffff880629da3c20 ffffffff812dca8b 0000000000000009
2023-12-05T04:13:20-06:00 thecus kernel: [  106.226767]  0000000000000000 ffff880629da3c58 ffffffff81050f82 ffff8806287f3040
2023-12-05T04:13:20-06:00 thecus kernel: [  106.234199]  0000000000001000 ffff880644b0a000 ffff8806440e0240 00000fc1a003c000
2023-12-05T04:13:20-06:00 thecus kernel: [  106.241632] Call Trace:
2023-12-05T04:13:20-06:00 thecus kernel: [  106.244073]  [<ffffffff812dca8b>] dump_stack+0x4d/0x72
2023-12-05T04:13:20-06:00 thecus kernel: [  106.249199]  [<ffffffff81050f82>] warn_slowpath_common+0x82/0xa0
2023-12-05T04:13:20-06:00 thecus kernel: [  106.255189]  [<ffffffff81051055>] warn_slowpath_null+0x15/0x20
2023-12-05T04:13:20-06:00 thecus kernel: [  106.261018]  [<ffffffffa0cd2a6d>] btree_io_failed_hook+0x12d/0x220 [btrfs]
2023-12-05T04:13:20-06:00 thecus kernel: [  106.267887]  [<ffffffffa0d055ba>] end_bio_extent_readpage+0x18a/0x910 [btrfs]
2023-12-05T04:13:20-06:00 thecus kernel: [  106.275009]  [<ffffffff812f0615>] ? find_next_bit+0x15/0x20
2023-12-05T04:13:20-06:00 thecus kernel: [  106.280574]  [<ffffffff812b3449>] bio_endio+0xa9/0x120
2023-12-05T04:13:20-06:00 thecus kernel: [  106.285710]  [<ffffffffa0cd1687>] end_workqueue_fn+0x27/0x40 [btrfs]
2023-12-05T04:13:20-06:00 thecus kernel: [  106.292063]  [<ffffffffa0d15466>] btrfs_worker_helper+0xc6/0x390 [btrfs]
2023-12-05T04:13:20-06:00 thecus kernel: [  106.298750]  [<ffffffff8107c935>] ? sched_clock_cpu+0xa5/0xb0
2023-12-05T04:13:20-06:00 thecus kernel: [  106.304492]  [<ffffffffa0d15799>] btrfs_endio_meta_helper+0x9/0x10 [btrfs]
2023-12-05T04:13:20-06:00 thecus kernel: [  106.311352]  [<ffffffff8107130a>] worker_run_work+0x9a/0xe0
2023-12-05T04:13:20-06:00 thecus kernel: [  106.316920]  [<ffffffffa0d15790>] ? btrfs_endio_helper+0x10/0x10 [btrfs]
2023-12-05T04:13:20-06:00 thecus kernel: [  106.323606]  [<ffffffff810692bb>] process_one_work+0x1db/0x4e0
2023-12-05T04:13:20-06:00 thecus kernel: [  106.329426]  [<ffffffff810695ed>] worker_thread+0x2d/0x4a0
2023-12-05T04:13:20-06:00 thecus kernel: [  106.334899]  [<ffffffff810695c0>] ? process_one_work+0x4e0/0x4e0
2023-12-05T04:13:20-06:00 thecus kernel: [  106.340890]  [<ffffffff8106d9a3>] kthread+0xd3/0xf0
2023-12-05T04:13:20-06:00 thecus kernel: [  106.345757]  [<ffffffff8106d8d0>] ? kthread_worker_fn+0x160/0x160
2023-12-05T04:13:20-06:00 thecus kernel: [  106.351839]  [<ffffffff81563baf>] ret_from_fork+0x3f/0x80
2023-12-05T04:13:20-06:00 thecus kernel: [  106.357227]  [<ffffffff8106d8d0>] ? kthread_worker_fn+0x160/0x160
2023-12-05T04:13:20-06:00 thecus kernel: [  106.363335] ---[ end trace 8daff8e233f959d2 ]---