DraconPern

Just got a very interesting error that I couldn't recover from. It looks like md device was unable to heal, but bad sector count also didn't show anything. Clean smart tests. DSM 7.1-42661 Update 4 running on a Thecus NAS with WD Red in a raid 5 config. Volume fall back to readonly on reboot. 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.617036] BTRFS critical (device dm-0): corrupt leaf: block=17324287705088 slot=68 extent bytenr=17301599223808 len=16384 unknown inline ref type: 144 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.630685] md2: [Self Heal] Retry sector [1099922016] round [1/2] start: sh-sector [137490272], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null] 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.643940] md2: [Self Heal] Retry sector [1099922024] round [1/2] start: sh-sector [137490280], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null] 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.657181] md2: [Self Heal] Retry sector [1099922032] round [1/2] start: sh-sector [137490288], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null] 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.670415] md2: [Self Heal] Retry sector [1099922040] round [1/2] start: sh-sector [137490296], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null] 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.683658] md2: [Self Heal] Retry sector [1099922016] round [1/2] choose d-disk 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.691054] md2: [Self Heal] Retry sector [1099922016] round [1/2] finished: return result to upper layer 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.700608] md2: [Self Heal] Retry sector [1099922024] round [1/2] choose d-disk 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.707992] md2: [Self Heal] Retry sector [1099922024] round [1/2] finished: return result to upper layer 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.717543] md2: [Self Heal] Retry sector [1099922032] round [1/2] choose d-disk 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.724926] md2: [Self Heal] Retry sector [1099922032] round [1/2] finished: return result to upper layer 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.734473] md2: [Self Heal] Retry sector [1099922040] round [1/2] choose d-disk 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.741857] md2: [Self Heal] Retry sector [1099922040] round [1/2] finished: return result to upper layer 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.751428] BTRFS critical (device dm-0): corrupt leaf: block=17324287705088 slot=68 extent bytenr=17301599223808 len=16384 unknown inline ref type: 144 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.765106] md2: [Self Heal] Retry sector [1099922016] round [1/2] start: sh-sector [137490272], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null] 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.778394] md2: [Self Heal] Retry sector [1099922024] round [1/2] start: sh-sector [137490280], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null] 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.791642] md2: [Self Heal] Retry sector [1099922032] round [1/2] start: sh-sector [137490288], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null] 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.804912] md2: [Self Heal] Retry sector [1099922040] round [1/2] start: sh-sector [137490296], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null] 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.848248] md2: [Self Heal] Retry sector [1099922016] round [1/2] choose d-disk 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.855666] md2: [Self Heal] Retry sector [1099922016] round [1/2] finished: get same result, retry next round 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.865663] md2: [Self Heal] Retry sector [1099922024] round [1/2] choose d-disk 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.873059] md2: [Self Heal] Retry sector [1099922024] round [1/2] finished: get same result, retry next round 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.883048] md2: [Self Heal] Retry sector [1099922032] round [1/2] choose d-disk 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.890448] md2: [Self Heal] Retry sector [1099922032] round [1/2] finished: get same result, retry next round 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.900442] md2: [Self Heal] Retry sector [1099922040] round [1/2] choose d-disk 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.907832] md2: [Self Heal] Retry sector [1099922040] round [1/2] finished: get same result, retry next round 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.917828] md2: [Self Heal] Retry sector [1099922040] round [2/2] start: sh-sector [137490296], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null] 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.931036] md2: [Self Heal] Retry sector [1099922032] round [2/2] start: sh-sector [137490288], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null] 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.944240] md2: [Self Heal] Retry sector [1099922024] round [2/2] start: sh-sector [137490280], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null] 2023-12-05T04:13:19-06:00 thecus kernel: [ 105.957466] md2: [Self Heal] Retry sector [1099922016] round [2/2] start: sh-sector [137490272], d-disk [3:sdc3], p-disk [7:sdh3], q-disk [-1:null] 2023-12-05T04:13:20-06:00 thecus kernel: [ 105.984666] md2: [Self Heal] Retry sector [1099922040] round [2/2] choose p-disk 2023-12-05T04:13:20-06:00 thecus kernel: [ 105.992065] md2: [Self Heal] Retry sector [1099922040] round [2/2] finished: get same result, give up 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.001269] md2: [Self Heal] Retry sector [1099922032] round [2/2] choose p-disk 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.008669] md2: [Self Heal] Retry sector [1099922032] round [2/2] finished: get same result, give up 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.017894] md2: [Self Heal] Retry sector [1099922024] round [2/2] choose p-disk 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.025279] md2: [Self Heal] Retry sector [1099922024] round [2/2] finished: get same result, give up 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.034491] md2: [Self Heal] Retry sector [1099922016] round [2/2] choose p-disk 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.041890] md2: [Self Heal] Retry sector [1099922016] round [2/2] finished: get same result, give up 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.051146] BTRFS critical (device dm-0): corrupt leaf: block=17324287705088 slot=68 extent bytenr=17301599223808 len=16384 unknown inline ref type: 144 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.064802] ------------[ cut here ]------------ 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.069428] WARNING: CPU: 0 PID: 17296 at fs/btrfs/disk-io.c:916 btree_io_failed_hook+0x12d/0x220 [btrfs]() 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.079167] Modules linked in: fuse 8021q ocs_fc_lio(PO) tcm_qla2xxx(O) qla2xxx(O) libfc scsi_transport_fc vhost_scsi(O) vhost(O) tcm_loop(O) iscsi_target_mod(O) target_core_user(O) target_core_ep(O) target_core_multi_file(O) target_core_file(O) target_core_iblock(O) target_core_mod(O) syno_extent_pool(PO) rodsp_ep(O) vfat fat udf isofs synoacl_vfs(PO) btrfs ecryptfs zstd_decompress zstd_compress xxhash raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq adt7475 zram aesni_intel glue_helper lrw gf128mul ablk_helper broadwellnk_synobios(PO) hid_generic usbhid hid usblp uhci_hcd etxhci_hcd bnxt_en(O) bnx2x(O) mdio qede(O) qed(O) mlx5_core(O) mlx4_en(O) mlx4_core(O) mlx_compat(O) atlantic_v2(O) atlantic(O) r8168(O) tn40xx(O) i40e(O) ixgbe(O) be2net(O) i2c_algo_bit igb(O) 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.150446] dca sg dm_snapshot dm_bufio crc_itu_t crc_ccitt psnap p8022 llc hfsplus md4 hmac sit tunnel4 ipv6 flashcache_syno(O) flashcache(O) syno_flashcache_control(O) dm_mod arc4 crc32c_intel cryptd ecb aes_x86_64 authenc des_generic ansi_cprng cts md5 cbc cpufreq_powersave cpufreq_performance processor cpufreq_stats vxlan ip6_udp_tunnel udp_tunnel ip_tunnel loop sha256_generic synorbd(O) synofsbd(O) e1000e(OE) usb_storage xhci_pci xhci_hcd ehci_pci ehci_hcd usbcore usb_common mv14xx(O) [last unloaded: adt7475] 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.195440] CPU: 0 PID: 17296 Comm: kworker/u16:13 Tainted: P OE 4.4.180+ #42661 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.203857] Hardware name: Intel Corporation SugarBay Platform/SCPT_CRB, BIOS TBML_T06 03/12/2012 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.212722] Workqueue: btrfs-endio-meta btrfs_endio_meta_helper [btrfs] 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.219335] 0000000000000000 ffff880629da3c20 ffffffff812dca8b 0000000000000009 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.226767] 0000000000000000 ffff880629da3c58 ffffffff81050f82 ffff8806287f3040 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.234199] 0000000000001000 ffff880644b0a000 ffff8806440e0240 00000fc1a003c000 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.241632] Call Trace: 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.244073] [<ffffffff812dca8b>] dump_stack+0x4d/0x72 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.249199] [<ffffffff81050f82>] warn_slowpath_common+0x82/0xa0 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.255189] [<ffffffff81051055>] warn_slowpath_null+0x15/0x20 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.261018] [<ffffffffa0cd2a6d>] btree_io_failed_hook+0x12d/0x220 [btrfs] 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.267887] [<ffffffffa0d055ba>] end_bio_extent_readpage+0x18a/0x910 [btrfs] 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.275009] [<ffffffff812f0615>] ? find_next_bit+0x15/0x20 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.280574] [<ffffffff812b3449>] bio_endio+0xa9/0x120 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.285710] [<ffffffffa0cd1687>] end_workqueue_fn+0x27/0x40 [btrfs] 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.292063] [<ffffffffa0d15466>] btrfs_worker_helper+0xc6/0x390 [btrfs] 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.298750] [<ffffffff8107c935>] ? sched_clock_cpu+0xa5/0xb0 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.304492] [<ffffffffa0d15799>] btrfs_endio_meta_helper+0x9/0x10 [btrfs] 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.311352] [<ffffffff8107130a>] worker_run_work+0x9a/0xe0 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.316920] [<ffffffffa0d15790>] ? btrfs_endio_helper+0x10/0x10 [btrfs] 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.323606] [<ffffffff810692bb>] process_one_work+0x1db/0x4e0 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.329426] [<ffffffff810695ed>] worker_thread+0x2d/0x4a0 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.334899] [<ffffffff810695c0>] ? process_one_work+0x4e0/0x4e0 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.340890] [<ffffffff8106d9a3>] kthread+0xd3/0xf0 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.345757] [<ffffffff8106d8d0>] ? kthread_worker_fn+0x160/0x160 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.351839] [<ffffffff81563baf>] ret_from_fork+0x3f/0x80 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.357227] [<ffffffff8106d8d0>] ? kthread_worker_fn+0x160/0x160 2023-12-05T04:13:20-06:00 thecus kernel: [ 106.363335] ---[ end trace 8daff8e233f959d2 ]---

Your etc/synoinfo.conf has a bad edit somewhere, may be changed encoding? Notice it says failed to patch.

Have you tried loading w/o any disks? If your loader is really working you should at least get ip and ping. Since you have at least one usb bootloader that works, you can boot it and enable ssh. Then login via ssh and then mount the working grub.cfg file to copy the working vid/pid/mac1. May be the usb stick is having problems? When I used rufus, I had to manually delete partitions off the usb stick for a write to be successful. sudo mount -o rw,offset=1048576 /dev/synoboot /mnt Some thoughts to try. Have you tried 1.03b and ds3615 image? Since your hardware is not that new that combo may work better instead of the ds3617 image.

You should probably backup whatever data first by manually using mdadm and mounting. Then do a clean install. Each time you change the bootloader/disk combination, it usually detects it as needing repair.

I was able to install DSM 6.2.1-23824 Update 4 using the 1.03b loader (ds3615sx) successfully. bootloader options and mods see my other post https://xpenology.com/forum/topic/13914-thecus-n10850-mods/

On DSM 6.2.1-23824 Update 4, I used the following settings for proper port mapping Note that you can edit the grub.cfg on the usb stick after installation through ssh and vi using sudo mount -o rw,offset=1048576 /dev/synoboot /mnt sudo vi /mnt/grub/grub.cfg edit your bootloader grub.cfg extra_args_3615='SataPortMap=22222 DiskIdxMap=0002040608' For /etc.defaults/synoinfo.conf I changed the following maxdisks="10" internalportcfg="0xffff" esataportcfg="0x0000"