Subtixx Posted January 24, 2021 Share #1 Posted January 24, 2021 (edited) So after updating some packages in my docker iobroker container I got a notification that my volume has crashed. This is weird since all HDDs show as normal but the volume shows crashed? EDIT: I can still access the files on there (I can see all folders etc) I also tried opening a mp4 file I have on there and it was playing fine. However becuase I am to afraid to break more I didn't check any other files only this one mp4... My first question is: how can this happen? What should I do now? Obviously I cannot transfer all files to an external drive because I'd need over 7TB of drives which I obviously don't have. output of 'cat /proc/mdstat' (no idea what all that means :S) Quote Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [raidF1] md2 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1] 11706589632 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU] md3 : active raid0 sde1[0] 244193984 blocks super 1.2 64k chunks [1/1] [U] md1 : active raid1 sda2[0] sdb2[1] sdc2[2] sdd2[3] 2097088 blocks [12/4] [UUUU________] md0 : active raid1 sda1[0] sdb1[1] sdc1[2] sdd1[3] 2490176 blocks [12/4] [UUUU________] unused devices: <none> output of 'vgdisplay': Quote --- Volume group --- VG Name vg1 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 5 VG Access read/write VG Status resizable MAX LV 0 Cur LV 2 Open LV 1 Max PV 0 Cur PV 1 Act PV 1 VG Size 10.90 TiB PE Size 4.00 MiB Total PE 2858053 Alloc PE / Size 2857987 / 10.90 TiB Free PE / Size 66 / 264.00 MiB VG UUID PX1920-Xorr-2zmD-Ua48-QqkI-MHlh-AnEZvO output of 'vgdisplay --verbose' Quote Using volume group(s) on command line. --- Volume group --- VG Name vg1 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 5 VG Access read/write VG Status resizable MAX LV 0 Cur LV 2 Open LV 1 Max PV 0 Cur PV 1 Act PV 1 VG Size 10.90 TiB PE Size 4.00 MiB Total PE 2858053 Alloc PE / Size 2857987 / 10.90 TiB Free PE / Size 66 / 264.00 MiB VG UUID PX1920-Xorr-2zmD-Ua48-QqkI-MHlh-AnEZvO --- Logical volume --- LV Path /dev/vg1/syno_vg_reserved_area LV Name syno_vg_reserved_area VG Name vg1 LV UUID oi96k9-z6Wv-rVKJ-i7FK-q0qp-qqNb-k00Nhw LV Write Access read/write LV Creation host, time , LV Status available # open 0 LV Size 12.00 MiB Current LE 3 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 768 Block device 253:0 --- Logical volume --- LV Path /dev/vg1/volume_1 LV Name volume_1 VG Name vg1 LV UUID fHa3p4-Ubwa-rQxy-DcPN-zp4e-2EyR-gTFg3Q LV Write Access read/write LV Creation host, time , LV Status available # open 1 LV Size 10.90 TiB Current LE 2857984 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 768 Block device 253:1 --- Physical volumes --- PV Name /dev/md2 PV UUID Lf7GbE-7Y6B-IMbz-NigR-Ekcu-KKQd-Pz1eG4 PV Status allocatable Total PE / Free PE 2858053 / 66 output of 'lvdisplay --verbose' Quote Using logical volume(s) on command line. --- Logical volume --- LV Path /dev/vg1/syno_vg_reserved_area LV Name syno_vg_reserved_area VG Name vg1 LV UUID oi96k9-z6Wv-rVKJ-i7FK-q0qp-qqNb-k00Nhw LV Write Access read/write LV Creation host, time , LV Status available # open 0 LV Size 12.00 MiB Current LE 3 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 768 Block device 253:0 --- Logical volume --- LV Path /dev/vg1/volume_1 LV Name volume_1 VG Name vg1 LV UUID fHa3p4-Ubwa-rQxy-DcPN-zp4e-2EyR-gTFg3Q LV Write Access read/write LV Creation host, time , LV Status available # open 1 LV Size 10.90 TiB Current LE 2857984 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 768 Block device 253:1 EDIT: After some looking through the folders I see that the "docker" folder seems empty dunno what else is missing though. Is this gone for good or is that just the filesystem being corrupt? If so is there any way I can restore those files? Edited January 24, 2021 by Subtixx Quote Link to comment Share on other sites More sharing options...
IG-88 Posted January 24, 2021 Share #2 Posted January 24, 2021 (edited) i's suggest the same as the dsm does, backup as long as you still have access, atm you might not need a recovery procedure it would be good to know in whats state the file system is and how it came to this to prevent is happening again but thats 2nd 1st priority are offloading the data asap if the disks, raid and volumes are intact then there might be damage in the btrfs system and if its self repair cant do the job then there might be problems beyond repair (i had that once when "experimenting" with ssd write cache combined with controller problems - ended up in deleting the volume and recreating it from backup) Edited January 24, 2021 by IG-88 1 Quote Link to comment Share on other sites More sharing options...
Subtixx Posted January 24, 2021 Author Share #3 Posted January 24, 2021 (edited) 10 minutes ago, IG-88 said: if the disks, raid and volumes are intact then there might be damage in the btrfs system and if its self repair cant do the job then there might be problems beyond repair Did it try to self-repair though? It didn't seem like it it just sent me an email "Volume crashed" and no self-repair failed message or anything like that.. The weird thing is the only thing that is showing any errors is the volume there are no protocols for the HDD for example. Btw backing up what I can with my limited HDD Space.. I'm probably going to lose 5TB of data now... Edited January 24, 2021 by Subtixx Quote Link to comment Share on other sites More sharing options...
IG-88 Posted January 24, 2021 Share #4 Posted January 24, 2021 (edited) 8 hours ago, Subtixx said: It didn't seem like it it just sent me an email "Volume crashed" and no self-repair failed message or anything like that.. you would need to have a look to the log files in /var/log/ but dsm usually schedules a check (scrubbing) oneth a month btrfs has some redundancy in itself, if thats broken you should offload as much as you can and should not write to the data volume maybe look up a btrfs file check that does not correct/repairs anything to know if there is a problem with the file system as long as you dont know what caused the problem you should assume the worst and backup the files Edited January 24, 2021 by IG-88 1 Quote Link to comment Share on other sites More sharing options...
Subtixx Posted January 24, 2021 Author Share #5 Posted January 24, 2021 Hmm okay. Can you tell me what I should check / execute via SSH when I'm finished copying my files? Btw strange thing is the docker folder shows empty in "File Station" but in WinSCP via SSH/SCP it does show and copy successfully Quote Link to comment Share on other sites More sharing options...
IG-88 Posted January 24, 2021 Share #6 Posted January 24, 2021 i'm not a expert on this, just done that once, as you are unmounting the volume in the process make sure you finished the backup sudo syno_poweroff_task -d sudo unmount /volume1 sudo btrfs check --readonly -p /dev/vg1 1 Quote Link to comment Share on other sites More sharing options...
Subtixx Posted January 24, 2021 Author Share #7 Posted January 24, 2021 Thanks, I'll try this as soon as the copy is finished. One thing though is that some files are indeed missing (show up as "? Unifi.syno.json" for example) and those cannot be copied aswell. But all of the missing files are just in the docker folder. Everything else is fine and can be successfully copied. This makes sense too since when the Volume crashed I updated some packages in the iobroker docker container. So it seems somehow docker fudged up its folder filesystem. Quote Link to comment Share on other sites More sharing options...
IG-88 Posted January 24, 2021 Share #8 Posted January 24, 2021 1 hour ago, Subtixx said: So it seems somehow docker fudged up its folder filesystem. i would not be so fast assuming anything like that, a simple write process should not do any damage to the file system you can already start looking into /var/log/ for clues what went wrong, you know roughly date and time so you might be able put things together from different logs proceeding with repair or restore will only make sense if you can pin the reason or you might end up with the same problem in a few days start with disk.log, kernel.log, messages, dmesg 1 Quote Link to comment Share on other sites More sharing options...
Subtixx Posted January 24, 2021 Author Share #9 Posted January 24, 2021 (edited) Quote 2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037039] BTRFS warning (device dm-2): csum failed ino 2404620 off 262144 csum 3001604934 expected csum 0 2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037333] BTRFS warning (device dm-2): csum failed ino 2404620 off 286720 csum 2248849372 expected csum 0 2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037340] BTRFS warning (device dm-2): csum failed ino 2404620 off 290816 csum 3009287732 expected csum 0 2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037344] BTRFS warning (device dm-2): csum failed ino 2404620 off 294912 csum 1097611249 expected csum 0 2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037348] BTRFS warning (device dm-2): csum failed ino 2404620 off 299008 csum 2610560654 expected csum 0 2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037352] BTRFS warning (device dm-2): csum failed ino 2404620 off 303104 csum 3633251318 expected csum 0 2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037356] BTRFS warning (device dm-2): csum failed ino 2404620 off 307200 csum 420833368 expected csum 0 2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037361] BTRFS warning (device dm-2): csum failed ino 2404620 off 311296 csum 2095865147 expected csum 0 2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037365] BTRFS warning (device dm-2): csum failed ino 2404620 off 315392 csum 411949296 expected csum 0 2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037369] BTRFS warning (device dm-2): csum failed ino 2404620 off 319488 csum 2093084079 expected csum 0 kern.log shows this, although I'm not able to look at the exact time it happen (it was around 13:07 since the synology sent me a message at that exact time) Quote 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612284] ------------[ cut here ]------------ 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612316] WARNING: at fs/btrfs/super.c:263 __btrfs_abort_transaction+0xdd/0x130 [btrfs]() 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612319] Modules linked in: openvswitch(O) gre tcm_loop(O) vhost_scsi(O) vhost(O) pci_stub vhost_net tun kvm_intel kvm snd_usb_hiface snd_pcm_oss snd_mixer_oss snd_usb_audio snd_pcm snd_timer snd_hwdep snd_usbmidi_lib snd_rawmidi snd_seq_device snd snd_page_alloc soundcore bridge stp aufs macvlan veth xt_conntrack xt_addrtype nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables ipt_MASQUERADE xt_REDIRECT xt_nat iptable_nat nf_nat_ipv4 nf_nat xt_recent xt_iprange xt_limit xt_state xt_tcpudp xt_multiport xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_filter ip_tables x_tables cifs udf isofs loop iscsi_target_mod(O) target_core_ep(O) target_core_multi_file(O) target_core_file(O) target_core_iblock(O) target_core_mod(O) syno_extent_pool(PO) rodsp_ep(O) hid_generic usbhid hid usblp flashcache_syno(O) flashcache(O) syno_flashcache_control(O) broadwell_synobios(PO) button(F) ax88179_178a(F) usbnet tg3(F) r8169(F) cnic(F) bnx2(F) vmxnet3(F) pcnet32(F) e1000(F) sfc(F) netxen_nic(F) qlge(F) qlcnic(F) qla3xxx(F) pch_gbe(F) ptp_pch(F) sky2(F) skge(F) jme(F) ipg(F) uio(F) alx(F) atl1c(F) atl1e(F) atl1(F) libphy(F) mii(F) exfat(O) btrfs synoacl_vfs(PO) zlib_deflate hfsplus md4 hmac bnx2x(O) libcrc32c mdio mlx5_core(O) mlx4_en(O) mlx4_core(O) mlx_compat(O) compat(O) qede(O) qed(O) atlantic(O) r8168(OF) tn40xx(O) i40e(O) ixgbe(O) be2net(O) igb(O) i2c_algo_bit e1000e(O) dca vxlan fuse vfat fat crc32c_intel aesni_intel glue_helper lrw gf128mul ablk_helper arc4 cryptd ecryptfs sha256_generic sha1_generic ecb aes_x86_64 authenc des_generic ansi_cprng cts md5 cbc cpufreq_conservative cpufreq_powersave cpufreq_performance cpufreq_ondemand mperf processor thermal_sys cpufreq_stats freq_table dm_snapshot crc_itu_t crc_ccitt quota_v2 quota_tree psnap p8022 llc sit tunnel4 ip_tunnel ipv6 zram(C) sg etxhci_hcd mpt3sas(O) mpt2sas(O) megaraid_sas(F) mptctl(F) mptsas(F) mptspi(F) mptscsih(F) mptbase(F) scsi_transport_spi(F) megaraid(F) megaraid_mbox(F) megaraid_mm(F) vmw_pvscsi(F) BusLogic(F) usb_storage xhci_hcd uhci_hcd ohci_hcd(F) ehci_pci(F) ehci_hcd(F) usbcore usb_common mv14xx(O) ouuueumuhn(OF) [last unloaded: gre] 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612415] CPU: 2 PID: 15395 Comm: node Tainted: PF C O 3.10.105 #23739 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612416] Hardware name: HP ProLiant MicroServer Gen8, BIOS J06 11/02/2015 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612417] ffffffff814b6c18 ffffffff81033f26 ffff8802ac9e01e0 ffff8800911f3bc8 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612420] 00000000fffffff4 ffffffffa0b491c0 0000000000001a36 ffffffff81033f87 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612423] ffffffffa0b4cde0 ffff880000000020 ffff8800911f3bd8 ffff8800911f3b98 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612425] Call Trace: 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612431] [<ffffffff814b6c18>] ? dump_stack+0xc/0x15 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612434] [<ffffffff81033f26>] ? warn_slowpath_common+0x56/0x70 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612438] [<ffffffff81033f87>] ? warn_slowpath_fmt+0x47/0x50 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612451] [<ffffffffa0aa756c>] ? update_block_group.isra.74+0x10c/0x390 [btrfs] 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612460] [<ffffffffa0a9396d>] ? __btrfs_abort_transaction+0xdd/0x130 [btrfs] 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612470] [<ffffffffa0aa7b27>] ? __btrfs_free_extent+0x337/0xd60 [btrfs] 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612482] [<ffffffffa0b132ea>] ? btrfs_merge_delayed_refs+0x3aa/0x460 [btrfs] 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612492] [<ffffffffa0aada07>] ? __btrfs_run_delayed_refs+0x847/0x11a0 [btrfs] 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612495] [<ffffffff810612c6>] ? dequeue_task_fair+0xc6/0xb00 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612498] [<ffffffff81058952>] ? finish_task_switch+0xa2/0xd0 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612508] [<ffffffffa0ab2c13>] ? btrfs_run_delayed_refs+0xb3/0x2c0 [btrfs] 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612519] [<ffffffffa0ac47ab>] ? btrfs_commit_transaction+0x3b/0xa90 [btrfs] 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612532] [<ffffffffa0b0a420>] ? btrfs_log_dentry_safe+0x60/0x80 [btrfs] 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612543] [<ffffffffa0adc079>] ? btrfs_sync_file+0x259/0x320 [btrfs] 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612547] [<ffffffff81128b8c>] ? do_fsync+0x4c/0x80 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612549] [<ffffffff81128e07>] ? SyS_fsync+0x7/0x10 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612552] [<ffffffff814bcc32>] ? system_call_fastpath+0x16/0x1b 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612553] ---[ end trace 50f3905ae30e560c ]--- 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612555] BTRFS: error (device dm-2) in __btrfs_free_extent:6710: errno=-12 Out of memory 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612754] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:2970: errno=-12 Out of memory 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.117159] parent transid verify failed on 6307284746240 wanted 1415403 found 1415400 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.117469] parent transid verify failed on 6307284779008 wanted 1415403 found 1415400 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.133002] parent transid verify failed on 6307284746240 wanted 1415403 found 1415400 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.133477] parent transid verify failed on 6307284779008 wanted 1415403 found 1415400 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171192] parent transid verify failed on 6307284779008 wanted 1415403 found 1415400 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171196] parent transid verify failed on 6307284779008 wanted 1415403 found 1415400 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171198] parent transid verify failed on 6307284779008 wanted 1415403 found 1415400 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171200] parent transid verify failed on 6307284779008 wanted 1415403 found 1415400 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171202] parent transid verify failed on 6307284746240 wanted 1415403 found 1415400 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171206] parent transid verify failed on 6307284746240 wanted 1415403 found 1415400 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171213] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307284779008, mirror = 1 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171213] 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171216] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307284746240, mirror = 1 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171216] 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.224076] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307284746240, mirror = 2 2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.224076] 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.299487] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307284779008, mirror = 2 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.299487] 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.299785] BTRFS warning (device dm-2): csum failed ino 8031 off 0 csum 138206697 expected csum 0 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.299788] BTRFS warning (device dm-2): csum failed ino 8031 off 4096 csum 3752213298 expected csum 0 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.299790] BTRFS warning (device dm-2): csum failed ino 8031 off 8192 csum 1106615519 expected csum 0 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.299794] BTRFS warning (device dm-2): csum failed ino 8031 off 12288 csum 3677950484 expected csum 0 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.340710] BTRFS warning (device dm-2): csum failed ino 1639144 off 8192 csum 34407137 expected csum 0 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.341612] BTRFS warning (device dm-2): csum failed ino 1639144 off 12288 csum 1776556878 expected csum 0 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.422076] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307298803712, mirror = 1 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.422076] 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.461747] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307295182848, mirror = 1 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.461747] 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.473046] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307298803712, mirror = 2 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.473046] 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.489659] BTRFS warning (device dm-2): csum failed ino 1638619 off 0 csum 3842319772 expected csum 0 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.489665] BTRFS warning (device dm-2): csum failed ino 1638619 off 4096 csum 624524293 expected csum 0 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.489667] BTRFS warning (device dm-2): csum failed ino 1638619 off 8192 csum 804132820 expected csum 0 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.489670] BTRFS warning (device dm-2): csum failed ino 1638619 off 12288 csum 1323862206 expected csum 0 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.500953] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307296526336, mirror = 1 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.500953] 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.522278] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307295920128, mirror = 1 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.522278] 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.575717] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307295182848, mirror = 2 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.575717] 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.575929] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307296526336, mirror = 2 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.575929] 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.576442] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307295920128, mirror = 2 2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.576442] 2021-01-24T13:07:07+01:00 mediaserver kernel: [10282724.234073] verify_parent_transid: 186 callbacks suppressed 2021-01-24T13:07:07+01:00 mediaserver kernel: [10282724.234081] parent transid verify failed on 6307284746240 wanted 1415403 found 1415400 2021-01-24T13:07:07+01:00 mediaserver kernel: [10282724.234346] parent transid verify failed on 6307284746240 wanted 1415403 found 1415400 Woah this might be the issue! I have no idea what to make with this though. I only see that some exception in a C file has occured and a function name. Quick googling shows however after this error the BTRFS is going read-only. Also no idea why it says out of memory. It has 16GB of memory installed and the memory utilization was never higher than 60% Quote BTRFS: error (device dm-2) in __btrfs_free_extent:6710: errno=-12 Out of memory Edited January 24, 2021 by Subtixx Quote Link to comment Share on other sites More sharing options...
IG-88 Posted January 24, 2021 Share #10 Posted January 24, 2021 (edited) sudo btrfs device stat /dev/vg1 what does this return Edited January 24, 2021 by IG-88 Quote Link to comment Share on other sites More sharing options...
IG-88 Posted January 24, 2021 Share #11 Posted January 24, 2021 also if you are done with backup and can reboot the system you should check the RAM for errors (memtest from a bootable live/recovery linux) Quote Link to comment Share on other sites More sharing options...
Subtixx Posted January 25, 2021 Author Share #12 Posted January 25, 2021 11 hours ago, IG-88 said: sudo btrfs device stat /dev/vg1 what does this return Quote ERROR: not a btrfs filesystem: /dev/vg1 11 hours ago, IG-88 said: also if you are done with backup and can reboot the system you should check the RAM for errors (memtest from a bootable live/recovery linux) Okay. I'll do that once my new NAS with the HDDs arrive Quote Link to comment Share on other sites More sharing options...
IG-88 Posted January 25, 2021 Share #13 Posted January 25, 2021 8 hours ago, Subtixx said: Quote ERROR: not a btrfs filesystem: /dev/vg1 works on my system (not SHR, just plain raid6, so no lvm2) btrfs device stat /dev/md2 [/dev/md2].write_io_errs 0 [/dev/md2].read_io_errs 0 [/dev/md2].flush_io_errs 0 [/dev/md2].corruption_errs 0 [/dev/md2].generation_errs 0 try it with /dev/vg1/volume_1 (logical volume) there are a threads about checking in repairing btrfs here in the forum and you can also use a search engine to find more about it, i dont have a tested workflow for recovering a btrfs file system, maybe @flyride has good advice how to proceed https://xpenology.com/forum/topic/30702-btrfs-volume-crashed-storage-pool-ok/#comment-151129 Quote Link to comment Share on other sites More sharing options...
flyride Posted January 25, 2021 Share #14 Posted January 25, 2021 Sorry I haven't been paying much attention to this thread. I don't fully understand the German screenshots so I am a little challenged. I am completely confused by the reports on the volume mount however. Based on the mdstat, it looks like @Subtixx has a straight RAID 5 (with a read cache), and not a SHR. So there should be no /dev/vg1 or /dev/vg1000 and the data volume must be directly contained within /dev/md2. But still you have posted volume group and logical volume screenshots. It was reported that your files are accessible right now. So the volume must be presently mounted. We need to be sure how things are set up. Please post the output of: # cat /etc/fstab and # df Also, Synology Docker creates and loads btrfs subvolumes. Most of that is temporal/transient data and the subvolumes are destroyed when the container is ended. If you go into Package Manager and stop Docker, do you see any different reports with your filesystems? 1 Quote Link to comment Share on other sites More sharing options...
Subtixx Posted January 26, 2021 Author Share #15 Posted January 26, 2021 8 hours ago, flyride said: Sorry I haven't been paying much attention to this thread. I don't fully understand the German screenshots so I am a little challenged. I am completely confused by the reports on the volume mount however. Based on the mdstat, it looks like @Subtixx has a straight RAID 5 (with a read cache), and not a SHR. So there should be no /dev/vg1 or /dev/vg1000 and the data volume must be directly contained within /dev/md2. But still you have posted volume group and logical volume screenshots. It was reported that your files are accessible right now. So the volume must be presently mounted. We need to be sure how things are set up. Please post the output of: # cat /etc/fstab and # df Also, Synology Docker creates and loads btrfs subvolumes. Most of that is temporal/transient data and the subvolumes are destroyed when the container is ended. If you go into Package Manager and stop Docker, do you see any different reports with your filesystems? Output of: Quote cat /etc/fstab Quote none /proc proc defaults 0 0 /dev/root / ext4 defaults 1 1 /dev/mapper/cachedev_0 /volume1 btrfs auto_reclaim_space,synoacl,relatime 0 0 Output of Quote df Quote Filesystem 1K-blocks Used Available Use% Mounted on /dev/md0 2385528 1140444 1126300 51% / none 8189516 0 8189516 0% /dev /tmp 8236708 996 8235712 1% /tmp /run 8236708 23460 8213248 1% /run /dev/shm 8236708 16 8236692 1% /dev/shm none 4 0 4 0% /sys/fs/cgroup cgmfs 100 0 100 0% /run/cgmanager/fs /dev/mapper/cachedev_0 11238050368 7842418632 3395631736 70% /volume1 /dev/sdu1 15136768 859984 14276784 6% /volumeUSB1/usbshare tmpfs 1073741824 2523136 1071218688 1% /dev/virtualization /dev/sdv1 961296496 456479868 504697844 48% /volumeUSB2/usbshare /dev/sdw1 976727036 564140872 412586164 58% /volumeUSB3/usbshare (I'm still waiting for the HDDs to transfer my files thats why it shows 3 USB Drives) Docker in the package manager shows as error and the only option I have is to repair :S Quote Link to comment Share on other sites More sharing options...
flyride Posted January 26, 2021 Share #16 Posted January 26, 2021 Well that tells us that the cache is intact and in use and is being used as the target to mount your volume. Can you post your Storage Pool status screen? This is one of those situations where the problem is harder to address because of the use of cache. And let's try to get your Docker shut down, use the manual command: # synoservice --stop pkgctl-Docker I would be interested to see if your volume error stops with Docker. If it does not, we can look into whether you can use the UI to remove the cache or try it manually as well. 1 Quote Link to comment Share on other sites More sharing options...
Subtixx Posted January 27, 2021 Author Share #17 Posted January 27, 2021 Storage Pool Status: Quote And let's try to get your Docker shut down, use the manual command: # synoservice --stop pkgctl-Docker This worked. However after looking at my DSM today docker is gone all together. Quote Link to comment Share on other sites More sharing options...
flyride Posted January 27, 2021 Share #18 Posted January 27, 2021 We stopped Docker, that was the point. Is your volume still corrupted? 1 Quote Link to comment Share on other sites More sharing options...
Subtixx Posted January 27, 2021 Author Share #19 Posted January 27, 2021 Sorry was at work and didn't had enough time to write something useful. Before I executed the command docker was already gone in DSM when I looked at it today. Aswell as all the other packages I installed. All vanished. In package manager they all now show up as "Error". When I executed the command it was sucessful, but the volume still shows up as crashed Quote Link to comment Share on other sites More sharing options...
flyride Posted January 27, 2021 Share #20 Posted January 27, 2021 Ok, don't mind your Docker for the moment. Before trying to troubleshoot the filesystem further, I would remove the cache from the Storage Pool. 1 Quote Link to comment Share on other sites More sharing options...
Subtixx Posted January 29, 2021 Author Share #21 Posted January 29, 2021 On 1/24/2021 at 11:25 PM, IG-88 said: sudo btrfs device stat /dev/vg1 what does this return Quote ERROR: '/dev/vg1/volume_1' is not a mounted btrfs device On 1/27/2021 at 6:01 PM, flyride said: Ok, don't mind your Docker for the moment. Before trying to troubleshoot the filesystem further, I would remove the cache from the Storage Pool. After transferring all my files to the new NAS and restarting the old one the volume is back again and doesnt show up as crashed anymore Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.