Recommended Posts

So after updating some packages in my docker iobroker container I got a notification that my volume has crashed. This is weird since all HDDs show as normal but the volume shows crashed? 

 

EDIT:

I can still access the files on there (I can see all folders etc) I also tried opening a mp4 file I have on there and it was playing fine. However becuase I am to afraid to break more I didn't check any other files only this one mp4...

 

chrome_hhm016GlGc.thumb.png.c6bb290631653abfaa88efe022ba6cc5.png

chrome_Ni0Ubu8JLi.thumb.png.78ef33561cf7cf2650eed6ffe5f41a5e.png

8jXzOdaU2N.thumb.png.dc6b376cba23ccbe3fa57bc25789653f.png

 

 

 

My first question is: how can this happen?

What should I do now? Obviously I cannot transfer all files to an external drive because I'd need over 7TB of drives which I obviously don't have.

 

output of 'cat /proc/mdstat' (no idea what all that means :S)

Quote

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [raidF1]
md2 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1]
      11706589632 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md3 : active raid0 sde1[0]
      244193984 blocks super 1.2 64k chunks [1/1] [U]

md1 : active raid1 sda2[0] sdb2[1] sdc2[2] sdd2[3]
      2097088 blocks [12/4] [UUUU________]

md0 : active raid1 sda1[0] sdb1[1] sdc1[2] sdd1[3]
      2490176 blocks [12/4] [UUUU________]

unused devices: <none>

 

output of 'vgdisplay':

Quote

  --- Volume group ---
  VG Name               vg1
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  5
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               10.90 TiB
  PE Size               4.00 MiB
  Total PE              2858053
  Alloc PE / Size       2857987 / 10.90 TiB
  Free  PE / Size       66 / 264.00 MiB
  VG UUID               PX1920-Xorr-2zmD-Ua48-QqkI-MHlh-AnEZvO

 

output of 'vgdisplay --verbose'

Quote

    Using volume group(s) on command line.
  --- Volume group ---
  VG Name               vg1
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  5
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               10.90 TiB
  PE Size               4.00 MiB
  Total PE              2858053
  Alloc PE / Size       2857987 / 10.90 TiB
  Free  PE / Size       66 / 264.00 MiB
  VG UUID               PX1920-Xorr-2zmD-Ua48-QqkI-MHlh-AnEZvO

  --- Logical volume ---
  LV Path                /dev/vg1/syno_vg_reserved_area
  LV Name                syno_vg_reserved_area
  VG Name                vg1
  LV UUID                oi96k9-z6Wv-rVKJ-i7FK-q0qp-qqNb-k00Nhw
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              available
  # open                 0
  LV Size                12.00 MiB
  Current LE             3
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     768
  Block device           253:0

  --- Logical volume ---
  LV Path                /dev/vg1/volume_1
  LV Name                volume_1
  VG Name                vg1
  LV UUID                fHa3p4-Ubwa-rQxy-DcPN-zp4e-2EyR-gTFg3Q
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              available
  # open                 1
  LV Size                10.90 TiB
  Current LE             2857984
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     768
  Block device           253:1

  --- Physical volumes ---
  PV Name               /dev/md2
  PV UUID               Lf7GbE-7Y6B-IMbz-NigR-Ekcu-KKQd-Pz1eG4
  PV Status             allocatable
  Total PE / Free PE    2858053 / 66
 

 

output of 'lvdisplay --verbose'

Quote

    Using logical volume(s) on command line.
  --- Logical volume ---
  LV Path                /dev/vg1/syno_vg_reserved_area
  LV Name                syno_vg_reserved_area
  VG Name                vg1
  LV UUID                oi96k9-z6Wv-rVKJ-i7FK-q0qp-qqNb-k00Nhw
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              available
  # open                 0
  LV Size                12.00 MiB
  Current LE             3
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     768
  Block device           253:0

  --- Logical volume ---
  LV Path                /dev/vg1/volume_1
  LV Name                volume_1
  VG Name                vg1
  LV UUID                fHa3p4-Ubwa-rQxy-DcPN-zp4e-2EyR-gTFg3Q
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              available
  # open                 1
  LV Size                10.90 TiB
  Current LE             2857984
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     768
  Block device           253:1

 

EDIT:

 

After some looking through the folders I see that the "docker" folder seems empty dunno what else is missing though. Is this gone for good or is that just the filesystem being corrupt? If so is there any way I can restore those files?

chrome_jKT3kjlUmN.png

Edited by Subtixx
Link to post
Share on other sites

i's suggest the same as the dsm does, backup as long as you still have access, atm you might not need a recovery procedure

it would be good to know in whats state the file system is and how it came to this to prevent is happening again but thats 2nd

1st priority are offloading the data asap

if the disks, raid and volumes are intact then there might be damage in the btrfs system and if its self repair cant do the job then there might be problems beyond repair

(i had that once when "experimenting" with ssd write cache combined with controller problems - ended up in deleting the volume and recreating it from backup)

 

Edited by IG-88
  • Like 1
Link to post
Share on other sites
10 minutes ago, IG-88 said:

if the disks, raid and volumes are intact then there might be damage in the btrfs system and if its self repair cant do the job then there might be problems beyond repair

 

Did it try to self-repair though? It didn't seem like it it just sent me an email "Volume crashed" and no self-repair failed message or anything like that..

The weird thing is the only thing that is showing any errors is the volume there are no protocols for the HDD for example.

 

Btw backing up what I can with my limited HDD Space.. I'm probably going to lose 5TB of data now...

Edited by Subtixx
Link to post
Share on other sites
8 hours ago, Subtixx said:

It didn't seem like it it just sent me an email "Volume crashed" and no self-repair failed message or anything like that..

 you would need to have a look to the log files in /var/log/ but dsm usually schedules a check (scrubbing) oneth a month

btrfs has some redundancy in itself, if thats broken you should offload as much as you can and should not write to the data volume

maybe look up a btrfs file check that does not correct/repairs anything to know if there is a problem with the file system

as long as you dont know what caused the problem you should assume the worst and backup the files

Edited by IG-88
  • Like 1
Link to post
Share on other sites

Hmm okay. Can you tell me what I should check / execute via SSH when I'm finished copying my files?

 

Btw strange thing is the docker folder shows empty in "File Station" but in WinSCP via SSH/SCP it does show and copy successfully

Link to post
Share on other sites

i'm not a expert on this, just done that once, as you are unmounting the volume in the process make sure you finished the backup

 

sudo syno_poweroff_task -d
sudo unmount /volume1
sudo btrfs check --readonly -p /dev/vg1

 

  • Like 1
Link to post
Share on other sites

Thanks, I'll try this as soon as the copy is finished.

 

One thing though is that some files are indeed missing (show up as "? Unifi.syno.json" for example) and those cannot be copied aswell. But all of the missing files are just in the docker folder. Everything else is fine and can be successfully copied. This makes sense too since when the Volume crashed I updated some packages in the iobroker docker container. So it seems somehow docker fudged up its folder filesystem.

 

WinSCP_4u30rHmUcL.png.68885c0b2d6b19861d16a90fdcc1add7.png

Link to post
Share on other sites
1 hour ago, Subtixx said:

So it seems somehow docker fudged up its folder filesystem.

i would not be so fast assuming anything like that, a simple write process should not do any damage to the file system

you can already start looking into /var/log/ for clues what went wrong, you know roughly date and time so you might be able put things together from different logs

proceeding with repair or restore will only make sense if you can pin the reason or you might end up with the same problem in a few days

start with disk.log, kernel.log, messages, dmesg

  • Like 1
Link to post
Share on other sites
Quote

2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037039] BTRFS warning (device dm-2): csum failed ino 2404620 off 262144 csum 3001604934 expected csum 0
2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037333] BTRFS warning (device dm-2): csum failed ino 2404620 off 286720 csum 2248849372 expected csum 0
2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037340] BTRFS warning (device dm-2): csum failed ino 2404620 off 290816 csum 3009287732 expected csum 0
2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037344] BTRFS warning (device dm-2): csum failed ino 2404620 off 294912 csum 1097611249 expected csum 0
2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037348] BTRFS warning (device dm-2): csum failed ino 2404620 off 299008 csum 2610560654 expected csum 0
2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037352] BTRFS warning (device dm-2): csum failed ino 2404620 off 303104 csum 3633251318 expected csum 0
2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037356] BTRFS warning (device dm-2): csum failed ino 2404620 off 307200 csum 420833368 expected csum 0
2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037361] BTRFS warning (device dm-2): csum failed ino 2404620 off 311296 csum 2095865147 expected csum 0
2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037365] BTRFS warning (device dm-2): csum failed ino 2404620 off 315392 csum 411949296 expected csum 0
2021-01-24T22:18:57+01:00 mediaserver kernel: [10315817.037369] BTRFS warning (device dm-2): csum failed ino 2404620 off 319488 csum 2093084079 expected csum 0
 

 

kern.log shows this, although I'm not able to look at the exact time it happen (it was around 13:07 since the synology sent me a message at that exact time)

 

Quote

2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612284] ------------[ cut here ]------------
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612316] WARNING: at fs/btrfs/super.c:263 __btrfs_abort_transaction+0xdd/0x130 [btrfs]()
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612319] Modules linked in: openvswitch(O) gre tcm_loop(O) vhost_scsi(O) vhost(O) pci_stub vhost_net tun kvm_intel kvm snd_usb_hiface snd_pcm_oss snd_mixer_oss snd_usb_audio snd_pcm snd_timer snd_hwdep snd_usbmidi_lib snd_rawmidi snd_seq_device snd snd_page_alloc soundcore bridge stp aufs macvlan veth xt_conntrack xt_addrtype nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables ipt_MASQUERADE xt_REDIRECT xt_nat iptable_nat nf_nat_ipv4 nf_nat xt_recent xt_iprange xt_limit xt_state xt_tcpudp xt_multiport xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_filter ip_tables x_tables cifs udf isofs loop iscsi_target_mod(O) target_core_ep(O) target_core_multi_file(O) target_core_file(O) target_core_iblock(O) target_core_mod(O) syno_extent_pool(PO) rodsp_ep(O) hid_generic usbhid hid usblp flashcache_syno(O) flashcache(O) syno_flashcache_control(O) broadwell_synobios(PO) button(F) ax88179_178a(F) usbnet tg3(F) r8169(F) cnic(F) bnx2(F) vmxnet3(F) pcnet32(F) e1000(F) sfc(F) netxen_nic(F) qlge(F) qlcnic(F) qla3xxx(F) pch_gbe(F) ptp_pch(F) sky2(F) skge(F) jme(F) ipg(F) uio(F) alx(F) atl1c(F) atl1e(F) atl1(F) libphy(F) mii(F) exfat(O) btrfs synoacl_vfs(PO) zlib_deflate hfsplus md4 hmac bnx2x(O) libcrc32c mdio mlx5_core(O) mlx4_en(O) mlx4_core(O) mlx_compat(O) compat(O) qede(O) qed(O) atlantic(O) r8168(OF) tn40xx(O) i40e(O) ixgbe(O) be2net(O) igb(O) i2c_algo_bit e1000e(O) dca vxlan fuse vfat fat crc32c_intel aesni_intel glue_helper lrw gf128mul ablk_helper arc4 cryptd ecryptfs sha256_generic sha1_generic ecb aes_x86_64 authenc des_generic ansi_cprng cts md5 cbc cpufreq_conservative cpufreq_powersave cpufreq_performance cpufreq_ondemand mperf processor thermal_sys cpufreq_stats freq_table dm_snapshot crc_itu_t crc_ccitt quota_v2 quota_tree psnap p8022 llc sit tunnel4 ip_tunnel ipv6 zram(C) sg etxhci_hcd mpt3sas(O) mpt2sas(O) megaraid_sas(F) mptctl(F) mptsas(F) mptspi(F) mptscsih(F) mptbase(F) scsi_transport_spi(F) megaraid(F) megaraid_mbox(F) megaraid_mm(F) vmw_pvscsi(F) BusLogic(F) usb_storage xhci_hcd uhci_hcd ohci_hcd(F) ehci_pci(F) ehci_hcd(F) usbcore usb_common mv14xx(O) ouuueumuhn(OF) [last unloaded: gre]
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612415] CPU: 2 PID: 15395 Comm: node Tainted: PF        C O 3.10.105 #23739
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612416] Hardware name: HP ProLiant MicroServer Gen8, BIOS J06 11/02/2015
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612417]  ffffffff814b6c18 ffffffff81033f26 ffff8802ac9e01e0 ffff8800911f3bc8
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612420]  00000000fffffff4 ffffffffa0b491c0 0000000000001a36 ffffffff81033f87
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612423]  ffffffffa0b4cde0 ffff880000000020 ffff8800911f3bd8 ffff8800911f3b98
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612425] Call Trace:
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612431]  [<ffffffff814b6c18>] ? dump_stack+0xc/0x15
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612434]  [<ffffffff81033f26>] ? warn_slowpath_common+0x56/0x70
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612438]  [<ffffffff81033f87>] ? warn_slowpath_fmt+0x47/0x50
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612451]  [<ffffffffa0aa756c>] ? update_block_group.isra.74+0x10c/0x390 [btrfs]
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612460]  [<ffffffffa0a9396d>] ? __btrfs_abort_transaction+0xdd/0x130 [btrfs]
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612470]  [<ffffffffa0aa7b27>] ? __btrfs_free_extent+0x337/0xd60 [btrfs]
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612482]  [<ffffffffa0b132ea>] ? btrfs_merge_delayed_refs+0x3aa/0x460 [btrfs]
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612492]  [<ffffffffa0aada07>] ? __btrfs_run_delayed_refs+0x847/0x11a0 [btrfs]
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612495]  [<ffffffff810612c6>] ? dequeue_task_fair+0xc6/0xb00
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612498]  [<ffffffff81058952>] ? finish_task_switch+0xa2/0xd0
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612508]  [<ffffffffa0ab2c13>] ? btrfs_run_delayed_refs+0xb3/0x2c0 [btrfs]
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612519]  [<ffffffffa0ac47ab>] ? btrfs_commit_transaction+0x3b/0xa90 [btrfs]
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612532]  [<ffffffffa0b0a420>] ? btrfs_log_dentry_safe+0x60/0x80 [btrfs]
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612543]  [<ffffffffa0adc079>] ? btrfs_sync_file+0x259/0x320 [btrfs]
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612547]  [<ffffffff81128b8c>] ? do_fsync+0x4c/0x80
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612549]  [<ffffffff81128e07>] ? SyS_fsync+0x7/0x10
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612552]  [<ffffffff814bcc32>] ? system_call_fastpath+0x16/0x1b
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612553] ---[ end trace 50f3905ae30e560c ]---
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612555] BTRFS: error (device dm-2) in __btrfs_free_extent:6710: errno=-12 Out of memory
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282717.612754] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:2970: errno=-12 Out of memory
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.117159] parent transid verify failed on 6307284746240 wanted 1415403 found 1415400
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.117469] parent transid verify failed on 6307284779008 wanted 1415403 found 1415400
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.133002] parent transid verify failed on 6307284746240 wanted 1415403 found 1415400
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.133477] parent transid verify failed on 6307284779008 wanted 1415403 found 1415400
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171192] parent transid verify failed on 6307284779008 wanted 1415403 found 1415400
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171196] parent transid verify failed on 6307284779008 wanted 1415403 found 1415400
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171198] parent transid verify failed on 6307284779008 wanted 1415403 found 1415400
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171200] parent transid verify failed on 6307284779008 wanted 1415403 found 1415400
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171202] parent transid verify failed on 6307284746240 wanted 1415403 found 1415400
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171206] parent transid verify failed on 6307284746240 wanted 1415403 found 1415400
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171213] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307284779008, mirror = 1
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171213]
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171216] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307284746240, mirror = 1
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.171216]
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.224076] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307284746240, mirror = 2
2021-01-24T13:07:01+01:00 mediaserver kernel: [10282718.224076]
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.299487] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307284779008, mirror = 2
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.299487]
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.299785] BTRFS warning (device dm-2): csum failed ino 8031 off 0 csum 138206697 expected csum 0
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.299788] BTRFS warning (device dm-2): csum failed ino 8031 off 4096 csum 3752213298 expected csum 0
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.299790] BTRFS warning (device dm-2): csum failed ino 8031 off 8192 csum 1106615519 expected csum 0
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.299794] BTRFS warning (device dm-2): csum failed ino 8031 off 12288 csum 3677950484 expected csum 0
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.340710] BTRFS warning (device dm-2): csum failed ino 1639144 off 8192 csum 34407137 expected csum 0
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.341612] BTRFS warning (device dm-2): csum failed ino 1639144 off 12288 csum 1776556878 expected csum 0
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.422076] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307298803712, mirror = 1
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.422076]
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.461747] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307295182848, mirror = 1
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.461747]
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.473046] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307298803712, mirror = 2
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.473046]
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.489659] BTRFS warning (device dm-2): csum failed ino 1638619 off 0 csum 3842319772 expected csum 0
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.489665] BTRFS warning (device dm-2): csum failed ino 1638619 off 4096 csum 624524293 expected csum 0
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.489667] BTRFS warning (device dm-2): csum failed ino 1638619 off 8192 csum 804132820 expected csum 0
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.489670] BTRFS warning (device dm-2): csum failed ino 1638619 off 12288 csum 1323862206 expected csum 0
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.500953] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307296526336, mirror = 1
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.500953]
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.522278] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307295920128, mirror = 1
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.522278]
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.575717] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307295182848, mirror = 2
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.575717]
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.575929] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307296526336, mirror = 2
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.575929]
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.576442] BTRFS error (device dm-2): BTRFS: dm-2 failed to repair parent transid verify failure on 6307295920128, mirror = 2
2021-01-24T13:07:02+01:00 mediaserver kernel: [10282718.576442]
2021-01-24T13:07:07+01:00 mediaserver kernel: [10282724.234073] verify_parent_transid: 186 callbacks suppressed
2021-01-24T13:07:07+01:00 mediaserver kernel: [10282724.234081] parent transid verify failed on 6307284746240 wanted 1415403 found 1415400
2021-01-24T13:07:07+01:00 mediaserver kernel: [10282724.234346] parent transid verify failed on 6307284746240 wanted 1415403 found 1415400
 

Woah this might be the issue! I have no idea what to make with this though. I only see that some exception in a C file has occured and a function name. Quick googling shows however after this error the BTRFS is going read-only.

 

Also no idea why it says out of memory. It has 16GB of memory installed and the memory utilization was never higher than 60%

Quote

BTRFS: error (device dm-2) in __btrfs_free_extent:6710: errno=-12 Out of memory

 

Edited by Subtixx
Link to post
Share on other sites
11 hours ago, IG-88 said:

sudo btrfs device stat /dev/vg1

what does this return

 

Quote

ERROR: not a btrfs filesystem: /dev/vg1

 

11 hours ago, IG-88 said:

also if you are done with backup and can reboot the system you should check the RAM for errors (memtest from a bootable live/recovery linux)

 

Okay. I'll do that once my new NAS with the HDDs arrive

Link to post
Share on other sites
8 hours ago, Subtixx said:

 

Quote

ERROR: not a btrfs filesystem: /dev/vg1

 

 

works on my system (not SHR, just plain raid6, so no lvm2)

 

btrfs device stat /dev/md2
[/dev/md2].write_io_errs   0
[/dev/md2].read_io_errs    0
[/dev/md2].flush_io_errs   0
[/dev/md2].corruption_errs 0
[/dev/md2].generation_errs 0

 

try it with  /dev/vg1/volume_1

(logical volume)

 

there are a threads about checking in repairing btrfs here in the forum and you can also  use a search engine to find more about it, i dont have a tested workflow for recovering a btrfs file system, maybe @flyride has good advice how to proceed

https://xpenology.com/forum/topic/30702-btrfs-volume-crashed-storage-pool-ok/#comment-151129

 

 

Link to post
Share on other sites

Sorry I haven't been paying much attention to this thread.  I don't fully understand the German screenshots so I am a little challenged.

 

I am completely confused by the reports on the volume mount however.  Based on the mdstat, it looks like @Subtixx has a straight RAID 5 (with a read cache), and not a SHR.  So there should be no /dev/vg1 or /dev/vg1000 and the data volume must be directly contained within /dev/md2.  But still you have posted volume group and logical volume screenshots. 

 

It was reported that your files are accessible right now.  So the volume must be presently mounted. We need to be sure how things are set up. Please post the output of:

 

# cat /etc/fstab

     and

# df

 

Also, Synology Docker creates and loads btrfs subvolumes.  Most of that is temporal/transient data and the subvolumes are destroyed when the container is ended.  If you go into Package Manager and stop Docker, do you see any different reports with your filesystems?

  • Like 1
Link to post
Share on other sites
8 hours ago, flyride said:

Sorry I haven't been paying much attention to this thread.  I don't fully understand the German screenshots so I am a little challenged.

 

I am completely confused by the reports on the volume mount however.  Based on the mdstat, it looks like @Subtixx has a straight RAID 5 (with a read cache), and not a SHR.  So there should be no /dev/vg1 or /dev/vg1000 and the data volume must be directly contained within /dev/md2.  But still you have posted volume group and logical volume screenshots. 

 

It was reported that your files are accessible right now.  So the volume must be presently mounted. We need to be sure how things are set up. Please post the output of:

 

# cat /etc/fstab

     and

# df

 

Also, Synology Docker creates and loads btrfs subvolumes.  Most of that is temporal/transient data and the subvolumes are destroyed when the container is ended.  If you go into Package Manager and stop Docker, do you see any different reports with your filesystems?

 

Output of:

Quote

cat /etc/fstab

 

Quote

none /proc proc defaults 0 0
/dev/root / ext4 defaults 1 1
/dev/mapper/cachedev_0 /volume1 btrfs auto_reclaim_space,synoacl,relatime 0 0

 

Output of

Quote

df

 

Quote

Filesystem               1K-blocks       Used  Available Use% Mounted on
/dev/md0                   2385528    1140444    1126300  51% /
none                       8189516          0    8189516   0% /dev
/tmp                       8236708        996    8235712   1% /tmp
/run                       8236708      23460    8213248   1% /run
/dev/shm                   8236708         16    8236692   1% /dev/shm
none                             4          0          4   0% /sys/fs/cgroup
cgmfs                          100          0        100   0% /run/cgmanager/fs
/dev/mapper/cachedev_0 11238050368 7842418632 3395631736  70% /volume1
/dev/sdu1                 15136768     859984   14276784   6% /volumeUSB1/usbshare
tmpfs                   1073741824    2523136 1071218688   1% /dev/virtualization
/dev/sdv1                961296496  456479868  504697844  48% /volumeUSB2/usbshare
/dev/sdw1                976727036  564140872  412586164  58% /volumeUSB3/usbshare

 

(I'm still waiting for the HDDs to transfer my files thats why it shows 3 USB Drives)

 

Docker in the package manager shows as error and the only option I have is to repair :S

BUadmIF.png

Link to post
Share on other sites

Well that tells us that the cache is intact and in use and is being used as the target to mount your volume.  Can you post your Storage Pool status screen?  This is one of those situations where the problem is harder to address because of the use of cache.

 

And let's try to get your Docker shut down, use the manual command:

# synoservice --stop pkgctl-Docker

 

I would be interested to see if your volume error stops with Docker.  If it does not, we can look into whether you can use the UI to remove the cache or try it manually as well.

  • Like 1
Link to post
Share on other sites

Storage Pool Status:

dp5Nd2F.png

 

Quote

And let's try to get your Docker shut down, use the manual command:

# synoservice --stop pkgctl-Docker

 

This worked. However after looking at my DSM today docker is gone all together.

Link to post
Share on other sites

Sorry was at work and didn't had enough time to write something useful. 

 

Before I executed the command docker was already gone in DSM when I looked at it today. Aswell as all the other packages I installed. All vanished. In package manager they all now show up as "Error".

 

When I executed the command it was sucessful, but the volume still shows up as crashed

Link to post
Share on other sites
On 1/24/2021 at 11:25 PM, IG-88 said:

sudo btrfs device stat /dev/vg1

what does this return

Quote

ERROR: '/dev/vg1/volume_1' is not a mounted btrfs device

 

On 1/27/2021 at 6:01 PM, flyride said:

Ok, don't mind your Docker for the moment.  Before trying to troubleshoot the filesystem further, I would remove the cache from the Storage Pool.

 

After transferring all my files to the new NAS and restarting the old one the volume is back again and doesnt show up as crashed anymore

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.