• 0

Volume crashed but Storage Pool is fine


Question

Hi together

i got a problem with a crashed volume and my knowledge is now on a point where I need some help

 

image.thumb.png.b13aa43172f1af4f133d2c1f02dc9077.png

 

image.thumb.png.a94762bff6abb87fcb0fa9318da06e2e.png

 

 

I already did a few lines to identify a failure

(attached because the forum doesn't like it)

 

I found in the above response these two error lines, but couldn't find anything else about it

Found dropping subvolume ID:10011501971644612608 dropping progress: key (259824214343680 UNKNOWN.0 697453126021545984) level: 0

Errors found in extent allocation tree or chunk allocation

 

 

commands.txt

Link to post
Share on other sites

12 answers to this question

Recommended Posts

  • 0

Looks like everything is good on the array side.  Your btrfs filesystem has some corruption.

 

btrfs attempts to heal itself in real-time and when it cannot, it throws errors or refuses to mount.  Your goal is to get it to mount read-only through combination of discovery and leveraging redundancy in btrfs.

 

btrfs is not like ext4 and fsck. You should not expect to "fix" the problem.  If you can make your data accessible through command-line directives, then COPY everything off, delete the volume entirely, and recreate it.

 

I'm no expert at ferreting out btrfs problems, but you might find some of the strategies in this thread helpful (starting with the linked post):

https://xpenology.com/forum/topic/14337-volume-crash-after-4-months-of-stability/?do=findComment&comment=107979

 

Post back here if you aren't sure what to do.

 

Edited by flyride
Link to post
Share on other sites
  • 0
On 9/24/2021 at 3:23 AM, flyride said:

starting with the linked post

i tried and got the following lines back

 

ash-4.3# mount /dev/vg1/volume_2 /volume2
mount: wrong fs type, bad option, bad superblock on /dev/vg1/volume_2,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

ash-4.3# mount -o clear_cache /dev/vg1/volume_2 /volume2
mount: wrong fs type, bad option, bad superblock on /dev/vg1/volume_2,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

ash-4.3# mount -o recovery /dev/vg1/volume_2 /volume2
mount: wrong fs type, bad option, bad superblock on /dev/vg1/volume_2,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

ash-4.3# dmesg | tail
[1897666.766324] parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
[1897666.766961] parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
[1897666.766964] BTRFS error (device dm-1): BTRFS: dm-1 failed to repair parenttransid verify failure on 6095662284800, mirror = 1
[1897666.768033] parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
[1897666.768484] parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
[1897666.769166] parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
[1897666.769170] BTRFS error (device dm-1): BTRFS: dm-1 failed to repair parenttransid verify failure on 6095662284800, mirror = 2
[1897666.795109] BTRFS: open_ctree failed

 

Link to post
Share on other sites
  • 0

Some more, i case needed

 

ash-4.3# btrfs inspect-internal dump-super -f /dev/vg1/volume_2
superblock: bytenr=65536, device=/dev/vg1/volume_2
---------------------------------------------------------
csum                    0xccc63a47 [match]
bytenr                  65536
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    dfeb27ca-82a4-4861-a849-4ffe88ea67be
label                   2017.04.27-14:46:20 v15047
generation              15520602
root                    29360128
sys_array_size          129
chunk_root_generation   15518618
root_level              1
chunk_root              21037056
chunk_root_level        1
log_root                0
log_root_transid        0
log_root_level          0
total_bytes             11544872091648
bytes_used              9558720299008
sectorsize              4096
nodesize                16384
leafsize                16384
stripesize              4096
root_dir                6
num_devices             1
compat_flags            0x8000000000000000
compat_ro_flags         0x0
incompat_flags          0x16b
                        ( MIXED_BACKREF |
                          DEFAULT_SUBVOL |
                          COMPRESS_LZO |
                          BIG_METADATA |
                          EXTENDED_IREF |
                          SKINNY_METADATA )
csum_type               0
csum_size               4
cache_generation        9
uuid_tree_generation    15520601
dev_item.uuid           69a7b70c-63dc-47e3-ad61-0c2b10285219
dev_item.fsid           dfeb27ca-82a4-4861-a849-4ffe88ea67be [match]
dev_item.type           0
dev_item.total_bytes    11544872091648
dev_item.bytes_used     10724532289536
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          1
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0
sys_chunk_array[2048]:
        item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520)
                chunk length 8388608 owner 2 stripe_len 65536
                type SYSTEM|DUP num_stripes 2
                        stripe 0 devid 1 offset 20971520
                        dev uuid: 69a7b70c-63dc-47e3-ad61-0c2b10285219
                        stripe 1 devid 1 offset 29360128
                        dev uuid: 69a7b70c-63dc-47e3-ad61-0c2b10285219
backup_roots[4]:
        backup 0:
                backup_tree_root:       6095759638528   gen: 15520599   level: 1
                backup_chunk_root:      21037056        gen: 15518618   level: 1
                backup_extent_root:     6095758278656   gen: 15520599   level: 2
                backup_fs_root:         29573120        gen: 7  level: 0
                backup_dev_root:        6095662284800   gen: 15520571   level: 1
                backup_csum_root:       6095760113664   gen: 15520600   level: 3
                backup_total_bytes:     11544872091648
                backup_bytes_used:      9558720221184
                backup_num_devices:     1

        backup 1:
                backup_tree_root:       6095763111936   gen: 15520600   level: 1
                backup_chunk_root:      21037056        gen: 15518618   level: 1
                backup_extent_root:     6095761719296   gen: 15520600   level: 2
                backup_fs_root:         29573120        gen: 7  level: 0
                backup_dev_root:        6095662284800   gen: 15520571   level: 1
                backup_csum_root:       6095763636224   gen: 15520601   level: 3
                backup_total_bytes:     11544872091648
                backup_bytes_used:      9558720270336
                backup_num_devices:     1

        backup 2:
                backup_tree_root:       6095763980288   gen: 15520601   level: 1
                backup_chunk_root:      21037056        gen: 15518618   level: 1
                backup_extent_root:     6095767175168   gen: 15520602   level: 2
                backup_fs_root:         29573120        gen: 7  level: 0
                backup_dev_root:        6095662284800   gen: 15520571   level: 1
                backup_csum_root:       6095764750336   gen: 15520602   level: 3
                backup_total_bytes:     11544872091648
                backup_bytes_used:      9558719885312
                backup_num_devices:     1

        backup 3:
                backup_tree_root:       6095756312576   gen: 15520598   level: 1
                backup_chunk_root:      21037056        gen: 15518618   level: 1
                backup_extent_root:     6095756132352   gen: 15520598   level: 2
                backup_fs_root:         29573120        gen: 7  level: 0
                backup_dev_root:        6095662284800   gen: 15520571   level: 1
                backup_csum_root:       6095756607488   gen: 15520599   level: 3
                backup_total_bytes:     11544872091648
                backup_bytes_used:      9558720253952
                backup_num_devices:     1

 

Link to post
Share on other sites
  • 0
2 hours ago, flyride said:

 

Output below...

 

ash-4.3# btrfs rescue super /dev/vg1/volume_2
All supers are valid, no need to recover

ash-4.3# btrfs-find-root /dev/vg1/volume_2
parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
Ignoring transid failure
parent transid verify failed on 6095751888896 wanted 15520595 found 15507981
parent transid verify failed on 6095751888896 wanted 15520595 found 15507981
parent transid verify failed on 6095751888896 wanted 15520595 found 15507981
parent transid verify failed on 6095751888896 wanted 15520595 found 15507981
Ignoring transid failure
leaf parent key incorrect 6095751888896
Superblock thinks the generation is 15520604
Superblock thinks the level is 1
Found tree root at 29360128 gen 15520604 level 1

ash-4.3# btrfs insp dump-s -f /dev/vg1/volume_2
superblock: bytenr=65536, device=/dev/vg1/volume_2
---------------------------------------------------------
csum                    0xcdb9092e [match]
bytenr                  65536
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    dfeb27ca-82a4-4861-a849-4ffe88ea67be
label                   2017.04.27-14:46:20 v15047
generation              15520604
root                    29360128
sys_array_size          129
chunk_root_generation   15518618
root_level              1
chunk_root              21037056
chunk_root_level        1
log_root                0
log_root_transid        0
log_root_level          0
total_bytes             11544872091648
bytes_used              9558720299008
sectorsize              4096
nodesize                16384
leafsize                16384
stripesize              4096
root_dir                6
num_devices             1
compat_flags            0x8000000000000000
compat_ro_flags         0x0
incompat_flags          0x16b
                        ( MIXED_BACKREF |
                          DEFAULT_SUBVOL |
                          COMPRESS_LZO |
                          BIG_METADATA |
                          EXTENDED_IREF |
                          SKINNY_METADATA )
csum_type               0
csum_size               4
cache_generation        9
uuid_tree_generation    15520601
dev_item.uuid           69a7b70c-63dc-47e3-ad61-0c2b10285219
dev_item.fsid           dfeb27ca-82a4-4861-a849-4ffe88ea67be [match]
dev_item.type           0
dev_item.total_bytes    11544872091648
dev_item.bytes_used     10724532289536
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          1
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0
sys_chunk_array[2048]:
        item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520)
                chunk length 8388608 owner 2 stripe_len 65536
                type SYSTEM|DUP num_stripes 2
                        stripe 0 devid 1 offset 20971520
                        dev uuid: 69a7b70c-63dc-47e3-ad61-0c2b10285219
                        stripe 1 devid 1 offset 29360128
                        dev uuid: 69a7b70c-63dc-47e3-ad61-0c2b10285219
backup_roots[4]:
        backup 0:
                backup_tree_root:       6095759638528   gen: 15520599   level: 1
                backup_chunk_root:      21037056        gen: 15518618   level: 1
                backup_extent_root:     6095758278656   gen: 15520599   level: 2
                backup_fs_root:         29573120        gen: 7  level: 0
                backup_dev_root:        6095662284800   gen: 15520571   level: 1
                backup_csum_root:       6095760113664   gen: 15520600   level: 3
                backup_total_bytes:     11544872091648
                backup_bytes_used:      9558720221184
                backup_num_devices:     1

        backup 1:
                backup_tree_root:       6095763111936   gen: 15520600   level: 1
                backup_chunk_root:      21037056        gen: 15518618   level: 1
                backup_extent_root:     6095761719296   gen: 15520600   level: 2
                backup_fs_root:         29573120        gen: 7  level: 0
                backup_dev_root:        6095662284800   gen: 15520571   level: 1
                backup_csum_root:       6095763636224   gen: 15520601   level: 3
                backup_total_bytes:     11544872091648
                backup_bytes_used:      9558720270336
                backup_num_devices:     1

        backup 2:
                backup_tree_root:       6095763980288   gen: 15520601   level: 1
                backup_chunk_root:      21037056        gen: 15518618   level: 1
                backup_extent_root:     6095767175168   gen: 15520602   level: 2
                backup_fs_root:         29573120        gen: 7  level: 0
                backup_dev_root:        6095662284800   gen: 15520571   level: 1
                backup_csum_root:       6095764750336   gen: 15520602   level: 3
                backup_total_bytes:     11544872091648
                backup_bytes_used:      9558719885312
                backup_num_devices:     1

        backup 3:
                backup_tree_root:       6095756312576   gen: 15520598   level: 1
                backup_chunk_root:      21037056        gen: 15518618   level: 1
                backup_extent_root:     6095756132352   gen: 15520598   level: 2
                backup_fs_root:         29573120        gen: 7  level: 0
                backup_dev_root:        6095662284800   gen: 15520571   level: 1
                backup_csum_root:       6095756607488   gen: 15520599   level: 3
                backup_total_bytes:     11544872091648
                backup_bytes_used:      9558720253952
                backup_num_devices:     1

 

Link to post
Share on other sites
  • 0

i did...

got after every command the "wrong fs type" error

 

ash-4.3# mount -o ro /dev/vg1/volume_2 /volume2
mount: wrong fs type, bad option, bad superblock on /dev/vg1/volume_2,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

ash-4.3# dmesg | tail
[  707.211487] parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
[  707.216825] parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
[  707.216828] BTRFS error (device dm-1): BTRFS: dm-1 failed to repair parent transid verify failure on 6095662284800, mirror = 1

[  707.235314] parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
[  707.245467] parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
[  707.254106] parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
[  707.254109] BTRFS error (device dm-1): BTRFS: dm-1 failed to repair parent transid verify failure on 6095662284800, mirror = 2

[  707.270652] BTRFS: open_ctree failed

 

Link to post
Share on other sites
  • 0

i'm already as root/sudo on the machine, but tried it too with sudo before...

 

ash-4.3# mount -o recovery,ro /dev/vg1/volume_2 /volume2
mount: wrong fs type, bad option, bad superblock on /dev/vg1/volume_2,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
ash-4.3# sudo mount -o recovery,ro /dev/vg1/volume_2 /volume2
mount: wrong fs type, bad option, bad superblock on /dev/vg1/volume_2,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
ash-4.3# dmesg | tail
[ 1303.154209] parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
[ 1303.154899] parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
[ 1303.154902] BTRFS error (device dm-1): BTRFS: dm-1 failed to repair parent transid verify failure on 6095662284800, mirror = 1

[ 1303.155769] parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
[ 1303.156247] parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
[ 1303.156910] parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
[ 1303.156912] BTRFS error (device dm-1): BTRFS: dm-1 failed to repair parent transid verify failure on 6095662284800, mirror = 2

[ 1303.177893] BTRFS: open_ctree failed

 

Link to post
Share on other sites
  • 0
Posted (edited)
20 hours ago, flyride said:

Try the alternate tree roots

i tried to use them with the script

 

it doesn't create a single "333..." WellBlock file

Since it doesn't find a single well block...

ash-4.3# btrfs-find-root /dev/vg1/volume_2
parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
parent transid verify failed on 6095662284800 wanted 15520571 found 15507911
Ignoring transid failure
parent transid verify failed on 6095751888896 wanted 15520595 found 15507981
parent transid verify failed on 6095751888896 wanted 15520595 found 15507981
parent transid verify failed on 6095751888896 wanted 15520595 found 15507981
parent transid verify failed on 6095751888896 wanted 15520595 found 15507981
Ignoring transid failure
leaf parent key incorrect 6095751888896
Superblock thinks the generation is 15520604
Superblock thinks the level is 1
Found tree root at 29360128 gen 15520604 level 1

 

Edited by frooop
Link to post
Share on other sites
  • 0

You're running low on options... please take a look at this post. If none of the information works for you, that is probably the extent of what I know to help you.

https://xpenology.com/forum/topic/14337-volume-crash-after-4-months-of-stability/?do=findComment&comment=108021

 

There might be some btrfs experts out there in the ether but I don't think you'll get any other options here.

 

Good luck and if something works, please report back!

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.