Hi guys,
I decided to make a call for help as right now I'm stuck on recovering data from my BTRFS drive.
I am using a hardware RAID 1 on the back-end [2x 4TB WD Red Drives], and on the front-end, on XPEnology I configured a Basic RAID Group with only "one drive" passed from ESXi.
Until this January I've been using EXT file system, but I read that BTRFS is better both in speed and stability terms, so I decided to give it a go :)
I run my system on UPS which can keep the system powered for more than 4 hours, in case of a blackout, so I though that my data was safe.
Two weeks ago I decided to power off the system after 3 months of continual usage, just for checking that everything is okay and to clean it's inside for dust, as a normal routine check.
Unfortunately, after I powered on my system, the main data drive from XPEnology was crashed, not even degraded.
I rebooted it thinking that it might run a fsck by itself, but unfortunately it remained crashed.
I ejected both drives from the system and ran extensive checks on them, both smart tests and surface checks, and everything looks just fine.
Given these, I decided to eject one of the drives from the system and to connect it to an external docking station, in order to perform other troubleshooting.
Unfortunately, seems that I'm unable to get it mounted, even as RO, with recovery parameters :(
I have the drive connected to a live Parted Magic OS running, for troubleshooting and hope so, recovery procedures :)
This is the output of some of the commands I issued, but with no luck to have it mounted:
root@PartedMagic:~# btrfs version
btrfs-progs v4.15
root@PartedMagic:~# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md3 : active raid1 sda3[0]
3902163776 blocks super 1.2 [1/1]
root@PartedMagic:~# lsblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT
loop253 squashfs
sr0
loop254 squashfs
loop252 squashfs
sda
├─sda2 linux_raid_member 7b661c20-ea68-b207-e8f5-bc45dd5a58c4
├─sda3 linux_raid_member DISKSTATION:3 be150a49-a5a3-f915-32bf-80e4732a20ac
│ └─md3 btrfs 2018.01.02-20:42:27 v15217 1457616f-5daf-4487-ba1c-07963a0c4723
└─sda1 linux_raid_member
root@PartedMagic:~# btrfs fi show -d
Label: '2018.01.02-20:42:27 v15217' uuid: 1457616f-5daf-4487-ba1c-07963a0c4723
Total devices 1 FS bytes used 2.15TiB
devid 1 size 3.63TiB used 2.21TiB path /dev/md3
root@PartedMagic:~# mount -t btrfs -oro,degraded,recovery /dev/md3 /mnt/temp1
mount: wrong fs type, bad option, bad superblock on /dev/md3,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
root@PartedMagic:~# dmesg | tail -n 100
[SNIP]
[39026.756985] BTRFS info (device md3): allowing degraded mounts
[39026.756988] BTRFS warning (device md3): 'recovery' is deprecated, use 'usebackuproot' instead
[39026.756989] BTRFS info (device md3): trying to use backup root at mount time
[39026.756990] BTRFS info (device md3): disk space caching is enabled
[39026.756992] BTRFS info (device md3): has skinny extents
[39027.082642] BTRFS error (device md3): bad tree block start 1917848902723122217 2473933062144
[39027.083051] BTRFS error (device md3): bad tree block start 5775457142092792234 2473933062144
[39027.083552] BTRFS error (device md3): bad tree block start 1917848902723122217 2473933062144
[39027.083936] BTRFS error (device md3): bad tree block start 5775457142092792234 2473933062144
[39027.097706] BTRFS error (device md3): bad tree block start 1917848902723122217 2473933062144
[39027.098146] BTRFS error (device md3): bad tree block start 5775457142092792234 2473933062144
[39027.114806] BTRFS error (device md3): bad tree block start 1917848902723122217 2473933062144
[39027.115410] BTRFS error (device md3): bad tree block start 5775457142092792234 2473933062144
[39027.133510] BTRFS error (device md3): bad tree block start 1917848902723122217 2473933062144
[39027.133941] BTRFS error (device md3): bad tree block start 5775457142092792234 2473933062144
[39027.136206] BTRFS error (device md3): open_ctree failed
root@PartedMagic:~# btrfsck /dev/md3
checksum verify failed on 2473933062144 found 908A6889 wanted 1646EB1A
checksum verify failed on 2473933062144 found 908A6889 wanted 1646EB1A
checksum verify failed on 2473933062144 found E41F2C90 wanted 90ED32B2
checksum verify failed on 2473933062144 found 908A6889 wanted 1646EB1A
bytenr mismatch, want=2473933062144, have=1917848902723122217
Couldn't setup device tree
ERROR: cannot open file system
root@PartedMagic:~# btrfs check --repair /dev/md3
enabling repair mode
checksum verify failed on 2473933062144 found 908A6889 wanted 1646EB1A
checksum verify failed on 2473933062144 found 908A6889 wanted 1646EB1A
checksum verify failed on 2473933062144 found E41F2C90 wanted 90ED32B2
checksum verify failed on 2473933062144 found 908A6889 wanted 1646EB1A
bytenr mismatch, want=2473933062144, have=1917848902723122217
Couldn't setup device tree
ERROR: cannot open file system
root@PartedMagic:~# btrfs restore -v -i /dev/md3 /mnt/temp1
checksum verify failed on 2473933062144 found 908A6889 wanted 1646EB1A
checksum verify failed on 2473933062144 found 908A6889 wanted 1646EB1A
checksum verify failed on 2473933062144 found E41F2C90 wanted 90ED32B2
checksum verify failed on 2473933062144 found 908A6889 wanted 1646EB1A
bytenr mismatch, want=2473933062144, have=1917848902723122217
Couldn't setup device tree
Could not open root, trying backup super
checksum verify failed on 2473933062144 found 908A6889 wanted 1646EB1A
checksum verify failed on 2473933062144 found 908A6889 wanted 1646EB1A
checksum verify failed on 2473933062144 found E41F2C90 wanted 90ED32B2
checksum verify failed on 2473933062144 found 908A6889 wanted 1646EB1A
bytenr mismatch, want=2473933062144, have=1917848902723122217
Couldn't setup device tree
Could not open root, trying backup super
checksum verify failed on 2473933062144 found 908A6889 wanted 1646EB1A
checksum verify failed on 2473933062144 found 908A6889 wanted 1646EB1A
checksum verify failed on 2473933062144 found E41F2C90 wanted 90ED32B2
checksum verify failed on 2473933062144 found 908A6889 wanted 1646EB1A
bytenr mismatch, want=2473933062144, have=1917848902723122217
Couldn't setup device tree
Could not open root, trying backup super
root@PartedMagic:~# btrfs rescue super-recover -v /dev/md3
All Devices:
Device: id = 1, name = /dev/md3
Before Recovering:
[All good supers]:
device name = /dev/md3
superblock bytenr = 65536
device name = /dev/md3
superblock bytenr = 67108864
device name = /dev/md3
superblock bytenr = 274877906944
[All bad supers]:
All supers are valid, no need to recover
root@PartedMagic:~# smartctl -a /dev/sda | grep PASSED
SMART overall-health self-assessment test result: PASSED
root@PartedMagic:~# btrfs inspect-internal dump-super /dev/md3
superblock: bytenr=65536, device=/dev/md3
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x91d56f51 [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 1457616f-5daf-4487-ba1c-07963a0c4723
label 2018.01.02-20:42:27 v15217
generation 31802
root 2473938698240
sys_array_size 129
chunk_root_generation 30230
root_level 1
chunk_root 20987904
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 3995815706624
bytes_used 2366072147968
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 1
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x16b
( MIXED_BACKREF |
DEFAULT_SUBVOL |
COMPRESS_LZO |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )
cache_generation 22402
uuid_tree_generation 31802
dev_item.uuid 8c0579fe-b94c-465a-8005-c55991b8727e
dev_item.fsid 1457616f-5daf-4487-ba1c-07963a0c4723 [match]
dev_item.type 0
dev_item.total_bytes 3995815706624
dev_item.bytes_used 2428820783104
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
Given these, do you have any other idea that I should apply to get my data recovered? In case you need the output of other troubleshooting commands, please, just let me know, and I will post them here.
Thank you in advance for your tips and help! :)