Jump to content
XPEnology Community

DANGER : Raid crashed. Help me restore my data!


Recommended Posts

 

5 minutes ago, jbesclapez said:

The 8TB is out of the system. I edidted teh fstab, did the reboot and the fstab goes back to its previous state without our work.

Did you note also that when I log on with putty i get this error > Could not chdir to home directory /var/services/homes/admin: No such file or directory
Any idea on what is happening?

 

The symlink to the homes root folder has been damaged.  I don't think you should worry about that now.

 

Can you mount your filesystem manually?

# mount -v -oro,noload,sb=1934917632 /dev/vg1000/lv /volume1

 

Link to comment
Share on other sites

Just now, flyride said:

 

 

The symlink to the homes root folder has been damaged.  I don't think you should worry about that now.

 

Can you mount your filesystem manually?


# mount -v -oro,noload,sb=1934917632 /dev/vg1000/lv /volume1

 

root@DiskStation:~# mount -v -oro,noload,sb=1934917632 /dev/vg1000/lv /volume1
mount: wrong fs type, bad option, bad superblock on /dev/vg1000/lv,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

 

root@DiskStation:~# dmesg | tail
[  116.513849] ata4.00: configured for UDMA/133
[  116.513857] ata4: EH complete
[  116.606535] init: pkg-php56-fpm main process (13020) terminated with status 1
[  116.606564] init: pkg-php56-fpm main process ended, respawning
[  116.642449] init: pkg-php56-fpm main process (13038) terminated with status 1
[  116.642485] init: pkg-php56-fpm main process ended, respawning
[  116.685181] init: pkg-php56-fpm main process (13046) terminated with status 1
[  116.685216] init: pkg-php56-fpm respawning too fast, stopped
[  762.662594] hfsplus: unable to parse mount options
[  762.667635] UDF-fs: bad mount option "noload" or missing value


 

Link to comment
Share on other sites

Just now, flyride said:

Just to make sure, 


# lvm lvmdiskscan
# vgchange -ay
# mount -v -oro,noload,sb=1934917632 /dev/vg1000/lv /volume1

 

root@DiskStation:~# lvm lvmdiskscan
  /dev/md2 [       8.17 TiB] LVM physical volume
  0 disks
  0 partitions
  0 LVM physical volume whole disks
  1 LVM physical volume
root@DiskStation:~# vgchange -ay
  1 logical volume(s) in volume group "vg1000" now active
root@DiskStation:~# mount -v -oro,noload,sb=1934917632 /dev/vg1000/lv /volume1
mount: wrong fs type, bad option, bad superblock on /dev/vg1000/lv,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
 

Link to comment
Share on other sites

root@DiskStation:~# mke2fs -n /dev/vg1000/lv
mke2fs 1.42.6 (21-Sep-2012)
Filesystem label=1.42.6-15266
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=16 blocks, Stripe width=48 blocks
274272256 inodes, 2194148352 blocks
25600 blocks (0.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
66961 block groups
32768 blocks per group, 32768 fragments per group
4096 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848, 512000000, 550731776, 644972544, 1934917632

Same output...

Link to comment
Share on other sites

Just now, flyride said:

# mount -v -oro,noload,sb=644972544 /dev/vg1000/lv /volume1
# dmesg | tail

root@DiskStation:~# mount -v -oro,noload,sb=644972544 /dev/vg1000/lv /volume1
mount: wrong fs type, bad option, bad superblock on /dev/vg1000/lv,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
root@DiskStation:~# dmesg | tail
[  880.920192] Got empty serial number. Generate serial number from product.
[  880.938475] input: Microsoft Wired Keyboard 600 as /devices/pci0000:00/0000:00:1d.1/usb7/7-2/7-2:1.0/input/input4
[  880.938490] hid-generic 0003:045E:0750.0003: input: USB HID v1.11 Keyboard [Microsoft Wired Keyboard 600] on usb-0000:00:1d.1-2/input0
[  880.969333] input: Microsoft Wired Keyboard 600 as /devices/pci0000:00/0000:00:1d.1/usb7/7-2/7-2:1.1/input/input5
[  880.969339] Get empty minor:104
[  880.969399] hid-generic 0003:045E:0750.0004: input,hiddev0: USB HID v1.11 Device [Microsoft Wired Keyboard 600] on usb-0000:00:1d.1-2/input1
[ 1469.210206] hfsplus: unable to parse mount options
[ 1469.215261] UDF-fs: bad mount option "noload" or missing value
[ 2685.853634] hfsplus: unable to parse mount options
[ 2685.858674] UDF-fs: bad mount option "noload" or missing value
root@DiskStation:~#


 

 

Link to comment
Share on other sites

It seems like it can't figure out the file system type at mount time now.  Add a space between "-o" and "ro" - it shouldn't matter but please do it anyway.

# mount -t ext4 -v -o ro,noload,sb=644972544 /dev/vg1000/lv /volume1
# dmesg | tail

 

Edited by flyride
Link to comment
Share on other sites

Just now, flyride said:

It seems like it can't figure out the file system type at mount time now.  Add a space between "-o" and "ro" - it shouldn't matter but please do it anyway.


# mount -t ext4 -v -o ro,noload,sb=644972544 /dev/vg1000/lv /volume1
# dmesg | tail

 

 


 

root@DiskStation:~# mount -t ext4 -v -o ro,noload,sb=644972544 /dev/vg1000/lv /volume1
mount: wrong fs type, bad option, bad superblock on /dev/vg1000/lv,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

 

Link to comment
Share on other sites

Just now, flyride said:

dmesg?

Sorry I thought you did not need it as the result was the same. Here it is :

 

root@DiskStation:~# dmesg | tail
[ 3737.954274] hid-generic 0003:045E:0750.0006: input,hiddev0: USB HID v1.11 Device [Microsoft Wired Keyboard 600] on usb-0000:00:1d.1-2/input1
[ 3740.204044] hub 7-0:1.0: port 2 disabled by hub (EMI?), re-enabling...
[ 3740.210578] usb 7-2: USB disconnect, device number 4
[ 3740.695011] usb 7-2: new low-speed USB device number 5 using uhci_hcd
[ 3740.881327] Got empty serial number. Generate serial number from product.
[ 3740.899617] input: Microsoft Wired Keyboard 600 as /devices/pci0000:00/0000:00:1d.1/usb7/7-2/7-2:1.0/input/input8
[ 3740.899630] hid-generic 0003:045E:0750.0007: input: USB HID v1.11 Keyboard [Microsoft Wired Keyboard 600] on usb-0000:00:1d.1-2/input0
[ 3740.930467] input: Microsoft Wired Keyboard 600 as /devices/pci0000:00/0000:00:1d.1/usb7/7-2/7-2:1.1/input/input9
[ 3740.930474] Get empty minor:104
[ 3740.930534] hid-generic 0003:045E:0750.0008: input,hiddev0: USB HID v1.11 Device [Microsoft Wired Keyboard 600] on usb-0000:00:1d.1-2/input1


 

Link to comment
Share on other sites

Just now, flyride said:

I don't really understand all the usb events in your dmesg.  Are you plugging and unplugging devices?

 

Anyway, please review dmesg for the last message that is related to the mount operation, as it is not shown in what you posted.

 

Flyride, I am not touching an USB. I am not touching the server basically.
I do not really understand what you ask me to do with the msg on the  mount operation?

Edited by jbesclapez
Link to comment
Share on other sites

root@DiskStation:~# mount -t ext4 -v -o ro,noload,sb=644972544 /dev/vg1000/lv /volume1
mount: wrong fs type, bad option, bad superblock on /dev/vg1000/lv,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
root@DiskStation:~# dmesg | tail
[ 3740.204044] hub 7-0:1.0: port 2 disabled by hub (EMI?), re-enabling...
[ 3740.210578] usb 7-2: USB disconnect, device number 4
[ 3740.695011] usb 7-2: new low-speed USB device number 5 using uhci_hcd
[ 3740.881327] Got empty serial number. Generate serial number from product.
[ 3740.899617] input: Microsoft Wired Keyboard 600 as /devices/pci0000:00/0000:00:1d.1/usb7/7-2/7-2:1.0/input/input8
[ 3740.899630] hid-generic 0003:045E:0750.0007: input: USB HID v1.11 Keyboard [Microsoft Wired Keyboard 600] on usb-0000:00:1d.1-2/input0
[ 3740.930467] input: Microsoft Wired Keyboard 600 as /devices/pci0000:00/0000:00:1d.1/usb7/7-2/7-2:1.1/input/input9
[ 3740.930474] Get empty minor:104
[ 3740.930534] hid-generic 0003:045E:0750.0008: input,hiddev0: USB HID v1.11 Device [Microsoft Wired Keyboard 600] on usb-0000:00:1d.1-2/input1
[ 4796.371660] EXT4-fs (dm-0): VFS: Can't find ext4 filesystem

 

Link to comment
Share on other sites

58 minutes ago, jbesclapez said:

[ 4796.371660] EXT4-fs (dm-0): VFS: Can't find ext4 filesystem

 

 

So that is where you are now.  48 hours ago, this was the state:

 

On 4/10/2020 at 12:35 PM, flyride said:

But here's the situation:

  1. Your RAID5 array is critical (no redundancy) and has mild corruption
  2. Your filesystem has some corruption but we have been able to get it to mount

My strong recommendation is that you not attempt to "fix" anything further, and do the following:

  1. Copy everything off your volume1 onto another device.  If you need to go buy an 8TB external drive, do it.
  2. Delete your volume1
  3. Delete your SHR
  4. Click the Fix System Partition options in Storage Manager to correct the DSM and swap replicas
  5. Remove/replace your bad drive #0/sda
  6. Create a new SHR
  7. Create a new volume1
  8. Copy your files back

 

I also said this:

 

On 4/10/2020 at 12:20 PM, flyride said:

I'm pretty sure Syno won't rewrite fstab as long as you don't make any changes to the GUI.

 

Up to this point, we had really made *NO* irreversible changes to the system, aside from rewriting the array superblocks when executing the array create.  The point of editing the fstab file is so we did not have to do a filesystem recovery or other action that would alter the data in the filesystem.  When the filesystem was finally mounted, it was read-only so that no other action could damage the data within.

 

Then you asked if you could attach an 8TB drive directly to the system. Aside from diverging from recommendation, I was not expecting you to use the GUI to configure it.  It could have been attached via usb, or mkfs on the new device. Unfortunately GUI was used and I believe that DSM tried to write to your lv device directly to build the new filesystem, which in all evidence has corrupted the filesystem that was already there.  So when dmesg says "can't find ext4 filesystem" that is probably the reason why.

 

I think that now there's no direct way to get to the data on the filesystem without heroic and/or irreversible filesystem check/recovery operations. That's the bad news.

The good news is that your array is still in the same state (critical, but intact). And very likely that most of your data is still physically present on the array.

 

You basically have three choices now, and to be blunt, you're on your own with the decision and how to proceed further.

 

1. Send your drives off for forensic file recovery. This is expensive, but they will find data and recover it for you.  If you do this, it's vitally important that you tell them the following:

  • You are sending a FOUR-drive Synology SHR, with one missing member
  • drive 0 is missing
  • drive 1 is /dev/sdc5:      Array UUID : 75762e2e:4629b4db:259f216e:a39c266d     Device UUID : 6ba575e4:53121f53:a8fe4876:173d11a9
  • drive 2 is /dev/sdb5:      Array UUID : 75762e2e:4629b4db:259f216e:a39c266d     Device UUID : 7eee55dc:dbbf5609:e737801d:87903b6c
  • drive 3 is /dev/sdd5:      Array UUID : 75762e2e:4629b4db:259f216e:a39c266d     Device UUID : fb417ce4:fcdd58fb:72d35e06:9d7098b5
  • The filesystem type is ext4, NOT BTRFS (there may be spurious btrfs signatures in various places in /dev/lv)

2. Attempt to recover the filesystem using e2fsck or variant

  • At its core, fsck is an irreversible process, which might invalidate a subsequent choice to send the drives for forensic recovery
  • If I were to do this, I would first clone /dev/lv to another device as a backup.  Unfortunately you don't possess a device large enough to do this (8.17TB > 7.27TB)

3. Use third-party utilities to attempt to recover data from the /dev/lv device directly

  • You would need to be able to safely install such utility via command line
  • You would need to connect storage. The best way to do that would be to connect your 8TB via usb which then would be accessible as a mounted filesystem.
  • These tools are not plug and play, may need some technical guidance, and may have varying degrees of effectiveness

Here's a Google search to help you with your decisions:

https://www.google.com/search?q=recover+data+from+unmountable+ext4&oq=recover+data+from+unmountable+ext4

 

Edited by flyride
Link to comment
Share on other sites

Just now, flyride said:

 

So that is where you are now.  48 hours ago, this was the state:

 

 

I also said this:

 

 

Up to this point, we had really made *NO* irreversible changes to the system, aside from rewriting the array superblocks when executing the array create.  The point of editing the fstab file is so we did not have to do a filesystem recovery or other action that would alter the data in the filesystem.  When the filesystem was finally mounted, it was read-only so that no other action could damage the data within.

 

Then you asked if you could attach an 8TB drive directly to the system. Aside from diverging from recommendation, I was not expecting you to use the GUI to configure it.  It could have been attached via usb, or mkfs on the new device. Unfortunately GUI was used and I believe that DSM tried to write to your lv device directly to build the new filesystem, which in all evidence has corrupted the filesystem that was already there.  So when dmesg says "can't find ext4 filesystem" that is probably the reason why.

 

I think that now there's no direct way to get to the data on the filesystem without heroic and/or irreversible filesystem check/recovery operations. That's the bad news.

The good news is that your array is still in the same state (critical, but intact). And very likely that most of your data is still physically present on the array.

 

You basically have three choices now, and to be blunt, you're on your own with the decision and how to proceed further.

 

1. Send your drives off for forensic file recovery. This is expensive, but they will find data and recover it for you.  If you do this, it's vitally important that you tell them the following:

  • You are sending a FOUR-drive Synology SHR, with one missing member
  • drive 0 is missing
  • drive 1 is /dev/sdc5:      Array UUID : 75762e2e:4629b4db:259f216e:a39c266d     Device UUID : 6ba575e4:53121f53:a8fe4876:173d11a9
  • drive 2 is /dev/sdb5:      Array UUID : 75762e2e:4629b4db:259f216e:a39c266d     Device UUID : 7eee55dc:dbbf5609:e737801d:87903b6c
  • drive 3 is /dev/sdd5:      Array UUID : 75762e2e:4629b4db:259f216e:a39c266d     Device UUID : fb417ce4:fcdd58fb:72d35e06:9d7098b5
  • The filesystem type is ext4, NOT BTRFS (there may be spurious btrfs signatures in various places in /dev/lv)

2. Attempt to recover the filesystem using e2fsck or variant

  • At its core, fsck is an irreversible process, which might invalidate a subsequent choice to send the drives for forensic recovery
  • If I were to do this, I would first clone /dev/lv to another device as a backup.  Unfortunately you don't possess a device large enough to do this (8.17TB > 7.27TB)

3. Use third-party utilities to attempt to recover data from the /dev/lv device directly

  • You would need to be able to safely install such utility via command line
  • You would need to connect storage. The best way to do that would be to connect your 8TB via usb which then would be accessible as a mounted filesystem.
  • These tools are not plug and play, may need some technical guidance, and may have varying degrees of effectiveness

Here's a Google search to help you with your decisions:

https://www.google.com/search?q=recover+data+from+unmountable+ext4&oq=recover+data+from+unmountable+ext4

 

I cant believe it! we were so close and yet now so far! I totally ******* up by using GUI!

I will do some reading and give it a bit of time.

I need to look at my other back ups to see what I lost exactly, but I am not ready to go to Forensic as it is too expensive I think!

Thanks for your time , I will come back to you!

 

 

Link to comment
Share on other sites

12 hours ago, flyride said:

 

Good morning Flyride!

 

I had a tough night to be honest, I was thinking of my mistake. All this comes from a big misundertanding from my side !

I did some reading like you recommended and I also contacted a FORENSIC to ask for a quote with all the details you gave me.

Then I did more reading to see what i could do. Now I need your opinion, as once again, I might misundertand things :-)

 

I was thinking of doing a RAW copy of the drives and then some analysis. I will have to buy a new drive - but anyway I was thinking of upgrading my NAS as it is pretty old, so the drive is not a waste. 

Before diving into this, I would like to know if you still would be OK to assist me as you did before? I will surely need your help in that, and I was wondering if you would be OK to continue this adventure with me :-). I would prefer having your help than the one of a Forensic! At least, I am learning a side of IT I did not know before... It is also kind of fun, i have to admit, even it plays with my nerves.

 

A good news is that I realised yesterday I had a drive with the backup of 99% of my personnal videos (kids, family). What is left on the NAS, I would still like to recover it...

 

Happy Easter to you!

 

 

 

 

 

Link to comment
Share on other sites

may I make a suggestion to you here.

If you plan is to upgrade your xpenology

I suggest to shutdown your actual version

remove your 4 disk array

Put only your 8TB disk and install newest DSM but Don't create a volume with your 8tb disk

shutdown your new dsm

Put back your 4 disk array, and start your dsm.

Let's see if you are now with  a array (still degraded) but without filesystem corruption.

as you said you got 99% of your data arleady saved into another backup, you have just 1% of possibility loosing stuff here.

Link to comment
Share on other sites

13 minutes ago, supermounter said:

may I make a suggestion to you here.

If you plan is to upgrade your xpenology

I suggest to shutdown your actual version

remove your 4 disk array

Put only your 8TB disk and install newest DSM but Don't create a volume with your 8tb disk

shutdown your new dsm

Put back your 4 disk array, and start your dsm.

Let's see if you are now with  a array (still degraded) but without filesystem corruption.

as you said you got 99% of your data arleady saved into another backup, you have just 1% of possibility loosing stuff here.

 

Hi Supermounter! With all due respect,  I do not really see the point in that as the array is arlready seen by my DSM if you look at the print screens. The filesystem will not recover on its own...

 

 

image.thumb.png.89f21e8022e6c4c698a7a7a273d52f64.png

 

Link to comment
Share on other sites

Yes you probabely right

I was thinking about a repair of your array with the spare 8TB disk...as I said, you have nothing to loose now to try different approach.

If FSTAB is now messy with the new disk addon, this will be maybe a solution to fix you file system with the disk 3 once the resync will perform

Link to comment
Share on other sites

3 minutes ago, supermounter said:

Yes you probabely right

I was thinking about a repair of your array with the spare 8TB disk...as I said, you have nothing to loose now to try different approach.

If FSTAB is now messy with the new disk addon, this will be maybe a solution to fix you file system with the disk 3 once the resync will perform

 

I see things a bit differently. I think I have everything to loose. I think everything can be forecasted and planned. There is no maybe. I was simply really stupid to use the GUI yesterday! And as Flyride said,  it should have been avoided if I did not miread/got confused. I am not ready to change anything without having at least a RAW backup of the disks. 
I am waiting for my guru to reply :-)

 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...