Jump to content
XPEnology Community

DANGER : Raid crashed. Help me restore my data!


Recommended Posts

Just now, flyride said:

That's good.  This next command may or may not work, and it may or may not take a long time.  Don't interrupt it.


# e2fsck -b 32768 /dev/vg1000/lv

 

 

 

Quote

 

root@DiskStation:/# e2fsck -b 32768 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
Journal transaction 20926119 was corrupt, replay was aborted.
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

 

 

I feel we are really unlucky!
 

Link to comment
Share on other sites

Repeat the last command, substituting each of the superblocks from this list in sequence (we already tried 32768)

Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848, 512000000, 550731776, 644972544, 1934917632

If one of them produces an output different than what you just received, stop and post the results.

Link to comment
Share on other sites

5 minutes ago, flyride said:

Repeat the last command, substituting each of the superblocks from this list in sequence (we already tried 32768)


Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848, 512000000, 550731776, 644972544, 1934917632

If one of them produces an output different than what you just received, stop and post the results.

 

None of them produces an output different. 

Link to comment
Share on other sites

6 minutes ago, flyride said:

Repeat the last command, substituting each of the superblocks from this list in sequence (we already tried 32768)


Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848, 512000000, 550731776, 644972544, 1934917632

If one of them produces an output different than what you just received, stop and post the results.

root@DiskStation:/# e2fsck -b 32768 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
Journal transaction 20926119 was corrupt, replay was aborted.
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# ^C
root@DiskStation:/# e2fsck -b 98304 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 163840 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 229376 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 294912 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 819200 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 884736 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 1605632 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 2654208 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 4096000 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 7962624 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 11239424 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 20480000 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 23887872 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 71663616 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 78675968 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 102400000 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 214990848 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 512000000 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 550731776 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 644972544 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

root@DiskStation:/# e2fsck -b 1934917632 /dev/vg1000/lv
e2fsck 1.42.6 (21-Sep-2012)
1.42.6-3202: recovering journal
e2fsck: unable to set superblock flags on 1.42.6-3202


1.42.6-3202: ***** FILE SYSTEM WAS MODIFIED *****

1.42.6-3202: ********** WARNING: Filesystem still has errors **********

 

Link to comment
Share on other sites

Just now, flyride said:

I must point out that all the superblocks you tried were different than the first output.  e2fsck did not report corrupted journal entries.


mount -v -b 1934917632 -oro,noload /dev/vg1000/lv /volume1

 

root@DiskStation:/# mount -v -b 1934917632 -oro,noload /dev/vg1000/lv /volume1
mount: invalid option -- 'b'

Usage:
 mount [-lhV]
 mount -a [options]
 mount [options] [--source] <source> | [--target] <directory>
 mount [options] <source> <directory>
 mount <operation> <mountpoint> [<target>]

Mount a filesystem.

Options:
 -a, --all               mount all filesystems mentioned in fstab
 -c, --no-canonicalize   don't canonicalize paths
 -f, --fake              dry run; skip the mount(2) syscall
 -F, --fork              fork off for each device (use with -a)
 -T, --fstab <path>      alternative file to /etc/fstab
 -h, --help              display this help text and exit
 -i, --internal-only     don't call the mount.<type> helpers
 -l, --show-labels       lists all mounts with LABELs
 -n, --no-mtab           don't write to /etc/mtab
 -o, --options <list>    comma-separated list of mount options
 -O, --test-opts <list>  limit the set of filesystems (use with -a)
 -r, --read-only         mount the filesystem read-only (same as -o ro)
 -t, --types <list>      limit the set of filesystem types
     --source <src>      explicitly specifies source (path, label, uuid)
     --target <target>   explicitly specifies mountpoint
 -v, --verbose           say what is being done
 -V, --version           display version information and exit
 -w, --rw, --read-write  mount the filesystem read-write (default)

 -h, --help     display this help and exit
 -V, --version  output version information and exit

Source:
 -L, --label <label>     synonym for LABEL=<label>
 -U, --uuid <uuid>       synonym for UUID=<uuid>
 LABEL=<label>           specifies device by filesystem label
 UUID=<uuid>             specifies device by filesystem UUID
 PARTLABEL=<label>       specifies device by partition label
 PARTUUID=<uuid>         specifies device by partition UUID
 <device>                specifies device by path
 <directory>             mountpoint for bind mounts (see --bind/rbind)
 <file>                  regular file for loopdev setup

Operations:
 -B, --bind              mount a subtree somewhere else (same as -o bind)
 -M, --move              move a subtree to some other place
 -R, --rbind             mount a subtree and all submounts somewhere else
 --make-shared           mark a subtree as shared
 --make-slave            mark a subtree as slave
 --make-private          mark a subtree as private
 --make-unbindable       mark a subtree as unbindable
 --make-rshared          recursively mark a whole subtree as shared
 --make-rslave           recursively mark a whole subtree as slave
 --make-rprivate         recursively mark a whole subtree as private
 --make-runbindable      recursively mark a whole subtree as unbindable

For more details see mount(8).

 

Link to comment
Share on other sites

Ok, I think we couldn't modify the superblock because we had set the array to read only (that only makes sense).

 

Check your files, they should be there (File Station).  I expect if files are there, there will be some data corruption.  ext4 won't really be able to tell you about it.

 

Can you map a drive or UNC to your data? (\\nasname\volume1 in Windows Explorer or Finder)?

 

 

Edited by flyride
Link to comment
Share on other sites

4 minutes ago, flyride said:

Ok, I think we couldn't modify the superblock because we had set the array to read only (that only makes sense).

 

Check your files, they should be there (File Manager).  I expect if files are there, there will be some data corruption.  ext4 won't really be able to tell you about it.

 

Can you map a drive or UNC to your data? (\\nasname\volume1 in Windows Explorer or Finder)?

 

 

 

I lost my syno config but i dont mind.

I am still a bit worried but when i go do this i feel better :

 

root@DiskStation:/# cd volume1/
root@DiskStation:/volume1# ls
@appstore      Download                     esynoscheduler.output.14154  esynoscheduler.output.14693  esynoscheduler.output.15257  music               synoquota.db
aquota.group   @eaDir                       esynoscheduler.output.14201  esynoscheduler.output.14698  esynoscheduler.output.15345  music-kids          Time Machine
aquota.user    Ebooks                       esynoscheduler.output.14285  esynoscheduler.output.14724  esynoscheduler.output.15665  NetBackup           @tmp
BackUp         esynoscheduler.output.12636  esynoscheduler.output.14312  esynoscheduler.output.14740  esynoscheduler.output.15909  photo               @tmp.del
BD             esynoscheduler.output.13578  esynoscheduler.output.14335  esynoscheduler.output.14744  esynoscheduler.output.17013  Plex                ubooquity.conf
books          esynoscheduler.output.13591  esynoscheduler.output.14348  esynoscheduler.output.14751  esynoscheduler.output.17064  Public              usbwebcam
@cloudstation  esynoscheduler.output.13674  esynoscheduler.output.14407  esynoscheduler.output.14777  esynoscheduler.output.18665  @quarantine         video
@cloudsync     esynoscheduler.output.13692  esynoscheduler.output.14462  esynoscheduler.output.14791  esynoscheduler.output.25932  @S2S                VIDEO STATION
comics         esynoscheduler.output.13827  esynoscheduler.output.14516  esynoscheduler.output.14811  @ffmpeg.core                 @smbd.core          Virtual Machines
data           esynoscheduler.output.13844  esynoscheduler.output.14524  esynoscheduler.output.14813  @ffmpeg-thumb.core           @spool              web
@database      esynoscheduler.output.13950  esynoscheduler.output.14525  esynoscheduler.output.14844  Google Back Up               @ssbackup
@ddnsd.core    esynoscheduler.output.13969  esynoscheduler.output.14590  esynoscheduler.output.14944  homes                        @sudo.core
docker         esynoscheduler.output.14014  esynoscheduler.output.14608  esynoscheduler.output.14981  @img_bkp_cache               surveillance
@docker        esynoscheduler.output.14061  esynoscheduler.output.14613  esynoscheduler.output.15135  @iSCSI                       @synoelasticd.core
@download      esynoscheduler.output.14144  esynoscheduler.output.14680  esynoscheduler.output.15185  lost+found                   @synonetbkp.core

So, as I have no services running on the syno, I do not have SAMBA. So I can not mount correct?
Is it safe to add SAMBA? Can I create a shared folder?

Is there a way to reload the config? 

 

 

Snag_1c0b2b3.png

Snag_1c0bb4e.png

Snag_1c0c418.png

Link to comment
Share on other sites

Ok, let's modify /etc/fstab to mount your filesystem with the good superblock and in read-only mode:

 

First, make a backup copy of fstab

 

# cp /etc/fstab /etc/fstab.bak

Right now your fstab looks like this:

none /proc proc defaults 0 0
/dev/root / ext4 defaults 1 1
/dev/vg1000/lv /volume1 ext4 usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,synoacl,relatime 0 0

You need to edit so that it looks like this:

none /proc proc defaults 0 0
/dev/root / ext4 defaults 1 1
/dev/vg1000/lv /volume1 ext4 ro,noload,sb=1934917632,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,synoacl,relatime 0 0

After you are done editing, cat /etc/fstab and confirm that it looks exactly like the above, then reboot the NAS.  I'm pretty sure Syno won't rewrite fstab as long as you don't make any changes to the GUI.

 

After reboot, report if you can get to files on the network, and post the output of:

# cat /proc/mdstat
# cat /etc/fstab
# df

 

Link to comment
Share on other sites

It's up to you.  I suspect that if you do the steps in the last post, you can access the files via Windows.  But here's the situation:

  1. Your RAID5 array is critical (no redundancy) and has mild corruption
  2. Your filesystem has some corruption but we have been able to get it to mount

My strong recommendation is that you not attempt to "fix" anything further, and do the following:

  1. Copy everything off your volume1 onto another device.  If you need to go buy an 8TB external drive, do it.
  2. Delete your volume1
  3. Delete your SHR
  4. Click the Fix System Partition options in Storage Manager to correct the DSM and swap replicas
  5. Remove/replace your bad drive #0/sda
  6. Create a new SHR
  7. Create a new volume1
  8. Copy your files back

The copy operations can be done on either platform (Mac or Windows).  So it's up to you.

Edited by flyride
Link to comment
Share on other sites

11 minutes ago, flyride said:

It's up to you.  I suspect that if you do the steps in the last post, you can access the files via Windows.  But here's the situation:

  1. Your RAID5 array is critical (no redundancy) and has mild corruption
  2. Your filesystem has some corruption but we have been able to get it to mount

My strong recommendation is that you not attempt to "fix" anything further, and do the following:

  1. Copy everything off your volume1 onto another device.  If you need to go buy an 8TB external drive, do it.
  2. Delete your volume1
  3. Click the Fix System Partition options in Storage Manager to correct the DSM and swap replicas
  4. Remove/replace your bad drive #0/sda
  5. Create a new volume1 and copy your files back

The copy operations can be done on either platform (Mac or Windows).  So it's up to you.

 

 

root@DiskStation:/volume1# cat /etc/fstab
none /proc proc defaults 0 0
/dev/root / ext4 defaults 1 1
/dev/vg1000/lv /volume1 ext4 ro,noload,sb=1934917632,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,synoacl,relatime 0 0


OK, I rebooted and will now buy a new drive. I will also get an online network backup for my most important files.

Please can you send me you P4yp4l account? You deserve a big beer from me :-)

Link to comment
Share on other sites

9 minutes ago, flyride said:

Thanks for the offer of a beer... but pay it forward to help someone else!  I hope you don't have too much data loss. Good luck.

Thanks again Flyride. I doubt that i have much data loss... i hardly touched it!

I reallly appreciate your time and effort, you are one of the good guys in this community.

Thanks also to IG-88 for this help, Love you guys!!

Link to comment
Share on other sites

The data loss would be due to the partial resync that affected /dev/sdc5.  It was enough to invalidate the filesystem superblock so there is definitely some damage.  However, you probably won't know until you open an affected file.  The number of files affected should be low.

Link to comment
Share on other sites

Hi @flyride do you mind you will have some time to help me too ?

My volume is crashed but this one is not an ext4 but btrfs.

Raid array looks good but lvm show me

 #lvs
  LV   VG     Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv   vg1000 -wi-a----- 7.26t

# lvm pvscan
  PV /dev/md3   VG vg1000   lvm2 [5.44 TiB / 0    free]
  PV /dev/md4   VG vg1000   lvm2 [1.82 TiB / 0    free]
  Total: 2 [7.26 TiB] / in use: 2 [7.26 TiB] / in no VG: 0 [0   ]

# lvm lvmdiskscan
  /dev/md2 [     163.08 GiB]
  /dev/md3 [       5.44 TiB] LVM physical volume
  /dev/md4 [       1.82 TiB] LVM physical volume
  0 disks
  1 partition
  0 LVM physical volume whole disks
  2 LVM physical volumes

 

my issue appair with a power cut after a non responding system when I was in a scan folder shearch in the entire \

all disk looks OK, and at the first restart the system made a resynchronisyng of my raid but volume stay into crashed status.

 

 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...