Jump to content
XPEnology Community

DANGER : Raid crashed. Help me restore my data!


Recommended Posts

3 minutes ago, flyride said:

Still quite perplexed about the refusal of this drive to play, but we're probably out of non-invasive options and need to do a create - the path that IG-88 charted out. Before we do that let's get a current state of your system. Please do not reboot or do anything else to change the system state once we start using this information or your data is at risk.  If anything changes at all, please advise.


# mdadm --detail /dev/md2 | fgrep "/dev/"
# mdadm --examine /dev/sdb5 /dev/sdc5 /dev/sdd5 | egrep "/dev|Role|Events|UUID"

 

 

root@DiskStation:~# mdadm --detail /dev/md2 | fgrep "/dev/"
/dev/md2:
       2       8       21        2      active sync   /dev/sdb5
       4       8       53        3      active sync   /dev/sdd5

And then 

 

root@DiskStation:~# mdadm --examine /dev/sdb5 /dev/sdc5 /dev/sdd5 | egrep "/dev|Role|Events|UUID"
/dev/sdb5:
     Array UUID : 75762e2e:4629b4db:259f216e:a39c266d
    Device UUID : 7eee55dc:dbbf5609:e737801d:87903b6c
         Events : 15417
   Device Role : Active device 2
/dev/sdc5:
     Array UUID : 75762e2e:4629b4db:259f216e:a39c266d
    Device UUID : 6ba575e4:53121f53:a8fe4876:173d11a9
         Events : 15357
   Device Role : Active device 1
/dev/sdd5:
     Array UUID : 75762e2e:4629b4db:259f216e:a39c266d
    Device UUID : fb417ce4:fcdd58fb:72d35e06:9d7098b5
         Events : 15417
   Device Role : Active device 3


Data at risk if I repeat. You have me worried now :-)

 

 

Link to comment
Share on other sites

1 minute ago, flyride said:

# mdadm -v --create --assume-clean -e1.2 -n4 -l5 /dev/md2 missing /dev/sdc5 /dev/sdb5 /dev/sdd5 -u75762e2e:4629b4db:259f216e:a39c266d
# cat /proc/mdstat

 

 

root@DiskStation:~# mdadm -v --create --assume-clean -e1.2 -n4 -l5 /dev/md2 missing /dev/sdc5 /dev/sdb5 /dev/sdd5 -u75762e2e:4629b4db:259f216e:a39c266d
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 64K
mdadm: /dev/sdc5 appears to be part of a raid array:
       level=raid5 devices=4 ctime=Mon May 13 08:39:01 2013
mdadm: super1.x cannot open /dev/sdb5: Device or resource busy
mdadm: /dev/sdb5 is not suitable for this array.
mdadm: super1.x cannot open /dev/sdd5: Device or resource busy
mdadm: /dev/sdd5 is not suitable for this array.
mdadm: create aborted
root@DiskStation:~# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [raidF1]
md2 : active raid5 sdb5[2] sdd5[4]
      8776594944 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/2] [__UU]

md1 : active raid1 sdb2[0] sdc2[1] sdd2[2]
      2097088 blocks [12/3] [UUU_________]

md0 : active raid1 sdb1[2] sdd1[3]
      2490176 blocks [12/2] [__UU________]

unused devices: <none>


 

 

Link to comment
Share on other sites

2 minutes ago, flyride said:

Try with the stop command

root@DiskStation:~# mdadm -v --create --assume-clean -e1.2 -n4 -l5 /dev/md2 missing /dev/sdc5 /dev/sdb5 /dev/sdd5 -u75762e2e:4629b4db:259f216e:a39c266d
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 64K
mdadm: /dev/sdc5 appears to be part of a raid array:
       level=raid5 devices=4 ctime=Mon May 13 08:39:01 2013
mdadm: /dev/sdb5 appears to be part of a raid array:
       level=raid5 devices=4 ctime=Mon May 13 08:39:01 2013
mdadm: /dev/sdd5 appears to be part of a raid array:
       level=raid5 devices=4 ctime=Mon May 13 08:39:01 2013
mdadm: size set to 2925531840K
 

And then continue to creating aray

Edited by jbesclapez
Link to comment
Share on other sites

1 minute ago, flyride said:

Yes, we are trying to create the array.

Sorry for such ridiculous question I am asking, but as you said before, I have to be really careful... (and i am a bit naturally sloppy).

root@DiskStation:~# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [raidF1]
md2 : active raid5 sdd5[3] sdb5[2] sdc5[1]
      8776595520 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [_UUU]

md1 : active raid1 sdb2[0] sdc2[1] sdd2[2]
      2097088 blocks [12/3] [UUU_________]

md0 : active raid1 sdb1[2] sdd1[3]
      2490176 blocks [12/2] [__UU________]

unused devices: <none>


 

Link to comment
Share on other sites

No problem, conservative is good when dealing with arrays.

 

I think your array is started and should have data.  We absolutely do not want it to rebuild, resync or any other operations.  So don't click on any "fix" buttons in the GUI.

It also has no redundancy and your drive #0 /dev/sda is presumed to be dead.

 

I advise to flag the whole array read-only:

# mdadm --misc -o /dev/md2

Then reboot and see if your data is there.  Report back on status.

Link to comment
Share on other sites

11 minutes ago, flyride said:

No problem, conservative is good when dealing with arrays.

 

I think your array is started and should have data.  We absolutely do not want it to rebuild, resync or any other operations.  So don't click on any "fix" buttons in the GUI.

It also has no redundancy and your drive #0 /dev/sda is presumed to be dead.

 

I advise to flag the whole array read-only:


# mdadm --misc -o /dev/md2

Then reboot and see if your data is there.  Report back on status.

I am now trying to make sense out of this. But i lost my config on the server so i can not see the data in shared folder.... trying to find a way.

Snag_16fac44.png

Snag_16fbbb5.png

Snag_16fc625.png

Link to comment
Share on other sites

There is no server config on Synology, it all comes from the array itself.  If your /dev/sdc5 got corrupted as part of a failed rebuild, then your volume won't mount and your data is probably lost.  We don't know whether that happened yet but let's investigate further.

# cat /etc/fstab
# cat /proc/mdstat
# df

 

Link to comment
Share on other sites

 

 

root@DiskStation:/volume1# cat /etc/fstab
none /proc proc defaults 0 0
/dev/root / ext4 defaults 1 1
/dev/vg1000/lv /volume1 ext4 usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,synoacl,relatime 0 0
root@DiskStation:/volume1# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [raidF                                                                                                       1]
md2 : active raid5 sdc5[1] sdd5[3] sdb5[2]
      8776595520 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [_UUU]

md1 : active raid1 sdb2[0] sdc2[1] sdd2[2]
      2097088 blocks [12/3] [UUU_________]

md0 : active raid1 sdb1[2] sdd1[3]
      2490176 blocks [12/2] [__UU________]

unused devices: <none>

 

root@DiskStation:/volume1# df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/md0         2385528 1076524   1190220  48% /
none             1022500       0   1022500   0% /dev
/tmp             1027768    2156   1025612   1% /tmp
/run             1027768    2784   1024984   1% /run
/dev/shm         1027768       4   1027764   1% /dev/shm
none                   4       0         4   0% /sys/fs/cgroup
cgmfs                100       0       100   0% /run/cgmanager/fs

Scary again then...😲
 

Link to comment
Share on other sites

1 minute ago, flyride said:

Ok, still investigating:


# vgdisplay
# lvs
# lvm vgscan
# lvm pvscan
# lvm lvmdiskscan

 

 

 

 

root@DiskStation:/volume1# vgdisplay
  --- Volume group ---
  VG Name               vg1000
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               8.17 TiB
  PE Size               4.00 MiB
  Total PE              2142723
  Alloc PE / Size       2142723 / 8.17 TiB
  Free  PE / Size       0 / 0
  VG UUID               YQVlVb-else-xKqP-OVtH-kU9e-WJPm-7ZWuWt

 

oot@DiskStation:/volume1# lvs
  LV   VG     Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv   vg1000 -wi-a----- 8.17t
root@DiskStation:/volume1# lvm vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "vg1000" using metadata type lvm2
root@DiskStation:/volume1# lvm pvscan
  PV /dev/md2   VG vg1000   lvm2 [8.17 TiB / 0    free]
  Total: 1 [8.17 TiB] / in use: 1 [8.17 TiB] / in no VG: 0 [0   ]
root@DiskStation:/volume1# lvm lvmdiskscan
  /dev/md2 [       8.17 TiB] LVM physical volume
  0 disks
  0 partitions
  0 LVM physical volume whole disks
  1 LVM physical volume

 

Link to comment
Share on other sites

1 minute ago, flyride said:

That all looks ok.


# mount -v

 

 

 

root@DiskStation:/volume1# mount -v
/dev/md0 on / type ext4 (rw,relatime,journal_checksum,barrier,data=ordered)
none on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=1022500k,nr_inodes=255625,mode=755)
none on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
none on /proc type proc (rw,nosuid,nodev,noexec,relatime)
none on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
/tmp on /tmp type tmpfs (rw,relatime)
/run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)
/dev/shm on /dev/shm type tmpfs (rw,nosuid,nodev,relatime)
none on /sys/fs/cgroup type tmpfs (rw,relatime,size=4k,mode=755)
cgmfs on /run/cgmanager/fs type tmpfs (rw,relatime,size=100k,mode=755)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,relatime,cpuset,release_agent=/run/cgmanager/agents/cgm-release-agent.cpuset,clone_children)
cgroup on /sys/fs/cgroup/cpu type cgroup (rw,relatime,cpu,release_agent=/run/cgmanager/agents/cgm-release-agent.cpu)
cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,relatime,cpuacct,release_agent=/run/cgmanager/agents/cgm-release-agent.cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,relatime,memory,release_agent=/run/cgmanager/agents/cgm-release-agent.memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices,release_agent=/run/cgmanager/agents/cgm-release-agent.devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,relatime,freezer,release_agent=/run/cgmanager/agents/cgm-release-agent.freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,relatime,blkio,release_agent=/run/cgmanager/agents/cgm-release-agent.blkio)
none on /proc/bus/usb type devtmpfs (rw,nosuid,noexec,relatime,size=1022500k,nr_inodes=255625,mode=755)
none on /sys/kernel/debug type debugfs (rw,relatime)
securityfs on /sys/kernel/security type securityfs (rw,relatime)
none on /config type configfs (rw,relatime)


 

 

Link to comment
Share on other sites

9 minutes ago, flyride said:

Hmm that didn't do anything useful.


# mount -v /dev/vg1000/lv /volume1

 

 

root@DiskStation:/volume1# mount -v /dev/vg1000/lv /volume1
mount: wrong fs type, bad option, bad superblock on /dev/vg1000/lv,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
 

 

Link to comment
Share on other sites

root@DiskStation:/volume1# dmesg | tail
[ 2815.739355] EXT4-fs (dm-0): group descriptors corrupted!
[ 3646.703050] hub 7-0:1.0: port 2 disabled by hub (EMI?), re-enabling...
[ 3646.709583] usb 7-2: USB disconnect, device number 2
[ 3647.186022] usb 7-2: new low-speed USB device number 3 using uhci_hcd
[ 3647.372757] Got empty serial number. Generate serial number from product.
[ 3647.391054] input: Microsoft Wired Keyboard 600 as /devices/pci0000:00/0000:00:1d.1/usb7/7-2/7-2:1.0/input/input4
[ 3647.391069] hid-generic 0003:045E:0750.0003: input: USB HID v1.11 Keyboard [Microsoft Wired Keyboard 600] on usb-0000:00:1d.1-2/input0
[ 3647.421895] input: Microsoft Wired Keyboard 600 as /devices/pci0000:00/0000:00:1d.1/usb7/7-2/7-2:1.1/input/input5
[ 3647.421902] Get empty minor:104
[ 3647.421953] hid-generic 0003:045E:0750.0004: input,hiddev0: USB HID v1.11 Device [Microsoft Wired Keyboard 600] on usb-0000:00:1d.1-2/input1


 

Link to comment
Share on other sites

1 minute ago, flyride said:

# mke2fs -n /dev/vg1000/vg

 

 

 

root@DiskStation:/volume1# mke2fs -n /dev/vg1000/vg
mke2fs 1.42.6 (21-Sep-2012)
Could not stat /dev/vg1000/vg --- No such file or directory

The device apparently does not exist; did you specify it correctly?


 

Link to comment
Share on other sites

1 minute ago, flyride said:

Woops it's /dev/vg1000/lv

 

 

root@DiskStation:/# mke2fs -n /dev/vg1000/lv
mke2fs 1.42.6 (21-Sep-2012)
Filesystem label=1.42.6-15266
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=16 blocks, Stripe width=48 blocks
274272256 inodes, 2194148352 blocks
25600 blocks (0.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
66961 block groups
32768 blocks per group, 32768 fragments per group
4096 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848, 512000000, 550731776, 644972544, 1934917632


 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...