Rihc0 Posted November 11, 2020 #1 Posted November 11, 2020 (edited) Hello everyone, So I have a Dell r720 rack server installed with ESXi 6.7 u3. the server is running virtual machines like xpenology, but my dsm crashed somehow (really dont know what happend). i have 5x 4tb sas drives configured in the raid controller as 1 disk each Virtual disks. so 5 virtual disks total. to push the whole disk to the virtual machine and use raw disk mapping. my expenology has 5 RDM (5x 4 tb sas drives) . my DSM handles the raid and it has been set in a raid 5 ext4 file system. but now when i booted xpenology VM i got this (Picture1). when when i try to repair the storage pool (picture 3,4,5) you can see the message :(. Can anyone help me with this PS: Now i remember i deleted the virtual disks in the raid controller, made a raid 5 virtual disk for windows, booted a windows vm with the virtual disk (for some data recovery) and made seperare virtual disk for each disk and added them to expenology again) Edited November 11, 2020 by Rihc0 Quote
IG-88 Posted November 12, 2020 #2 Posted November 12, 2020 On 11/11/2020 at 3:30 AM, Rihc0 said: Now i remember i deleted the virtual disks in the raid controller, made a raid 5 virtual disk for windows, booted a windows vm with the virtual disk (for some data recovery) and made separate virtual disk for each disk and added them to xpenology again) then there was/is not data on that volume? the simple way to get rid of the mass is to delete the storage pool (incl. the volume) and create a new with the now free unused disks if there are any messages about system partitions not working let dsm repair it (system is a raid1 over all disks, as long as one is still working it will copy to the others by raid1) Quote
Rihc0 Posted November 12, 2020 Author #3 Posted November 12, 2020 There is data on it that I want to keep :(, I can't repair it because dam says I need more disks for some reason Quote
IG-88 Posted November 12, 2020 #4 Posted November 12, 2020 24 minutes ago, Rihc0 said: There is data on it that I want to keep :(, I can't repair it because dam says I need more disks for some reason if you had 5 raid0 disks on the physical controller then destroyed 3 disks, moved these 3 disks into a raid5 (on then hardware controller) , installed windows on it and after that you destroyed that raid5 and created new raid0 disks, how will there be any usable data left that can fit into a raid? if you remove 3 disks from raid5 disk set of 5 disks and destroy the data on that 3 disks, then there is no way to fix it, raid5 has one disk redundancy, there are cases where more disks drop out of a raid set and you can force them back in as long as there is not much difference but having 3 disks reconfigured and written to in that new configuration (raid5 of 3 disks) i'd guess there is nothing left of you old data that you can use thats what i took from your comment, maybe i got it wrong? Quote
billat29 Posted November 12, 2020 #5 Posted November 12, 2020 I think he is saying -> My DSM crashed. How to recover data? I know! I will mount the disks under Windows and pull them off. Hmmm. It's RAID 5. How do I do that? I know! I'll define the disks as RAID 5 using the storage controller and mount the entire RAID to a running Windows machine and pull the data off. Whoops! that didn't work. I'll redefine the disks using the storage controller and mount them individually again under DSM and let DSM go back to using them as a RAID. Ah! Nothing works! Quote
Rihc0 Posted November 12, 2020 Author #6 Posted November 12, 2020 okay, so the physical controller had the 5 disks in raid 0 each disk working on the dsm (everything was fine from here and smoothly. I made a raid 5 disk and put it in windows because I wanted to try data recovery on it because I used the disk as raid 5 on a virtual machine before I used it for DSM, when I scanned the disk and recovered the data. I configured the 5 disks as raid 0 each disk. and connected them to dsm and now I have the error, do you guys get it, might be a bit confusing xD Quote
IG-88 Posted November 12, 2020 #7 Posted November 12, 2020 26 minutes ago, Rihc0 said: I made a raid 5 disk and put it in windows from what drives? you had 5 disks raid0 in your system used for dsm you added 3-5 disks from another system to create the raid5 for windows? having a closer look at your pictures it looks really strange, pool1 is raid5 but volume 1 says raid1 and its 2.28 GIGABYTE not TB 2nd picure of the storage pool, right below storage pool 1 it says raid5 and on the right with the error its raid1 also a small GB number and THAT number from the size 2.37 GB looks like the size of a dsm system partition (these are raid1) the for some reason the pool thinks these raid1 partitions are part of the pool that looks very strange maybe you should list some basics from the command line like list of disk (/dev/sda-sd... and list of partitions on every disk, this should do it sudo fdisk -l /dev/sd* also the output of sudo cat /proc/mdstat Quote
Rihc0 Posted November 12, 2020 Author #8 Posted November 12, 2020 Disk /dev/sdb: 16 GiB, 17179869184 bytes, 33554432 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0xa43eb840 Device Boot Start End Sectors Size Id Type /dev/sdb1 2048 4982527 4980480 2.4G fd Linux raid autodetect /dev/sdb2 4982528 9176831 4194304 2G fd Linux raid autodetect /dev/sdb3 9437184 33349631 23912448 11.4G fd Linux raid autodetect Disk /dev/sdb1: 2.4 GiB, 2550005760 bytes, 4980480 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdb2: 2 GiB, 2147483648 bytes, 4194304 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdb3: 11.4 GiB, 12243173376 bytes, 23912448 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdc: 16 GiB, 17179869184 bytes, 33554432 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x7a551fe7 Device Boot Start End Sectors Size Id Type /dev/sdc1 2048 4982527 4980480 2.4G fd Linux raid autodetect /dev/sdc2 4982528 9176831 4194304 2G fd Linux raid autodetect /dev/sdc3 9437184 33349631 23912448 11.4G fd Linux raid autodetect Disk /dev/sdc1: 2.4 GiB, 2550005760 bytes, 4980480 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdc2: 2 GiB, 2147483648 bytes, 4194304 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdc3: 11.4 GiB, 12243173376 bytes, 23912448 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdd: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: AB3D10CC-2A35-4075-AF8B-135C47B30870 Device Start End Sectors Size Type /dev/sdd1 2048 4982527 4980480 2.4G Linux RAID /dev/sdd2 4982528 9176831 4194304 2G Linux RAID /dev/sdd3 9437184 7812734975 7803297792 3.6T Linux RAID Disk /dev/sdd1: 2.4 GiB, 2550005760 bytes, 4980480 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdd2: 2 GiB, 2147483648 bytes, 4194304 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdd3: 3.6 TiB, 3995288469504 bytes, 7803297792 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes The primary GPT table is corrupt, but the backup appears OK, so that will be used. Disk /dev/sde: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: E3D1C69D-D406-4A97-BF00-168262F1025C Device Start End Sectors Size Type /dev/sde1 2048 4982527 4980480 2.4G Linux RAID /dev/sde2 4982528 9176831 4194304 2G Linux RAID /dev/sde3 9437184 7812734975 7803297792 3.6T Linux RAID Disk /dev/sdf: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: EB6537B4-CC88-4A1B-99A4-C235A327CFB6 Device Start End Sectors Size Type /dev/sdf1 2048 4982527 4980480 2.4G Linux RAID /dev/sdf2 4982528 9176831 4194304 2G Linux RAID /dev/sdf3 9437184 7812734975 7803297792 3.6T Linux RAID Disk /dev/sdf1: 2.4 GiB, 2550005760 bytes, 4980480 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdf2: 2 GiB, 2147483648 bytes, 4194304 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdf3: 3.6 TiB, 3995288469504 bytes, 7803297792 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdg: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdh: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: F681C773-F2AE-452F-8E71-55511FA5AEE0 Device Start End Sectors Size Type /dev/sdh1 2048 4982527 4980480 2.4G Linux RAID /dev/sdh2 4982528 9176831 4194304 2G Linux RAID /dev/sdh3 9437184 7812734975 7803297792 3.6T Linux RAID Disk /dev/sdh1: 2.4 GiB, 2550005760 bytes, 4980480 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdh2: 2 GiB, 2147483648 bytes, 4194304 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdh3: 3.6 TiB, 3995288469504 bytes, 7803297792 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdm3: 4 MiB, 4177408 bytes, 8159 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes the 5 drives in my Synology right now, are the drives I had in the first place. Quote
flyride Posted November 12, 2020 #9 Posted November 12, 2020 (edited) 2 hours ago, Rihc0 said: okay, so the physical controller had the 5 disks in raid 0 each disk working on the dsm (everything was fine from here and smoothly. I made a raid 5 disk and put it in windows because I wanted to try data recovery on it because I used the disk as raid 5 on a virtual machine before I used it for DSM, when I scanned the disk and recovered the data. I configured the 5 disks as raid 0 each disk. and connected them to dsm and now I have the error, do you guys get it, might be a bit confusing xD RAID5 on a hardware controller is going to use the full amount of the disk for the array. There probably is no metadata on the drives at all; it's usually on NVRAM in the controller. With a hardware controller RAID, the array member numbering will correspond to the physical ports on the controller. RAID5 on DSM is over a data partition (actually the third data partition) across all three drives. Each member has metadata information that explains how it fits into the array, and there is no guarantee they are in port order. There is no way to interchange between hardware raid and mdm, and it's incredibly likely that critical data structures have been corrupted by trying to do so. I think this is a basket case. Edited November 12, 2020 by flyride Quote
Rihc0 Posted November 12, 2020 Author #10 Posted November 12, 2020 you think I won't be able to repair this? Quote
IG-88 Posted November 12, 2020 #11 Posted November 12, 2020 (edited) 19 minutes ago, Rihc0 said: you think I won't be able to repair this? if you are a data recovery specialist working for kroll ontrack you could you could asks a professional recovery company and describe your case in details but my guess would be the recovery would cost you at least a one digit thousand bugs beside this, you fdisk listing shows two 16GB disks and four 4TB disks you stated its 5 x 4TB - somehow even that does not match maybe you should think about doing backups and maybe a original synology nas Edited November 12, 2020 by IG-88 Quote
Rihc0 Posted November 13, 2020 Author #12 Posted November 13, 2020 I created the 2x 16gb so I could boot the Synology Nas. I had a ds918+ but the hardware was slow :(. I asked if you think I won't be able to repair this because if not then, I know it is pointless on working on this problem. I should learn more abut the raid and how it works, I am too unfamiliar with it Quote
IG-88 Posted November 13, 2020 #14 Posted November 13, 2020 3 minutes ago, Rihc0 said: I asked if you think I won't be able to repair this because if not then, I know it is pointless on working on this problem. I should learn more abut the raid and how it works, I am too unfamiliar with it 1 hour ago, flyride said: think this is a basket case. if he say's so i#m pretty sure you wont be able to do that yourself just by reading some articles about mdadm he is the one helping people with such problems and might be the one with the most experience here you would need to have a deep knowledge about the the data structures on that madam disks and would need to reconstruct some data manually and even then there will be some losses depending on how much was written to the disks - that was the meaning of my comment about working for a world known recovery company if you just initialized the hardware raid and the controller did not write the whole disk then most sectors of the disk still have the original content and if the controller only wrote to the first few MB of the disk then your data might be untouched as they are on the 3rd partition (2.4GB dsm system, 2GB swap, your date partition form the raid5) you would be needed to know what the controller did when making the disks into a raid5 set and what happened after this (what you did with windows) i guess a recovery professional would do some kind of interview to reconstruct what happened have a rough estimate if its even worth to start, like if the controller did a full cycle of zero the disks in the raid (would at least take 1-2 hours i guess) then its pointless to even start but the fact that we still see the old partitions indicate that the zeroing out part did not happen Quote
IG-88 Posted November 13, 2020 #15 Posted November 13, 2020 (edited) 15 minutes ago, Rihc0 said: forgot this one at least there is still a md3 raid5 that would be the part with your data, only 2 of 5 disks are recognized maybe its not that hopeless? @flyride Edited November 13, 2020 by IG-88 Quote
flyride Posted November 13, 2020 #16 Posted November 13, 2020 (edited) Argh. We'll see. # mdadm --detail /dev/md3 # mdadm --examine /dev/sd[defgh]3 | egrep 'Event|/dev/sd' Edited November 13, 2020 by flyride Quote
Rihc0 Posted November 13, 2020 Author #17 Posted November 13, 2020 Okay, I had to do this on an another virtual machine because I deleted the other one because i thought it was hopeless :P. output of sudo fdisk -l /dev/sd* ash-4.3# sudo fdisk -l /dev/sd* Disk /dev/sdb: 16 GiB, 17179869184 bytes, 33554432 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x22d5f435 Device Boot Start End Sectors Size Id Type /dev/sdb1 2048 4982527 4980480 2.4G fd Linux raid autodetect /dev/sdb2 4982528 9176831 4194304 2G fd Linux raid autodetect /dev/sdb3 9437184 33349631 23912448 11.4G fd Linux raid autodetect Disk /dev/sdb1: 2.4 GiB, 2550005760 bytes, 4980480 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdb2: 2 GiB, 2147483648 bytes, 4194304 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdb3: 11.4 GiB, 12243173376 bytes, 23912448 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdc: 16 GiB, 17179869184 bytes, 33554432 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x8504927a Device Boot Start End Sectors Size Id Type /dev/sdc1 2048 4982527 4980480 2.4G fd Linux raid autodetect /dev/sdc2 4982528 9176831 4194304 2G fd Linux raid autodetect /dev/sdc3 9437184 33349631 23912448 11.4G fd Linux raid autodetect Disk /dev/sdc1: 2.4 GiB, 2550005760 bytes, 4980480 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdc2: 2 GiB, 2147483648 bytes, 4194304 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdc3: 11.4 GiB, 12243173376 bytes, 23912448 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdd: 3.7 TiB, 3999688294400 bytes, 7811891200 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 88AB940C-A74C-425C-B303-3DD15285C607 Device Start End Sectors Size Type /dev/sdd1 2048 7811889152 7811887105 3.7T unknown Disk /dev/sdd1: 3.7 TiB, 3999686197760 bytes, 7811887105 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sde: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: EB6537B4-CC88-4A1B-99A4-C235A327CFB6 Device Start End Sectors Size Type /dev/sde1 2048 4982527 4980480 2.4G Linux RAID /dev/sde2 4982528 9176831 4194304 2G Linux RAID /dev/sde3 9437184 7812734975 7803297792 3.6T Linux RAID Disk /dev/sde1: 2.4 GiB, 2550005760 bytes, 4980480 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sde2: 2 GiB, 2147483648 bytes, 4194304 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sde3: 3.6 TiB, 3995288469504 bytes, 7803297792 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdf: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: AB3D10CC-2A35-4075-AF8B-135C47B30870 Device Start End Sectors Size Type /dev/sdf1 2048 4982527 4980480 2.4G Linux RAID /dev/sdf2 4982528 9176831 4194304 2G Linux RAID /dev/sdf3 9437184 7812734975 7803297792 3.6T Linux RAID Disk /dev/sdf1: 2.4 GiB, 2550005760 bytes, 4980480 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdf2: 2 GiB, 2147483648 bytes, 4194304 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdf3: 3.6 TiB, 3995288469504 bytes, 7803297792 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdg: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: F681C773-F2AE-452F-8E71-55511FA5AEE0 Device Start End Sectors Size Type /dev/sdg1 2048 4982527 4980480 2.4G Linux RAID /dev/sdg2 4982528 9176831 4194304 2G Linux RAID /dev/sdg3 9437184 7812734975 7803297792 3.6T Linux RAID Disk /dev/sdg1: 2.4 GiB, 2550005760 bytes, 4980480 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdg2: 2 GiB, 2147483648 bytes, 4194304 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdg3: 3.6 TiB, 3995288469504 bytes, 7803297792 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes The primary GPT table is corrupt, but the backup appears OK, so that will be used. Disk /dev/sdh: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: E3D1C69D-D406-4A97-BF00-168262F1025C Device Start End Sectors Size Type /dev/sdh1 2048 4982527 4980480 2.4G Linux RAID /dev/sdh2 4982528 9176831 4194304 2G Linux RAID /dev/sdh3 9437184 7812734975 7803297792 3.6T Linux RAID Disk /dev/sdi: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdm3: 4 MiB, 4177408 bytes, 8159 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes ash-4.3# output of cat /proc/mdstat ash-4.3# cat /proc/mdstat Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [raidF1] md3 : active raid5 sdf3[0] sde3[1] 15606591488 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/2] [UU___] md2 : active raid1 sdb3[0] sdc3[1] 11955200 blocks super 1.2 [2/2] [UU] md127 : active raid1 sde1[1] sdf1[0] 2490176 blocks [12/2] [UU__________] md1 : active raid1 sdb2[0] sdc2[1] sde2[2] sdf2[3] sdg2[4] 2097088 blocks [12/5] [UUUUU_______] md0 : active raid1 sdb1[0] sdc1[1] 2490176 blocks [12/2] [UU__________] unused devices: <none> ash-4.3# output of: mdadm --detail /dev/md3 ash-4.3# mdadm --detail /dev/md3 /dev/md3: Version : 1.2 Creation Time : Sat Jun 20 00:46:08 2020 Raid Level : raid5 Array Size : 15606591488 (14883.61 GiB 15981.15 GB) Used Dev Size : 3901647872 (3720.90 GiB 3995.29 GB) Raid Devices : 5 Total Devices : 2 Persistence : Superblock is persistent Update Time : Fri Nov 13 02:22:44 2020 State : clean, FAILED Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K Name : Dabadoo:2 UUID : ff64862b:9edfe233:c498ea84:9d4b9ffd Events : 39091 Number Major Minor RaidDevice State 0 8 83 0 active sync /dev/sdf3 1 8 67 1 active sync /dev/sde3 - 0 0 2 removed - 0 0 3 removed - 0 0 4 removed ash-4.3# output of: mdadm --examine /dev/sd[defgh]3 | egrep 'Event|/dev/sd' ash-4.3# mdadm --examine /dev/sd[defgh]3 | egrep 'Event|/dev/sd' mdadm: No md superblock detected on /dev/sdg3. /dev/sde3: Events : 39091 /dev/sdf3: Events : 39091 ash-4.3# Quote
flyride Posted November 13, 2020 #18 Posted November 13, 2020 # mdadm --examine /dev/sdg3 # mdadm --examine /dev/sdh3 Quote
Rihc0 Posted November 17, 2020 Author #19 Posted November 17, 2020 Okay, Sorry for the late reply, the lifecycle controller on my server was not doing great and had to troubleshoot it. Ill redo all the commands because there might have been some changes. output of sudo fdisk -l /dev/sd* ash-4.3# sudo fdisk -l /dev/sd* Disk /dev/sdb: 16 GiB, 17179869184 bytes, 33554432 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x22d5f435 Device Boot Start End Sectors Size Id Type /dev/sdb1 2048 4982527 4980480 2.4G fd Linux raid autodetect /dev/sdb2 4982528 9176831 4194304 2G fd Linux raid autodetect /dev/sdb3 9437184 33349631 23912448 11.4G fd Linux raid autodetect Disk /dev/sdb1: 2.4 GiB, 2550005760 bytes, 4980480 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdb2: 2 GiB, 2147483648 bytes, 4194304 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdb3: 11.4 GiB, 12243173376 bytes, 23912448 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdc: 16 GiB, 17179869184 bytes, 33554432 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x8504927a Device Boot Start End Sectors Size Id Type /dev/sdc1 2048 4982527 4980480 2.4G fd Linux raid autodetect /dev/sdc2 4982528 9176831 4194304 2G fd Linux raid autodetect /dev/sdc3 9437184 33349631 23912448 11.4G fd Linux raid autodetect Disk /dev/sdc1: 2.4 GiB, 2550005760 bytes, 4980480 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdc2: 2 GiB, 2147483648 bytes, 4194304 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdc3: 11.4 GiB, 12243173376 bytes, 23912448 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdd: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sde: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: EB6537B4-CC88-4A1B-99A4-C235A327CFB6 Device Start End Sectors Size Type /dev/sde1 2048 4982527 4980480 2.4G Linux RAID /dev/sde2 4982528 9176831 4194304 2G Linux RAID /dev/sde3 9437184 7812734975 7803297792 3.6T Linux RAID Disk /dev/sde1: 2.4 GiB, 2550005760 bytes, 4980480 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sde2: 2 GiB, 2147483648 bytes, 4194304 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sde3: 3.6 TiB, 3995288469504 bytes, 7803297792 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes The primary GPT table is corrupt, but the backup appears OK, so that will be used. Disk /dev/sdf: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: E3D1C69D-D406-4A97-BF00-168262F1025C Device Start End Sectors Size Type /dev/sdf1 2048 4982527 4980480 2.4G Linux RAID /dev/sdf2 4982528 9176831 4194304 2G Linux RAID /dev/sdf3 9437184 7812734975 7803297792 3.6T Linux RAID Disk /dev/sdg: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: AB3D10CC-2A35-4075-AF8B-135C47B30870 Device Start End Sectors Size Type /dev/sdg1 2048 4982527 4980480 2.4G Linux RAID /dev/sdg2 4982528 9176831 4194304 2G Linux RAID /dev/sdg3 9437184 7812734975 7803297792 3.6T Linux RAID Disk /dev/sdg1: 2.4 GiB, 2550005760 bytes, 4980480 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdg2: 2 GiB, 2147483648 bytes, 4194304 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdg3: 3.6 TiB, 3995288469504 bytes, 7803297792 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdh: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: F681C773-F2AE-452F-8E71-55511FA5AEE0 Device Start End Sectors Size Type /dev/sdh1 2048 4982527 4980480 2.4G Linux RAID /dev/sdh2 4982528 9176831 4194304 2G Linux RAID /dev/sdh3 9437184 7812734975 7803297792 3.6T Linux RAID Disk /dev/sdh1: 2.4 GiB, 2550005760 bytes, 4980480 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdh2: 2 GiB, 2147483648 bytes, 4194304 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdh3: 3.6 TiB, 3995288469504 bytes, 7803297792 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdm3: 4 MiB, 4177408 bytes, 8159 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes output of cat /proc/mdstat ash-4.3# cat /proc/mdstat Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [raidF1] md3 : active raid5 sdg3[0] sde3[1] 15606591488 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/2] [UU___] md2 : active raid1 sdb3[0] sdc3[1] 11955200 blocks super 1.2 [2/2] [UU] md127 : active raid1 sde1[1] sdg1[0] 2490176 blocks [12/2] [UU__________] md1 : active raid1 sdb2[0] sdc2[1] sde2[2] sdg2[3] sdh2[4] 2097088 blocks [12/5] [UUUUU_______] md0 : active raid1 sdb1[0] sdc1[1] 2490176 blocks [12/2] [UU__________] unused devices: <none> output of: mdadm --detail /dev/md3 ash-4.3# mdadm --detail /dev/md3 /dev/md3: Version : 1.2 Creation Time : Sat Jun 20 00:46:08 2020 Raid Level : raid5 Array Size : 15606591488 (14883.61 GiB 15981.15 GB) Used Dev Size : 3901647872 (3720.90 GiB 3995.29 GB) Raid Devices : 5 Total Devices : 2 Persistence : Superblock is persistent Update Time : Tue Nov 17 03:41:41 2020 State : clean, FAILED Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K Name : Dabadoo:2 UUID : ff64862b:9edfe233:c498ea84:9d4b9ffd Events : 39101 Number Major Minor RaidDevice State 0 8 99 0 active sync /dev/sdg3 1 8 67 1 active sync /dev/sde3 - 0 0 2 removed - 0 0 3 removed - 0 0 4 removed output of: mdadm --examine /dev/sd[defgh]3 | egrep 'Event|/dev/sd' ash-4.3# mdadm --examine /dev/sd[defgh]3 | egrep 'Event|/dev/sd' mdadm: No md superblock detected on /dev/sdh3. /dev/sde3: Events : 39101 /dev/sdg3: Events : 39101 mdadm --examine /dev/sdg3 ash-4.3# mdadm --examine /dev/sdg3 /dev/sdg3: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : ff64862b:9edfe233:c498ea84:9d4b9ffd Name : Dabadoo:2 Creation Time : Sat Jun 20 00:46:08 2020 Raid Level : raid5 Raid Devices : 5 Avail Dev Size : 7803295744 (3720.90 GiB 3995.29 GB) Array Size : 15606591488 (14883.61 GiB 15981.15 GB) Data Offset : 2048 sectors Super Offset : 8 sectors Unused Space : before=1968 sectors, after=0 sectors State : clean Device UUID : e0c37824:42d56226:4bb0cdcc:d29cca2f Update Time : Tue Nov 17 03:41:41 2020 Checksum : cf00f943 - correct Events : 39101 Layout : left-symmetric Chunk Size : 64K Device Role : Active device 0 Array State : AA... ('A' == active, '.' == missing, 'R' == replacing) mdadm --examine /dev/sdh3 ash-4.3# mdadm --examine /dev/sdh3 mdadm: No md superblock detected on /dev/sdh3. Quote
flyride Posted November 17, 2020 #20 Posted November 17, 2020 As you suggest, it seems like disk assignments have moved around since your first post. Again, a hardware raid will always use the port position to determine array member order, but mdraid does not because it maintains an on-drive superblock. In this second run, md knows that the drive assigned to /dev/sdg3 is array member 0, and /dev/sde3 is #1. We need four disks out of five to recover data from the array. There are two really big problems with the array at the moment. /dev/sdd doesn't seem to have a recognizable partition table at all, /dev/sdf and /dev/sdh have RAID partition structures but their superblocks are no longer intact. As mentioned before this is very likely to have occurred when the disks were introduced to a foreign controller and array structure. It is possible usable data still exists on the inaccessible disks. We still could try and start the array from sde/f/g/h but the data available doesn't tell us how to order sdf and sdh. Might you have any fdisk -l dumps from prior to the crash (or any other array information saved) so we can match the drive uuids to the array member positions? Quote
Rihc0 Posted November 17, 2020 Author #21 Posted November 17, 2020 I don't, I guess that is something that shows how the raid is configured? Quote
flyride Posted November 17, 2020 #22 Posted November 17, 2020 (edited) This is the step in the process where you must understand that your best chances for recovery are to send your disks to a forensic lab. It's expensive and there is no guarantee they can recover data, but once we start to try brute-force methods to overcome the corruption that has occurred on your disks, some of the information that a lab uses to figure out how to work will be overwritten and their job will become harder or impossible. Again, the issue is that three of your five disks are now missing array administrative information, and /dev/sdd doesn't even have a partition table. Because the information defining the order of those three disks is lost, we have to work through the six possible permutations and see if any of them result in the recovery of a filesystem. Logically, I recommend we omit /dev/sdd, because it is the most likely to be corrupted into the array partition. If we include it in the array and it has corrupted data, the array will be inaccessible. If we leave it out and the remaining disks are okay, the array will function due to parity redundancy. Here's a table of the different array member permutations that will need to be attempted for recovery: The process will be to manually override the array with each of these configurations, then attempt to access it in a read-only mode, and if successful, STOP. Don't reboot, don't click on any repair options in the DSM UI. Your best result will be to copy data off immediately. Once you have recovered as much data as possible, only then consider whether it makes sense to repair the array in place, or to delete and rebuild it. If you wish to proceed, please confirm whether you chose RAID5 or SHR when you created the array in DSM. Also is the filesystem ext4 or btrfs? EDIT: I now realize that you posted that information in the beginning. RAID5 over SHR makes this job easier - that's good. Unfortunately a data recovery like this is enhanced by btrfs over ext4, as btrfs would be able to detect corruption if the checksumming features were properly enabled on the filesystem. In any case, it is irrelevant for this recovery. Edited November 17, 2020 by flyride Quote
Rihc0 Posted November 17, 2020 Author #23 Posted November 17, 2020 I see, I thought I used raid 5, but not sure, and filesystem is ext4 for sure. lets try. Btw, how did you configure your nas? I want to do it the best way but at the moment I learn by making mistakes, and I don't know what is the best way to set it up. Quote
flyride Posted November 18, 2020 #24 Posted November 18, 2020 Haven't forgotten about this, just let me assemble the commands and I will provide a suggested plan to try the matrix of permutations shortly. Quote
flyride Posted November 21, 2020 #25 Posted November 21, 2020 Work emergency took my idle time away, yay for the weekend. On 11/17/2020 at 7:50 AM, flyride said: Here's a table of the different array member permutations that will need to be attempted for recovery: The process will be to manually override the array with each of these configurations, then attempt to access it in a read-only mode, and if successful, STOP. Don't reboot, don't click on any repair options in the DSM UI. Your best result will be to copy data off immediately. Once you have recovered as much data as possible, only then consider whether it makes sense to repair the array in place, or to delete and rebuild it. We have to model the array in sequence according to the table above. Only one of the six permutations will be successful, so it is important that we do not write to the array until we are sure that it is good. The following will create the array in the configuration of the first line of the table (note that we are root): # cat /etc/fstab # mdadm --stop /dev/md3 # mdadm -v --create --assume-clean -e1.2 -n5 -l5 /dev/md3 /dev/sdg3 /dev/sde3 /dev/sdf3 /dev/sdh3 missing -uff64862b:9edfe233:c498ea84:9d4b9ffd # cat /proc/mdstat # mount -o ro,noload /dev/md3 /volume1 I expect that all the commands will execute without error except the mount command. If the mount command succeeds, we may have found the correct sequence of drives in the array. If it fails, we need to investigate further before proceeding to the next sequence. Remember NOT to use any DSM UI features to edit the array, fix the System Partition, or make any other changes. Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.