flyride

- Outcome of the update: SUCCESSFUL (but see comments) - DSM version prior update: DSM 6.2.2-24922 Update 6 - Loader version and model: Jun's v1.04b - DS918+ - Using custom extra.lzma: YES - real3x mod (but see comments) - Installation type: BAREMETAL - ASRock J4105-ITX - Comments: no /dev/dri (missing Gemini Lake firmware) NVMe code is new. The original NVMe patch does not work. I uninstalled NVMe cache as a prerequisite prior to upgrade and recommend that you do too. NVMe cache is reinstalled and working fine after applying the updated patch here. ASRock J/Q-series motherboards require extra.lzma or real3x mod to boot DSM 6.2.2. 6.2.3 is verified to install on J4105-ITX with no mods. So I chose to revert real3x mod immediately prior to invoking the upgrade with the following procedure:

Loader version and type (918+): 1.04b DS918+ DSM version in use (including critical update): DSM 6.2.3-25423 Using custom modules/ramdisk? If yes which one?: None Hardware details: Motherboard: ASRock J4105-ITX Comments: A clean baremetal install of 6.2.3 installs on this motherboard and supports the RealTek NIC Gemini Lake i915 drivers and firmware will need to be added in order to transcode NVMe cache support requires an updated patch

NOTE: 6.2.3 upgrade overwrites the nvme lib file. I strongly recommend deleting your r/w cache prior to installing 6.2.3 or risk corrupting your volume on reboot. The patch installs (meaning the search strings are found) but it is not working completely. There is an additional PCI port check that is in the new library that is not addressed by this patch.

That doesn't work the way you think it does. If you mounted it there, it would not be storing any data on the filesystem mounted on /volume1. You will only confuse yourself by doing so. But it doesn't matter, as far as I can see, your filesystem is quite damaged. Do the restore and see if you can get your files out.

Just for fun, # mkdir /mnt/volume1 # mount -o recovery,ro /dev/md2 /mnt/volume1 and see if you can mount the filesystem somewhere else. If nothing works, you may want to do a btrfs file restore https://btrfs.wiki.kernel.org/index.php/Restore

Stale file handle is an odd error message. Have you rebooted this NAS? Are you using NFS? If you are, disable it.

In that case you only have filesystem repair options, or extract files in a recovery mode. If you have no other plan, start here and follow the subsequent posts until resolution.

Bay Trail precedes Haswell, but it's one of those hybrid architectures. The GPU is from Ivy Bridge. I've previously suggested that the processor feature DS918+ is dependent upon is FMA3, which showed up in Haswell, and really isn't anything that we should use in the NAS world, but it seems to be how the kernel is compiled. I'm not sure how Bearcat got a J1900 to work in that case, but here are the architecture comparisons nonetheless. https://en.wikipedia.org/wiki/Silvermont https://en.wikipedia.org/wiki/Haswell_(microarchitecture)

NOTE: This problem is consistently manifested when running Jun's loader on a virtual machine with 6.2.3, but some also have problems on baremetal, and under certain conditions, other 6.2.x versions. The fix can be implemented safely on all Jun loader installs. You can verify if you have the issue by launching SSH and issuing the following command: $ ls /dev/synoboot* /dev/synoboot /dev/synoboot1 /dev/synoboot2 If these files are not present, your Synoboot devices are not being configured correctly, and you should install the fix script. If synoboot3 exists that is okay. TL;DR: When running DSM 6.2.3 as a virtual machine (and sometimes on baremetal), Jun's 1.03b and 1.04b bootloaders fail to build /dev/synoboot Bootloader SATA device normally mapped beyond the MaxDisks limit becomes visible if /dev/synoboot is not created DSM 6.2.3 update rewrites the synoinfo.cfg disk port bitmasks which may break arrays >12 disks, and cause odd behavior with the bootloader device Background: Setting the PID/VID for a baremetal install allows Jun's loader to pretend that the USB key is a genuine Synology flash loader. On an ESXi install, there is no USB key - instead, the loader runs a script to find its own boot device, and then remakes it as /dev/synoboot. This was very reliable on 6.1.x and Jun's loader 1.02b. But moving to DSM 6.2.x and loaders 1.03b and 1.04b, there are circumstances when /dev/synoboot is created and the original boot device is not suppressed. The result is that sometimes the loader device is visible in Storage Manager. Someone found that if the controller was mapped beyond the maximum number of disk devices (MaxDisks), any errant /dev/sd boot device was suppressed. Adjusting DiskIdxMap became an alternative way to "hide" the loader device on ESXi and Jun's latest loaders use this technique. Now, DSM 6.2.3: The upgrade changes at least two fundamental DSM behaviors: SATA devices that are mapped beyond the MaxDisks limit no longer are suppressed, including the loader (appearing as /dev/sdm if DiskIdxMap is set to 0C) The disk port configuration bitmasks are rewritten in synoinfo.conf: internalportcfg, usbportcfg and esataportcfg and on 1.04b, do not match up with the default MaxDisks=16 anymore (or if you have modified MaxDisks). NOTE: If you have more than 12 disks, it will probably break your array and you will need to restore the values of those parameters Also, when running as a virtual machine (and sometimes on baremetal), DSM 6.2.3 breaks Jun's loader synoboot script such that /dev/synoboot is not created at all. Negative impacts: The loader device might be accidentally configured in Storage Manager, which will crash the system The loader partitions may inadvertently be mapped as USB or eSata folders in File Station and become corrupted Absence of /dev/synoboot devices may cause future upgrades to fail, when the upgrade wants to modify rd.gz in the loader (often, ERROR 21 or "file corrupt") Unpacking Jun's synoboot script reveals that it harvests the device nodes, deletes the devices altogether, and remakes them as /dev/synoboot. It tries to identify the boot device by looking for a partition smaller than the smallest array partition allowed. It's an ambiguous strategy to identify the device, and something new in 6.2.3 is causing it to fail during early boot system state. There are a few technical configuration options can can cause the script to select the correct device, but they are difficult and dependent upon loader version, DSM platform, and BIOS/EFI boot. Fix: However, if Jun's script is re-run after the system is fully started, everything is as it should be. So extracting the script from the loader, and adding it to post-boot actions appears to be a solution to this problem: Download the attached FixSynoboot.sh script (if you cannot see it attached to this post, be sure you are logged in) Copy the file to /usr/local/etc/rc.d chmod 0755 /usr/local/etc/rc.d/FixSynoboot.sh Thus, Jun's own code will re-run after the initial boot after whatever system initialization parameters that break the first run of the script no longer apply. This solution works with either 1.03b or 1.04b and is simple to install. This should be considered required for a virtual system running 6.2.3, and it won't hurt anything if installed or ported to another environment. FixSynoboot.sh

ls -la /sys/bus/cpu/devices/cpu* | wc -l This will show you the count of active cores.

- Outcome of the update: SUCCESSFUL - DSM version prior update: DSM 6.2.2-24922 Update 6 - Loader version and model: Jun's v1.03b - DS3615xs - Using custom extra.lzma: NO - Installation type: VM - ESXi 6.7U2 - Comments: Test VM. Boot loader no longer aliased as /dev/synoboot but displayed as /dev/sdm. "satashare-x" appeared in Control Panel and File Station - Outcome of the update: SUCCESSFUL - DSM version prior update: DSM 6.2.2-24922 Update 5 - Loader version and model: Jun's v1.04b - DS918+ - Using custom extra.lzma: NO - Installation type: VM - ESXi 6.7U2 - Comments: Test VM. Boot loader no longer aliased as /dev/synoboot but displayed as /dev/sdm /dev/synoboot is broken on ESXi and DSM 6.2.3. This has a number of negative implications. See here for resolution and a warning for arrays larger than 12 disks

You don't need more drive slots, you need to correctly configure and test your SataPortMap/DiskIdxMap for your controllers so that all active ports map to active slots.

I have no issue with Jumbo frames and Mellanox Connect-X 3 installed on both NAS and Windows 10 1903 and direct connected via DAS. I get full-speed transfers >1.1 GBps I would always keep a standard 1Gbps Intel card for out-of-band management and emergency access.

There is a big difference between directing someone on a technical recovery, and offering opinion on a course of action. I have done a lot of mdadm array recoveries in my past work life, a fair number of my own, and those you see online. DSM cannot repair but the most simple array failures, which leaves folks otherwise in the cold, while I am pretty confident that if there is a reasonable way to recover mdadm, I can help find it. I don't know anything about working with a forensic recovery service, and I have no direct knowledge of how to run recovery tools. So while I am happy to offer my opinion if you describe a situation and a decision point, I can't offer step-by-step, detailed direction on how to proceed. In short, I can't continue to be a guru. I post online here because I want to provide good examples on how to recover broken arrays so that XPenology users can help each other and resolve their own problems. I think it would be worth documenting your upcoming experience here for the same reason, and I am willing to participate in and facilitate that. I agree that if you want to attempt filesystem repairs, making a full copy of your array to another (larger) drive is a good move before proceeding. You will need to decide if you want to work on the original or the copy (I advise the copy). Be careful when doing this operation as it is operating directly on storage devices with no safeguards and it's not impossible to accidentally damage your original or some other storage device (like the root filesystem).

So that is where you are now. 48 hours ago, this was the state: I also said this: Up to this point, we had really made *NO* irreversible changes to the system, aside from rewriting the array superblocks when executing the array create. The point of editing the fstab file is so we did not have to do a filesystem recovery or other action that would alter the data in the filesystem. When the filesystem was finally mounted, it was read-only so that no other action could damage the data within. Then you asked if you could attach an 8TB drive directly to the system. Aside from diverging from recommendation, I was not expecting you to use the GUI to configure it. It could have been attached via usb, or mkfs on the new device. Unfortunately GUI was used and I believe that DSM tried to write to your lv device directly to build the new filesystem, which in all evidence has corrupted the filesystem that was already there. So when dmesg says "can't find ext4 filesystem" that is probably the reason why. I think that now there's no direct way to get to the data on the filesystem without heroic and/or irreversible filesystem check/recovery operations. That's the bad news. The good news is that your array is still in the same state (critical, but intact). And very likely that most of your data is still physically present on the array. You basically have three choices now, and to be blunt, you're on your own with the decision and how to proceed further. 1. Send your drives off for forensic file recovery. This is expensive, but they will find data and recover it for you. If you do this, it's vitally important that you tell them the following: You are sending a FOUR-drive Synology SHR, with one missing member drive 0 is missing drive 1 is /dev/sdc5: Array UUID : 75762e2e:4629b4db:259f216e:a39c266d Device UUID : 6ba575e4:53121f53:a8fe4876:173d11a9 drive 2 is /dev/sdb5: Array UUID : 75762e2e:4629b4db:259f216e:a39c266d Device UUID : 7eee55dc:dbbf5609:e737801d:87903b6c drive 3 is /dev/sdd5: Array UUID : 75762e2e:4629b4db:259f216e:a39c266d Device UUID : fb417ce4:fcdd58fb:72d35e06:9d7098b5 The filesystem type is ext4, NOT BTRFS (there may be spurious btrfs signatures in various places in /dev/lv) 2. Attempt to recover the filesystem using e2fsck or variant At its core, fsck is an irreversible process, which might invalidate a subsequent choice to send the drives for forensic recovery If I were to do this, I would first clone /dev/lv to another device as a backup. Unfortunately you don't possess a device large enough to do this (8.17TB > 7.27TB) 3. Use third-party utilities to attempt to recover data from the /dev/lv device directly You would need to be able to safely install such utility via command line You would need to connect storage. The best way to do that would be to connect your 8TB via usb which then would be accessible as a mounted filesystem. These tools are not plug and play, may need some technical guidance, and may have varying degrees of effectiveness Here's a Google search to help you with your decisions: https://www.google.com/search?q=recover+data+from+unmountable+ext4&oq=recover+data+from+unmountable+ext4

Do #135 again please.

I don't really understand all the usb events in your dmesg. Are you plugging and unplugging devices? Anyway, please review dmesg for the last message that is related to the mount operation, as it is not shown in what you posted.

dmesg?

It seems like it can't figure out the file system type at mount time now. Add a space between "-o" and "ro" - it shouldn't matter but please do it anyway. # mount -t ext4 -v -o ro,noload,sb=644972544 /dev/vg1000/lv /volume1 # dmesg | tail

# mount -v -oro,noload,sb=644972544 /dev/vg1000/lv /volume1 # dmesg | tail

Do the step in post #75. I am concerned that trying to add the new volume may have corrupted your old one.

Just to make sure, # lvm lvmdiskscan # vgchange -ay # mount -v -oro,noload,sb=1934917632 /dev/vg1000/lv /volume1

The symlink to the homes root folder has been damaged. I don't think you should worry about that now. Can you mount your filesystem manually? # mount -v -oro,noload,sb=1934917632 /dev/vg1000/lv /volume1

Is the 8TB drive out of the system? If so, edit fstab again and reboot.

Ok, just edit your /etc/fstab to look exactly like post #93. Shut down the NAS. Remove the 8TB drive altogether. Power back up and hopefully you should be able to get to your data again.