flyride

NVMe optimization & baremetal to ESXi report

Recommended Posts

USE NVMe AS VIRTUAL DISK / HOW TO LIMIT DSM TO SPECIFIC DRIVES

Just thought to share some of the performance I'm seeing after converting from baremetal to ESXi in order to use NVMe SSDs.

My hardware: SuperMicro X11SSH-F with E3-1230V6, 32GB RAM, Mellanox 10GBe, 8-bay hotplug chassis, with 2x WD Red 2TB (sda/sdb) in RAID1 as /volume1 and 6x WD Red 4TB (sdc-sdh) in RAID10 as /volume2

 

I run a lot of Docker apps installed on /volume1. This worked the 2TB Reds (which are not very fast) pretty hard, so I thought to replace them with SSD.  I ambitiously acquired NVMe drives (Intel P3500 2TB) to try and get those to work in DSM. I tried many tactics to get them running in the baremetal configuration. But ultimately, the only way was to virtualize them and present them as SCSI devices.

 

After converting to ESXi, /volume1 is on a pair of vmdk's (one on each NVMe drive) in the same RAID1 configuration. This was much faster, but I noted that Docker causes a lot of OS system writes which spanned all the drives (since Synology replicates the system and swap partitions across all devices). 32GB RAM is enough to avoid swap activity so that was irrelevant, so I isolated DSM I/O to the NVMe drives by disabling the system partition from the RAID10 disks:

mdadm /dev/md0 -f /dev/sdc1 .../dev/sdh1

then

mdadm -grow -n 2  /dev/md0

then repair from DSM Storage Manager (which converts the "failed" drives to hotspares)

 

After this, no Docker or DSM system I/O ever touches a spinning disk. Results: The DSM VM now boots in about 15 seconds. Docker used to take a minute or more to start and launch all the containers, now about 5 seconds.  Copying to/from the NVMe volume maxes out the 10GBe interface (1 gigaBYTE per second) and it cannot fill the DSM system cache; the NVMe disks can sustain the write rate indefinitely.  This is some serious performance, and a system configuration only possible because of XPEnology!

 

Just as a matter of pointing out what is possible with Jun's boot loader, I was able to move the DSM directly from baremetal to ESXi, without reinstalling, by passthru of the SATA controller and the 10GBe NIC to the VM.  I also was able to switch back and forth between USB boot using the baremetal bootloader menu option and ESXi boot image using the ESXi bootloader menu option. Without the correct VM settings, this will result in hangs, crashes and corruption, but it can be done.

 

I did have to shrink /volume1 to convert it to the NVMe drives (because some space was lost by virtualizing them), but ultimately was able to retain all aspects of the system configuration and many months of btrfs snapshots converting from baremetal to ESXi. For those who are contemplating such a conversion, it helps to have a mirror copy to fall back on, because it took many iterations to learn the ideal ESXi configuration.

Edited by flyride
  • Like 3

Share this post


Link to post
Share on other sites
On 3/20/2018 at 9:01 AM, flyride said:

 


mdadm /dev/md0 -f /dev/sdc1 .../dev/sdh1

then


mdadm -grow -n 2  /dev/md0

then repair from DSM Storage Manager (which converts the "failed" drives to hotspares)

interesting move, not just for VM use

how does it look in storage manager? the drives are spare an also part of a data volume (volume2 in your case i guesss as volume1 is on nvme together with docker and packages)

Share this post


Link to post
Share on other sites

Just to clarify the layout:

 

/dev/md0 is the system partition, /dev/sda.../dev/sdn.  This is a n-disk RAID1

/dev/md1 is the swap partition, /dev/sda../dev/sdn.  This is a n-disk RAID1

/dev/md2 is /volume1 (on my system /dev/sda and /dev/sdb RAID1)

/dev/md3 is /volume2 (on my system /dev/sdc.../dev/sdh RAID10)

and so forth

 

Failing RAID members manually in /dev/md0 will cause DSM to initially report that the system partition is crashed as long as the drives are present and hotplugged.  But it is still functional and there is no risk to the system, unless you fail all the drives of course.

 

At that point cat /proc/mdstat will show failed drives with (F) flag but the number of members of the array will still be n.

 

mdadm -grow -n 2  /dev/md0 forces the system RAID to the first two available devices (in my case, /dev/sda and /dev/sdb).  Failed drives will continue to be flagged as (F) but the member count will be two.

 

After this, in DSM it will still show a crashed system partition, but if you click on Repair, the failed drives will change to hotspares and the system partition state will be Normal.  If one of the /dev/md0 RAID members fails, DSM will instantly activate one of the other drives as the hotspare, so you always have two copies of DSM.

 

FYI, this does NOT work with the swap partition.  No harm in trying, but DSM will always report it crashed and fixing it will restore swap across all available devices.

 

It's probably worth mentioning that I do not use LVM (no SHR) but I think this works fine in systems with SHR arrays.

 

  • Like 1

Share this post


Link to post
Share on other sites

yes i got the mdadm part (-n 2) but if dsm sees the whole drive as spare then it might try to use the "whole" drive as spare, like using a drive thats part of volume2 as spare drive for volume1 (when one of the nvme's or vmware vmfs fails), but is guess if dsm tries to use the partiton that is still part of md3 as spare for md2 then it will hit the wall and fail to do anything harmful to md3

nice solution for the disks keep spinning problem i guess, i will keep that in mind when people complain about that or wanting ssd only system partiton(s)

pity i dont have any sata ports left, that method would justify two cheap small ssd's, maybe  end of the year when i might replace my 4TB disks with 12TB disk then i will have sata ports open

 

Share this post


Link to post
Share on other sites

This doesn't get any space back, just avoiding disk access.  The system drive partition structure is intact on all drives even after the adjustment.  So if DSM "reclaims" via hotspare activity or other, it only operates within the preallocated system partition.  So no possibility of damage to any other existing RAID partition on the drives.

 

If the system or swap partitions are deleted on any disk, DSM will call the drive Not Initialized.  Any activity that initializes a drive will create them, no exceptions

Edited by flyride

Share this post


Link to post
Share on other sites

USING PHYSICAL RDM TO ENABLE NVMe (or any other ESXi accessible disk device) AS REGULAR DSM DISK

Summary:

  1. Heretofore, XPEnology DSM under ESXi using virtual disks is unable to retrieve SMART information from those disks. Disks connected to passthrough controllers work, however.

  2. NVMe SSDs are now verified to work with XPEnology using ESXi physical Raw Device Mapping (RDM). pRDM allows the guest to directly read/write to the device, while still virtualizing the controller.

  3. NVMe SSDs configured with pRDM are about 10% faster than as a VMDK, and the full capacity of the device is accessible.

  4. Configuring pRDM using the ESXi native SCSI controller set specifically to use the "LSI Logic SAS" dialect causes DSM to generate the correct smartctl commands for SCSI drives. SMART temperature, life remaining, etc are then properly displayed from DSM, /var/log/messages is not filled with spurious errors, and drive hibernation should now be possible. EDIT: SAS controller dialect works on 6.1.7 only (see this post).

Like many other posters, I was unhappy with ESXi filling the logfiles with SMART errors every few seconds, mostly because it made the logs very hard to use for other things.  Apparently this also prevents hibernation from working. I was able to find postings online using ESXi and physical RDM to enable SMART functionality under other platforms, but this didn't seem to work with DSM, which apparently tries to query all drives as ATA devices. This is also validated by synodisk --read_temp /dev/sdn returning "-1"

 

I also didn't believe that pRDM would work with NVMe, but in hindsight I should have known better, as pRDM is frequently used to access SAN LUNs, and it is always presented as SCSI to the ESXi guests.  Here's how pRDM is configured for a local device: https://kb.vmware.com/s/article/1017530  If you try this, understand that pRDM presents the whole drive to the guest - you must have a separate datastore to store your virtual machine and the pointer files to the pRDM disk!  By comparison, a VMDK and the VM that uses it can coexist on one datastore.  The good news is that none of the disk capacity is lost to ESXi, like it is with a VMDK.

 

Once configured as a pRDM, the NVMe drive showed up with its native naming and was accessible normally.  Now, the smartctl --device=sat,auto -a /dev/sda syntax worked fine!  Using smartctl --device=test, I found that the pRDM devices were being SMART-detected as SCSI, but as expected, DSM would not query them.

 

NVMe device performance received about a 10% boost, which was unexpected based on VMWare documentation.  Here's the mirroring operation results:

root@nas:/proc/sys/dev/raid# echo 1500000 >speed_limit_min
root@nas:/proc/sys/dev/raid# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [raidF1]
<snip>
md2 : active raid1 sdb3[2] sda3[1]
      1874226176 blocks super 1.2 [3/1] [__U]
      [==>..................]  recovery = 11.6% (217817280/1874226176) finish=20.8min speed=1238152K/sec
<snip>

Once the pRDM drive mirrored and fully tested, I connected the other drive to my test VM to try a few device combinations.  Creating a second ESXi SATA controller has never tested well for me.  But I configured it anyway to see if I could get DSM to use SMART correctly.  I tried every possible permutation and the last one was the "LSI Logic SAS" controller dialect associated with the Virtual SCSI controller... and it worked!  DSM correctly identified the pRDM drive as a SCSI device, and both smartctl and synodisk worked!

 

root@testnas:/dev# smartctl -a /dev/sdb
smartctl 6.5 (build date Jan  2 2018) [x86_64-linux-3.10.102] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor:               NVMe
Product:              INTEL SSDPE2MX02
Revision:             01H0
Compliance:           SPC-4
User Capacity:        2,000,398,934,016 bytes [2.00 TB]
<snip>
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Disabled or Not Supported
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Current Drive Temperature:     26 C
Drive Trip Temperature:        85 C
<snip>
root@testnas:/dev# synodisk --read_temp /dev/sdb
disk /dev/sdb temp is 26

Finally, /var/log/messages is now quiet.  There is also a strong likelihood that drive hibernation is also possible, although I can't really test that with NVMe SSD's.

Edited by flyride
  • Like 1

Share this post


Link to post
Share on other sites

I have tried this, added SCSI Controller as LSI Logic SAS, setup the RDMs as pRDM in ESXi 6.5 but DSM doesn't then see the drives at all. It only seems to see them when they are on the SATA Controller which SMART obviously doesn't work for.

 

What disk mode did you use?

And which version of DSM?

Share this post


Link to post
Share on other sites

This was written for 6.1.7.  The system that I have configured in this way is still on that version.

 

Note that pRDM does still work fine on 6.2.1 to map ESXi physical drives to virtual SATA drives.

 

I've done some preliminary testing of 6.2.1 for SCSI/SAS.  LSI Logic SAS dialect doesn't seem to work at all.  The other two SCSI options do work. However, you can't initially configure your system with a virtual SCSI controller or the drive won't be found.  Once the drives are initialized and part of an array, they can be moved from the vSATA controller to the vSCSI controller, probably demanding a system "Recover" operation, but no system or data impact so far.

Edited by flyride

Share this post


Link to post
Share on other sites

Hi, again. (Your post is very interesting for me. I think I should follow you. :-) )

 

Have you ever tried NVMe passthrough with 918+ VM ? 

As you know, some 918+ baremetal users successfully setup NVMe as a SSD cache.

 

I wanted to know is it possible to work 918+VM with NVMe pass through like other OS's VM.

 

I think pRDM is a bit more cumbersome than passthrough because of vmdk setup.

(But I appreciate your sharing experience about those pRDM. :-) )

 

p.s.

My configuration is something like below.

I've read btrfs performance is very bad for VM workload on some Synology forum.
So I choose btrfs VM over ext4 fs. I don't measure performance seriously, but I'm satisfied using that.


With this setup, you can safely upgrade child DSM VM using snapshot, so it's easier to follow new DSM.
 

It's not recommended for everyone, but for those who uses many VMs or who loves flexibility than simplicity.

(But I'm thinking recently this design is not so good because of inefficient energy usage, complexity, can't directly migrate to Synology...)

 

p.s. 2

It's related only to esxi, but It might be possible to passthrough iGPU for transcoding to child DSM VM . (with some recent XEON, or Core i)

I don't have hardware(& budget :() to try NVMe, iGPU passthrough now, but I want to know if is there someone who has tried those.

 

Sorry for a bit thread hijacking .. ;-) 

Share this post


Link to post
Share on other sites
2 hours ago, benok said:

Have you ever tried NVMe passthrough with 918+ VM ? 

I wanted to know is it possible to work 918+VM with NVMe pass through like other OS's VM.

 

My conclusion is that flash cache has little value for the workloads likely to be delivered by a DSM installation. There are many instances of cache corruption and data loss so the risk isn't worth it for me. My goal was to get NVMe drives running as regular disk for DSM.  Presently the ONLY way to do this is to have ESXi convert them (via virtual disk, or pRDM) to SATA or SCSI.  By doing so I get 1-1.5MBps read and write speed on NVMe RAID 1.  So what purpose is the cache?

 

During testing, I did notice that I could select my NVMe drives (as SCSI via pRDM) as a SSD cache target.  This was on DSM 6.1.7 on DS3615, which does not support NVMe natively.  So DSM was identifying them as regular SSDs, eligible for cache.  This would be fine and equivalent to running DS918 code to use NVMe natively.

 

DSM for DS918 has drivers, utilities and custom udev code to differentiate SATA SSD from NVMe. However it's a policy decision by Synology to treat NVMe differently as there is no technical reason it can't act as regular disk. That is why NVMe can be used as cache but not as a regular disk.  ESXi passthrough actually presents the hardware to the VM.  All it needs to do so is to be able to identify the PCI resource (you may have to edit hardware map to force ESXi to do it though).  So if you know you are able to run NVMe cache baremetal, it is very likely to work as a passthrough device on ESXi.  But again, only as cache.

 

2 hours ago, benok said:

I've read btrfs performance is very bad for VM workload on some Synology forum.

 

I'm pretty sure that is referring to running child VM's within Synology's Virtual Machine Manager package.  But it does offer advantages for hot snapshots etc.

Running btrfs on XPenology running on ESXi VM should have no performance difference versus baremetal.

 

Share this post


Link to post
Share on other sites
2 hours ago, flyride said:

DSM for DS918 has drivers, utilities and custom udev code to differentiate SATA SSD from NVMe. However it's a policy decision by Synology to treat NVMe differently as there is no technical reason it can't act as regular disk.

Ah really, so that screenshot says it's "Cache Device".

I didn't notice such limitation exists in NVMe on DS918.

If NVMe is brought to higher models like DS3619(?),  they would remove such a limit.

 

2 hours ago, flyride said:

My conclusion is that flash cache has little value for the workloads likely to be delivered by a DSM installation. There are many instances of cache corruption and data loss so the risk isn't worth it for me.

I have no experience of cache corruption, using SSD cache several years, but it's heavy access wares SSD more, 

and SSD cache doesn't accelerate sequential read/write.

Umm, I was attracted huge datastore & good random access performance for < ~10 VMs online.

I agree that is not efficient resource usage. (But it's not so bad job for old SATA SSD.)

 

I also use SSD only datastore (via DSM nfs) on another env,  but I should consider my setup again.

(I just migrating data & app from slow DSM5.2 baremetal NAS to my esxi server with new external enclosure.)

 

2 hours ago, flyride said:

I'm pretty sure that is referring to running child VM's within Synology's Virtual Machine Manager package. 

As far as my memory is correct, several business vSphere users heavily complain about bad performance of btrfs on DSM6 beta (& Syno's support). (but I'm not sure it was nfs or iSCSI.)

That thread continues after DSM6 release. So I thought I should not choose btrfs for VM datastore.

I googled Syno's forum again, but no luck. (It might be deleted by Syno because of such a content and it was beta thread.)

 

Edited by benok

Share this post


Link to post
Share on other sites
Posted (edited)
On 3/21/2018 at 5:59 AM, flyride said:

Just to clarify the layout:

 

/dev/md0 is the system partition, /dev/sda.../dev/sdn.  This is a n-disk RAID1

/dev/md1 is the swap partition, /dev/sda../dev/sdn.  This is a n-disk RAID1

/dev/md2 is /volume1 (on my system /dev/sda and /dev/sdb RAID1)

/dev/md3 is /volume2 (on my system /dev/sdc.../dev/sdh RAID10)

and so forth

 

Failing RAID members manually in /dev/md0 will cause DSM to initially report that the system partition is crashed as long as the drives are present and hotplugged.  But it is still functional and there is no risk to the system, unless you fail all the drives of course.

 

At that point cat /proc/mdstat will show failed drives with (F) flag but the number of members of the array will still be n.

 

mdadm -grow -n 2  /dev/md0 forces the system RAID to the first two available devices (in my case, /dev/sda and /dev/sdb).  Failed drives will continue to be flagged as (F) but the member count will be two.

 

After this, in DSM it will still show a crashed system partition, but if you click on Repair, the failed drives will change to hotspares and the system partition state will be Normal.  If one of the /dev/md0 RAID members fails, DSM will instantly activate one of the other drives as the hotspare, so you always have two copies of DSM.

 

FYI, this does NOT work with the swap partition.  No harm in trying, but DSM will always report it crashed and fixing it will restore swap across all available devices.

 

It's probably worth mentioning that I do not use LVM (no SHR) but I think this works fine in systems with SHR arrays.

 

 

Hello All!

 

I know it have been some time since this post has been active 

but have been reading on the forums for some time but never posted anything so i hope that someone will be able to point me in the right direction

 

I have xpenology running on DSM 6.2.2-24922 on loader 1.04 with DS918+

and trying to accomplish what @flyride have on the count of limiting DSM system to specific drives but I am unsuccessful on the following command:

mdadm -grow -n 2  /dev/md0

I am able to fail the drives just fine and DSM storage manager reports los of system partition on the drives i want to bee excluded 

i have 2 samsung 860pro ssd´s as "sda" and "sdb" and would like to keep DSM only SSD storage and 2 seagate ironwolf drives as "sdc" and "sdd"

but every time I trie it get

admin@BALOO:/$ sudo mdadm --grow -n 2 /dev/md0
mdadm: /dev/md0: no change requested

hope anyone can point me to the next steps as i am no Linux guy

 

thanks from a neewbie

/martine

Edited by Polanskiman
Please use code tags when posting command line, code or logs.

Share this post


Link to post
Share on other sites
13 hours ago, martine said:

 


admin@BALOO:/$ sudo mdadm --grow -n 2 /dev/md0
mdadm: /dev/md0: no change requested

hope anyone can point me to the next steps as i am no Linux guy

 

This could be a bit of a high risk operation if you don't understand what is happening.  But in any case, if you are seeking assistance, post output of

cat /proc/mdstat

and

df

 

Share this post


Link to post
Share on other sites

This is a new system that will eventually take over from my trusted DS214play that are getting to slow, but at the moment there is NO important data on so if it fails i will just start over i am also in this for the learning this is what my

cat /proc/mdstat

 displays when i have failed the drive sdc1 and sdd1

admin@BALOO:/$ cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md3 : active raid1 sdc3[0] sdd3[1]
      13667560448 blocks super 1.2 [2/2] [UU]

md2 : active raid1 sda3[0] sdb3[1]
      245237056 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda2[0] sdb2[1] sdc2[2] sdd2[3]
      2097088 blocks [16/4] [UUUU____________]

md0 : active raid1 sdd1[2](F) sdc1[3](F) sda1[0] sdb1[1]
      2490176 blocks [2/2] [UU]

unused devices: <none>

and the DF command

admin@BALOO:/$ df
Filesystem       1K-blocks   Used   Available Use% Mounted on
/dev/md0           2385528 939504     1327240  42% /
none               8028388      0     8028388   0% /dev
/tmp               8074584    816     8073768   1% /tmp
/run               8074584   4252     8070332   1% /run
/dev/shm           8074584      4     8074580   1% /dev/shm
none                     4      0           4   0% /sys/fs/cgroup
cgmfs                  100      0         100   0% /run/cgmanager/fs
/dev/md2         235427576 978700   234448876   1% /volume1
/dev/md3       13120858032  17844 13120840188   1% /volume2

hope this is the info you need @flyride

 

/Martine

Share this post


Link to post
Share on other sites

It looks to me that you have already made the change and are trying to repeat the grow event.  Now go into Disk Manager and repair the System Partition to convert the failed members to spares, per the instructions cited.

Share this post


Link to post
Share on other sites

@flyride

After the "grow command" the DSM just reports that it have fixed the partitions and the DSM system partition is back on all drives and no hot spare. please see attached screen dumps..

but correct i have tried the command multiple times

 

 

Storage manager after repair.PNG

hot spare after repair.PNG

Storage manager before repair.PNG

system partition failed.png

Share this post


Link to post
Share on other sites
Posted (edited)

I think there are two issues here.  First, the GUI "hot spare" refers to /dev/md2, /dev/md3... and is not the same thing as manually configured hotspares on /dev/md0.  I am unsure if you are trying additional commands in order to get something to appear in the hotspares GUI screen.  If you successfully convert /dev/sdc1 and /dev/sdd1 to hotspares, it will NOT be reflected there.

 

Second, this technique was tested and documented for DSM 6.1.x and you are running 6.2.x.  I just tested on a DSM 6.2 system and it does just what you say it does - clicking repair from the GUI restores the failed partitions to active status.  This is a change in behavior from the GUI in 6.1.x.

 

You can still accomplish what you want to do from the command line.  Using your configuration example:

root@dsm62x:~# mdadm /dev/md0 -f /dev/sdc1 /dev/sdd1
mdadm: set /dev/sdc1 faulty in /dev/md0
mdadm: set /dev/sdd1 faulty in /dev/md0

root@dsm62x:~# mdadm --manage /dev/md0 --remove faulty
mdadm: hot removed 8:49 from /dev/md0
mdadm: hot removed 8:33 from /dev/md0

root@dsm62x:~# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md2 : active raid5 sdc3[5] sda3[0] sdd3[4] sdb3[2]
      23427613632 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md1 : active raid1 sdc2[1] sda2[0] sdb2[3] sdd2[2]
      2097088 blocks [16/4] [UUUU____________]

md0 : active raid1 sda1[0] sdb1[1]
      2490176 blocks [2/2] [UU]

unused devices: <none>

root@dsm62x:~# mdadm --add /dev/md0 /dev/sdc1
mdadm: added /dev/sdc1
root@dsm62x:~# mdadm --add /dev/md0 /dev/sdd1
mdadm: added /dev/sdd1

root@dsm62x:~# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md2 : active raid5 sdc3[5] sda3[0] sdd3[4] sdb3[2]
      23427613632 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md1 : active raid1 sdc2[1] sda2[0] sdb2[3] sdd2[2]
      2097088 blocks [16/4] [UUUU____________]

md0 : active raid1 sdd1[2](S) sdc1[3](S) sda1[0] sdb1[1]
      2490176 blocks [2/2] [UU]
root@dsm62x:~#

 

Edited by flyride

Share this post


Link to post
Share on other sites
Posted (edited)

After trying what it think is the right commands

admin@BALOO:/$ sudo mdadm /dev/md0 -f /dev/sdc1 /sdd1
Password:
mdadm: set /dev/sdc1 faulty in /dev/md0
mdadm: Cannot find /sdd1: No such file or directory
admin@BALOO:/$ sudo mdadm /dev/md0 -f /dev/sdd1
mdadm: set /dev/sdd1 faulty in /dev/md0
admin@BALOO:/$ sudo mdadm --manage /dev/md0 --remove faulty
mdadm: hot removed 8:49 from /dev/md0
mdadm: hot removed 8:33 from /dev/md0
admin@BALOO:/$ cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md3 : active raid1 sdc3[0] sdd3[1]
      13667560448 blocks super 1.2 [2/2] [UU]

md2 : active raid1 sda3[0] sdb3[1]
      245237056 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda2[0] sdb2[1] sdc2[2] sdd2[3]
      2097088 blocks [16/4] [UUUU____________]

md0 : active raid1 sda1[0] sdb1[1]
      2490176 blocks [2/2] [UU]

unused devices: <none>
admin@BALOO:/$ sudo mdadm --add /dev/md0 /dev/sdc1
mdadm: added /dev/sdc1
admin@BALOO:/$ sudo mdadm --add /dev/md0 /dev/sdd1
mdadm: added /dev/sdd1
admin@BALOO:/$ cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md3 : active raid1 sdc3[0] sdd3[1]
      13667560448 blocks super 1.2 [2/2] [UU]

md2 : active raid1 sda3[0] sdb3[1]
      245237056 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda2[0] sdb2[1] sdc2[2] sdd2[3]
      2097088 blocks [16/4] [UUUU____________]

md0 : active raid1 sdd1[2](S) sdc1[3](S) sda1[0] sdb1[1]
      2490176 blocks [2/2] [UU]

unused devices: <none>
admin@BALOO:/$

this is what i get from my system

when i use the first part of the command the DSM storage manager reports faulty system partition as before, but after the second set of commands it it back to "normal" with what seems like it is having DSM system partition on all disks once more without me having used the DSM storage manger to fix the system partition

Edited by martine
not allowed to post new comment

Share this post


Link to post
Share on other sites

All looks good from here.  After the "second set," the DSM partition is not on all disks, it's only on /dev/sda1 and /dev/sda2, which is what you were trying to do.

Share this post


Link to post
Share on other sites

Okay. 😀

well DSM reports that the system partition is still on the hard drives and when i log in to DSM when the hard drives is in hibernation i do not get the system health widget "that my system is Good" before i can hear the physical hard drives have been completely waken up.

/Martine

DSM Storage manager still reports system partition on the HDDs.PNG

DSM system health.PNG

Share this post


Link to post
Share on other sites

I think you are declaring success... ?

 

This technical information is earlier in the thread, but I'll restate here.

 

Synology's reference to the "system partition" is a generic term for two physical partitions that are each in RAID 1.  These arrays span across all regular drives in the system.  When a disk is called "Initialized" it means that DSM has created the partition structures on the drive.  The first partition is for DSM itself and is mapped to /dev/md0.  The second is for Linux swap and is mapped to /dev/md1.  A "failed" system partition means that either or both of the arrays (/dev/md0 or /dev/md1) are degraded.  Repairing the system partition resyncs those arrays.

 

What we have done is changed two of your four drives to hotspares in /dev/md0, the RAID array for the DSM partition.  The hotspare drives must have the partition configured (or DSM will call the disk "Not Initialized"), and must be in the array (or DSM will call the system partition "Failed") but are idle unless one of the remaining members fails. 

 

Unfortunately DSM does not tolerate hotspares on the swap partition (/dev/md1) so that RAID 1 will always span all the drives actively.

 

Share this post


Link to post
Share on other sites

@flyride  Great now it makes more sense in my head thanks for "spelling it out" 😀 then it is a success.

 

18 hours ago, flyride said:

  What we have done is changed two of your four drives to hotspares in /dev/md0, the RAID array for the DSM partition.  The hotspare drives must have the partition configured (or DSM will call the disk "Not Initialized"), and must be in the array (or DSM will call the system partition "Failed") but are idle unless one of the remaining members fails. 

 

Unfortunately DSM does not tolerate hotspares on the swap partition (/dev/md1) so that RAID 1 will always span all the drives actively.

 

 

i had the idea that the "system partition" would not show up i DSM storage manager, but now i get it

 

Thanks a lot for helping out a Newbie with very limited Linux knowhow 👍🥇

/Martine

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.