36 Bay NAS, drives starting at 11 instead of 1 (1.03b with DS3617)

[Jay Y]

 

I've been using Xpenology boxes for a few years now and decided to build a new, bigger box. So now I'm building a 36 Bay unit, plus another 24 bay and 12 bay on the way next. 

 

This is the problem: I started to plug in drives on my NAS and realized that HDD numbers start from 11 and go on sequentially, irrespective of which bay I put the next drive in. To be clear, I put my first drive in Bay 1 and that gets recorded as disk 11. Now, if I put the next drive in Bay 10 that is still shown as disk 12 in storage manager.

 

This is where it gets worse: I then put 10 drives in from Bay 1 through Bay 10, and disk numbers went up sequentially. If I then remove a drive from Bay 5, (Disk 15) and add a new drive in Bay 11 that shows up as disk 15 in storage manager. If I then go back and put a drive in Bay 5, that now shows as disk 16! 

 

I have searched through this forum and attempted lots of google-fu to figure this out with combinations of SataPortMap, DiskIdxMap, SasIdxMap and Sata_remap. I've restarted, re-configured and rebuilt this box over a dozen times in the last 4 days. For the sake of completeness I will include a list of forum posts, at this end of this post, that I've referenced in trying to solve this. I started with setting this up as a 918+ box but couldn't get that to install so switched over to 3617xs.

 

I am asking for help with these two things:

1. Consistent, predictable drive numbers (Preferably starting at disk 1): I know there are lots of posts about not caring for sequence numbers and that's what I did in the past with smaller boxes. However, this is a large NAS and I want to build it for uptime and need the ability to be able to identify correct drives to be able to pull out, replace or add new drives as needed without bringing the machine down. At minimum, I need consistent drive numbers (So disk 15 remains disk 15 and does not become disk 16, etc). I worry a lot about pulling the wrong drive out since the sequence numbers have a lot of potential to get jumbled up. Finally, I will obviously run out of disk numbers as I keep adding disks!
2. Setting up the boot loader with defaults for 36 bays: Right now I am editing config files (/etc/synoinfo.conf and /etc.defaults/synoinfo.conf) to convert this to a 36 Bay unit after DSM has installed. Is it possible to configure the boot loader so it gets configured as a 36 Bay unit from the start? I found this post for doing exactly this for a 918+:Can I somehow use the same approach with my DS3617XS? I opened up the extra.lzma but that does not have synoinfo_override.conf and jun.patch also does not have the relevant data.

 

 

My config:

• DSM Version: DSM 6.2.2-24922 Update 4
• Loader: 1.03b for DS3617xs
• CPU: Intel Xeon E5-2630L V3
• RAM: 128 GB, DDR4
• Motherboard: Supermicro X10SRH-CF
• HBAs:
  • LSI 3008 (Built into motherboard - 8 SAS Ports)
  • LSI 3224 (LSI 9305-16i) (Add on with 64 SAS Ports)
  • LSI 3008 (Dell H330 HBA) (Add on with 8 SAS Ports)
  • Motherboard also has 10 SATA ports (Only 4 used in my config)
• Backplanes: Supermicro BPN-SAS-846A and BPN-SAS-826A

 

Relevant outputs from lspci -v

0000:00:11.4 SATA controller: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] (rev 05) (prog-if 01)
	Subsystem: Super Micro Computer Inc Device 0838
	Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 65
	I/O ports at f110 [size=8]
	I/O ports at f100 [size=4]
	I/O ports at f0f0 [size=8]
	I/O ports at f0e0 [size=4]
	I/O ports at f020 [size=32]
	Memory at fbd38000 (32-bit, non-prefetchable) [size=2K]
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [70] Power Management version 3
	Capabilities: [a8] SATA HBA v1.0
	Kernel driver in use: ahci

0000:00:1f.2 SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05) (prog-if 01)
	Subsystem: Super Micro Computer Inc Device 0838
	Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 66
	I/O ports at f070 [size=8]
	I/O ports at f060 [size=4]
	I/O ports at f050 [size=8]
	I/O ports at f040 [size=4]
	I/O ports at f000 [size=32]
	Memory at fbd32000 (32-bit, non-prefetchable) [size=2K]
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [70] Power Management version 3
	Capabilities: [a8] SATA HBA v1.0
	Kernel driver in use: ahci

0000:01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
	Subsystem: Super Micro Computer Inc AOC-S3008L-L8e
	Flags: bus master, fast devsel, latency 0, IRQ 26
	I/O ports at e000 [size=256]
	Memory at fbb40000 (64-bit, non-prefetchable) [size=64K]
	Memory at fbb00000 (64-bit, non-prefetchable) [size=256K]
	Expansion ROM at fba00000 [disabled] [size=1M]
	Capabilities: [50] Power Management version 3
	Capabilities: [68] Express Endpoint, MSI 00
	Capabilities: [a8] MSI: Enable- Count=1/1 Maskable+ 64bit+
	Capabilities: [c0] MSI-X: Enable+ Count=96 Masked-
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [1e0] #19
	Capabilities: [1c0] Power Budgeting <?>
	Capabilities: [190] #16
	Capabilities: [148] Alternative Routing-ID Interpretation (ARI)
	Kernel driver in use: mpt3sas

0000:03:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3224 PCI-Express Fusion-MPT SAS-3 (rev 01)
	Subsystem: Broadcom / LSI Device 3190
	Flags: bus master, fast devsel, latency 0, IRQ 34
	I/O ports at d000 [size=256]
	Memory at fb900000 (64-bit, non-prefetchable) [size=64K]
	Expansion ROM at fb800000 [disabled] [size=1M]
	Capabilities: [50] Power Management version 3
	Capabilities: [68] Express Endpoint, MSI 00
	Capabilities: [d0] Vital Product Data
	Capabilities: [a8] MSI: Enable- Count=1/1 Maskable+ 64bit+
	Capabilities: [c0] MSI-X: Enable+ Count=96 Masked-
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [1e0] #19
	Capabilities: [1c0] Power Budgeting <?>
	Capabilities: [190] #16
	Capabilities: [148] Alternative Routing-ID Interpretation (ARI)
	Kernel driver in use: mpt3sas

0000:04:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
	Subsystem: Dell HBA330 Adapter
	Flags: bus master, fast devsel, latency 0, IRQ 40
	I/O ports at c000 [size=256]
	Memory at fb700000 (64-bit, non-prefetchable) [size=64K]
	Memory at fb600000 (64-bit, non-prefetchable) [size=1M]
	Expansion ROM at fb500000 [disabled] [size=1M]
	Capabilities: [50] Power Management version 3
	Capabilities: [68] Express Endpoint, MSI 00
	Capabilities: [a8] MSI: Enable- Count=1/1 Maskable+ 64bit+
	Capabilities: [c0] MSI-X: Enable+ Count=96 Masked-
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [1e0] #19
	Capabilities: [1c0] Power Budgeting <?>
	Capabilities: [190] #16
	Capabilities: [150] Single Root I/O Virtualization (SR-IOV)
	Capabilities: [148] Alternative Routing-ID Interpretation (ARI)
	Kernel driver in use: mpt3sas

0001:08:00.0 Non-VGA unclassified device: Marvell Technology Group Ltd. 88SE9235 PCIe 2.0 x2 4-port SATA 6 Gb/s Controller (rev ff) (prog-if ff)
	!!! Unknown header type 7f

 

These are the settings I've tried in grub.cfg

==> Currently in use
set sata_args='sata_uid=1 sata_pcislot=5 synoboot_satadom=1 DiskIdxMap=01 SataPortMap=4' 

==> DSM fails and reinstalls when I attempt to increase drive count to 36
set sata_args='sata_uid=1 sata_pcislot=5 synoboot_satadom=1 DiskIdxMap=0C SataPortMap=2421 SasIdxMap=0' 

==> Similar behavior to current state 
set sata_args='sata_uid=1 sata_pcislot=5 synoboot_satadom=1 DiskIdxMap=2428 SataPortMap=9 SasIdxMap=0'
set sata_args='sata_uid=1 sata_pcislot=5 synoboot_satadom=1 DiskIdxMap=01 SataPortMap=4 SasIdxMap=0xfffffff8 sata_remap=10>0'

I also attempted lots of combinations of sata_remap via the grub command line after reading this post: 

I specifically tried various combinations of this command via grub after DSM had already been installed

append "sata_remap=10>0:0>10"

These are some of the other posts I've gone through:

https://hedichaibi.com/fix-xpenology-problems-viewing-internal-hard-drives-as-esata-hard-drives/

https://gugucomputing.wordpress.com/2018/11/11/experiment-on-sata_args-in-grub-cfg/

 

Edited March 21, 2020 by Jay Y

[IG-88]

5 hours ago, Jay Y said:

Consistent, predictable drive numbers (Preferably starting at disk 1): I know there are lots of posts about not caring for sequence numbers and that's what I did in the past with smaller boxes. However, this is a large NAS and I want to build it for uptime and need the ability to be able to identify correct drives to be able to pull out, replace or add new drives as needed without bringing the machine down. At minimum, I need consistent drive numbers (So disk 15 remains disk 15 and does not become disk 16, etc). I worry a lot about pulling the wrong drive out since the sequence numbers have a lot of potential to get jumbled up. Finally, I will obviously run out of disk numbers as I keep adding disks!

 

i guess your Broadcom / LSI SAS controler (drivers) are the problem, the always fill gaps and drives will always show up in a row when it comes to /dev/sdX can be really annoying when you d data recovery and after a reboot the disks change sdX names

it will be hard to impossible to find good replacements for lsi sas controllers

 

5 hours ago, Jay Y said:

Setting up the boot loader with defaults for 36 bays: Right now I am editing config files (/etc/synoinfo.conf and /etc.defaults/synoinfo.conf) to convert this to a 36 Bay unit after DSM has installed. Is it possible to configure the boot loader so it gets configured as a 36 Bay unit from the start? I found this post for doing exactly this for a 918+:

 

Can I somehow use the same approach with my DS3617XS? I opened up the extra.lzma but that does not have synoinfo_override.conf and jun.patch also does not have the relevant data.

 

just to "use" higher numbers you dont need to mod the loader, just edit synoinfo.conf's and you are good to go, the only problem are big (full ~250MB pat file) updates where the whole system partition get renewed and the modded synoinfo.conf is overwritten, that not too often the case

the (synology) default for 918+ is 4 so jun had to include something for a higher number, 4 is just to low -> the patch for the loader contains a section to handle this and that can be changed

the 3615/17 default is 12 drives already, that was good enough so the loader does not mod it and does not contain code you could change to increase the number of drives

you would need to create a new patch file with "diff", patch the original file with jun's part, change your part and make a diff between the original file and your new file, then insert/replace the section for synoinfo.conf in jun's patch and repack extra.lzma - at least that is my plan to do it when i have time (its low on my list as i dont need it and most people dont need it, next are new drivers made with the newly released 24922 kernel source)

 

i can't remember a case where someone had more then 24 drives safely running

quicknick had a part in his loader that only used "safe" numbers, so at least try 35 or 40 instead of 36

12, 16, 20, 24, 25, 26, 28, 30, 32, 35, 40, 45, 48, 50, 55, 58, 60, 64

https://xpenology.com/forum/topic/8057-physical-drive-limits-of-an-emulated-synology-machine/page/3/?tab=comments#comment-87213

 

[mosaati]

Can I ask how many cores reported on your setup?

[Jay Y]

On 3/22/2020 at 1:18 AM, mosaati said:

Can I ask how many cores reported on your setup?

I turned off hyper threading on the CPU. On command line cores was 8, in DSM it was 4 if I recall right 

[Jay Y]

On 3/21/2020 at 3:25 PM, IG-88 said:

https://xpenology.com/forum/topic/8057-physical-drive-limits-of-an-emulated-synology-machine/page/3/?tab=comments#comment-87213

 

 

Ultimately this turned out to be a problem bigger than the others! I tried some combinations of disk counts but the system was just not stable and would occasionally lose all config and start setup on reboot. That risk was not acceptable so I gave up on using Xpenology for this box and went with FreeNAS instead. 

 

I did build out one 24 bay Xpenology device next. Did that initially with the 918+ loader but that kept having issues as well so I switched to DS3617 and that runs wonderfully. I've used the 3617 in the past as well and now just trust it to work stably long term. 

 

Thanks for the help. Hopefully a fix will be found by someone creating a loader for the more pro syno devices out there. 

[bearcat]

On 3/21/2020 at 5:11 PM, Jay Y said:

 

 

My config:

• HBAs:
  • LSI 3008 (Built into motherboard - 8 SAS Ports)
  • LSI 3224 (LSI 9305-16i) (Add on with 64 SAS Ports)
  • LSI 3008 (Dell H330 HBA) (Add on with 8 SAS Ports)
  • Motherboard also has 10 SATA ports (Only 4 used in my config)

 

 

 

Just out of curiosity, if one of your HBA's have 64 SAS ports, why do you use any of the others?

 

btw: are they all in IT-mode?

 

[Jay Y]

7 hours ago, bearcat said:

 

Just out of curiosity, if one of your HBA's have 64 SAS ports, why do you use any of the others?

 

btw: are they all in IT-mode?

 

 

Yup, all in IT mode.

 

Sorry, I wrote that wrong; not sure what I was thinking then! Its 16 ports and the others have 8 ports each. In addition the board has 10 SATA ports of which I used 4.