Richard Hopton Posted November 16, 2020 Share #1 Posted November 16, 2020 I've setup a 3617xs with DSM 6.2.3 on my ESXi 6.7 machine and passed through a Avago/LSI 9305-16i card with 10 drives attached (mostly old drives for testing before moving the drives from my old 3615xs VM) In the LSI BIOS the drives appear in the correct slots (1-16) based on their location in the enclosure, but in DSM they appear totally out of order and spread out 1-8 & 17-24. Anyone have any tips to fix this? I've no idea why 9-16 don't show up! Quote Link to comment Share on other sites More sharing options...
flyride Posted November 16, 2020 Share #2 Posted November 16, 2020 Every controller that DSM can see will reserve slots. It enumerates the controllers based on PCIe slot number, lowest to highest. If the total number of reservations exceeds the number of slots then the ports are truncated. If you have extra/idle virtual controllers in your VM definition, it will create what you see here. A secondary issue is that DSM doesn't know how many ports are on a controller, so you may need to do some tweaking in grub.cfg to help it know how to arrange the controllers and ports. For baremetal this tends not to be an issue, but under ESXi where hardware is free, it comes into play more frequently. Here are the command directives that help manage this: https://xpenology.com/forum/topic/35937-configuring-sataportmap/?do=findComment&comment=172654 Quote Link to comment Share on other sites More sharing options...
Richard Hopton Posted November 16, 2020 Author Share #3 Posted November 16, 2020 (edited) Here's my config from ESXi And if I do `lsscsi -C` it shows two controllers (0 = sata controller, 1=mpt3sas) Additionally the drives in the list are out of order:- Syno Drive # SAS Slot 1 3 3 0 4 1 5 7 7 4 17 11 19 8 21 15 22 14 23 12 This is the sata args from my grub.cfg... set sata_args='sata_uid=1 sata_pcislot=5 synoboot_satadom=1 DiskIdxMap=1800 SataPortMap=1 SasIdxMap=0' Since the controller has 16 ports I can't specify that in the SataPortMap due to it only supporting one digit Edited November 16, 2020 by Richard Hopton Quote Link to comment Share on other sites More sharing options...
Richard Hopton Posted November 18, 2020 Author Share #4 Posted November 18, 2020 On 11/16/2020 at 2:36 PM, flyride said: Here are the command directives that help manage this: https://xpenology.com/forum/topic/35937-configuring-sataportmap/?do=findComment&comment=172654 I added some additional context in my message yesterday. It sounds like I might need to play with the sata-remap but am a little unclear how I identify the device numbers, I only see ata1 in dmesg which is for the synoboot Virtual SATA Hard Drive. Quote Link to comment Share on other sites More sharing options...
flyride Posted November 18, 2020 Share #5 Posted November 18, 2020 The grub.cfg looks okay to me. But the HBA already uses more ports than the loader is configured by default (12 slots). So you will probably need to override MaxDisks and adjust the bitmaps defining the drive types (internalportcfg, usbportcfg, esataportcfg) in /etc.defaults/synoinfo.conf I think the inconsistent drive naming is characteristic of the driver. This is very difficult for technical support but DSM handles constantly changing disk assignments okay. You might find this thread illuminating: https://xpenology.com/forum/topic/27070-36-bay-nas-drives-starting-at-11-instead-of-1-103b-with-ds3617/?tab=comments#comment-138321 @IG-88 you advised on this type of HBA recently? Any further advice? Quote Link to comment Share on other sites More sharing options...
Richard Hopton Posted November 18, 2020 Author Share #6 Posted November 18, 2020 On 11/16/2020 at 2:36 PM, flyride said: Here are the command directives that help manage this: https://xpenology.com/forum/topic/35937-configuring-sataportmap/?do=findComment&comment=172654 I added some additional context in my message yesterday. It sounds like I might need to play with the sata_remap setting but i'm unclear how to find how each drive is identified. The only ata# in dmesg is ata1 for the Virtual HDD for synoboot; any additional guidance would be appreciated. I've extracted some pertinent entries from dmesg and attached... Interestingly dmesg shows the drives in the right order that they were in the enclosure so I'm still confused why the drives are showing out of order. Oddly I've wiped the partition table on every drive using an Ubuntu VM (where the drives appear in the right order) and have even recreated the VM over and over. dmesg.txt Quote Link to comment Share on other sites More sharing options...
Richard Hopton Posted November 18, 2020 Author Share #7 Posted November 18, 2020 I have an LSI 9300-16i that I was planning to try; maybe that will work the way I'd expect... It works fine on my existing Xpenology with a LSI 9205-8i & LSI 9205-4i4e so was surprised it doesn't work with the LSI 9305-16i. I really enjoy the simplicity of Xpenology but it sounds like, based on the difficulties faced by the user with a 36 bay NAS drives, that I might need to try something different if I can't get it to work sadly. Quote Link to comment Share on other sites More sharing options...
Richard Hopton Posted November 18, 2020 Author Share #8 Posted November 18, 2020 22 minutes ago, flyride said: you will probably need to override MaxDisks and adjust the bitmaps defining the drive types (internalportcfg, usbportcfg, esataportcfg) in /etc.defaults/synoinfo.conf Yeah I did that too... setting the MaxDisks to 24, and the drive type bitmaps accordingly - that worked just fine. Quote Link to comment Share on other sites More sharing options...
bearcat Posted November 18, 2020 Share #9 Posted November 18, 2020 On 11/16/2020 at 11:53 PM, Richard Hopton said: Since the controller has 16 ports I can't specify that in the SataPortMap due to it only supporting one digit I wonder, can 16 be represented by "F" ? Quote Link to comment Share on other sites More sharing options...
IG-88 Posted November 18, 2020 Share #10 Posted November 18, 2020 (edited) 6 hours ago, Richard Hopton said: dmesg.txt 12.13 kB · 0 downloads if you remove everything apart from what you already found/know then its not very likely we might see anything you might have overlooked please provide the whole dmesg from the numbering it looks like something take up the fist 16 drive positions that it behaves differently from a normal linux might be normal as synology customizes the kernel and driver and thats one of the reasons the sata_* and pata_* are not working and there are strange problems with the sas drivers in 918+ the grub.cfg is not default (jun's) anymore to, did you read about the values and what they are good for? there where references to that in older kernels and i did some documentation about this https://xpenology.com/forum/topic/32867-sata-and-sas-config-commands-in-grubcfg-and-what-they-do/ edit: to give an impression about synology's way's with its units here the original grub config of the 3617: Spoiler serial --port=0x3F8 --speed=115200 terminal_input serial terminal_output serial set default='1' set timeout='3' set fallback='0' menuentry "SYNOLOGY_1" { search --fs-uuid --no-floppy --set=root 10EE-589C linux /zImage root=/dev/md0 ihd_num=0 netif_num=4 SataPortMap=44 DiskIdxMap=100c acpi_enforce_resources=lax syno_hw_version=DS3617xs vender_format_version=2 initrd /rd.gz } menuentry "SYNOLOGY_2" { search --fs-uuid --no-floppy --set=root 45E5B07D-4783-4867-A369-F99C0CD1E610 cksum /grub_cksum.syno vender /vender -s linux /zImage root=/dev/md0 ihd_num=0 netif_num=4 ahci_remap="0>16:16>0:1>17:17>1:2>18:18>2:3>19:19>3:4>12:12>4:5>13:13>5:6>14:14>6:7>15:15>7" mv14xx_remap="0>0" opt_pci_slot=000000032 acpi_enforce_resources=lax syno_hw_version=DS3617xs vender_format_version=2 initrd /rd.gz } Edited November 18, 2020 by IG-88 Quote Link to comment Share on other sites More sharing options...
flyride Posted November 18, 2020 Share #11 Posted November 18, 2020 49 minutes ago, bearcat said: I wonder, can 16 be represented by "F" ? It is, for grub it should be 0F. However, 18 in hex is correct to move the loader drive to position 24 Quote Link to comment Share on other sites More sharing options...
IG-88 Posted November 18, 2020 Share #12 Posted November 18, 2020 DSM knows different ways of "counting" the disks and we have not tried if they can be used for our loaders but synology seems to use 16port sas controllers like in the FS6400, controllers with high counts of ports only seem to be used in sas mode if the switch supportsas="yes" is set in synoinfo.conf then there is no mapping through the usual way we use it atm with the bitmask and there are not variables set like internalportcfg, only maxdisks is present (like in synoinfo.conf of FS3017) there is also a additional variable supportportmappingv2="yes" its use can be seen in the synoinfo.conf of the FS6400 (also a unit using supportsas="yes") it might need some experiments so see what happens when the sas mode is used, might end with a unusable system, so that should only be tested with a fresh test vm or a baremetal test system (having a lsi sas controller) it can't be tested on installing as the synoinfo.conf used by dsm on install (3615/17) does not contain this and the patch in our loader does not set it (but the patch can be easily extended to add it) , so atm the only way would be to have a 3617 fresh installed system, setting it manually in the synoinfo.conf and reboot (having a serial console might be good to see what happens on boot) any guinea ... ahhh ... volunteers? Quote Link to comment Share on other sites More sharing options...
IG-88 Posted November 18, 2020 Share #13 Posted November 18, 2020 11 hours ago, Richard Hopton said: I've extracted some pertinent entries from dmesg and attached... Interestingly dmesg shows the drives in the right order that they were in the enclosure so I'm still confused why the drives are showing out of order. what can be seen in the snippet ist that the sata boot disk get "sdy" so the, 25th letter so from "DiskIdxMap=1800" the 18 seems to be working but the 00 does not and there seems to be something wrong scsi 1:0:0:0 -> sdc ??? thats supposed to be sda as we defined 00 in DiskIdxMap for the second controller to start with the 1st drive to count scsi 1:0:1:0 -> sda ok but why only starting with the 2nd drive found on the controller? scsi 1:0:2:0 -> sde ??? whats going on, what happened to "b" and "d" scsi 1:0:3:0 -> sdg can't see why its skipping scsi 1:0:4:0 -> sdq ??? i have no idea how the kernel (code from synology) comes up with these sdX values @Richard Hopton any relation to the connection of the cables and the controller posts (it should be 4 ports with 4 disks per port)? with the older lsi sas2 8 port controllers there is usually no relation to the ports of the controller, the disks are counted in a row how they where found, empty places on the controller will simply be skipped as if not existent and they appear in that row in dsm one odd thing in the snippet is that the sata controller announces a huge number of ports (30) [1.182039] ahci 0000:02:01.0: AHCI 0001.0300 32 slots 30 ports 6 Gbps 0x3fffffff impl SATA mode as i've never used esxi with dsm i can't tell if thats normal, in virtialbox i have the option to set the number of ports a sata controller provides, try to reduce that to a usual number like 2,4,6 or 8, maybe try to reduce the version of the virtual hardware of the vm used for dsm to have that number for sata ports on a low count Quote Link to comment Share on other sites More sharing options...
flyride Posted November 18, 2020 Share #14 Posted November 18, 2020 1 hour ago, IG-88 said: one odd thing in the snippet is that the sata controller announces a huge number of ports (30) [1.182039] ahci 0000:02:01.0: AHCI 0001.0300 32 slots 30 ports 6 Gbps 0x3fffffff impl SATA mode as i've never used esxi with dsm i can't tell if thats normal, in virtialbox i have the option to set the number of ports a sata controller provides, try to reduce that to a usual number like 2,4,6 or 8, maybe try to reduce the version of the virtual hardware of the vm used for dsm to have that number for sata ports on a low count That is a normal message for the virtual SATA controller under ESXi. Quote Link to comment Share on other sites More sharing options...
Richard Hopton Posted November 19, 2020 Author Share #15 Posted November 19, 2020 (edited) 10 hours ago, IG-88 said: any guinea ... ahhh ... volunteers? I'm happy to try stuff out... If I can follow your suggestions I'll give it a go 10 hours ago, IG-88 said: having a serial console might be good to see what happens on boot) How do I turn this on? Edited November 19, 2020 by Richard Hopton Quote Link to comment Share on other sites More sharing options...
Richard Hopton Posted November 20, 2020 Author Share #16 Posted November 20, 2020 (edited) looks like someone spotted this problem 4 years ago and had a solution in mind but doesn't look like it came to anything... I tried to add supportsas to the synoinfo.conf and after a reboot it forced a migrate/recover which overwrote these changes. Edited November 20, 2020 by Richard Hopton Quote Link to comment Share on other sites More sharing options...
IG-88 Posted November 23, 2020 Share #17 Posted November 23, 2020 On 11/20/2020 at 8:02 AM, Richard Hopton said: I tried to add supportsas to the synoinfo.conf and after a reboot it forced a migrate/recover which overwrote these changes. started with a fresh 1.03b, 3617, dsm 6.2.3, sas9211-8i with 2 disks (there are also 6 sata ports but not yet used/tested, all disks on sas) and added supportsas="yes" to both synoinfo.conf, rebooted system did not came up anymore but offered migration and keep settings, so i did after that it started and still had the change in /etc/synoinfo.conf but does not seem to change anything in that state, disks names are assigned the usual way on closer look to rc files in /etc, the state of "supportsas" is read from /etc.defaults/synoinfo.conf, so the entry in /etc/synoinfo.conf is not used it was supposed to be a simple test and as if its not working then thats it for now it would be a better use of time to look into quicknicks loader to see how did the >24 drives so if anyone wants to do something about that https://xpenology.com/forum/topic/8057-physical-drive-limits-of-an-emulated-synology-machine/page/4/#comments XPEnology_DSM_6.1.x-quicknick-3.0.img quicknick.img\.quicknick\quicknick.lzma\.quicknick\etc\rc.d\post Quote Link to comment Share on other sites More sharing options...
Richard Hopton Posted November 24, 2020 Author Share #18 Posted November 24, 2020 Okay - My LSI 9300-16i turned up (which is two 8 port controllers on a single card) and it works a treat. Adding new drives get a higher number, but upon a reboot the drive numbers reset to 1-16. Looks like it's an oddity with the 9305-16i that I was using before. 1 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.