Drives out of order DSM 6.2.3 with LSI 9305-16i


Recommended Posts

I've setup a 3617xs with DSM 6.2.3 on my ESXi 6.7 machine and passed through a Avago/LSI 9305-16i card with 10 drives attached (mostly old drives for testing before moving the drives from my old 3615xs VM)

 

In the LSI BIOS the drives appear in the correct slots (1-16) based on their location in the enclosure, but in DSM they appear totally out of order and spread out 1-8 & 17-24. Anyone have any tips to fix this? I've no idea why 9-16 don't show up!

 

 

Screenshot 2020-11-15 121828.png

Link to post
Share on other sites

Every controller that DSM can see will reserve slots.

It enumerates the controllers based on PCIe slot number, lowest to highest.  If the total number of reservations exceeds the number of slots then the ports are truncated.

 

If you have extra/idle virtual controllers in your VM definition, it will create what you see here.

 

A secondary issue is that DSM doesn't know how many ports are on a controller, so you may need to do some tweaking in grub.cfg to help it know how to arrange the controllers and ports.  For baremetal this tends not to be an issue, but under ESXi where hardware is free, it comes into play more frequently.

 

Here are the command directives that help manage this: https://xpenology.com/forum/topic/35937-configuring-sataportmap/?do=findComment&comment=172654

 

Link to post
Share on other sites

Here's my config from ESXi

1395272820_ScreenShot2020-11-16at2_39_28PM.thumb.png.c7f3f915dc8b38ac6f01bc4c0c6b0649.png

 

And if I do `lsscsi -C` it shows two controllers (0 = sata controller, 1=mpt3sas)

 

Additionally the drives in the list are out of order:- 
 

Syno Drive # SAS Slot
1 3
3 0
4 1
5 7
7 4
17 11
19 8
21 15
22 14
23 12
   

 

This is the sata args from my grub.cfg...

set sata_args='sata_uid=1 sata_pcislot=5 synoboot_satadom=1 DiskIdxMap=1800 SataPortMap=1 SasIdxMap=0'


Since the controller has 16 ports I can't specify that in the SataPortMap due to it only supporting one digit

Edited by Richard Hopton
Link to post
Share on other sites
On 11/16/2020 at 2:36 PM, flyride said:

I added some additional context in my message yesterday. It sounds like I might need to play with the sata-remap but am a little unclear how I identify the device numbers, I only see ata1 in dmesg which is for the synoboot Virtual SATA Hard Drive.

Link to post
Share on other sites

The grub.cfg looks okay to me.  But the HBA already uses more ports than the loader is configured by default (12 slots).  So you will probably need to override MaxDisks and adjust the bitmaps defining the drive types (internalportcfg, usbportcfg, esataportcfg) in /etc.defaults/synoinfo.conf

 

I think the inconsistent drive naming is characteristic of the driver.  This is very difficult for technical support but DSM handles constantly changing disk assignments okay.

 

You might find this thread illuminating:  https://xpenology.com/forum/topic/27070-36-bay-nas-drives-starting-at-11-instead-of-1-103b-with-ds3617/?tab=comments#comment-138321

 

 

@IG-88 you advised on this type of HBA recently?  Any further advice?

 

 

Link to post
Share on other sites
On 11/16/2020 at 2:36 PM, flyride said:

I added some additional context in my message yesterday. It sounds like I might need to play with the sata_remap setting but i'm unclear how to find how each drive is identified. The only ata# in dmesg is ata1 for the Virtual HDD for synoboot; any additional guidance would be appreciated.  I've extracted some pertinent entries from dmesg and attached... Interestingly dmesg shows the drives in the right order that they were in the enclosure so I'm still confused why the drives are showing out of order.

 

Oddly I've wiped the partition table on every drive using an Ubuntu VM (where the drives appear in the right order) and have even recreated the VM over and over.

 

 

dmesg.txt

Link to post
Share on other sites

I have an LSI 9300-16i that I was planning to try; maybe that will work the way I'd expect... It works fine on my existing Xpenology with a LSI 9205-8i & LSI 9205-4i4e so was surprised it doesn't work with the LSI 9305-16i. I really enjoy the simplicity of Xpenology but it sounds like, based on the difficulties faced by the user with a 36 bay NAS drives, that I might need to try something different if I can't get it to work sadly.

Link to post
Share on other sites
22 minutes ago, flyride said:

you will probably need to override MaxDisks and adjust the bitmaps defining the drive types (internalportcfg, usbportcfg, esataportcfg) in /etc.defaults/synoinfo.conf

Yeah I did that too... setting the MaxDisks to 24, and the drive type bitmaps accordingly - that worked just fine.

Link to post
Share on other sites
On 11/16/2020 at 11:53 PM, Richard Hopton said:

Since the controller has 16 ports I can't specify that in the SataPortMap due to it only supporting one digit

 

I wonder, can 16 be represented by "F" ?

Link to post
Share on other sites
6 hours ago, Richard Hopton said:

 

if you remove everything apart from what you already found/know then its not very likely we might see anything you might have overlooked

please provide the whole dmesg

from the numbering it looks like something take up the fist 16 drive positions

 

that it behaves differently from a normal linux might be normal as synology customizes  the kernel and driver and thats one of the reasons the sata_*  and pata_* are not working and there are strange problems with the sas drivers in 918+

 

the grub.cfg is not default (jun's) anymore to, did you read about the values and what they are good for? there where references to that in older kernels and i did some documentation about this

https://xpenology.com/forum/topic/32867-sata-and-sas-config-commands-in-grubcfg-and-what-they-do/

 

edit: to give an impression about synology's way's with its units here the original grub config of the 3617:

Spoiler

 

serial --port=0x3F8 --speed=115200
terminal_input serial
terminal_output serial
set default='1'
set timeout='3'
set fallback='0'

menuentry "SYNOLOGY_1" {
    search --fs-uuid --no-floppy --set=root 10EE-589C
    linux /zImage root=/dev/md0 ihd_num=0 netif_num=4 SataPortMap=44 DiskIdxMap=100c acpi_enforce_resources=lax syno_hw_version=DS3617xs vender_format_version=2
    initrd /rd.gz
}

menuentry "SYNOLOGY_2" {
    search --fs-uuid --no-floppy --set=root 45E5B07D-4783-4867-A369-F99C0CD1E610
    cksum /grub_cksum.syno
    vender /vender -s
    linux /zImage root=/dev/md0 ihd_num=0 netif_num=4 ahci_remap="0>16:16>0:1>17:17>1:2>18:18>2:3>19:19>3:4>12:12>4:5>13:13>5:6>14:14>6:7>15:15>7" mv14xx_remap="0>0" opt_pci_slot=000000032 acpi_enforce_resources=lax syno_hw_version=DS3617xs vender_format_version=2
    initrd /rd.gz
}

 

Edited by IG-88
Link to post
Share on other sites

DSM knows different ways of "counting" the disks and we have not tried if they can be used for our loaders but synology seems to use 16port sas controllers like in the FS6400, controllers with high counts of ports only seem to be used in sas mode

 

if the switch

supportsas="yes"

is set in synoinfo.conf then there is no mapping through the usual way we use it atm with the bitmask and there are not variables set like internalportcfg, only maxdisks is present (like  in synoinfo.conf of FS3017)

 

there is also a additional variable

supportportmappingv2="yes"

its use can be seen in the synoinfo.conf of the FS6400 (also a unit using supportsas="yes")

 

it might need some experiments so see what happens when the sas mode is used, might end with a unusable system, so that should only be tested with a fresh test vm or a baremetal test system (having a lsi sas controller)

it can't be tested on installing as the synoinfo.conf used by dsm on install (3615/17) does not contain this and the patch in our loader does not set it (but the patch can be easily extended to add it) , so atm the only way would be to have a 3617 fresh installed system, setting it manually in the synoinfo.conf and reboot (having a serial console might be good to see what happens on boot)

 

any guinea ... ahhh ... volunteers? ;-)

 

Link to post
Share on other sites
11 hours ago, Richard Hopton said:

I've extracted some pertinent entries from dmesg and attached... Interestingly dmesg shows the drives in the right order that they were in the enclosure so I'm still confused why the drives are showing out of order.

what can be seen in the snippet ist that the sata boot disk get "sdy" so the, 25th letter so from "DiskIdxMap=1800" the 18 seems to be working

but the 00 does not and there seems to be something wrong

 scsi 1:0:0:0 -> sdc

??? thats supposed to be sda as we defined 00 in DiskIdxMap for the second controller to start with the 1st drive to count

 

scsi 1:0:1:0 -> sda

ok but why only starting with the 2nd drive found on the controller?

 

scsi 1:0:2:0 -> sde

??? whats going on, what happened to "b" and "d"

 

scsi 1:0:3:0 -> sdg

can't see why its skipping

 

scsi 1:0:4:0 -> sdq

???

 

i have no idea how the kernel (code from synology) comes up with these sdX values

@Richard Hopton any relation to the connection of the cables and the controller posts (it should be 4 ports with 4 disks per port)?

with the older lsi sas2 8 port controllers there is usually no relation to the ports of the controller, the disks are counted in a row how they where found, empty places on the controller will simply be skipped as if not existent and they appear in that row in dsm

 

one odd thing in the snippet is that the sata controller announces a huge number of ports (30)

[1.182039] ahci 0000:02:01.0: AHCI 0001.0300 32 slots 30 ports 6 Gbps 0x3fffffff impl SATA mode

as i've never used esxi with dsm i can't tell if thats normal, in virtialbox i have the option to set the number of ports a sata controller provides, try to reduce that to a usual number like 2,4,6 or 8, maybe try to reduce the version of the virtual hardware of the vm used for dsm to have that number for sata ports on a low count

 

Link to post
Share on other sites
1 hour ago, IG-88 said:

one odd thing in the snippet is that the sata controller announces a huge number of ports (30)


[1.182039] ahci 0000:02:01.0: AHCI 0001.0300 32 slots 30 ports 6 Gbps 0x3fffffff impl SATA mode

as i've never used esxi with dsm i can't tell if thats normal, in virtialbox i have the option to set the number of ports a sata controller provides, try to reduce that to a usual number like 2,4,6 or 8, maybe try to reduce the version of the virtual hardware of the vm used for dsm to have that number for sata ports on a low count

 

That is a normal message for the virtual SATA controller under ESXi.

Link to post
Share on other sites
10 hours ago, IG-88 said:

any guinea ... ahhh ... volunteers?

I'm happy to try stuff out... If I can follow your suggestions I'll give it a go

 

10 hours ago, IG-88 said:

having a serial console might be good to see what happens on boot)

How do I turn this on?

Edited by Richard Hopton
Link to post
Share on other sites

looks like someone spotted this problem 4 years ago and had a solution in mind but doesn't look like it came to anything... 

 

I tried to add supportsas to the synoinfo.conf and after a reboot it forced a migrate/recover which overwrote these changes.

Edited by Richard Hopton
Link to post
Share on other sites
On 11/20/2020 at 8:02 AM, Richard Hopton said:

I tried to add supportsas to the synoinfo.conf and after a reboot it forced a migrate/recover which overwrote these changes.

 

started with a fresh 1.03b, 3617,  dsm 6.2.3, sas9211-8i with 2 disks (there are also 6 sata ports but not yet used/tested, all disks on sas)

and added supportsas="yes" to both synoinfo.conf, rebooted

system did not came up anymore but offered migration and keep settings, so i did

after that it started and still had the change in /etc/synoinfo.conf

but does not seem to change anything in that state, disks names are assigned the usual way

on closer look to rc files in /etc, the state of "supportsas" is read from /etc.defaults/synoinfo.conf, so the entry in /etc/synoinfo.conf is not used

 

it was supposed to be a simple test and as if its not working then thats it for now

it would be a better use of time to look into quicknicks loader to see how did the >24 drives

so if anyone wants to do something about that

 

https://xpenology.com/forum/topic/8057-physical-drive-limits-of-an-emulated-synology-machine/page/4/#comments

 

XPEnology_DSM_6.1.x-quicknick-3.0.img

quicknick.img\.quicknick\quicknick.lzma\.quicknick\etc\rc.d\post

 

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.