Proxmox: HBA passthrough & second virtual controller not detected


Recommended Posts

I'm running DSM 6.2.3-25426 Update 3 / DS3615xs on Proxmox. I have my actual HBA card (JMB585-based one) connected to the VM using PCI passthrough and it works perfectly. However I added another disk using Proxmox:

 

image.thumb.png.57d7137bc6d8c725dc260d7f23c0a0a3.png

 

However no matter what I do DSM doesn't see that disk in the UI. I tried changing "SCSI Controller":

  • VMWare PVSCSI - disk is visible in CLI only (fdisk)
  • VirtIO - not detected
  • LSI 53C895A - not detected

 

Some people suggested that VirtIO but DSM doesn't seem to support it. LSI 53C895A was mentioned in multiple tutorials for DSM but 6.2.3-25426 is not detecting disks connected to it. The only one which is sort-of working is the VMWare PVSCSI. However the disk is not visible in Storage Manager. DSM sees it as an eSATA disk to an extent:

417613557_ScreenShot2021-05-29at2_37_48PM.png.bb0d706f4ea887fd60245a2a7f64e8a7.png   1128653015_ScreenShot2021-05-29at2_38_04PM.thumb.png.b348bb732f3331368a1bc2642ed613b7.png  

 

I checked my sata_args but I have no idea if I should change anything in them to make it working: 

set sata_args='sata_uid=1 sata_pcislot=5 synoboot_satadom=1 DiskIdxMap=0C SataPortMap=1 SasIdxMap=0'

 

 

Can someone suggest how to make that additional disk working?

 

 

Link to post
Share on other sites
Posted (edited)

Thank you for the link. That image indeed contains the virtio driver but doesn't solve the problem. Now I can use VirtIO as the bus and the SMART shows something (i.e. emulated SMART data) when I use sata0 instead of scsi0. However the disk is still being seen as external and thus being unusable for the array :(

Edited by kiler129
Link to post
Share on other sites
Posted (edited)

What is your sata_args? No matter what I do with it there's no change in the UI and it still shows QEMU disk as external.

For me it will be logical to have DiskIdxMap=0000 SataPortMap=57

DiskIdxMap=0000 SataPortMap=57 SasIdxMap=0

as it is consistent with https://github.com/cake654326/xpenology/blob/master/synoconfigs/Kconfig.devices but after rebooting nothing changed.

 

 

EDIT

I just realized "sata_args" is NOT used when booting as baremetal (which is also, AFAIK, recommended for KVM). Thus my config from above results in semi-sensible configuration:

image.thumb.png.85e0d63169f648e5bc283279ec99da5e.png

As the first bay is currently physically empty in my setup this makes sense for disk numbering. However the QEMU disk is still seen as eSATA by DSM grrrr.... 

Edited by kiler129
Link to post
Share on other sites
# Sata0 is dsm port0, Sata1 = dsm Port1, port 2-5 is reserved for sata, scsi drives start at port 6
set extra_args_918='DiskIdxMap=0F00 SataPortMap=15'

btw im booting synoboot from usb.

Proxmox config line:

args: -device 'nec-usb-xhci,id=usb-bus0,multifunction=on' -drive 'file=/var/lib/vz/images/150/synoboot.img,media=disk,format=raw,if=none,id=synoboot' -device 'usb-storage,bus=usb-bus0.0,port=1,drive=synoboot,id=synoboot,bootindex=999,removable=on'

 

Link to post
Share on other sites
Posted (edited)

Did you modify synoinfo.conf (in /etc and /etc.defaults) by any chance? I finally managed to "brute-force" DSM to show that QEMU disk as internal one. It shows as 13th disk (which is over 12 max disk set), which is obviously invalid (for now):

 

1315453503_ScreenShot2021-05-29at7_48_27PM.thumb.png.853a691b0d2ab6c188698887e2673fe6.png

1353884137_ScreenShot2021-05-29at7_48_22PM.thumb.png.5c0f52081fddb929b21e0626ed04c1f0.png

 

 

I did force it by just telling DSM to treat all storage devices as internal:

# cat /etc.defaults/synoinfo.conf | grep portcfg
esataportcfg="0"
internalportcfg="0xffffffffff"
usbportcfg="0"

 

 

However the defaults which they provide should actually work (but somehow they don't):

# cat /etc.defaults/backup_synoinfo.conf | grep portcfg
esataportcfg="0xff000"
internalportcfg="0xfff"
usbportcfg="0x300000"

 

I saw https://hedichaibi.com/fix-xpenology-problems-viewing-internal-hard-drives-as-esata-hard-drives/ and from my dmesg I see that it sees 12 ATAs (7+5) and then 8 USB buses:

0000 0000 0000 0000 0000 0000 1111 1111 0000 0000 0000 ==> Usb ports   => 0xff000
0000 0000 0000 0000 0000 0000 0000 0000 1111 1111 1111 ==> Sata ports  => 0xfff

However setting them like that makes the QEMU disk land as eSATA again. I need to dig more into how it actually works here...

 

 

 

 

That aside: 

1 hour ago, loomes said:


# Sata0 is dsm port0, Sata1 = dsm Port1, port 2-5 is reserved for sata, scsi drives start at port 6
set extra_args_918='DiskIdxMap=0F00 SataPortMap=15'

btw im booting synoboot from usb.

I'm booting from USB too and have a similar config for the USB image. However your map is strange or I don't get how it works. Since 0F00 will mean that it will start numbering first controller from 15 (0Fh=1d, O is the 15th letter) and the first one will be sdp and the second controller will number from sda. Then SataPortMap=15 will indicate that 1st controller has 1 port and the second controller has 5 ports...

Edited by kiler129
Link to post
Share on other sites
Posted (edited)

It works!

I think I finally understood how this whole thing comes together and achieved exactly the configuration I wanted:

 

image.thumb.png.7616fce553eb1657240ee603a209e0db.png

 

No synoinfo.conf modification is needed here. For this to work I had to set the boot config to include:

DiskIdxMap=0C0005 SataPortMap=157

 

 

 

But HOW did I get these values

What didn't make sense to me was how "maxdisks" (in synoinfo.conf, patched to 12 by Jun's loader), "SataPortMap", and "DiskIdxMap" play together. Often times posts just list these values as correct ones without explanation. The kconfig is often shown as an explanation (https://github.com/cake654326/xpenology/blob/master/synoconfigs/Kconfig.devices). While it does explain things it has a significant typo in DiskIdxMap and doesn't take Xpenology synoboot hacking into consideration. However it is actually pretty simple:

  • maxdisks tells DSM how many disks the UI should enumerate and show in Storage Manager, it seems to have nothing to do with disk detection by the OS. That's why most configs just default to 12 as having this number higher than the actual number of slots will not cause problems.
  • SataPortMap is a list of digits (one character = one entry) which are read in order to tell DSM how many ports per controller to initialize (max 9; in my example 1 port at 1st controller, 5 ports at 2nd, and 7 at 3rd)
  • DiskIdxMap is a list of hex pairs (two characters = one entry) which instructs DSM how to map numbered ports from controllers to sda-sdz devs. The order of values here is the same as in SataPortMap. In my example above it maps like so:
    • 0C (dec: 12)
      • 1st controller starts mapping from 13th position
      • 1st controller had 1 port in SataPortMap
      • Result: [sdm]
    • 00 (dec: 0)
      • 2nd controller starts mapping from 1st position
      • 2nd controller had 5 ports in SataPortMap
      • Result: [sda] [sdb] [sdc] [sdd] [sde]
    • 05 (dec: 5)
      • 3rd controller starts mapping from 6th position
      • 3rd controller had 7 ports in SataPortMap
      • Result: [sdf] [sdg] [sdh] [sdi] [sdj] [sdk] [sdl]

With such configuration as above (and on the screenshot) the system sees a direct mapping between bays (even if empty) and sdX:

# fdisk -l /dev/sd? | grep 'Disk /'
Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Disk /dev/sdc: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Disk /dev/sdd: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Disk /dev/sdf: 111.8 GiB, 120034123776 bytes, 234441648 sectors

(sda and sde are missing as the ports 1 and 5 of the PCI controller aren't connected; sdf is the 1st port of the QEMU controller)

 

sdm is missing from the list as it is "renamed" to synoboot:

# fdisk -l /dev/synoboot | grep 'Disk /'
Disk /dev/synoboot: 50 MiB, 52428800 bytes, 102400 sectors

 

 

But I said I have two controllers... or do I?

An important missing bit which doesn't seem to be explained fully anywhere is that the first controller is (always?) the one where DSM boots from. Since USB devices are also SCSI-based (like SATA):

[    5.500034] sd 13:0:0:0: [synoboot] Attached SCSI removable disk

This means they will appear as sdX too. This is exactly why DiskIdxMap maps the first controller to start from 13th and why SataPortMap should list a single port in the first controller. In such configuration kernel will always put the boot device (residing on the 1st controller) outside of the 12 maxdisks making it invisible in the UI. Then the 2nd controller (which in my case is the PCI controller) starts normally from sda, making the UI numbering sane (starting from "Disk 1"):

image.thumb.png.a185a7e690d8279985df0f9c8e7f9644.png

This can be easily confirmed by setting SataPortMap to 057 - it will instantly cause a nice KP :D

image.thumb.png.d8c2e1a62020cadf9fc8b40b5f07a15a.png 

 

 

What about eSATA & synoinfo.conf?

On top of these above "internalportcfg" / "esataportcfg" / "usbportcfg" masks are used to decide which of the sda-sdz devices are treated as internal / eSATA / USB. When something is connected to the DSM. With settings above the defaults for DS3615xs make perfect sense:

  • internalportcfg=0xfff -> first 12 devices (sda-sdl) should be treated as internal

  • esataportcfg=0xff000 -> devices 13-20 (sdm-sdt) should be treated as eSATA

  • usbportcfg=0x300000 -> devices 21-22 (sdu-sdv) should be treated as USB

The DS3615xs according to the data sheet has 12 bays and 2 USB ports.

 

Any unmapped devices or controllers will be attached with letters after the mapped boundary. So connecting a USB stick in my configuration results in sdu:

# fdisk -l /dev/sd? | grep 'Disk /'
Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Disk /dev/sdc: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Disk /dev/sdd: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Disk /dev/sdf: 111.8 GiB, 120034123776 bytes, 234441648 sectors
Disk /dev/sdu: 14.6 GiB, 15682240512 bytes, 30629376 sectors

 

...and a correctly mapped entry in the UI:

image.thumb.png.d02242e4c90f376a937036bc810c5fda.png

Edited by kiler129
Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.