HDD sleeping in passthrough with ESXI

waazdakka · November 15, 2023

Hello guys,

Just an new question / update on the disks sleep with ESXI, but in a different configuration than I used on this topic:

I just bought an HPE Microserver gen10+ v2 and I think I won't use a SAS card since I don't need a lot of storage, so the setup will be like this:

- 1 PCIe card with 2 NVMe drives: 1 for the ESXI and the datastore, I have not decided yet for what I'm going to use the second NVMe slot

- 4 HDD on the SATA bays

I'm going to install ESXI on 1 SSD NVme of the PCIe card, with the Synology VM (And probably other VMs, but not using the HDD)

I will to configure the 4 HDD in passthrough to manage the RAID with Synology.

Since it's directly connected to the mainboard, do I need to configure anything else to get a real passthrough? I can perform multiple installations to test it but I'm sure you'll be able to provide a lot of good advices

Thanks for your help!

dj_nsk · November 16, 2023

13 часов назад, waazdakka сказал:

will to configure the 4 HDD in passthrough to manage the RAID with Synology.

You need to configure the passthrough of the SATA controller, and not each HDD separately

waazdakka · November 16, 2023

Oh wow so it's a full passthrough and I will get all informations about HDD (SMART, serials etc) directly into DSM?

Are they any pro and cons about this configuration?

Thanks a lot!!!

dj_nsk · November 17, 2023

15 часов назад, waazdakka сказал:

Oh wow so it's a full passthrough and I will get all informations about HDD (SMART, serials etc) directly into DSM?

Yes

15 часов назад, waazdakka сказал:

Are they any pro and cons about this configuration?

the disadvantage is that all disks connected to this controller become inaccessible to ESXi datastore

waazdakka · November 20, 2023

Ok ! Well I'm confused now if I'm using VM or going back to baremetal...

dj_nsk · November 20, 2023

3 часа назад, waazdakka сказал:

Ok ! Well I'm confused now if I'm using VM or going back to baremetal...

If you need other virtual machines (for example, Windows or several different DSMs, including test ones), then ESXi is a great solution. If you only need one DSM, then ESXi is not required.

waazdakka · November 29, 2023

Well I followed the tutorial but the 2 disks disappeared 🤣 (and the SATA controller is still here)
I think I made a mistake somewhere..

To be clear, I need to add both part of configuration, or just the first one? (Obviously personnalized with the right vendor ID)

And if I need to put the second part, is there not a problem on the second line, where there is not the device ID mentionned? And do I need to remove the "(in hex) (ffff can be used for wildchar match)" part since it seems to be just a comment?

Thanks a lot... (I feel like a zero 🤦‍♂️)

dj_nsk · November 29, 2023

2 часа назад, waazdakka сказал:

Well I followed the tutorial but...

what exactly did you do?

What is the result of the following command?

lspci -v | grep "Class 0106" -B 1

2 часа назад, waazdakka сказал:

... I need to add both part of configuration, or just the first one?

the line "#Intel Corporation Lynx Point AHCI Controller" from the article on the link is just a comment that you need yourself. The system does not use it in any way, the string "8086 8d02 d3d0 false" is enough for it.

Спойлер

file format: vendor-id device-id resetMethod fptShareable
vendor/device id: xxxx (in hex) (ffff can be used for wildchar match)
reset methods: flr, d3d0, link, bridge, default
fptShareable: true/default, false

waazdakka · November 30, 2023

So I entered

lspci -v | grep "Class 0106" -B 1

The result of the commande gives me

0000:00:17.0 Mass storage controller SATA controller: Intel Corporation Device 43d2 
	 Class 0106: 8086:43d2

I added this line at the end of the file /etc/vmware/passthru.map to add the controller into passtrough list

8086   43d2    d3d0    false

So the final file looks like this:

# Intel 82579LM Gig NIC can be reset with d3d0
8086  1502  d3d0     default
# Intel 82598 10Gig cards can be reset with d3d0
8086  10b6  d3d0     default
8086  10c6  d3d0     default
8086  10c7  d3d0     default
8086  10c8  d3d0     default
8086  10dd  d3d0     default
# Broadcom 57710/57711/57712 10Gig cards are not shareable
14e4  164e  default  false
14e4  164f  default  false
14e4  1650  default  false
14e4  1662  link     false
# Qlogic 8Gb FC card can not be shared
1077  2532  default  false
# LSILogic 1068 based SAS controllers
1000  0056  d3d0     default
1000  0058  d3d0     default
# NVIDIA (FLR issue on Ampere and Hopper GPUs)
10de  ffff  bridge   false
# AMD FCH SATA Controller [AHCI mode]
1022  7901  d3d0     default
8086   43d2    d3d0    false

Before activating "Relay" on the manage / hardware tab, I do have all my disks but all the disks disappear when I activate it and reboot the server.

BUT... I do have an error when activating relay:

And if I hit refresh, the message is different...

Edited November 30, 2023 by waazdakka

waazdakka · November 30, 2023

Hmm I've found some leads which could explain my issue.

I'm using a QM2-2P10G1TB QNAP card (2 NVMe + 10Gb) and My ESXI is installed on the NVMe disk.

I found someone who seems to have the same issue:

And I don't have any IOMMU option on my BIOS (I'm using a HPE Proliant Microserver Gen10+ v2)

dj_nsk · December 1, 2023

I don't know the specifics of HP, I can't help you anymore.

Perhaps you need a specialized version of ESXi with drivers for HP servers? (VMware-ESXi-***-HPE-***)

waazdakka · December 1, 2023

That's the ESXI version I used, but I didn't installed the HPE addon pack, I'll try tomorrow but I'm not sure it will change anything.

Quote

HPE Addon for ESXi 8.0U2

Contains:
Name--Version
--------------------------- -----------------------
Broadcom-bnxt-Net-RoCE--226.0.221.0-1OEM.800.1.0.20613240
Intel-Volume-Mgmt-Device--3.2.0.1008-1OEM.800.1.0.20613240
Intel-i40en--2.4.2.0-1OEM.700.1.0.15843807
Intel-icen--1.11.3.0-1OEM.800.1.0.20613240
Intel-igbn--1.11.2.0-1OEM.800.1.0.20613240
Intel-ixgben--1.15.1.0-1OEM.800.1.0.20613240
MRVL-E3-Ethernet-iSCSI-FCoE--3.0.221.0-1OEM.700.1.0.15843807
MRVL-E4-CNA-Driver-Bundle--6.0.367.0-1OEM.800.1.0.20613240
MRVL-QLogic-FC--5.4.81.2-1OEM.800.1.0.20613240
Mellanox-MFT-Tools--4.25.0.802-1OEM.802.0.0.21974771
Mellanox-NATIVE-NMST--4.25.0.802-1OEM.802.0.0.21974771
Microchip-smartpqi--80.4532.0.108-1OEM.800.1.0.20613240
amsdComponent--701.11.9.5.16-1
amsdvComponent--701.11.5.0.2-1
fc-enablement-component--800.3.9.0.30-1
hpe-upgrade-component--901.2.0.5-1OEM.800.0.0.20172892
ilo-driver--700.10.8.1.6-1OEM.700.1.0.15843807
ilorest-component--800.4.5.0.0.4-20613240
ssacli2-component--6.25.9.0-8.0.0
sutComponent--800.4.5.0.22-0

But I'm pretty sure the issue is on the server side since I found some "interesting" elements on the technical sheets:

The Storage Controller controller is an Intel VROC SATA Software RAID. Maybe I have to dig on how to deactivate it for good, IF it's possible since I just discovered on the specs that...

Quote

Intel VROC SATA RAID supports Windows Server and Linux but does not support VMware.

🤦‍♂️OMG.

But I'm not sure on how VROC works with the "Essential RAID Controller: HPE Smart Array E208i-p SR Gen10 Controller". I have to try some BIOS settings I guess.

Anyway, I'm a it disapointed of the lack of support between the different components. As I said, I'm using a QNAP QM2-2P10G1TB with a 10Gb port and the 10 Gb chipset,

AQC113C, is not supported by VMware and will probably never be...

I guess I missed some critical points when building this 🤣

HDD sleeping in passthrough with ESXI

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Join the conversation