Tutorial: Install DSM 6.2.x on ESXi 6.7/7.0 as BAREMETAL


Recommended Posts

I know there are tutorials for installing DSM on ESXi as "virtualization" by creating virtual SATA drive. I ran into DSM upgrade problem (Error 13) using that method. Finally, I've dicovered a way to let ESXi create an emulated USB storage for the loader (the BAREMETAL way)!

 

1. Get the loader.

Either take Jun's loader 1.04b and do some customization, or take the modified loader here.

 

1a. Modify VID/PID grub.cfg

If you are using the modified loader, the VID/PID has already been fixed. Skip to the next step.

If you are using original 1.04b loader, follow the baremetal guide to mount the second partition in the loader, edit grub.cfg using the following VID/PID set.
 

# VMware virtual USB mass storage
set vid=0x0e0f
set pid=0x0005

 

DO NOT remove any boot options, ONLY update VID/PID (and SN/MAC if you have them).

 

Then, use StarWind V2V Converter to covert the .bin to .vmdk. (Note: it will generate a .vmdk and a -flat.vmdk file. Both files are required)

 

2. Create VM in ESXi

While creating VM, add a USB 3.0 controller

esxi-create-vm-usb-controller.thumb.png.3203ca24815edd9a2fb74a1aaf4f0143.png

 

In the VM options, select OS type "Other 4.x or later Linux (64-bit)".

esxi-create-vm-os-version.thumb.png.1ab4aabfc20e6e92a79fc50261919814.png

 

In the boot option, select EFI firmware, uncheck Secure boot, and select Force BIOS setup.

esxi-create-vm-boot.thumb.png.0efa88a8705ef6bf25feebbd95ec98b2.png

 

After VM is created, DO NOT power it up for now.

 

3. Attach loader as emulated USB storage.

Go to ESXI datastore.

 

First, update synoboot.vmdk + synoboot.bin to the VM folder (synoboot.vmdk + synoboot-flat.vmdk if you follow Step 1a).

 

Then, download the VM's .vmx file (VM configuration file). Edit it with a text editor.

 

Make sure the following line present in the .vmx file.

usb_xhci.present = "TRUE"

 

Then, add the following line at the bottom of the file.

usb_xhci:3.present = "TRUE"
usb_xhci:3.deviceType = "disk"
usb_xhci:3.fileName = "synoboot.vmdk"
usb_xhci:3.readonly = "FALSE"

 

Re-upload the .vmx file to datastore.

 

Now it is ready to power up the VM.

 

4. Setup boot device

As soon as the VM is powered on, it should go into the BIOS settings.

 

Navigate to Enter setup -> Configure boot option -> Enable or disable boot option.

 

Leave "EFI USB device" as ENABLED and set the rest options to DISABLED.

esxi-bios-setup.thumb.png.0a854b9e30e5343d979427f399dedba8.png

 

Commit change and exit, and reboot the VM.

 

 

 

Now the DSM is running in ESXi in the same way as a "baremetal" installation. No more Error 13 while upgrade DSM!

  • Thanks 1
Link to post
Share on other sites

This actually works great. I am using a passed through Dell H200 HBA flashed to LSI IT-Mode. Also i am using a virtual SATA-Controller from ESXi with one disk in addition.

 

The only thing i noticed is that you have to use common_args instead of sata_args to define your port mappings.

The variable sata_args is ignored it seems.

 

My mapping is this:

Bay 1-8 Dell H200

Bay 9 VMware Disk

Link to post
Share on other sites
On 5/3/2021 at 6:21 PM, tvon said:

I gave this a shot and it seemed to work but after going through the find.synology tool, it said the VM had no HD attached. 

 

If you have configured virtual disk, you'd better using a virtual SATA controller.

 

I'm actually using PCIe passthrough to attach my LSI 9211-8i to the VM, so the VM natively see 10 disks.

Link to post
Share on other sites
  • 3 weeks later...

Hi,

Just trying to get this working on ESXi 7, but no success. I see you have "Then, use StarWind V2V Converter to covert the .bin to .vmdk" - are you referring to the synoboot.img file from the "modified loader" or any other bin-file that I have missed? Thanks!

Link to post
Share on other sites

 

On 5/26/2021 at 6:54 AM, Grevlingkroken said:

Hi,

Just trying to get this working on ESXi 7, but no success. I see you have "Then, use StarWind V2V Converter to covert the .bin to .vmdk" - are you referring to the synoboot.img file from the "modified loader" or any other bin-file that I have missed? Thanks!

 

The ".bin" file was a typo, but it seems I cannot edit the post. Here the ".bin" means synoboot.img you just patched in Step 1a.

 

On 5/28/2021 at 10:46 AM, Marawan said:

This tutorial is meant for DS918+ (Since you're using loader v1.04b and UEFI boot)

Have you tried this for DS3615xs/DS3617xs with loader 1.03b in CSM mode?

 

I don't think ESXi VM can boot from virtual USB storage in BIOS mode. It has to be in EFI mode.

Also, there is no CSM in ESXi's EFI mode.

I haven't tried v1.03b loader, but I don't think it will work.

Link to post
Share on other sites
8 hours ago, snailium said:

 

I don't think ESXi VM can boot from virtual USB storage in BIOS mode. It has to be in EFI mode.

Also, there is no CSM in ESXi's EFI mode.

I haven't tried v1.03b loader, but I don't think it will work.

 

Not true (at least with VM built in 6.7), set BIOS mode here:

 

image.thumb.png.f59088ac48777af0fbb4f497b3906030.png

 

This particular VM is running 1.03b happily with ESXi 7.  I would be very surprised if ESXi 7 VM's did not have BIOS option, but if that happens, create the VM with an earlier ESXi version.

Link to post
Share on other sites
  • 2 weeks later...
On 6/4/2021 at 9:25 AM, KeyMag said:

Hi! Can you help with DS3617xs? When i use your method all loader start correct but network doesn't work.

ESXi 6.7

 

It sounds the network driver problem. Did you choose "e1000e" for the virtual network adapter? Or maybe try IG-88's extension for additional network driver?

Link to post
Share on other sites
On 5/31/2021 at 11:00 AM, flyride said:

 

Not true (at least with VM built in 6.7), set BIOS mode here:

 

image.thumb.png.f59088ac48777af0fbb4f497b3906030.png

 

This particular VM is running 1.03b happily with ESXi 7.  I would be very surprised if ESXi 7 VM's did not have BIOS option, but if that happens, create the VM with an earlier ESXi version.

 

It is not the boot option. I mean if you use BIOS mode, there is no support on booting from USB drive. At least, I didn't have it work that way. Maybe you can find some other solution to make the VM boot from USB using BIOS mode.

Link to post
Share on other sites
  • 1 month later...
# VMware virtual USB mass storage
set vid=0x0e0f
set pid=0x0005

May I ask how to get this parameter?

i think the vid and pid parameters have been changed on ESXi 7.0 so i can't use the loder file you created.

 

Here is the content of the file .vmx

.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "14"
nvram = "XPEnology918.nvram"
pciBridge0.present = "TRUE"
svga.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "TRUE"
hpet0.present = "TRUE"
floppy0.present = "FALSE"
RemoteDisplay.maxConnections = "-1"
numvcpus = "4"
memSize = "4096"
bios.bootRetry.delay = "10"
firmware = "efi"
powerType.powerOff = "default"
powerType.suspend = "soft"
powerType.reset = "default"
tools.upgrade.policy = "manual"
sched.cpu.units = "mhz"
sched.cpu.affinity = "all"
sched.cpu.latencySensitivity = "normal"
vm.createDate = "1628805320629596"
sata0.present = "TRUE"
usb.present = "TRUE"
ehci.present = "TRUE"
usb_xhci.present = "TRUE"
svga.autodetect = "TRUE"
serial0.fileType = "file"
serial0.fileName = "serial.out"
serial0.yieldOnMsrRead = "TRUE"
serial0.present = "TRUE"
sata0:1.fileName = "XPEnology918.vmdk"
sched.sata0:1.shares = "normal"
sched.sata0:1.throughputCap = "off"
sata0:1.present = "TRUE"
ethernet0.virtualDev = "e1000e"
ethernet0.networkName = "VM Network"
ethernet0.addressType = "static"
ethernet0.address = "00:11:32:12:34:56"
ethernet0.wakeOnPcktRcv = "FALSE"
ethernet0.present = "TRUE"
displayName = "XPEnology918"
guestOS = "other4xlinux-64"
bios.forceSetupOnce = "TRUE"
toolScripts.afterPowerOn = "TRUE"
toolScripts.afterResume = "TRUE"
toolScripts.beforeSuspend = "TRUE"
toolScripts.beforePowerOff = "TRUE"
tools.syncTime = "FALSE"
uuid.bios = "56 4d 3b af 2c 82 0a 64-b0 ed 5e 70 14 f4 99 39"
uuid.location = "56 4d 3b af 2c 82 0a 64-b0 ed 5e 70 14 f4 99 39"
vc.uuid = "52 21 7d 5a 86 30 d7 c8-a0 ec 4b ee 8b b9 93 dc"
sched.cpu.min = "0"
sched.cpu.shares = "normal"
sched.mem.min = "0"
sched.mem.minSize = "0"
sched.mem.shares = "normal"
usb_xhci:3.present = "TRUE"
usb_xhci:3.deviceType = "disk"
usb_xhci:3.fileName = "synoboot.vmdk"
usb_xhci:3.readonly = "FALSE"

 

Edited by Pantigon
Link to post
Share on other sites
7 hours ago, Pantigon said:

@Marawan

I used his mod loader file HERE. The problem is that I boot ok but XPEN doesn't seem to get the IP when installing on ESXi 7, but ESXi 6.7 is ok.

Are you running it on ESXi 7?

 

I use 3615x VM in ESXi 7.02u with passthrough of a NIC and a SAS HBA .

I passed both Intel and Realtek types of NICs and both were recognized properly.

 

Have you set your card mac correctly in grub.cfg? set mac1=?

May be your problem is that you are reusing an ESXi 6.7 VM in 7 ??!, Try create a new fresh one in 7 and see what happens.

Try experiment with a different model with a different loader and see what happens.

  • Thanks 1
Link to post
Share on other sites
9 hours ago, Pantigon said:

@Marawan

I used his mod loader file HERE. The problem is that I boot ok but XPEN doesn't seem to get the IP when installing on ESXi 7, but ESXi 6.7 is ok.

Are you running it on ESXi 7?

Which network adapter are you using? You should use e1000e, not vmxnet3, because DSM doesn't have driver for vmxnet3.

  • Thanks 1
Link to post
Share on other sites
4 hours ago, snailium said:

Which network adapter are you using? You should use e1000e, not vmxnet3, because DSM doesn't have driver for vmxnet3.

As I said I don't use virtual network adapter e1000et/vmxnet3,, I passthrough the physical PCIe network adapter to Synology VM.

That means the VM sees the adapter as is not as e1000et/vmxnet3.

 

You need to enable passthrough in ESXi, go to Host->Manage->Hardware->PCI Devices

After you enable the passthrough for the network adapter you need to add it your VM.

From VM settings choose Add other device->PCI device

and select the network adapter you just enabled passthrough for it.

You will not be able to share a passthrough device with other VMs, Only one VM can use it at a time.

Link to post
Share on other sites
10 hours ago, snailium said:

Which network adapter are you using? You should use e1000e, not vmxnet3, because DSM doesn't have driver for vmxnet3.

I using e1000e. But it doesn't seem to get the IP. But with your instructions, I run on ESXi 6.7 and it's ok

Link to post
Share on other sites
12 hours ago, Marawan said:

 

I use 3615x VM in ESXi 7.02u with passthrough of a NIC and a SAS HBA .

I passed both Intel and Realtek types of NICs and both were recognized properly.

 

Have you set your card mac correctly in grub.cfg? set mac1=?

May be your problem is that you are reusing an ESXi 6.7 VM in 7 ??!, Try create a new fresh one in 7 and see what happens.

Try experiment with a different model with a different loader and see what happens.

you are booting DS3617 as virtual USB synoboot.vmdk or sata synoboot.vmdk? Of course I fixed the card's MAC. I booting DS3167 as sata synoboot.vmdk is ok but virtual USB synoboot.vmdk isn't IP received. And of course my ESXi 6.7 and ESXi 7 on 2 different physical servers.

Edited by Pantigon
Link to post
Share on other sites
On 8/15/2021 at 8:40 AM, Pantigon said:

I using e1000e. But it doesn't seem to get the IP. But with your instructions, I run on ESXi 6.7 and it's ok

 

I would recommend adding a serial port in the VM options, dump the output to a log file. After booting your VM, keep checking the serial log file for hints (you can access the log file from datastore).

 

It may tell you which driver has issue.

 

P.S. double check "secure boot" option, it needs to remain disabled.

  • Like 1
  • Thanks 1
Link to post
Share on other sites
2 hours ago, snailium said:

 

I would recommend adding a serial port in the VM options, dump the output to a log file. After booting your VM, keep checking the serial log file for hints (you can access the log file from datastore).

 

It may tell you which driver has issue.

 

P.S. double check "secure boot" option, it needs to remain disabled.

Here is the configuration on ESXi 7.0 U2

yM3QB2e.png

wTDAEzZ.png

 

Boot ok, but not getting IP....

xBdLikA.png

 

P/s: Same settings, I run it on ESXi 6.7 server, it works 😀

Link to post
Share on other sites
On 8/17/2021 at 4:32 AM, Pantigon said:

Here is the configuration on ESXi 7.0 U2

yM3QB2e.png

wTDAEzZ.png

 

Boot ok, but not getting IP....

xBdLikA.png

 

P/s: Same settings, I run it on ESXi 6.7 server, it works 😀

 

Can you share the "serial.out" file under your VM directory?

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.