Wanted to add.. running esxi 6.5 and using 3617 latest 6.1.6.
Choosing the option ESXI in the bootloader, at most I can move forward with the DSM upload but after that reboot, it throws out the error and cannot see the hard drives. I've tried it with SCSI, SATA and even IDE.
Choosing bare metal, will work but will show the 50 MB VMDK file as an actual hard drive plus any hard drives you added. That worked using IDE, SATA, (and if I remember, SCSI). Using IDE, SATA, (maybe SCSI), does show you different hard drive numbers/configurations when you go look at your drives under storage FYI.
I read somewhere that masking the VMDK hard drive file was working with earlier version of the 6.1 DSM but something must of changed and broke that.
So there is a way to get it to work, but my OCD side says gaining that ONE sata drive space back should be possible.
I have been experimenting with maybe somehow emulating (like in a real Bare Metal installed) booting the CD-ROM/Floppy Drive with the IMG/ISO file and having the bootloader load that way. No luck so far. Any attempts of mounting the IMG file or converting the IMG to ISO, ESXI just does not boot and/or recognize the file(s) correctly (i'm assuming).