Read up a little on hypervisors. ESXi, ProxMox VE, KVM, XenServer, etc are all hypervisors, which run as the initial boot environment (before an operating system). Hypervisors manage virtual machines, each of which are completely seperate and independent of each other. DSM is then run as a virtual machine workload within the hypervisor environment. We can support NVMe services because they are managed completely outside of the DSM VM.
DSM has the ability to act as a hypervisor (VMM), but this is not useful to support NVMe drives (since DSM doesn't support them in the first place).
Docker is not a hypervisor, it's an aliased environment within Linux. Docker apps share the same Linux working environment. You might think of it working in the same way as a hypervisor, but it is quite different.
It is possible to import your baremetal XPe DSM system into an ESXi VM, but that is a technically demanding task and you should be very familiar with XPe and running it under ESXi before attempting it. So yes, you would probably need to plan to "start over" and import your data once everything was set up and tested under ESXi.