Technically there isn't such a thing as "hardware RAID" just a primitive CPU on a controller that doesn't have much else to do. Some time in the past, that card was faster than the CPUs that were then available. That just isn't true any more. And your "hardware RAID" is the one in the motherboard BIOS right? That's just software my friend, and not nearly as sophisticated as MDRAID in linux.
The very fastest enterprise flash SANs in my own data centers are NVMe connected to bus, and Xeon... totally software based.
I'm not sure why you don't want to passthrough your SATA controller, but if you must, you can try to RDM the specific drives you want XPenology to see, while leaving the controller to ESXi.