How do you get hardware transcoding to work on ESXi?

[ilovepancakes]

Just installed DSM 6.2.2 Update 3 1.04 918+. Video Station has the "hardware acceleration" box checked but videos won't transcode, the wheel just spins forever, and if I do an offline transcode it says "Error". If I uncheck hardware acceleration in Video Station settings menu, transcoding works, but even on high its poor quality and all of the codecs that I get activated on 3617 1.03 loader don't show up on this DSM. mpeg4 and hevc are missing, see below. I am using a real 918+ serial and mac pair.

 

root@dsm-test3:/usr/syno/etc# cat /usr/syno/etc/codec/activation.conf

{"success":true,"activated_codec":["h264_dec","h264_enc","ac3_dec","aac_dec","aac_enc"],"token":"2db1d7306789e41b5e3d5b0a70d702db"}

I know nothing about hardware transcoding on 918, except that it supposedly can work. Do I need to do anything else to make it not return "Error" when transcoding videos? And, if there is no way for it to work, how do I get all of the codecs activated like on 3617?

[ilovepancakes]

Update: Realized that more codecs appear as you try to convert them. If hardware transcoding can't work, I'm fine with software however the transcoding quality seems way worse on 918 than my 3617 loader. I am using the same original file but when I change quality to "High" on 918 loader in Video Station, the quality is pixelated and bad, compared to changing the same video to "High" on 3617 loader where quality remains good. Any idea why 918 loader transcodes at lower quality? How can this be fixed?