What the hell. That worked. I didn't even have to enable transcoding, as it was already on. It took me longer to install because I lost my flash drive for 3 hours, than it did to just install Plex and... Wow.
I think what happened was, my first install was on a spare AMD that I put 3615 on, then migrated to a DS920 install, but it must've not installed the drivers because it didn't work, and neither did a clean 918 install, but this one sure did. Thank you so much. I've been pulling my hair for 2 weeks on this.
Deep down, I knew the solution was going to be super simple, as it so often is. I'm just glad it's over.