Usually when the install fails, you get the chance to telnet with user root to the system and check a few known things. One is the existence of /dev/synoboot device that actually proves that you have successfully shim the usb. If not which I think is the case, check with the following command the VID/PID   # cat /proc/cmdline    and that you have selected USB boot for physical/qemu or SATA for VMware during boot time. The default is set to USB