DSM 6.2.4 / 7 loader - major kernel changes?


Recommended Posts

After seeing a sea of report for 6.2.4-25554 and 6.2.4-25556 I fired up my test rig and got the same result as for for v7 release:

 

va not found
Failed to process header

 

This seems to happen way before the kernel is actually loaded into memory. I compared before & after boot images and not surprisingly the kernel did change. From my limited knowledge of Linux boot process I started poking around and even binwalk doesn't recognize the new image:

 

Scan Time:     2021-05-30 13:01:52
Target File:   /mnt/_synoboot-old/zImage
MD5 Checksum:  e3d859f75819dda05e065c5da23187f2
Signatures:    386

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
15966         0x3E5E          LZMA compressed data, properties: 0x5D, dictionary size: 67108864 bytes, uncompressed size: -1 bytes
3045174       0x2E7736        LZMA compressed data, properties: 0xCF, dictionary size: 0 bytes, uncompressed size: 2156462080 bytes


Scan Time:     2021-05-30 13:01:52
Target File:   /mnt/_synoboot-new/zImage
MD5 Checksum:  d6e6929c303ff7e60834ea4b60564fd7
Signatures:    386

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
15966         0x3E5E          LZMA compressed data, properties: 0x5D, dictionary size: 67108864 bytes, uncompressed size: -1 bytes

 

 

However extract-vmlinux is able to unpack the kernel no problem. Both images seem to contain the same kernel version thou:

# binwalk vmlinux-* | grep 'Linux k'
8388800       0x8000C0        Linux kernel version 3.10.1
8388800       0x8000C0        Linux kernel version 3.10.1

 

As a dirty test I replaced the zImage & extra.lzma and was able to boot. However most of the things aren't working ;)

image.thumb.png.c98140ad03b889a7be8591d392f037e1.png

 

Looking into console it looks like most of the core processes are crashing (did they move a new toolchain internally?):

[   23.169638] init: syno-check-disk-compatibility main process (13391) terminated with status 255
[  100.656475] init: hotplugd main process (9595) killed by TERM signal
[  100.731591] init: smbd main process (11593) killed by TERM signal
[  101.515728] init: scsi_plugin_server main process (11456) killed by TERM signal
[  102.500297] init: synonetd main process (5542) killed by TERM signal
[  102.984653] init: synostoraged main process (9228) terminated with status 15
[  108.353693] init: Disconnected from D-Bus system bus

 

What I tried next:

  • Initially I wanted to recompress the new kernel into a bare standard zImage but it looks like it's easier said than done.
  • Since GRUB should be able to boot a raw image too. I tried that by passing the vmlinux image from the second partition/(hd0,gpt2) directly but it complains about a bad magic number for it (as normally it will boot bzImage from the first partition/(hd0,gpt1) which somehow boots zImage from the second one)
  • Next I tried to put uncompressed kernel in (hd0,gpt2)/zImage. This goes further but ends up in "Wrong file type bzimage64, file matches type elf-x86_64"

 

 

Does anybody have any ideas how to progress from here? ;)  Maybe even @jun himself? :D 

  • Like 5
Link to post
Share on other sites

<va not found> it is the kexec's error.

it is located in the ramdisk in the primary kernel. it loads the secondary (synology) kernel and patches it.

although the patches are "fuzzy" some of them do not applies due to a _minor_ changes in the kernel.

I know what the reason is, but I can't fix it yet because of the complex kexec packaging inside the ramdisk inside the primary kernel.

it won't be easy to do without source code.

Link to post
Share on other sites
6 hours ago, Vortex said:

Those scripts create a bootloader and extras.

What we are talking about is much much deeper.

Unfortunately, Jun didn't share the source :-(

Yeah, I'm aware it is much more complex but this gives a glimpse of what the process was. I saw there were some (successful?) userland attempts too.

I'm trying to piece-out what is his module doing/what it has to do without jumping to IDA. In other words what "security" did Syno put into place (or rather what they check because if they wanted to block it they would put a proper security layer). So far I see two big areas:

- Synobios emulation

- Making sure our USB drive is seen as /dev/synoboot

 

 

 

4 hours ago, Aigor said:

Almost off topic, why regular synology boot doesn't work into regular hardware without patching? 

 

I tried this but I got nowhere as the kernel stops after earlycon is switched to a regular one:

grub> boot
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.10.105 (root@build1) (gcc version 4.9.3 20150311 (prerelease) (crosstool-NG 1.20.0) ) #25556 SMP Thu Mar 18 12:51:03 CST 2021
[    0.000000] Command line: BOOT_IMAGE=/zImage earlycon=uart8250,io,0x3f8,115200n8 earlyprintk loglevel=15 console=ttyS0,115200n8 syno_hdd_powerup_seq=0 HddHotplug=0 syno_hw_version=DS3615xs vender_format_version=2 withefi elevator=elevator quiet syno_port_thaw=1 root=/dev/md0 sn=1430LWN023446 mac1=7E458095B027 netif_num=1 vid=0x46f4 pid=0x0001
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007ffdcfff] usable
[    0.000000] BIOS-e820: [mem 0x000000007ffdd000-0x000000007fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] extended physical RAM map:
[    0.000000] reserve setup_data: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] reserve setup_data: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] reserve setup_data: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] reserve setup_data: [mem 0x0000000000100000-0x0000000001d4cd87] usable
[    0.000000] reserve setup_data: [mem 0x0000000001d4cd88-0x0000000001d4cd97] usable
[    0.000000] reserve setup_data: [mem 0x0000000001d4cd98-0x000000007ffdcfff] usable
[    0.000000] reserve setup_data: [mem 0x000000007ffdd000-0x000000007fffffff] reserved
[    0.000000] reserve setup_data: [mem 0x00000000b0000000-0x00000000bfffffff] reserved
[    0.000000] reserve setup_data: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved
[    0.000000] reserve setup_data: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] reserve setup_data: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] Early serial console at I/O port 0x3f8 (options '115200n')
[    0.000000] bootconsole [uart0] enabled
--R--R-4
-9
--R

 

I can't find any docummented debugging attempts 🤔 

  • Like 1
Link to post
Share on other sites
4 minutes ago, kiler129 said:

 

- Making sure our USB drive is seen as /dev/synoboot

 

This is a simple script to manually create the devs after the kernel boots, not very hard to find.

Link to post
Share on other sites
32 minutes ago, kiler129 said:

I'm trying to piece-out what is his module doing

 

I believe I saw a post by @jun on this forum from a while ago about the current loader that said part of the check is DSM checks for a certain combination of components to verify if it is booting on legit hardware so he found a way to modify each piece of the code that does so to trick Xpenology into thinking it is seeing legit syno components/hardware.

 

Also, since you are @Vortex seem to know what you're talking about.... I'll bring up a point I made previously on another thread. Do you think there is anything that can be gathered from Synology's Virtual Machine Manager in terms of emulating syno hardware to let Xpenology or even Virtual DSM boot outside of actual syno hardware? My thought is, VMM obviously has some code somewhere in it that let's Virtual DSM boot, and VDSM was obviously designed to not be run directly on actual syno hardware. What I'm wondering is if the easier method of making Xpenology work with DSM 7 is to somehow use Virtual DSM as the base DSM and emulating whatever method their VMM uses to boot Virtual DSM on ESXi.

 

I would think VMM has built in some sort of "bootloader" or "BIOS" for VDSM it emulates. Not sure the underlying tech behind VMM but considering it also boot windows, linux, etc. I am assuming it is built in a way that the choice of OS in VMM determines a type of module used to boot that OS. Extract or emulate that module for VDSM and maybe we have something we can use to boot VDSM on a different hypervisor.

Link to post
Share on other sites
1 hour ago, flyride said:

This is a simple script to manually create the devs after the kernel boots, not very hard to find.

Ok, that will sound stupid but is that script built into the DSM itself or something the loader does? Normally when I see kernels there's a separate kernel image + a ramdisk to fiddle with. The bzImage supplied with Jun's loader seems to only have the kernel image (but Jun wrote that there IS a ramdisk)... the only separate initrd I see is the Synology's one.

 

Is there any place with dev resources? I have some experience with kernel hacking (e.g. https://github.com/kiler129/relax-intel-rmrr) and low-level patching for hackintosh but my knowledge of Xpenology is limited. 

 

50 minutes ago, ilovepancakes said:

I believe I saw a post by @jun on this forum from a while ago about the current loader that said part of the check is DSM checks for a certain combination of components to verify if it is booting on legit hardware so he found a way to modify each piece of the code that does so to trick Xpenology into thinking it is seeing legit syno components/hardware.

I think you may be thinking about these two:

 

 

1 hour ago, ilovepancakes said:

Also, since you are @Vortex seem to know what you're talking about.... I'll bring up a point I made previously on another thread. Do you think there is anything that can be gathered from Synology's Virtual Machine Manager in terms of emulating syno hardware to let Xpenology or even Virtual DSM boot outside of actual syno hardware? My thought is, VMM obviously has some code somewhere in it that let's Virtual DSM boot, and VDSM was obviously designed to not be run directly on actual syno hardware. What I'm wondering is if the easier method of making Xpenology work with DSM 7 is to somehow use Virtual DSM as the base DSM and emulating whatever method their VMM uses to boot Virtual DSM on ESXi.

Hm, that may be a good approach actually. Without checking, based on other implementations like MetaRouter on MikroTik, such things however usually just proxy communication between stuff like synobios and VM shimming parts of the communication.

The VMM itself is nothing special - it's just a wrapper around an old version of KVM/QEMU with some things added here and there.

  • Like 1
Link to post
Share on other sites
22 minutes ago, kiler129 said:

I think you may be thinking about these two:

 

Ahh okay, yeah it was the first one. So seems to do with DSM 6.1, although I'm assuming unless major changes were made to how they check, it still checks in some similar way by verifying components, devices, some parts of the code, etc. I guess the question is..... where is the check function implemented and when does it run?

 

24 minutes ago, kiler129 said:

The VMM itself is nothing special - it's just a wrapper around an old version of KVM/QEMU with some things added here and there.

 

I don't have a deep understanding of virtualization back ends but wouldn't a VMM based on KVM/QEMU have some sort of "loader" that boots the VDSM images? And that loader would have to emulate DSM hardware (assuming VDSM uses same checking mechanisms as regular bare metal DSM)? Here's why I am thinking that.... right now, on DSM 6.2.3 on ESXi (loader 918+ 1.04b) if I install VMM and boot a VDSM 7 pat file, DSM 7 works and I can use it through VMM, even though my underlying hardware is not true syno hardware.

 

Now, you mention VMM may just be proxying info between VDSM and the host DSM bios/kernel.... although if that were true, why does DSM 7 work on VMM (installed on xpenology) but not DSM 7 does NOT work when installed via xpenology? Again, just thinking out loud and curious for your input (I'm far from an expert), but common sense would make me think VMM really is emulating actual syno hardware directly to VDSM 7 via some sort of "loader".

Link to post
Share on other sites
36 minutes ago, ilovepancakes said:

 

Ahh okay, yeah it was the first one. So seems to do with DSM 6.1, although I'm assuming unless major changes were made to how they check, it still checks in some similar way by verifying components, devices, some parts of the code, etc. I guess the question is..... where is the check function implemented and when does it run?

IMHO it is not relevant where they check things but how they do that. Patching checks can get you only so far, it's a better approach to emulate responses. I don't think Syno nor Apple are interested in deliberately trying to block unauthorized installs and only REALLY protect access to their services ($$$, security). With a slight OT I can say Syno got six-figures in hardware because I was able to poke around with Xpenology privately to later recommend it for business deployments. I'm sure they're fully aware of that - nobody in a right mind would use Xpenology commercially (legal issues aside).

 

 

40 minutes ago, ilovepancakes said:

I don't have a deep understanding of virtualization back ends but wouldn't a VMM based on KVM/QEMU have some sort of "loader" that boots the VDSM images? And that loader would have to emulate DSM hardware (assuming VDSM uses same checking mechanisms as regular bare metal DSM)? Here's why I am thinking that.... right now, on DSM 6.2.3 on ESXi (loader 918+ 1.04b) if I install VMM and boot a VDSM 7 pat file, DSM 7 works and I can use it through VMM, even though my underlying hardware is not true syno hardware.

 

Now, you mention VMM may just be proxying info between VDSM and the host DSM bios/kernel.... although if that were true, why does DSM 7 work on VMM (installed on xpenology) but not DSM 7 does NOT work when installed via xpenology? Again, just thinking out loud and curious for your input (I'm far from an expert), but common sense would make me think VMM really is emulating actual syno hardware directly to VDSM 7 via some sort of "loader".

It is probably just a modified TianoCore. Writing such things from scratch doesn't make sense. The reason WHY vDSM works on Xpenology is because Jun's module seems to emulate synobios and create fake PCI devs - again, instead of monkey-patching checks (which got other communities into big troubles) it just shims them (so that emulated behaviors are close enough to fool the OS). Any commercial OS I've ever seen runs great on VMs as developers aren't going to work on real hardware all the time.

 

Link to post
Share on other sites
Posted (edited)
1 hour ago, kiler129 said:
3 hours ago, flyride said:

This is a simple script to manually create the devs after the kernel boots, not very hard to find.

Ok, that will sound stupid but is that script built into the DSM itself or something the loader does? Normally when I see kernels there's a separate kernel image + a ramdisk to fiddle with. The bzImage supplied with Jun's loader seems to only have the kernel image (but Jun wrote that there IS a ramdisk)... the only separate initrd I see is the Synology's one.

 

 

In jun.patch, the following code creates synoboot from the USB key.

Incidentally, the patcher fails this chunk initializing a 6.2.3 boot - so this never runs, and that is where the post-install FixSynoboot script comes in.

 

+FixSynoboot()
+{
+    tail -n+3 /proc/partitions | while read major minor sz name
+    do
+        if echo $name | grep -q "^sd[[:alpha:]]*$";then
+            basename=$name
+            minor0=$minor
+            synoboot1=""
+            synoboot2=""
+            continue
+        fi
+        if [ $name = "${basename}1" -a $sz -le 512000 ]; then
+            synoboot1="$name"
+            minor1=$minor
+        elif [ $name = "${basename}2" -a $sz -le 512000 ]; then
+            synoboot2="$name"
+            minor2=$minor
+        else
+            continue
+        fi
+        if [ -n "$synoboot1" -a -n "$synoboot2" ]; then
+            rm "/dev/$basename"
+            rm "/dev/$synoboot1"
+            rm "/dev/$synoboot2"
+            # leave other partitions as is for now
+            mknod /dev/synoboot b $major $minor0
+            mknod /dev/synoboot1 b $major $minor1
+            mknod /dev/synoboot2 b $major $minor2
+            break
+        fi
+    done
+}

Edited by flyride
  • Thanks 1
Link to post
Share on other sites
1 hour ago, kiler129 said:

The reason WHY vDSM works on Xpenology is because Jun's module seems to emulate synobios and create fake PCI devs

 

OK, any idea why DSM v7 works then on VMM running on Xpenology but we can't boot DSM 7 directly with loaders? Is it because the issue with DSM 7 not booting directly with 1.04 loader is the changed kernel rather than potential changed "checks" for real syno hardware?

Link to post
Share on other sites
Posted (edited)
13 hours ago, flyride said:

In jun.patch, the following code creates synoboot from the USB key.

Incidentally, the patcher fails this chunk initializing a 6.2.3 boot - so this never runs, and that is where the post-install FixSynoboot script comes in.

This is not necessary if you have specified the correct vid/pid of usb flash drive inside grub.cfg

 

Here is the kexec from Jun's loader v1.04b You can tinker with it.

To invoke, just grab any x86-64 VM and run:

./kexec -d -z --args-linux ./zImage --type=bzimage64 --reuse-cmdline --initrd=./rd.gz

zImage and rd.gz are from syno fw. Test both 6.2.3 and 6.2.4/7.0 beta  (DS918+) sets.

With 6.2.3 you'll get a pretty big load log then KP (normal behavior)

but with 6.2.4/7.0 you'll get the <va not found> error and immediate return.

Now you can investigate and try to fix this kexec from this point.

kexec.zip

Edited by Vortex
  • Like 1
Link to post
Share on other sites
6 hours ago, Vortex said:

This is not necessary if you have specified the correct vid/pid of usb flash drive inside grub.cfg

 

AFAIK the vid/pid simply hides the device and defeats the boot check.  That the device is manifested as /dev/synobootx is not required for booting.  Upgrades, however, will fail as the upgrade PATs look for a modify files on the synoboot device.

Link to post
Share on other sites
19 hours ago, flyride said:

Baremetal DSM v7 doesn't work on VMM.

 

I meant, any DSM 7 won't boot with jun's loaders directly.... but, virtual DSM 7 DOES boot when run through VMM even when the VMM host is in fact a 6.2.3 xpenology install. My point being, why does DSM 7 (in the form of Virtual DSM) work on non-syno hardware through VMM, but DSM 7 won't boot by one of jun's loaders. I know it's a complicated answer and if we knew where something was going wrong already we could work on a fix, but it's interesting to me and makes me wonder what exactly VMM is emulating/doing when booting Virtual DSM images.

 

Obviously whatever it does makes Virtual DSM think it's all on legit hardware, or maybe the code of Virtual DSM itself just doesn't have the same checks that regular DSM for bare metal pat files have. Which makes me then wonder, is Virtual DSM a better base image to try and build xpenology off of (especially for running on ESXi)?

Link to post
Share on other sites
Posted (edited)
11 hours ago, Vortex said:
On 6/1/2021 at 9:06 PM, flyride said:

In jun.patch, the following code creates synoboot from the USB key.

Incidentally, the patcher fails this chunk initializing a 6.2.3 boot - so this never runs, and that is where the post-install FixSynoboot script comes in.

This is not necessary if you have specified the correct vid/pid of usb flash drive inside grub.cfg

 

 

the fixsynoboot is about the "normal" patch (jun.patch) for files like synoinfo.conf inside extra.lzma, because it fails at a (more or less) important part since 6.2.3, not really a problem to fix this as its just about readjusting the patch (with diff or even manually) to match again,  no new code needed and we already have @flyridepatch to fix it after installing

@Vortex is about the stuff that happens with jun's "bzImage" from the 1st partition, the part where the magic happens like making the usb look like the original usb dom (vid/pid f400/f400 if you check the usb after boot device with lsusb) and the dummy devices you see when using lspci (devices present but no drivers loaded)

 

i already unpacked the kexec form bzImage about a year ago but had (and have) no clue how to deal with it, initially i wanted to confirm how extra/extra2 in 918+ loader are used (but the "run" only is about extra.lzma)

./kexec -d -z --args-linux /boot/zImage --type=bzimage64 --reuse-cmdline --initrd=/boot/rd.gz --initrd=/boot/extra.lzma

even with more logs about how the kernel is patched, for me its like with "Humpty Dumpty"

https://www.poetryfoundation.org/poems/46951/humpty-dumpty-sat-on-a-wall

i'm really not that much of a linux expert and no coding skills

as i have no further use for things like this, i'm not venturing deeper into linux as i have to and from my point of view omv is still a valid alternative and at some point i will just jump ship

Edited by IG-88
  • Like 1
Link to post
Share on other sites
4 minutes ago, ilovepancakes said:

irtual DSM 7 DOES boot when run through VMM even when the VMM host is in fact a 6.2.3 xpenology install. My point being, why does DSM 7 (in the form of Virtual DSM) work on non-syno hardware through VMM, but DSM 7 won't boot by one of jun's loaders. I know it's a complicated answer and if we knew where something was going wrong already we could work on a fix, but it's interesting to me and makes me wonder what exactly VMM is emulating/doing when booting Virtual DSM images

 

Fundamentally, a VM does not know what host OS is running.  There is no relationship between DSM 6.2.3 and VDSM 7 running as a guest on VMM.

 

EXCEPT - that undoubtedly there is a BIOS string or other "hardware" checks that make sure VDSMs only run on a Synology-branded version of KVM (VMM).  If not, we would all have run VDSM a long time ago as a guest on ESXi and XPe loaders would not be necessary.  The other issue is that VDSM is crippleware compared to regular DSM.  Hardware drivers and much of the disk management/cache/array functionality is (or at least has been) disabled as it expects to get these services from the host DSM.

 

Subject of many other discussions, if Syno would competitively license a VM-compatible DSM that did all the things that baremetal DSM did, there would be a lot of paid takers here on this forum.  But I don't think they will do that because it would compete too much with their baremetal offerings.  So VDSM will be limited to a test/prototype capacity, niche app services (why someone wants to do this on DSM I don't know, but Synology has delusions of grandeur) and possibly VDS/AWS/Azure functionality, similar to what QNAP is marketing.

Link to post
Share on other sites

Ok, this is gonna be a looooooong post. I spent some time digging around and I'm back. First some responses:

 

3 hours ago, ilovepancakes said:

OK, any idea why DSM v7 works then on VMM running on Xpenology but we can't boot DSM 7 directly with loaders? Is it because the issue with DSM 7 not booting directly with 1.04 loader is the changed kernel rather than potential changed "checks" for real syno hardware?

To my knowledge the problem is the same as described in the first post - "va not found" (aka the patcher cannot execute the main DSM kernel). It doesn't seem like Synology did anything on purpose, they changed the kernel just enough for the Jun's loader to fail. Synology uses a separate "distro" for VMM.

 

 

23 hours ago, Vortex said:

Now you can investigate and try to fix this kexec from this point.

But why do we even need a kexec? VMM boots the kernel directly and just insmod couple of other modules for serial and virtio stuff.

 

 

17 hours ago, flyride said:

AFAIK the vid/pid simply hides the device and defeats the boot check.  That the device is manifested as /dev/synobootx is not required for booting.  Upgrades, however, will fail as the upgrade PATs look for a modify files on the synoboot device.

IMHO the current loader does this properly - relying on mknod is not the best idea (as described later in this post). I think the most vanilla approach will be to not use kexec and jump to another kernel but rather patch init to load additional modules after every upgrade. Comparing multiple init scripts they hardly changed through many versions.

 

 

11 hours ago, flyride said:

(...) undoubtedly there is a BIOS string or other "hardware" checks that make sure VDSMs only run on a Synology-branded version of KVM (VMM).  If not, we would all have run VDSM a long time ago as a guest on ESXi and XPe loaders would not be necessary.  The other issue is that VDSM is crippleware compared to regular DSM.  Hardware drivers and much of the disk management/cache/array functionality is (or at least has been) disabled as it expects to get these services from the host DSM.

If ain't broken don't fix it ;) This is why nobody seems to bother to run VDSM outside VMM. There're aren't really DRM-like checks from what I can see. VDSM asks the host for network config, S/N, and license validity. The other stuff is present in the kernel and VDSM distro itself but just disabled since you cannot realistically use it with VMM (as you cannot OOB do PCI-passthru and stuff like that).

 

 

----------------------------------------------------------------------------------------------------------------------------------------------------------------

 

Investigating further...

Note: this is more a brain-dump as I'm investigating the possibly of VDSM working on other HVs than VMM rather than any instruction. Feedback is welcomed.

 

Regarding the VMM approach a special PAT is used (using "VirtualDSM" platform instead e.g. DS3615xs). After unpacking there're a couple differences:

image.png.25fa22c0c5da8877c395f5db0857a5c5.png

 

Ignoring what's not included in the VDSM b/c it's not a physical system, the notable changes here are I think:

  • Presence of grub.conf
  • Presence of synology_kvmx64_virtualdsm_25426_918C5640__.bin.zip (~20MB)

The zip file contains a disk image which looks like a boot drive for the VDSM:

# fdisk -l ./synology_kvmx64_virtualdsm_25426_918C5640__.bin
Disk ./synology_kvmx64_virtualdsm_25426_918C5640__.bin: 110 MiB, 115343360 bytes, 225280 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x28abac75

Device                                             Boot Start    End Sectors  Size Id Type
./synology_kvmx64_virtualdsm_25426_918C5640__.bin1 *       63  32129   32067 15.7M 83 Linux
./synology_kvmx64_virtualdsm_25426_918C5640__.bin2      32130 224909  192780 94.1M 83 Linux

 

Both partitions contain a... familiar structure... weirdly familiar:

../part1
├── boot
│   └── grub
│       ├── e2fs_stage1_5
│       ├── grub.conf
│       ├── menu.lst -> grub.conf
│       ├── stage1
│       └── stage2
├── GRUB_VER
├── rd.gz
└── zImage


../part2
├── grub_cksum.syno
├── rd.gz
├── vender
└── zImage

 

I don't want to say Synology uses a technique similar to Jun's loader to make VDSM work but I will leave the rest as an exercise for the reader ;) Coming back to the interesting parts from synology_kvmx64_virtualdsm_25426_918C5640__.bin:

  • both part1/zImage & part2/zImage (kernel) and part1/rd.gz & part2/rd.gz (initrd) are identical
  • both part1/boot/grub/grub.conf and PAT/grub.conf are identical
  • zImage contains a pretty new kernel: Linux kernel x86 boot executable bzImage, version 4.4.59+ (root@build1) #25426 SMP PREEMPT Tue May 12 04:41:18 CST 2020, RO-rootFS, swap_dev 0x3, Normal VGA
  • stage1/1_5 files are bare standard GRUB files
  • stage2 is a modified GRUB with support for vender & files signing (side note: GPL again... no sources)
    image.thumb.png.a0a185c156eabf4216b6b8b932312f97.png

----------------------------------------------------------------------------------------------------------------------------------------------------------------

 

VMM analysis

After static analysis I went into the active VM analysis. As the VMM uses KVM backend I first started a normal Linux machine and then started VDSM:

  1. The VM uses VirtIO drivers for everything so the VDSM has drivers specifically for libvirt like we have for Jun's loader
  2. The runtime of the machine is nothing special:
    1. Config like for any Linux VM
    2. Disks mounted via iSCSI LUN
    3. PAT is not mounted to the VM at any moment - host DSM mounts everything using iSCSI LUN
  3. The kernel in VDSM is compiled for both Intel and AMD
  4. Package contains a... synobios emulator?: 
    [    2.746361] kvmx64_synobios: module license 'Synology Inc.' taints kernel.
    [    2.747399] Disabling lock debugging due to kernel taint
    [    2.748581] 2021-6-2 1:9:54 UTC
    [    2.749193] synobios: load, major number 201
    [    2.749635] Brand: Synology
    [    2.751196] Model: VirtualDSM
    [    2.751497] set group disks wakeup number to 4, spinup time deno 7
    [    2.752661] synobios cpu_arch proc entry initialized
    [    2.753745] synobios crypto_hw proc entry initialized
    [    2.754295] synobios syno_platform proc entry initialized
    [    2.754831] synobios ioctl TCGETS /dev/ttyS1 failed
    [    2.755347] synobios unable to set termios of /dev/ttyS1
    [    2.761050] synobios: unload


    The bios kernel module (kvmx64_synobios) is much smaller than DS3615xs one:

    # file *.ko
    synobios-ds3615xs.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=b51232d7ad1bbb93bf868cafd962bb747cbf2ab8, not stripped
    synobios-vdsm.ko:     ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=b7baabd787acec8716fac8d85901cf984555b974, not stripped
    
    # ls -sh *.ko
    120K synobios-ds3615xs.ko
    80K synobios-vdsm.ko
  5. When VM is exported as OVA three different disks are exported:

    - 10G thin provisioned disk with just the contents of synology_kvmx64_virtualdsm_25426_918C5640__.bin
    - 10G thin provisioned disk with just the contents the OS installed (standard ~2GB two partition arrangement OS+SWAP)
    - 10G thin provisioned disk w/o partition (empty disk assigned to the VM to be used for data storage)

 

----------------------------------------------------------------------------------------------------------------------------------------------------------------

 

Reusing VMM image in a different HV?

Well, that was the next logical step. I used something close to VMM - Proxmox. Suspecting that iSCSI part is not important I simply attached Synology bootloader (synology_kvmx64_virtualdsm_25426_918C5640__.bin) via loop driver using SCSI-VirtIO driver & SATA (as the SCSI itself will not boot using SeaBIOS).

The VGA output successfully shows kernel decompression:

image.thumb.png.f80f58c97b57367de79d5dcf156c8d77.png

 

Since I was booting without the pre-prepared disk from VM I've got to the crashed state. The standard root/daily password works like a charm. Since that build most likely uses vid/pid check for specific iSCSI device from VMM (I didn't bother emulating that at this time) root is normally present in /dev/synoboot. This gave me the obvious "Exit on error [3] init not health...." while booting on Proxmox. I took a peek at the original Jun's patch for the init scripts and modified it a bit:

  • It fails with new version -> I moved the function to a more robust place
  • It originally calls FixSynoboot on Exit() when the health check fails -> while it probably works it may be fragile as some steps of the boot process may be skipped
  • I added more info to make debugging easier
  • However while this patch may be a viable option it will break other things (sdb will have to be renamed to sda etc) - I moved further as the kernel "sources" (or what's left from them) are available
  • I ended up not using FixSynoboot to keep stuff vanilla

 

While digging I found that the ramdisk has some quirks for running under a HV:

  • There are three different HV-related modes: kvmx64 (VMM), nextkvmx64, and kvmcloud (?!). The second one uses /dev/synoboot3 (normally there are only 2) as a root device. I didn't dig deeper into what nextkvmx64 actually is ¯\_(ツ)_/¯
  • The support for VirtIO is intentional and is not custom:
     319 if [ "$UniqueRD" = "kvmx64" -o \
     320                 "$UniqueRD" = "nextkvmx64" -o \
     321                 "$UniqueRD" = "kvmcloud" ]; then
     322         insmod /lib/modules/virtio.ko
     323         insmod /lib/modules/virtio_ring.ko
     324         insmod /lib/modules/virtio_pci.ko
     325         insmod /lib/modules/virtio_blk.ko
     326         insmod /lib/modules/virtio_net.ko
     327         insmod /lib/modules/virtio_scsi.ko
     328         insmod /lib/modules/virtio_console.ko
     329 fi

 

 

Conclusion - running under non-VMM should be possible. Additionally things like disabled RAID and such aren't a problem. They can be enabled back (kernel has all the modules needed).

 

----------------------------------------------------------------------------------------------------------------------------------------------------------------

 

Running VDSM on non-VMM KVM

 

Next I started digging to run VDSM on a standard KVM (and not the Syno VMM flavor) in Proxmox: 

  • Normally when VDSM is run under VMM it hand-picks PCI slots which is... unusual (normally you let your HV do that):
    # UUIDs are stripped for readability & replaced w/*UUID#*
    
     -device vhost-scsi-pci,virtqueue_size=256,cmd_per_lun=250,wwpn=naa.*UUID1*,addr=0xa,id=vdisk_*UUID1*,set_driverok=off,num_queues=1,max_sectors=16384,boot_tpgt=1,bootindex=1
     -device vhost-scsi-pci,virtqueue_size=256,cmd_per_lun=250,wwpn=naa.*UUID2*,addr=0xb,id=vdisk_*UUID2*,set_driverok=off,num_queues=1,max_sectors=16384,boot_tpgt=1,bootindex=2
     -device vhost-scsi-pci,virtqueue_size=256,cmd_per_lun=250,wwpn=naa.*UUID3*,addr=0xc,id=vdisk_*UUID3*,set_driverok=off,num_queues=1,max_sectors=16384,boot_tpgt=1,bootindex=3
    

     

  • Kernel has special provisioning for running VDSM. Normally DSM boot device/loader is a USB key or SATA-DOM. However the KVM kernel contains additional checks for exactly these special PCI slots:
    # synoconfigs/kvmx64
    CONFIG_SYNO_KVMX64_PCI_SLOT_SYSTEM=11
    CONFIG_SYNO_KVMX64_PCI_SLOT_BOOT=10
    
    # drivers/scsi/sd.c
    		while (virtdev) {
    			if (virtdev->driver && virtdev->driver->name && !strcmp(virtdev->driver->name, "virtio-pci")) {
    				pcidev = to_pci_dev(virtdev);
    				break;
    			}
    			virtdev = virtdev->parent;
    		}
    		if (pcidev && PCI_SLOT(pcidev->devfn) == CONFIG_SYNO_KVMX64_PCI_SLOT_BOOT) {
    			return SYNO_DISK_SYNOBOOT;
    		}
            
            
    # drivers/scsi/virtio_scsi.c
    # contains a hack to make sure system partition is always first
    	if (pcidev && PCI_SLOT(pcidev->devfn) >= CONFIG_SYNO_KVMX64_PCI_SLOT_SYSTEM) {
    		index = PCI_SLOT(pcidev->devfn) - CONFIG_SYNO_KVMX64_PCI_SLOT_SYSTEM;
    	}
    	return index;

    -> 0xa = 10 = CONFIG_SYNO_KVMX64_PCI_SLOT_BOOT = /dev/synoboot
    -> 0xb= 11 = CONFIG_SYNO_KVMX64_PCI_SLOT_SYSTEM = /dev/sda
     

  • Using a standard Linux KVM on Proxmox the drives can be easily emulated and it works with no modifications to kernel nor ramdisk:
    [    3.411295] scsi host0: Virtio SCSI HBA
    [    3.419226] scsi 0:0:0:0: Direct-Access     QEMU     QEMU HARDDISK            2.5+ PQ: 0 ANSI: 5
    [    3.421672] sd 0:0:0:0: [synoboot] 225280 512-byte logical blocks: (115 MB/110 MiB)
    [    3.424059] sd 0:0:0:0: [synoboot] Write Protect is off
    [    3.426034] sd 0:0:0:0: [synoboot] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
    [    3.434378]  synoboot: synoboot1 synoboot2
    [    3.442227] sd 0:0:0:0: [synoboot] Attached SCSI disk
    [    3.532075] usb 1-1: new full-speed USB device number 2 using uhci_hcd
    [    5.437846] scsi host1: Virtio SCSI HBA
    [    5.446323] scsi 1:0:0:0: Direct-Access     QEMU     QEMU HARDDISK            2.5+ PQ: 0 ANSI: 5
    [    5.455375] sd 1:0:0:0: [sda] 16777216 512-byte logical blocks: (8.59 GB/8.00 GiB)


    It literally needs just:
     

    -device virtio-scsi-pci,id=hw-synoboot,bus=pci.0,addr=0xa -drive file=<file>,if=none,id=drive-synoboot,format=raw,cache=none,aio=native,detect-zeroes=on -device scsi-hd,bus=hw-synoboot.0,channel=0,scsi-id=0,lun=0,drive=drive-synoboot,id=synoboot0,bootindex=1
    
    -device virtio-scsi-pci,id=hw-synosys,bus=pci.0,addr=0xb -drive file=<file>,if=none,id=drive-synosys,format=raw,cache=none,aio=native,detect-zeroes=on -device scsi-hd,bus=hw-synosys.0,channel=0,scsi-id=0,lun=0,drive=drive-synosys,id=synosys0,bootindex=2
      


     

  • After attempted install on non-VMM the (expected) lack of communication between guest & host stops the process:
    install.cgi: guest_command.c:86 Failed to retrieve message from host [timeout]
    install.cgi: vdsm_get_update_deadline.c:41 Failed to get vdsm update deadline.
    install.cgi: The vdsm is newer than license expire time.
    install.cgi: ninstaller.c:2730(ErrFHOSTDoUpgrade) err=[-1]
    install.cgi: ninstaller.c:2746(ErrFHOSTDoUpgrade) retv=[-65]
    install.cgi: install.c:988 Upgrade for Installation by the download patch fail.
    install.cgi: install.c:1290 Upgrade by the uploaded patch fail.
    install.cgi: ninstaller.c:253 umount partition /tmpData
    synoagentregisterd: guest_command.c:86 Failed to retrieve message from host [timeout]
    synoagentregisterd: vdsm_serial.c:84 Failed to get vdsm sn
    synoagentregisterd: vdsm_serial.c:128 Failed to get vdsm sn
    <last 3 messages repeated forever>

     

  • The guest<>host communication was previously handled by /usr/syno/sbin/synoagentregisterd. Now that binary is symlinked to /usr/syno/bin/synosearchagent. The library responsible for handling S/N transfer between guest and host is most likely /usr/lib/libsynovdsmcore.so (it's the only file with "vdsm_serial" string present).
  • There's overall very little regarding VDSM in the image (see hda1.tgz in PAT):
    ./usr/share/init/vdsm_dummy_tty.conf
    ./usr/lib/libsynovdsmcore.so
    ./usr/lib/libsynovdsm.so
    ./usr/syno/synovdsm
    ./usr/syno/synovdsm/scripts
    ./usr/syno/synovdsm/scripts/initializeVdsm.sh
    ./usr/syno/synovdsm/conf
    ./usr/syno/synovdsm/conf/vdsm_service_soft_limit.conf
    ./usr/syno/synovdsm/conf/cdsm_profile.conf
    ./usr/syno/synovdsm/conf/cdsm_service_soft_limit.conf
    ./usr/syno/synovdsm/conf/vdsm_profile.conf
    ./usr/syno/synovdsm/conf/vdsm_service_strict_limit.conf
    ./usr/syno/synovdsm/conf/cdsm_service_strict_limit.conf
    ./usr/syno/bin/synovdsmtool
    ./etc.defaults/apparmor.d/abstractions/libsynovdsmcore

    There's also /usr/lib/libsynovmcommguest.so.6.2 for generic host<>guest communication using /usr/syno/bin/synoguestcommd binary. The serial communication is done via /dev/vport1p1 (standard virtio port, see drivers/char/virtio_console.c in kernel). 

  • While I poked around the communication (it's binary, so to get anything useful decompilation of the comm library is needed) I would rather not go around license checks - this is a tricky ground. Synology sells them and I'm here for a good'ol hacking puzzle, not for piracy.
  • Alternatively (which I didn't explore yet) we can try running VDSM kernel with normal DSM distribution. This should work actually ;) While it may sound crazy at first, this is essentially how docker runs the world.
     

----------------------------------------------------------------------------------------------------------------------------------------------------------------

 

Jun's patch/loader reconstruction?

  • It doesn't look like a sophisticated hack or monkey-patching VID+PID in memory - it is a proper KISS approach and I love it.
  • I think recreating it is possible as the most important part of it isn't hacky. I didn't properly look into the loader itself yet... yet ;) 
  • His patch loads a couple of modules for console (so we can see serial output after earlycon), usb storage (so more USB controllers can be used), and some helpers
  • The most important part is getting the synoboot really - all other functions like "emulation" of the hw_version/disk mapping/serial#/MACs is already present in the Synology kernel (see kernel/syno_bootargs.c). They use that stuff themselves as they support a gazzilion devices per platform (and platform are varied to enable extra optimizations as the CPU has to be a least common denominator)
  • There are a couple DRM-ish things checking for PCI devices but since these are known they can be nulled pretty easily
  • Looking at the kernel code DSM has a couple of methods to find its bootdisk and they don't seem to be geared towards authenticating the actual hardware (i.e. not like SMBIOS in Hackintosh) but to actually find the correct disk. The bootdisk can present in multiple places (USB stick, SATA DOM, network, you name it). 
  • VID+PID of a genuine Synology boot device is hardcoded and compiled into the kernel:
    # include/linux/syno.h
    
    #define IS_SYNO_USBBOOT_ID_VENDOR(VENDOR) (0xF400 == (VENDOR))
    #define IS_SYNO_USBBOOT_ID_PRODUCT(PRODUCT) (0xF400 == (PRODUCT))

    I don't think these get patched - that would be too fragile.
     

  • These must match for a device to be considered a valid synoboot:
    		if (IS_SYNO_USBBOOT_ID_VENDOR(le16_to_cpu(usbdev->descriptor.idVendor)) &&
    			IS_SYNO_USBBOOT_ID_PRODUCT(le16_to_cpu(usbdev->descriptor.idProduct))) {
    			#....
    			if (!syno_find_synoboot()) {
    				return SYNO_DISK_SYNOBOOT;
    			}
    		}

    In this case making an early-loaded module iterate over USB devices and replace idVendor & idProduct in the structs to 0xF400 is the simplest solution. 
    There is also the SATA boot device support - it gets checked based on the vendor name and model (CONFIG_SYNO_SATA_DOM_*). Patching this is similar really to VID/PID - a module needs to patch it after devices tree is built and before SCSI (sd.c) builds /dev/ based on these info.

 

TBC...

 

  • Like 4
  • Thanks 3
Link to post
Share on other sites

If Synology use proprietary hardware naming, it comes from DSDT, why we can't use same technique of hackintosh bootloader to inject DSDT edited to match synology naming? 
I'm writing bullshit or what?  

Link to post
Share on other sites
Posted (edited)
11 hours ago, kiler129 said:

kvmx64_virtualdsm

i just had a look into the default kernel config for kvmx64 (dsm 6.2) and there is amd and intel support (needs to be that way as synology uses both cpu types now and vdsm is expected to run on all units) and "CONFIG_NR_CPUS=128", so no limit when it comes to newer cpu's with more cores

hyper-v would still not be working but full amd cpu support and no cpu core limit sounds promising for newer hardware

 

ds.kvmx64-6.2.dev.txz

\usr\local\x86_64-pc-linux-gnu\x86_64-pc-linux-gnu\sys-root\usr\lib\modules\DSM-6.2\build\.config

 

the base for scsi/sas seems to be active too, so adding mpt sas drivers from kernel 4.4.59 like mpt3sas should be no problem when its possible to run the kernel on baremetal

CONFIG_SCSI_ISCSI_ATTRS=y
CONFIG_SCSI_SAS_ATTRS=y
CONFIG_SCSI_SAS_LIBSAS=y
CONFIG_SCSI_SAS_ATA=y
CONFIG_SCSI_SAS_HOST_SMP=y
CONFIG_SCSI_LOWLEVEL=y

ahci is there too, so sata ootb

CONFIG_SATA_AHCI=y

 

i guess most nic and storage drivers as we have for 3615/17 now would be possible with the vdsm kernel (4.4.59 for 6.2 and 4.4.180 for 7.0)

 

edit:

the kernel config for kvmx64 for 7.0 seems not to support amd

CONFIG_CPU_SUP_INTEL=y
# CONFIG_CPU_SUP_AMD is not set

Edited by IG-88
Link to post
Share on other sites
2 hours ago, Aigor said:

If Synology use proprietary hardware naming, it comes from DSDT, why we can't use same technique of hackintosh bootloader to inject DSDT edited to match synology naming? 
I'm writing bullshit or what?  

It is certainly possible to do that. However DSDT is such a painful can of worms that people want to avoid that at all cost. x86 & IBM/PC architecture is a messy ratnest held with a duct tape and 30 or so years of hacks upon hacks. With macOS there are maaanyy things which need to be changed and few hook points (wrt to Linux) - here it's quite the opposite. We have almost the same hardware as the genuine one with just minor differences. In the hackintosh world it's not uncommon to run into Apple using undocumented CPU instructions which are only working properly on their SKUs.

In case you want to try poking with it a fair bit of warning: incorrect DSDT tables can physically damage the hardware. I don't know the details here but there were multiple instances of that in the hackintosh community (one of my motherboards included where the soundcard stopped working forever).

 

 

42 minutes ago, IG-88 said:

i just had a look into the default kernel config for kvmx64 (dsm 6.2) and there is amd and intel support (needs to be that way as synology uses both cpu types now and vdsm is expected to run on all units) and "CONFIG_NR_CPUS=128", so no limit when it comes to newer cpu's with more cores

hyper-v would still not be working but full amd cpu support and no cpu core limit sounds promising for newer hardware

Realistically I think running DSM on a hypervisor is a much better option overall. DSM is a great NAS platform but using it for Docker or VMs is a pretty painful experience... and when you have a good hardware you're most likely running something more than storage ;)

 

 

48 minutes ago, IG-88 said:

i guess most nic and storage drivers as we have for 3615/17 now would be possible with the vdsm kernel (4.4.59 for 6.2 and 4.4.180 for 7.0)

 

edit:

the kernel config for kvmx64 for 7.0 seems not to support amd

CONFIG_CPU_SUP_INTEL=y
# CONFIG_CPU_SUP_AMD is not set

To me kvmx64 backend looks like something which developers of DSM use, as it has many things compiled in which don't make sense for a VMM VDSM scenario. A pure speculation for lack of AMD support in 7.0 is just limiting the scope of changes - their kernel (and not only that) is highly customized, so while doing large changes for a new major releases it makes sense to first beta test it with a single platform, get it right, and then start adding back more exotic things. I cannot imagine Syno ignoring AMD in the current landscape ;)

Link to post
Share on other sites
11 hours ago, kiler129 said:

Package contains a... synobios emulator?: 

i do  remember a comment from @Trantorabout missing kernel source for synology's "own" kernel modules related to the old 5.2 custom kernel and that they had to reverse engineer around that bios module and without that stuff a 6.x custom kernel would not be possible

 

i also remember some trouble with the protection (24h) kicking in with loader 1.02a and that loader 1.02a2 fixed it and it was about some bios serial device (just from memory could not find that one)

  • Thanks 2
Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.