RedPill - the new loader for 6.2.4 - Discussion


Recommended Posts

Update time! ;)

 

Support for v4.4 kernel
TL;DR: it now works with v4.4
After some of you took a stab at compiling the LKM for Linux v4.4 (presumably for DS918+ support) we did too. We didn't see any PRs besides a small patch from @jumkey (thanks!). However, the reality came to be much MUCH more complex, especially if we want to keep the consistent codebase for all kernels.

So, it took 400 lines over 12 commits by 3 people: https://github.com/RedPill-TTG/redpill-lkm/compare/7f85cb2ceff44d6914135bf3841eb3cb2148af11...1abcbe13df39055dc8b925f74a00a75163157d60

 

However, the story is more complex here. The kernel development velocity is huge, to add to that DSM's kernel has its own peculiarities (more on that below ;)) too. The following things took us the time:

  1. Adjusting to build with a new source tree (very easy)
  2. Running the actual kernel in an isolated VM (quite challenging).
    This posed a problem because the first round of patches for v4.4 one of us made weren't stable. Currently, the ramdisk patch works (oh irony, the one generated by the automatic tool we hacked together) but the "boot_params" causes random KPs. Additionally, some of our HVs don't have CPUs with hardware MOVBE support, or they crash/freeze the machine after snapshot restore. This made debugging challenging as you don't know if it's your CPU, hypervisor, configuration, kernel patch, LKM, or a combination of multiple ones causing the problem.
    Another challenge here is a bug in Synology's v4.4 for apollolake (possibly others too). In essence, they modified ramdisk unpacking routine and broke it in the process. The effect of it is no matter what tools we used the ramdisk wouldn't unpack making the system unbootable. We have no idea how they pack their ramdisks but we failed to recreate that. This took us finally to a flat CPIO archive (more in docs: https://github.com/RedPill-TTG/dsm-research/blob/master/quirks/ramdisk-checksum.md#recreating-ramdisks)
  3. Getting cmd sanitization working on v4 (easy): they've changed the format of /proc linklists, so we had to adjust. However, we saw that @jumkey posted a possibly better (muuuch shorter) solution to the problem. We didn't try it yet, but by all means, please make a PR! It should actually work for both v3 & v4 lines.
  4. Chasing an annoying bug with preempts: it seems like v3 silently fixes these but v4 traps them and crashes.
  5. Refactoring how the re-export of functions work to make it much easier for others to contribute to (and make our life more pleasant). This was part of the fix for set_memory_rw/ro not being officially available anymore. Now you can just use 2 macros and call it a day in 30 seconds.
  6. Making vPCI stuff working on v4 (difficulty: 11/10) : this was probably the hardest thing to debug and the easiest things to fix (it took a bloody SINGLE line!). There are many levels of hell in PCI. In around v4.1 they've added a couple more. Now all PCI stuff assumes ACPI compliance and static DSDTs on x86. In the same time they broke dynamic bus scanning for new busses which aren't PC-BIOS based. We're assuming it was accidental as what they did was a refactor only. Here comes the syno kernel too: in v3 they have dynamic debugging enabled (but not the full kdebug). This gives at least a possibility of seeing some more messages
    On v4 the debugging cycle is: stub a method with some debug prints, override the method in kernel using override_symbol.c, compile the module, reboot/restore the test VM, download the module to a test VM, load the module and see what you wanted. Repeat 500x times ;)
  7. Fixing execve() shimming wasn't as hard as vPCI stuff but quite challenging as well. Since v3.18 the call structures of ASM shims changed. In essence they use different function call convention than GCC generates (there's a good article about that:L https://my.oschina.net/macwe/blog/603583). This cannot be achieved (correct us if we're wrong) using C but requires custom ASM. Once we realized what's the problem we decided to make route the call differently by skipping the shim altogether. This is less clean and may be a tad bit slower (nothing noticeable thou).

 

What is an elevator?
For long time we were wondering why Jun's loader sets "elevator=elevator" parameter to the kernel. Normally it sets the I/O scheduling, but obviously "elevator" is not a valid value. We think we accidentally found what this is for. Linux automatically loads a custom elevator module with the name specified (with some pattern, we didn't dig into that deeply yet) before ANY other modules are loaded. And viola, that's how the LKM can load as early as possible ;)

It will be included in RedPill as well as this is a cool trick and ensures the module loads earlier than with init script.


Anybody want the Synobios source code?
It's public. It has been for years in fact. The struct posted by Vortex lead us to explore older dumps. While lurking through them we found that Syno (accidentally?) used to publish full Synobios source code. They fixed their mistake but didn't remove these old files. You can find it as a normal kernel driver in all GPL archives till 5004 (e.g. synogpl-3776-bromolow/source/linux-2.6.32/drivers/synobios). After talking with @Vortex we learned that the full headers (but not the code this time) for synobios are present in DSMv7 dev env (more details: https://github.com/RedPill-TTG/dsm-research/blob/master/quirks/mfgbios.md#appendix-official-synobios_ops-structure)

 

Synobios ioctl emulation
We found out what the previous author meant in the PMU document. It seems like there are two part of the synobios:

  •  shimming the synobios_ops (done)
  •  responding to proxy communication

It turns out that synobios not only performs its own calls but other system components (e.g. secmd) can use ioctl to ask it to send some arbitrary commands to the PMU. These commands are actually what causes these logged messages which @Vortex explained as beeps etc. We need to respond to them in software (i.e. in LKM).


Making the GPL sources more accessible
Let us preface this by saying this isn't anything we're prioritizing. However, we're putting this onto the radar. While browsing through the code we realized two things:

  • it's annoying hard to find what you're looking for and/or compare, as they publish a sea of large archives
  • they remove things

Because of these we made, currently by hand, a rundown of architectures and formats over the versions. We are also planning to replicate the whole SF archive in a form of GH repos.

 

Full details: https://github.com/RedPill-TTG/dsm-research/blob/master/quirks/gpl.md


DSM v7 sources
Synology did already publish v7 sources and.... they pulled them quickly: https://sourceforge.net/p/dsgpl/discussion/862835/thread/a519b80124/. They used to be in the "41890branch" as you can see from the project's history: https://sourceforge.net/p/dsgpl/activity/?page=0&limit=100

We weren't lucky enough to get them. Did anyone maybe was able to grab them?

 


-----------------------------------------

 

 

On 7/15/2021 at 9:25 PM, pocopico said:

Why not also use an arduino/esp or even a PIC16 etc to actually perform some of the actions on real hw ? e.g. power on/off schedules led control etc...

Quite doable. Code-wise we need a similar interface for ioctl emulation (see above). Hardware-wise... well, *I* personally don't see the point as I virtualize what I can (and some of my colleagues disagree with me) but it's an easy addition. Currently, we're trying to get the software side of all the things to a non-developer-usable state. However once we do the ioctl interface patches for adding support for submitting events to some external source are welcomed. Then we can start another topic discussing making a small module for that. The cheapest AVR/arduino/PIC will do here.

I know you meant to control the LEDs but in case you're not aware: the actual power off via ACPI just lacks a kernel module and doesn't need any custom stuff.

 

 

On 7/15/2021 at 9:38 PM, Aigor said:

@pocopico i think there is a problem with bus, as far i know there is a I2C bus on the modo, but it's hard to connect to.
You need a sort of pcb slot with wire soldered on for I2C

Correct us if we're wrong but the architecture here is such that the kernel uses ASCII-based interface over UART to a bridge. Then the bridge/PMU has I2C for controlling other chips for like LEDs, fans etc. So strictly speaking we don't need I2C for anything.

 

 

On 7/17/2021 at 8:21 PM, yanjun said:

So whether we can establish a developer discussion group, similar to discord, telegram or others, I don’t know if this is compliant, I will try to establish one first, and welcome everyone to join so that we can iterate this project more efficiently.
telegram: https://t.me/joinchat/eYcaYV4ywDY4MWU9

As I see the forum doesn't have any Telegram channel nor IRC. We believe the latter is actually better in many ways (we're old I think ;)) but Telegram is more modern. We will join for sure.

 

 

On 7/19/2021 at 7:58 AM, calimansi said:
  1. Are all of the DSM apps built into the kernel?  This is an interesting way to get this done... no wonder they don't want to just give it away.
  2. Since we're compiling from the actual DSM kernel, does this mean we can build this for any model that Synology makes (more than just what is offered by Jun)?  I can see some awesome projects coming out of this possibility.  
  1. The DSM consists of a heavily modified kernel, modified userland applications, and a completely custom web UI to manage all that. Since they have to should abide by the GPL they agreed to when modifying the kernel & libs they do should publish the full source with modifications for GPL-licensed part. In practice this means we get kernel, some libs, and some userland apps code. The whole DSM web management is their own custom stuff.
     
  2. So, here's actually the issue. The source code for the kernel is readable-ish but it cannot be compiled. It used to be but it's not anymore. The biggest problem is they replaced a lot of constants with "MY_ABC_HERE". In our view this is a violation of the GPL but good luck pressuring them to do anything about that. It's probably possible to compile the kernel if someone goes through and manually decides which "MY_ABC_HERE" to turn on and off but even we aren't crazy enough to do so.

    What the kernel sources are good for, besides reading, is compiling modules against them. You cannot use vanilla v4.4 or v3.10 sources for this. DSM kernels are heavily modified (so e.g. structs have new fields added to them, sometimes in the middle). Additionally, their v3.10 is not a v3.10 mainline + their patches but v3.10+DSM patches+a ton of backported changes from newer kernels.
    Supporting many devices is difficult because of the standard matrix of permutations problem, as in any software project, AND on top of this because Synology releases the same but different "v3.10" for different platform. For example apollolake v4.4 has broken ramdisk support :D

 

On 7/19/2021 at 11:30 AM, UnknownO said:

I am an amateur, I spent a whole day trying to compile the other day, but only got redpill.ko. And I don’t even know how to use redpill.ko and add it to the core of Synology

It is also expected that the author of this patch will announce how to use or compile the kernel. But... he seems to have not logged in to this forum for a long time, he last logged in on Friday

We don't compile the kernel. We use DSMs kernel + binary patches from the dsm-research repo patches. You can load the patched image using kexec or repack it to boot from grub. You CANNOT use Jun's loader for this. Currently, we only published sources for the LKM. This is not because we have some secret one but because the kernel modification process is MUCH easier to do.
 

To give you a glimpse of how we do the development we will just say that it's a manual process. We deliberately did not write a step-by-step instruction because we don't want regular folks making YT videos how to hack this around. It is NOT production safe yet. If you want to play with it as a developer simply compile GRUB (or even just copy one from any Linux distro from /boot). Then get the kernel + rd.gz from 25556 PAT file. Next apply binary patches from dsm-research repo and put redpill ko in the rd.gz. Next modify the init script to include redpill loading at the very top.

 

We are looking here and discussing what you guys bring but we publish updates in batches as we don't want to rush with conclusions or publish FUD. It takes some times as we have full time jobs, families, kids, and socialize sometimes too (despite the rumors developers do sometimes leave their caves ;)).

 

 

On 7/19/2021 at 1:34 PM, mcdull said:

They modified it to 1019+ simply because it has the exact same core hardware.  And it is NOT an easy task to support whatever emulated model because of those pci address issues as discussed.

PCI addresses are very easy with RedPill. We can most likely prepare a patch in 5 minutes if someone posts "lspci -tvv" from a real DS/RS (ok, not just right now because we're currently trying to come up with a since runtime-level model-dependent configuration switching. The other things like testing and potentially different hardware requirements are the issue. Once we get a stable-stable version we can play with other ones. Another issue are S/Ns. They need to validate. We saw the validation somewhere in the kernel but the community developed serial generator only supports a handful of models.
 

If someone wants to take a peek / knows how the previous serial generator was developed we're open. However, we think @flyride nailed it. The number of disks is not limited and other features are already present in DS3615/7xs & DS918+. DS1621+ may be worth looking into due to the AMD support.

Edited by ThorGroup
  • Like 17
  • Thanks 2
Link to post
Share on other sites
6 hours ago, ThorGroup said:

Correct us if we're wrong but the architecture here is such that the kernel uses ASCII-based interface over UART to a bridge. Then the bridge/PMU has I2C for controlling other chips for like LEDs, fans etc. So strictly speaking we don't need I2C for anything.

 

Thats my understanding also. /dev/synobios is a character device, that should accept, kernel and user space communication altough it seems to be locked by secmd.

 

 

Link to post
Share on other sites
On 7/13/2021 at 7:16 AM, ThorGroup said:

The RedPill is back!

As some of you may be aware a lot of research materials as well as the code for parts of the kernel module were pulled from GH. We're happy to report it's back and fully public! Before further ado we have a small sneak-peek of the current state:

Untitled4.thumb.png.c658eaed8c849cdea82a72d25396973a.png

 

Repositories
Both LKM code and the research materials are present in two repositories. Both are automatic forks from our internal serves and are updated few times a day.
 - RedPill LKM: contains the current version of the Linux kernel module source code along with implementation details description
 - DSM Research/Docs: hosts documentation for developers regarding the inner details of DSM boot process


While the dates and authors in both repos are anonymized, the history is preserved. Thus, your forks and PRs will work properly.

 

The Current State
As of now the DSM installs & boots properly (sort of, continue reading). We are currently working on a toolset for generating the loader image automatically so that testing new iterations is easier for people not familiar with full inner workings of the kernel component & the bootloader itself. The tool with instructions will be published in a separate repo.

The kernel module is currently missing the PCI-IDs shimming and RTC emulation. While the latter is most likely not crucial, the former must be implemented. However, it's not really straight-forward as naturally the kernel doesn't have a high-level API to lie about nonexisting hardware ;)

The current revision of the LKM causes some errors to be sent to the PMU. If anyone in the community (@Vortex? @IG-88?) has an idea of what is the source of these we will be grateful for some pointers.
 


--R--R-p--R-4
-9
--R-r-K-8-3-8


As of now we're working on a robust PCI emulation layer. This isn't hard in theory but has many pitfalls if we want to do it properly and none of us ever studied inner workings of PCI on x86 ;) As described in the PCI document in the research repo there are three methods. We picked the third one (full PCI emulation) as it allows for creation of devices which are indistinguishable from real ones. While this is the hardest to pull off properly, it doesn't rely on a hack but rather an official and documented Linux API.


Q&A

  1. Who are you?
    We're a group of passionates dating back to the (great) phreaking times. If you know where to look you will find us on IRC ;)
     
  2. Can I get involved in the development?
    Yes! As this project took a lot from the community we strongly believe it should continue to be shared and developed under GPL. We greatly appreciate any PRs on GH.
     
  3. I'm not a developer, can I help?
    At this stage most likely not. However, we wish to have some testing version not too far in time. For various reasons we cannot (and not willing to) accept any donations. If you want to make us feel better leave a like and a good word for us, as naturally this isn't our full-time job :)))
     
  4. Why is making the code public matters?
    We believe that the code of the loader MUST be public. We aren't sure if the general community is aware of the degree of control the "loader" has over their box. Despite the name it is not just a load-and-leave situation. The majority of the loader code is active in the system for the whole time (you can check that by doing lsmod and looking for an entry which doesn't look like a proper module name but one or more random characters).
    The kernel module can do literally anything you can as root... and more. It can read files, send them in the background somewhere, hide files from you, execute programs with higher-than-root privileges without showing them in any tools, use your CPU while showing 0% in htop etc... and the worst part is that you will never know that it happened (unless you're monitoring your device from the outside).

    However, after this scary paragraph we can say two things: Jun's loader doesn't seem to do any evil things, and the actions any loader needs to perform in the system after the initial load are minimal (e.g. fake responses to "turn on HDD led"). We've also reviewed the code we cloned and it's a solid base.
    Additionally, making the code open means anybody can tinker with it and adjust to new scenarios instead of relying to bit-patching a .ko.
     
  5. What happened to previous repos? Are you crediting the previous author?
    The author of the original code wishes to distance himself from the project and we are respecting that. That's all we know.
     
  6. Do you/anyone have the code of Jun's loader?
    We saw that there's some confusion on the forum regarding Jun's loader and why the work had to start from scratch. Neither the Jun's loader code nor any deeper implementation details regarding inner working of his amazing loader were ever shared with the public. We weren't able to obtain the code through our sources either. There's a good chance he never shared the with anybody.
     
  7. Is Syno trying to block the loader?
    While we cannot comment on any actions, we can surely talk about the code. The new kernel contains something which isn't present before 25556: https://github.com/RedPill-TTG/dsm-research/blob/master/quirks/boot_params-validation.md
    It is true that the "va not found" error triggered by the Jun's loader when used with >6.2.3 is indeed related to offsets which changed in the new build. However, the rabbit hole doesn't stop there. The new "boot_params" check doesn't seem to have any other purpose than detecting violation of the chain of trust. So did the new kernel build broke the loader intentially? Most likely not (it's probably a by-product of the new validation code present very early in the image) but why the boot params validation was placed in 6.2.4 in the first place? We leave the interpretation to the reader.
     
  8. When we can expect a stable release? Will it work on v7?
    We cannot promise any date for two reasons: 1) we can hit an unexpected roadblock (e.g. see errors mentioned above), and 2) we will like to test it and have it working on v6.2.4 and v7 as well (as of now v7 is available for selected devices only and from our tests it is not fully stable even on the devices it was officially released).
    Some of the protections found in v6 were pulled from v7... but don't worry, they will be back as soon as they port them... it's a carrot and a stick situation.

 


cc for people who followed the original topic:

@AleAmadoC @alexku44 @Amoureux @Balrog @blindspot @Bobbenoster @Bobur @coolinx @dimcheff @Fede @FiberInternetUser @gadreel @ilovepancakes @impala_84 @intrax @jarugut @juud @kiwiuk @lemon55 @loomes @minigranis @NeoID @Nuno @Piteball @pkdick1 @pro_info @profet @rufik @s2k7 @scoobdriver@setsunakawa @smilenkovski @smileyworld @smoojay @snakefox666 @Snyaify @SpiRe @T-REX-XP @The Chief @toolazy @vasiliy_gr 

 

Untitled4.png

Thanks for your job!

Link to post
Share on other sites
23 hours ago, flyride said:

 

No, the RS3621xs+ is indirectly supporting NVMe cache through add-in card, but there is no current Synology device that supports all of the desired image-specific features that are available in either DS918+ or DS3617xs: NVMe, Quicksync, RAIDF1 and 16-thread.

What?!?! Are you trying to say the DS3617xs has quicksync? or your just listing desired features that apply to one or the other?

Link to post
Share on other sites
20 hours ago, ThorGroup said:

We don't compile the kernel. We use DSMs kernel + binary patches from the dsm-research repo patches. You can load the patched image using kexec or repack it to boot from grub. You CANNOT use Jun's loader for this. Currently, we only published sources for the LKM. This is not because we have some secret one but because the kernel modification process is MUCH easier to do.
 

To give you a glimpse of how we do the development we will just say that it's a manual process. We deliberately did not write a step-by-step instruction because we don't want regular folks making YT videos how to hack this around. It is NOT production safe yet. If you want to play with it as a developer simply compile GRUB (or even just copy one from any Linux distro from /boot). Then get the kernel + rd.gz from 25556 PAT file. Next apply binary patches from dsm-research repo and put redpill ko in the rd.gz. Next modify the init script to include redpill loading at the very top.

 

We are looking here and discussing what you guys bring but we publish updates in batches as we don't want to rush with conclusions or publish FUD. It takes some times as we have full time jobs, families, kids, and socialize sometimes too (despite the rumors developers do sometimes leave their caves ;)).

Hello, I followed your method and successfully added redpill.ko to the ramdisk. However, when adding a binary patch to zImage, I successfully decompressed zImage and obtained vmlinux, and used the php file in the GH repo dsm-research for kernel patching, but I don’t know how to package vmlinux into zImage, So, currently I am stuck ramdisk corrupt

Looking forward for your reply, thank you!

11111111111111111111111.png

1222222222222222222.png

Link to post
Share on other sites
5 minutes ago, UnknownO said:

but I don’t know how to package vmlinux into zImage

I tried to search for "vmlinux packaged as zImage" on Google, Baidu, and Bing, but I only found the Android zImage packaging tutorial or Linux compilation tutorial

Edited by UnknownO
Link to post
Share on other sites
18 minutes ago, intrax said:

I found that this does not seem to be zImage, it seems to be bzImage. Because when I checked the information, I found that zImage seems to be compressed with gzip,but this kernel uses lzma compression. And using the file command also shows that this is bzImage(see the upload image).

 

I rummaged through the search engine results, but didn't find a useful one. Basically it is about how to compile Linux to get bzImage...

 

PS:I tried to cut the head of this zImage and merge it with the modified vmlinux compressed by lzma to get a new zImage, but when booted with grub, the virtual machine crashed directly

22222222222222222222222.png

Link to post
Share on other sites
18 minutes ago, UnknownO said:

 

I rummaged through the search engine results, but didn't find a useful one. Basically it is about how to compile Linux to get bzImage...

maybe you can try make bzImage with exist vmlinux.bin

something from jun's scripts

cp $WORKDIR/vmlinux.bin ${OUTDIR}/arch/x86/boot/compressed/vmlinux.bin

$SCRIPT_DIR/genzImage

 

or waitting for original author answer on  https://reverseengineering.stackexchange.com/questions/27803/repacking-vmlinux-into-zimage-bzimage

Link to post
Share on other sites
5 minutes ago, jumkey said:

vmlinux.bin

Does vmlinux.bin here refer to the vmlinux of objcopy?and Where can I download the jun's script?

emmm,In addition, I think no one should reply to this question and answer. This is a question and answer in June.

 

 

Edited by UnknownO
Link to post
Share on other sites
On 7/20/2021 at 2:16 AM, ThorGroup said:

DSM v7 sources
Synology did already publish v7 sources and.... they pulled them quickly: https://sourceforge.net/p/dsgpl/discussion/862835/thread/a519b80124/. They used to be in the "41890branch" as you can see from the project's history: https://sourceforge.net/p/dsgpl/activity/?page=0&limit=100

We weren't lucky enough to get them. Did anyone maybe was able to grab them?

 

Not the folder you mention but anything in here help? https://sourceforge.net/projects/dsgpl/files/toolkit/DSM7.0/

 

Link to post
Share on other sites
On 7/21/2021 at 10:44 PM, UnknownO said:

I can't get genzImage, So now I have no way, wait for ThorGroup to share more details

 

I remember @kiler129 posted that it is very complicated to repack the kernel to bzimage. But I cannot find the thread now as it seems to be deleted. Maybe he can give us some hints.

 

 

Link to post
Share on other sites
On 7/21/2021 at 10:01 PM, jumkey said:

maybe you can try make bzImage with exist vmlinux.bin

something from jun's scripts


cp $WORKDIR/vmlinux.bin ${OUTDIR}/arch/x86/boot/compressed/vmlinux.bin

$SCRIPT_DIR/genzImage

 

or waitting for original author answer on  https://reverseengineering.stackexchange.com/questions/27803/repacking-vmlinux-into-zimage-bzimage

I'm not gonna go into details of how it is done (as I said, it's ~10 pages of notes) as when I clean it up it will be shared as an open-source tool with docs. However I will only say that:

Flow cannot be directly applied to an unpacked kernel, as some files are missing (information is stripped while building the kernel and cannot be recovered from bzImage, yet the process makes calculations based on these pieces)

Some ASM modification is required to the sources to make it buildable

The kernel build process has parts which weren't touched since 90s', and it is at times a very spaghetti code (e.g. bash generating C which generates ASM with printf()s)

Kernel uses non-standard hacky Makefile tricks

Only some objects in the kernel source tree can be MAKEd without building the whole thing

I discovered that the 10-years old script present in the kernel source doesn't ACTUALLY produce a correct file but I had no energy to develop a patch for it

this is kill129 said before

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.