Jump to content
XPEnology Community

helixzz

Rookie
 View Profile  See their activity

  • Posts

    6

  • Joined

  • Last visited

 Content Type 

Posts posted by helixzz

    RedPill - the new loader for 6.2.4 - Discussion

    in Developer Discussion Room

    Posted

    On 10/1/2021 at 9:33 AM, ThorGroup said:

     

    Finally we've got some time to respond to the questions. We will try to respond to as many as possible so stay tight :)
    =======

     

    Why there's a problem with compiling drivers when syno didn't publish kernel sources?

     

     

    This is going to be a separate post addressing "why you cannot randomly grab drivers and compile them" so we can link to it. Let us explain where's the problem, this applies to many questions really.

    There are few major issue with compiling drivers against a different sources. The three major ones are:
    1) Versioning of syno kernel's is broken due to backporting
    2) Drivers modification
    3) Kernel <=> driver modification
    4) Data structures missing in toolchains for in-tree drivers

     

    Now let us explain each one of them separately and why this is an issue. There's a lot to unpack here. First lets address few terms which are used in kernel development:

    • "in-tree" = code which exists within the source of the kernel; in terms of drivers it means you will find it under "drivers/" (e.g. in drivers/scsi/mpt2sas)
    • "vanilla kernel" = source of the kernel which is the official version from kernel.org
    • "kernel headers" = .h files present in the toolchain. These are NOT all kernel's .h files but only a subset of public API kernel provides. You will find these under "include" or in the syno toolchain. What's important they DO NOT contain private interaction between modules. That's why if e.g. mpt3sas driver wants something from mpt2sas you cannot use toolchain to compile mpt3sas and you NEED full sources.
    • "upstream it" = make your custom patch for the kernel and get it accepted into the normal kernel sources

     

    1) Versioning & backporting
    Syno, like many other companies, take the kernel and modify it in house. This is something which kernel devs don't really like but since it's an open-source project everyone can. However, after some time these companies realize they want new features from new kernel but cannot get them as their modifications no longer apply to a new version. What they do is "backporting". They grab their e.g. 3.10.108 which was modified to oblivion, then download the e.g. 3.19.0 and check commits which add features they want. They get that commit and apply to their 3.10.108 making it 3.10.109. This is incorrect and broken. But that't not the biggest problem.
    In case you don't see it: this poses a HUGE problem for using drivers from the vanilla kernel. Often times if you take a driver from vanilla 3.10 you will realize that suddenly doesn't work at all... as in fact they previously lifted a newer one from 3.20 and stuffed it in 3.10. Good luck loading such driver and using it.
    For that reason alone you CANNOT use vanilla kernel code as you don't know which version of the code has to be used. Sure, there are ways to figure this out, but the only sane approach is to use the source provided by syno. In case of small-ish bumps in kernel it's usually safe to get e.g. driver from kernel from 3.10.105 and fake the version compiling for 3.10.108 hoping not much changed.

     

    2) Drivers modification
    Syno modifies in-tree drivers directly. To be honest they have no way around it as they apply vendor-specific hacks for e.g. timing bugs in certain chips. They should upstream it but... that's a different story and it's not as easy as it seems. Depending on WHAT was changed in the driver sometimes you can use kernel from a different kernel version. How do you know it's safe? Well... it's hard. If you don't have the sources for the new kernel (since if you did you wouldn't bother lifting the driver from an older version) you cannot know if the data structures changed. If they did you're pretty much toasted.
    Languages like C or ASM (or hardware in general) doesn't have magical way of discovering sizes of things. Everything is based on static assignment (ok, not everything, but lets make a simplification here). This means if syno decided to add a field to a struct [it's like an object] in mpt3sas driver and that field was added to a struct used ONLY in mpt3sas driver you can probably load the old driver and be fine. However, if they added a new argument to a "public" function (function used outside of the mpt3sas driver) or a new field to a struct used externally from the mpt3sas driver things may APPEAR to work. Every time such function is called or struct is used more and more memory and/or stack gets corrupted. Eventually you will crash something very badly and you will be lucky if that was at least the driver and not something unrelated which was just in the same area. In kernel there's virtually no protections of one piece of code breaking memory of the other. There are small protection for the kernel CODE itself and some sensitive areas but it's not like in the user space where one process trying to write into memory of another process will simply be killed - in the kernel you just corrupt the memory silently.


    3) Kernel <=> driver interactions
    Scrolling through the questions we saw that some of you load drivers and see missing symbols. Well, this is the effect of 2). When they modify the drivers they often add their own hooks by just placing a f... gSynoFoo() call somewhere in the driver. Honestly we think it's very sloppy but it is how it is. The issue here is two-fold:

    • if you compile a driver from vanilla kernel you will not have these calls so things may look correctly... but these calls were placed there for a reason and other things exepect them to happen. Sometimes the issue may be really small like performance drop or hibernation not working correctly. Sometimes you may step into a data loss with a certain controller which syno fixed in their kernel after getting a possible fix from the manufacturer... you don't know.
    • if you try to add e.g. SAS to a non-SAS model things break badly. This is because it's not as simple as flipping the flag in synoinfo. By doing so you're telling the DSM "this model supports SAS". But the issue here is that models which REALLY support SAS have A LOT of code which is compiled only and only when the kernel config option for SAS was enabled. This config option being disabled literally REMOVED a bunch of code from many places when they compiled the kernel without SAS support. That's also the reason why getting SATA boon on 918+ was so challenging as the SATA-DOM code is literally removed on 918+. Bottom line is don't try to activate functions which depend on the kernel features being present. You can tweak number of disks, supported number of cameras and other user-land features but not re-enable features which are partially kernel and partially userland. It's like trying to pump diesel fuel into gasoline car after drilling a larger hole to fit the bigger diesel nozzle into the tank.

    There's no straightforward way to KNOW if the feature you want to activate has a kernel component. However, going through ALL options in synoconfigs/KConfig.* files will give anybody an idea what sort of things can or cannot be activated. In general anything involving in-kernel drivers is a no-no (e.g. iGPU acceleration will not work on 3615xs most likely but NVidia GPU probably will).


    4) Data structures missing in toolchains for in-tree drivers
    You cannot get the code of a in-tree driver and compile it against the toolchain. This is because in-kernel drivers have access to many more things and link internal kernel files. Should they? Most likely no. However, the kernel is
    30 YEARS OLD codebase and only recently they started to encapsulate drivers from using internal things. In most cases it's better to stick to the hardware which is officially supported.


    We have v7 sources for geminilake which we took some hints from (e.g. about synobios structures). If there's a particular driver we can look and compare with v6. However, if there's anyone who has experience with drivers (@IG-88?) we can share the archive in a PM as syno pulled it from SF. Why they did pull it? It's possible it contains code from e.g. broadcomm which is under an NDA and should never be published as it's not GPL.

     

    Many thanks to @ThorGroup. Really fantastic explanations for noobs like me.

    RedPill - the new loader for 6.2.4 - Discussion

    in Developer Discussion Room

    Posted

    1 hour ago, Piteball said:

    I think you misread ThorGroups latest post. SAS support is not there yet, it's one of the things on the list to be addressed before they will release first beta. No SAS no LSI support. Also official support for adding unsupported hardware is also on the list of things that will be solved.

    Skickat från min GM1913 via Tapatalk
     

     

    Thanks, Piteball.

    Really excited and can't wait to try out the new loader. I thought there is already some way to 'inject' driver file into the loader and DSM installer as well.

    Then I will wait and also discover myself to see if I can help.

    RedPill - the new loader for 6.2.4 - Discussion

    in Developer Discussion Room

    Posted

    Hi @ThorGroup ! Many thanks to your contribution. 

    I was trying to build a new bare-metal setup using DSM 7.0, and then migrate data from my existing DSM 6.2.3 (using Jun's mod).

     

    Hardware brief:

    - Intel Core i7-6700K (Skylake) with ASUS Z170 motherboard

    - 2 SATA SSDs connected to onboard SATA ports

    - LSI 9300-8i HBA card (requires mpt3sas driver) with 2 SATA, 2 NLSAS drives

    - Intel X710-DA2 10GbE NIC (requires i40e driver)

    - RedPill-Loader image burned to USB flash disk using Rufus

     

    After tried both DS918+ and DS3615xs RedPill-LKM images, I stucked:

    - When using DS918+ image, the DSM installer can only recognize SSDs on onboard SATA ports

    - When using DS3615xs, the installer cannot recognize any disks at all (maybe the kernel too old?)

    - Onboard Intel i219-V NIC works well (since I can access DSM installer web page), while I hadn't confirm whether X710 works or not yet

     

    Seems it just lack some recent drivers in the image. And luckily I can confirm which driver is used by these hardware.

    So is there any way to inject / include / copy some driver files (probably *.ko) into the RP image?

×
×
  • Create New...