TinyCore RedPill Loader Build Support Tool ( M-Shell )

[djlongy]

On 3/20/2023 at 3:24 PM, Peter Suh said:

 

That file already contains the PID just fine as shown below.
It sounds like you've already had success using the HBA. Congratulations.

{
     "name": "mpt3sas",
     "alias": "pci:v00001000d000000C4sv*sd*bc*sc*i*"
},

It's weird, I tried it multiple times over and over but it didn't work no matter how many times I built it.

It might have been fixed now with updates. 🤷‍♂️

 

Another issue:

One thing I have also noticed is that the drive numbers are way out of whack when SataPortMap and DiskIdxMap are blank/ignored.

It jumps all the way to disk 33-44 instead of starting from 2 and upwards (number 1 being taken from the sataboot)

The only way I can get around this is to edit the user_config.json manually and run the old build method (

./rploader.sh build ds3622xsp-7.1.1-42962 auto

) instead of the menu.sh FRIEND/JOT

I am running another ESXi VM, this time with a LSI 9300-16i HBA passthrough and SATA0:0 for the boot tcrp image.

The values that work for me where the disk number start at 2 are:

SataPortMap=1

DiskIdxMap=10

If this is blank it will start at 33 and limit the disks to maximum 12.

If the disk starts from 2, it allows me to see more than 12 disks, (14 in my screenshot below)

Is there anyway we can override the blanking of these when building from menu.sh? When I edit the user_config.json manually it doesn't seem to make any different at next boot and still shows blank values for SataPortMap and DiskIdxMap.

 

Update:

I've added

"SasIdxMap": "0"

And rebuilt the loader, the setting has taken at boot, but the disks still start from 33, so "SasIdxMap=0" seems to have made no difference to the numbering.

 

Edited March 21, 2023 by djlongy
Added screenshots

[Peter Suh]

1 hour ago, djlongy said:

It's weird, I tried it multiple times over and over but it didn't work no matter how many times I built it.

It might have been fixed now with updates. 🤷‍♂️

 

Another issue:

One thing I have also noticed is that the drive numbers are way out of whack when SataPortMap and DiskIdxMap are blank/ignored.

It jumps all the way to disk 33-44 instead of starting from 2 and upwards (number 1 being taken from the sataboot)

The only way I can get around this is to edit the user_config.json manually and run the old build method (

./rploader.sh build ds3622xsp-7.1.1-42962 auto

) instead of the menu.sh FRIEND/JOT

I am running another ESXi VM, this time with a LSI 9300-16i HBA passthrough and SATA0:0 for the boot tcrp image.

The values that work for me where the disk number start at 2 are:

SataPortMap=1

DiskIdxMap=10

If this is blank it will start at 33 and limit the disks to maximum 12.

If the disk starts from 2, it allows me to see more than 12 disks, (14 in my screenshot below)

Is there anyway we can override the blanking of these when building from menu.sh? When I edit the user_config.json manually it doesn't seem to make any different at next boot and still shows blank values for SataPortMap and DiskIdxMap.

 

Update:

I've added

"SasIdxMap": "0"

And rebuilt the loader, the setting has taken at boot, but the disks still start from 33, so "SasIdxMap=0" seems to have made no difference to the numbering.

 

 

 

to fix the satadom
It seems necessary to record the values in SataPortMap and DiskIdxMap.

In HBA, to start from disk Index 1, If the value of "SasIdxMap":"0" is meaningful, it was confirmed in bare metal.
VM seems to have another variable.

The reason for forcibly blanking the two values of SataPortMap / DiskIdxMap in the case of a VM in M SHELL was a deliberate action taken by light users who do not use a lot of disks to increase the success rate of loader installation within the range of 6 disks or less.

I recently found out that I need to adjust the SataPortMap / DiskIdxMap values to fix the satadom to Index 1 while using a lot of disk in the VM.

It seems that M SHELL needs to think about how to adjust SataPortMap / DiskIdxMap to satisfy light users and heavy users at the same time.

Edited March 21, 2023 by Peter Suh

[Peter Suh]

[NOTICE]

 

Multilingual support for M SHELL for TCRP menu.

Currently, nine languages are supported: Korean/Chinese/Japanese/Russian/French/German/Spanish/Italian/Brazilian.

English is included as standard.

Because of this, the version has been raised to 0.9.4.3-1 .

This is a pre-built version of the Tinycore Unicode package for multilingual support.

https://github.com/PeterSuh-Q3/tinycore-redpill/releases/tag/v0.9.4.3-1

 

 

[Rantanplan]

On 3/10/2023 at 5:22 PM, Erik29gamer said:

Thanks for the help.

I ended up moving to a different motherboard (same cpu/chipset) for unrelated reasons, and the issue is no longer occurring. I'm going to assume it was a hardware issue at this point, but I appreciate the advice! 

I had the same problem. I ran DSM 6.2 with Jun´s Loader on a Gigabyte C246M-WU4 for "years" without a problem. Then I used the excellent 👍 M-Shell-Tool to build a loader and update my system to 7.1.1. Only problem was that DSM wouldn´t boot on the first try after it had been shut off for some time. I had to use the long-pressed power-button trick to shut it down the hard way and the second boot afterwards went through fine. As a test rig I´m using a Fujitsu D3644-B mainboard (same chipset) and this system was booting fine every time.

 

I already prepared a long "cry for help" post for this thread 😆, but yesterday I gave it a last shot and tried out a couple of BIOS settings of the C246M-WU4. Unfortunately I didn´t take notes what I changed, but it obviously it did help. Now I can shut down the rig for hours and it comes up to a WOL command just fine. I´ll try to check/remember what settings I did change and update this post, but for the meantime my advice to people with similar problems would be to "play" with their BIOS settings.

[Peter Suh]

[NOTICE]

 

We release TCRP Friend v0.0.5.

The original version of pocopico is staying at v0.0.4.

 

Only my M Shell for TCRP Friend is upgraded to v0.0.5 because new features have been added.

 

When TCRP Friend boots, it will be automatically updated to v0.0.5 version as shown below.

 

The newly added feature is the ability to directly edit the grub.cfg file, like the old Jot mode or jun's loader in the old DSM 6, as shown below.

Made in the form of a menu.

 

TCRP FRIEND used to have to rebuild the loader to fix the cmdline options, but now you don't have to.

 

Press the e key within 9 seconds during booting to enter this menu.

 

TCRP Friend does not refer to the cmdline of the existing grub.cfg file, so in the user_config.json file that tcrp has separately

You need to modify the cmdline defined as usb_line or sata_line.

 

It is the same as the part defined in the grub.cfg file.

 

You can select one of the USB Baremetal or SATA VM environment as you want, as shown on the screen.

You can modify it and continue booting with Continue Boot.

 

Once modified, it is permanently saved in /mnt/tcrp/user_config.json.

 

However, the /home/tc/user_config.json file used when building the original loader becomes an old version that retains the contents at the time of build.

 

Also added Force_junior boot option for those who want to reinstall DSM.

 

The text on the screen has also changed slightly.

 

 

Edited April 1, 2023 by Peter Suh

[shibby]

@Peter Suh I`m trying to us Mellanox ConnectX-3 dual sfp+ NIC but only "mlx4_core.ko" has been loaded. There is missing "mlx4_en.ko" module

 

 

latest tcrp-m-shell, ds920+ with Friend

 

[Peter Suh]

9 hours ago, shibby said:

@Peter Suh I`m trying to us Mellanox ConnectX-3 dual sfp+ NIC but only "mlx4_core.ko" has been loaded. There is missing "mlx4_en.ko" module

 

 

latest tcrp-m-shell, ds920+ with Friend

 

 

Here is the VID / PID matching table information from mlx4_core.ko / mlx4_en.ko.

 

What PID does your nic have?

 

You can check with lspci -nn 

 

https://cateee.net/lkddb/web-lkddb/MLX4_CORE.html

https://cateee.net/lkddb/web-lkddb/MLX4_EN.html

 

[shibby]

5 hours ago, Peter Suh said:

What PID does your nic have?

 

You can check with lspci -nn 

 

Here you go

 

 

 

I had to revert to "Jun`s Mod" so i dont have even mlx_core module build-in now. But i downloaded "mlx4_core-4.4.180plus-geminilake.tgz" package from pocopico`s repository and i can load modules manually. This is the result:

 

 

eth2/eth3 doesn`t appear. I have to "up" them manually

 

PS Is there any way to add drivers to my exist redpill-loader without compile newone?

[Peter Suh]

3 hours ago, shibby said:

 

Here you go

 

 

 

I had to revert to "Jun`s Mod" so i dont have even mlx_core module build-in now. But i downloaded "mlx4_core-4.4.180plus-geminilake.tgz" package from pocopico`s repository and i can load modules manually. This is the result:

 

 

eth2/eth3 doesn`t appear. I have to "up" them manually

 

PS Is there any way to add drivers to my exist redpill-loader without compile newone?

 

 

According to your test results, it seems that fabio's ARPL mlx4_core module is not working properly.
Pocopico's TCRP mlx4_core module also requires manual activation of some ports, but at least it seems to work.
The mlx4_core and mlx4_en modules seem to have dependencies on each other.

Currently, I am looking for something that works correctly by referring to both the ARPL and TCRP kernel module sources.
Basically, it is based on ARPL modules, but modules not in ARPL refer to the source of TCRP.
If the module does not work correctly in ARPL, replace it with the TCRP module.
Since there is an integrated module compilation environment created by fabio, I also compile the module myself recently.
We will compare ARPL / TCRP for Metrox module and release it as v1.04, the next version of M SHELL module after improvement.

M SHELL / ARPL / TCRP, the kernel sources of the three loaders are referenced in the following locations.
https://github.com/PeterSuh-Q3/arpl-modules
https://github.com/fbelavenuto/arpl-modules
https://github.com/pocopico/redpill-modules

[Peter Suh]

ARPL/TCRP both have errors in the Makefile definitions and I have checked the differences.

fabio defined CONFIG_MLX4_EN_DCB=n but this doesn't seem to be recommended by Kconfig.

https://github.com/pocopico/redpill-modules/blob/master/src/mlx4/4.x/Kconfig#L14

config MLX4_EN_DCB
bool "Data Center Bridging (DCB) Support"
default y
depends on MLX4_EN && DCB
---help---
Say Y here if you want to use Data Center Bridging (DCB) in the
driver.
If set to N, will not be able to configure QoS and ratelimit attributes.
This flag is depended on the kernel's DCB support.
If unsure, set to Y

https://github.com/fbelavenuto/arpl-modules/blob/main/src/4.x/defines.geminilake#L106

https://github.com/fbelavenuto/arpl-modules/blob/main/src/4.x/drivers/net/ethernet/mellanox/mlx4/Makefile#L11

-------------------------------------------------- ---------------------------------------

On the other hand, pocopico defined CONFIG_MLX4_EN_DCB value as m in Makefile.

https://github.com/pocopico/redpill-modules/blob/master/src/mlx4/4.x/Makefile#L11

It seems that the default y is used because CONFIG_MLX4_EN_DCB is incorrectly defined as m even though Kconfig guides you to use y or n.

So, as a result, the dependencies of the two modules mlx4_core and mlx4_en modules seem to have disappeared.

I'll switch to the recommended y option and compile.
If you give me back the same forced injection success result, I will upgrade it to the integrated version of v1.04 version.

[Peter Suh]

I think I know what fabio and pocopico were thinking about compiling the mlx4 module.

 

Eventually, when 'y' was used, a compilation error occurred, and when 'n' was used, there was no compilation error, but the module did not work.

 

I followed the 'm' setup the way pocopico used.

 

As shown below, there seems to have been some modification of the source accordingly.

 

https://github.com/PeterSuh-Q3/arpl-modules/commit/0722490e4d092fca3af021791d822d37d5f76e33

 

I have now finished compiling the module and updated the module on GitHub.

 

Please give feedback on the result of manually injecting the mlx4_core.ko file.

 

https://github.com/PeterSuh-Q3/arpl-modules/blob/main/geminilake-4.4.180/mlx4_core.ko

Edited April 2, 2023 by Peter Suh

[shibby]

i recompiled DS920+F but result is the same: mlx4_core is loaded on boot but mlx4_en does not

 

 

 

[shibby]

"if something is stupid but it works, that`s mean that is not a stupid"

 

I got it!! There must be some problem with order of loaded modules in all-modules package: mlx4_core module must be loaded first and then mlx4_en. It does not work with all-modules package, so... I wan run menu.sh script then in second console i added extension from pocopico to redpill-loader
 

https://raw.githubusercontent.com/pocopico/rp-ext/main/mlx4_core/rpext-index.json

 

because in this package we have kmod order set

 

  "kmods": {
    "mlx4_core.ko": "",
    "mlx4_en.ko": ""
  },

after compilation my Mellanox NIC boot up automatically

 

 

PS It will be nice to have a possibiity to add external extension via M-shell menu

Edited April 2, 2023 by shibby

[Peter Suh]

11 hours ago, shibby said:

"if something is stupid but it works, that`s mean that is not a stupid"

 

I got it!! There must be some problem with order of loaded modules in all-modules package: mlx4_core module must be loaded first and then mlx4_en. It does not work with all-modules package, so... I wan run menu.sh script then in second console i added extension from pocopico to redpill-loader
 

https://raw.githubusercontent.com/pocopico/rp-ext/main/mlx4_core/rpext-index.json

 

because in this package we have kmod order set

 

  "kmods": {
    "mlx4_core.ko": "",
    "mlx4_en.ko": ""
  },

after compilation my Mellanox NIC boot up automatically

 

 

PS It will be nice to have a possibiity to add external extension via M-shell menu

 

1. The module loading method of ARPL is EUDEV.
2. TCRP's module loading method is insmod (dependencies between modules are not considered)
3. The module loading method of M SHELL for TCRP is EUDEV / DDSML (Detected Device Static Module Loading).

The main commands used to process dependencies between modules are as follows.
depmod (create a module's dependencies)
modprobe (loads other modules related to the module as dependencies)
insmod (module single loading, inter-module dependencies are not considered)

2 TCRP sequentially processes mlx4_core and mlx4_en 2 times using insmod.
However, when ARPL 1 or M SHELL 3 was EUDEV used, it seemed that dependency processing between modules would be handled well, but it seems not to be the case.

If so, it seems that DDSML of M SHELL in number 3 can be the last solution.
In DDSML both depmod / modprobe are used.
Please do this test one more time.

EUDEV / DDSML module loading method switching exists at the top of the menu as shown below.

 

 

 

 

Edited April 3, 2023 by Peter Suh

[nemesis122]

Hi Peter 

I think i know now why ARPL bigger as 1.03b always failed at the gen8 because all time i have the error in the console eudev failed also woth Mshell eudev get failed only  DDSML (Detected Device Static Module Loading) is working.

So my Questions

ARPL 1.03b is working with DDSML ?because it is working without any issues

ARPL bigger as 1.03b goes with eudev is this the case ?

 

Do you know why the Gen8 is only workking with  DDSML (Detected Device Static Module Loading). ??

thank you very much 

 

 

[Peter Suh]

12 minutes ago, nemesis122 said:

Hi Peter 

I think i know now why ARPL bigger as 1.03b always failed at the gen8 because all time i have the error in the console eudev failed also woth Mshell eudev get failed only  DDSML (Detected Device Static Module Loading) is working.

So my Questions

ARPL 1.03b is working with DDSML ?because it is working without any issues

ARPL bigger as 1.03b goes with eudev is this the case ?

 

Do you know why the Gen8 is only workking with  DDSML (Detected Device Static Module Loading). ??

thank you very much 

 

 

 

I wonder if the eudev and modprobe already mentioned above have some key in the processing sequence.

My M SHELL safely uses only modprobe considering dependencies between modules in DDSML.

 

https://github.com/PeterSuh-Q3/tcrp-modules/search?q=modprobe

Until now, I believed that eudev alone would automatically handle module loading considering dependencies between modules by itself without modprobe.

However, in recent cases, this belief has been broken.

As a result of analyzing ARPL recently, I found that ARPL is used not only by eudev but also by modprobe.

 

https://github.com/fbelavenuto/arpl/search?q=modprobe

Since I don't have the ability to fully analyze fabio's ARPL program, I couldn't analyze it accurately, but I think ARPL is processed in the order of eudev -> modprobe.
In the process, eudev broke module dependencies, and it seems that modprobe fell into a situation where it couldn't cover it.

I'm thinking of merging M SHELL with DDSML (modprobe) and EUDEV like ARPL.
However, modprobe takes precedence, and after that, I plan to use EUDEV to handle unrecognized hardware.

Edited April 3, 2023 by Peter Suh

[shibby]

On 4/3/2023 at 6:57 AM, Peter Suh said:

Please do this test one more time.

ok, i will but probably over the weekend

[Dreadnought]

@Peter Suh I was trying your last release of M-shell when I booted the VM image no menu is shown and the tc folder under home is empty. 

Do I have to initiate your loader somehow?

[Peter Suh]

2 hours ago, Dreadnought said:

@Peter Suh I was trying your last release of M-shell when I booted the VM image no menu is shown and the tc folder under home is empty. 

Do I have to initiate your loader somehow?

 

I don't understand what context your screen is in exactly.
Can you give me a capture?

[Dreadnought]

54 minutes ago, Peter Suh said:

 

I don't understand what context your screen is in exactly.
Can you give me a capture?

this is how it looks like direct after the boot process:

 

and this is the content of /home/tc:

 

Now my question is, what should I do to open the m-shell menu?

Edited April 5, 2023 by Dreadnought

[Peter Suh]

42 minutes ago, Dreadnought said:

this is how it looks like direct after the boot process:

 

and this is the content of /home/tc:

 

Now my question is, what should I do to open the m-shell menu?

 

 

Tinycore has a boot entry like this:

menuentry 'Tiny Core Image Build' {
         savedefault
         search --set=root --fs-uuid $usbpart3uuid --hint hd0,msdos3
         echo Loading Linux...
         linux /vmlinuz64 loglevel=3 cde waitusb=5 vga=791
         echo Loading initramfs...
         initrd /corepure64.gz
         echo Booting TinyCore for loader creation

}

 

And the 3rd partition of the vmdk image contains the following files.

drwxrwxrwx 3 root root 4096 Feb 18 2021 cde/
-rwxrwxrwx 1 root root 12031520 Jan 5 2022 corepure64.gz
-rwxrwxrwx 1 root root 87030106 Mar 25 10:45 mydata.tgz
-rwxrwxrwx 1 root root 5366304 Feb 18 2021 vmlinuz64

hd0,msdos3 means this 3rd partition.

Among them, the files listed above
vmlinuz64 , corepure64.gz , mydata.tgz
These 3 files are directly related to Tinycore booting,
It seems that the restore of the last one, mydata.tgz, failed.

This file contains all the information under /home/tc related to TCRP and M SHELL.

You seem to be loading this process abnormally for some reason.

At that tc@box prompt you showed me
Show me the contents of /mnt/sd#3.

 

 

Edited April 5, 2023 by Peter Suh

[Dreadnought]

58 minutes ago, Peter Suh said:

 

 

Tinycore has a boot entry like this:

menuentry 'Tiny Core Image Build' {
         savedefault
         search --set=root --fs-uuid $usbpart3uuid --hint hd0,msdos3
         echo Loading Linux...
         linux /vmlinuz64 loglevel=3 cde waitusb=5 vga=791
         echo Loading initramfs...
         initrd /corepure64.gz
         echo Booting TinyCore for loader creation

}

 

And the 3rd partition of the vmdk image contains the following files.

drwxrwxrwx 3 root root 4096 Feb 18 2021 cde/
-rwxrwxrwx 1 root root 12031520 Jan 5 2022 corepure64.gz
-rwxrwxrwx 1 root root 87030106 Mar 25 10:45 mydata.tgz
-rwxrwxrwx 1 root root 5366304 Feb 18 2021 vmlinuz64

hd0,msdos3 means this 3rd partition.

Among them, the files listed above
vmlinuz64 , corepure64.gz , mydata.tgz
These 3 files are directly related to Tinycore booting,
It seems that the restore of the last one, mydata.tgz, failed.

This file contains all the information under /home/tc related to TCRP and M SHELL.

You seem to be loading this process abnormally for some reason.

At that tc@box prompt you showed me
Show me the contents of /mnt/sd#3.

 

 

 

there is nothing mounted as it looks like.

I used the vmdk maybe an error in the vm-image?

[Peter Suh]

40 minutes ago, Dreadnought said:

 

there is nothing mounted as it looks like.

I used the vmdk maybe an error in the vm-image?

 

There is nothing wrong with the VMDK image.

 

This is an image that many users have been using without problems so far.

 

I finished version 0.9.4.3-2 a while ago.

 

I don't know which VM program you are using, but in some cases you can use VMDK directly and in others it is not.

 

Please refer to the user guide for each VM program and try again.

[Dreadnought]

1 hour ago, Peter Suh said:

 

There is nothing wrong with the VMDK image.

 

This is an image that many users have been using without problems so far.

 

I finished version 0.9.4.3-2 a while ago.

 

I don't know which VM program you are using, but in some cases you can use VMDK directly and in others it is not.

 

Please refer to the user guide for each VM program and try again.

I am using Parallels for mac. I have to convert the vmdk to hdd via qemu but with this approach I am able to ARPL and tinycore redpill successful. 

[Dreadnought]

1 hour ago, Dreadnought said:

I am using Parallels for mac. I have to convert the vmdk to hdd via qemu but with this approach I am able to ARPL and tinycore redpill successful. 

I have tested your new version but even here /mnt/ is empty.