RedPill - the new loader for 6.2.4 - Discussion

Schyzo · October 12, 2021

Il y a 21 heures, pocopico a dit :

what is the last message you see in serial ? Try adding the latest tg3.ko

https://raw.githubusercontent.com/pocopico/rp-ext/master/tg3/rpext-index.json

Just tried with it, fresh build completely with jumkey develop branch, but same result.

I see the server while 30 seconds, and it restart.

I think I have to forget DSM 7 on my machine 😢

pippobimbo · October 12, 2021

2 hours ago, Mixpower said:

As @WiteWulfsaid you need to include the tg3 driver in the image.

My fault! I forgot the NIC driver. Now, it's working fine.

Thanks !!!!

Kouill · October 12, 2021

Il y a 7 heures, WiteWulf a dit :

Can you confirm the above? Your system no longer kernel panics under load when using the tg3 drivers? If so this is very important.

Could you please try running the docker influx-test on your system as described by TTG here? This test is guaranteed to crash my Gen8 100% of the time with NMI watchdog enabled. If it turns out to be an interaction with the PCIe bus or the NC360T card and drivers I'm using this could help solve the problem.

I'm running the influx-test container wait and see.

20 minutes after :

Edited October 12, 2021 by Kouill

WiteWulf · October 12, 2021

45 minutes ago, Kouill said:

I'm running the influx-test container wait and see.

20 minutes after :

Amazing! I'm going to build a 7.0.1 bromolow image with tg3 now and see if I can replicate.

Kouill · October 12, 2021

il y a 7 minutes, WiteWulf a dit :

Amazing! I'm going to build a 7.0.1 bromolow image with tg3 now and see if I can replicate.

Nice I can stop the container now

My (ugly) edits :

Makefile :

@if [ "$(TARGET_PLATFORM)" == "bromolow" ]; then \

pushd $(REDPILL_LOAD_SRC) && \

./ext-manager.sh add 'https://raw.githubusercontent.com/pocopico/rp-ext/master/tg3/rpext-index.json'; \

./build-loader.sh 'DS3615xs' '$(TARGET_VERSION)-$(TARGET_REVISION)'; \

global_config.json :

{

"id": "bromolow-7.0.1-42218",

"platform_version": "bromolow-7.0.1-42218",

"user_config_json": "bromolow_user_config.json",

"docker_base_image": "debian:8-slim",

"compile_with": "toolkit_dev",

"redpill_lkm_make_target": "dev-v7",

"downloads": {

"kernel": {

"url": "https://sourceforge.net/projects/dsgpl/files/Synology%20NAS%20GPL%20Source/25426branch/bromolow-source/linux-3.10.x.txz/download",

"sha256": "18aecead760526d652a731121d5b8eae5d6e45087efede0da057413af0b489ed"

},

"toolkit_dev": {

"url": "https://sourceforge.net/projects/dsgpl/files/toolkit/DSM7.0/ds.bromolow-7.0.dev.txz/download",

"sha256": "a5fbc3019ae8787988c2e64191549bfc665a5a9a4cdddb5ee44c10a48ff96cdd"

}

},

"redpill_lkm": {

"source_url": "https://github.com/RedPill-TTG/redpill-lkm.git",

"branch": "master"

},

"redpill_load": {

"source_url": "https://github.com/420Xnu/redpill-load.git",

"branch": "7.0.1"

}

Dockerfile :

ARG REDPILL_LKM_REPO=https://github.com/RedPill-TTG/redpill-lkm.git

ARG REDPILL_LKM_BRANCH=master

ARG REDPILL_LKM_SRC=/opt/redpill-lkm

ARG REDPILL_LOAD_REPO=https://github.com/420Xnu/redpill-load.git

ARG REDPILL_LOAD_BRANCH=7.0.1

ARG REDPILL_LOAD_SRC=/opt/redpill-load

Edited October 12, 2021 by Kouill

haydibe · October 12, 2021

40 minutes ago, Kouill said:

ARG REDPILL_LKM_REPO=https://github.com/RedPill-TTG/redpill-lkm.git

ARG REDPILL_LKM_BRANCH=master

ARG REDPILL_LKM_SRC=/opt/redpill-lkm

ARG REDPILL_LOAD_REPO=https://github.com/420Xnu/redpill-load.git

ARG REDPILL_LOAD_BRANCH=7.0.1

ARG REDPILL_LOAD_SRC=/opt/redpill-load

No need to touch those. The values are just "fallback values" if they are not provided by build-args, which they always are by the values from global_config.json.

WiteWulf · October 12, 2021

2 hours ago, WiteWulf said:

Amazing! I'm going to build a 7.0.1 bromolow image with tg3 now and see if I can replicate.

Okay, mixed results for me, I'm sad to say. @ThorGroup may find the results useful, however.

I built a new bromolow 7.0.1-42218 image using redpill-helper-v.0.12 and jumkey's "develop" branch of redpoll-load, with pocopico's tg3 extension.

I disabled the PCIe NC360T NIC and re-enabled the internal NIC from the BIOS and booted from the new USB stick.

Initially the server looked good: all docker containers started, as did Plex Media Server, with no kernel panic.

I ran a metadata update on a large library in Plex with no crashes and updated my influxdb docker container, also without a crash.

Next I ran the test TTG had previously asked me to do over on GitHub:

docker run --name influx-test -d -p 8086:8086 -v $PWD:/var/lib/influxdb influxdb:1.8
docker exec -it influx-test sh

# inside of the container:
wget https://golang.org/dl/go1.17.1.linux-amd64.tar.gz &&
    tar -C /usr/local -xzf go1.17.1.linux-amd64.tar.gz &&
    rm go1* &&
    export PATH=$PATH:/usr/local/go/bin &&
    go get -v 'github.com/influxdata/influx-stress/cmd/...'
/root/go/bin/influx-stress insert -f --stats

This time I got a kernel panic:

[  518.193477] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 6
[  518.228126] CPU: 6 PID: 28191 Comm: influx-stress Tainted: PF          O 3.10.108 #42218
[  518.267191] Hardware name: HP ProLiant MicroServer Gen8, BIOS J06 04/04/2019
[  518.301214]  ffffffff814a2759 ffffffff814a16b1 0000000000000010 ffff880409b88d60
[  518.337704]  ffff880409b88cf8 0000000000000000 0000000000000006 0000000000000001
[  518.374045]  0000000000000006 ffffffff80000001 0000000000000030 ffff8803f4c4bc00
[  518.410480] Call Trace:
[  518.422504]  <NMI>  [<ffffffff814a2759>] ? dump_stack+0xc/0x15
[  518.451063]  [<ffffffff814a16b1>] ? panic+0xbb/0x1df
[  518.475140]  [<ffffffff810a9eb8>] ? watchdog_overflow_callback+0xa8/0xb0
[  518.508166]  [<ffffffff810db7d3>] ? __perf_event_overflow+0x93/0x230
[  518.539141]  [<ffffffff810da612>] ? perf_event_update_userpage+0x12/0xf0
[  518.571601]  [<ffffffff810152a4>] ? intel_pmu_handle_irq+0x1b4/0x340
[  518.603124]  [<ffffffff814a9d06>] ? perf_event_nmi_handler+0x26/0x40
[  518.634926]  [<ffffffff814a944e>] ? do_nmi+0xfe/0x440
[  518.659908]  [<ffffffff814a8a53>] ? end_repeat_nmi+0x1e/0x7e
[  518.688056]  <<EOE>>
[  518.698239] Rebooting in 3 seconds..

So this is a *huge* improvement on where I was before, but it still crashes if I really push it, and I'm not sure why @Kouill's server *didn't* crash running the same test 🤔

One thing to note is that the NC360T card is still physically installed in my machine, but disabled in the BIOS.

Mixpower · October 12, 2021

2 hours ago, WiteWulf said:
Okay, mixed results for me, I'm sad to say. @ThorGroup may find the results useful, however.

I built a new bromolow 7.0.1-42218 image using redpill-helper-v.0.12 and jumkey's "develop" branch of redpoll-load, with pocopico's tg3 extension.

I disabled the PCIe NC360T NIC and re-enabled the internal NIC from the BIOS and booted from the new USB stick.

Initially the server looked good: all docker containers started, as did Plex Media Server, with no kernel panic.

I ran a metadata update on a large library in Plex with no crashes and updated my influxdb docker container, also without a crash.

Next I ran the test TTG had previously asked me to do over on GitHub:
docker run --name influx-test -d -p 8086:8086 -v $PWD:/var/lib/influxdb influxdb:1.8
docker exec -it influx-test sh

# inside of the container:
wget https://golang.org/dl/go1.17.1.linux-amd64.tar.gz &&
    tar -C /usr/local -xzf go1.17.1.linux-amd64.tar.gz &&
    rm go1* &&
    export PATH=$PATH:/usr/local/go/bin &&
    go get -v 'github.com/influxdata/influx-stress/cmd/...'
/root/go/bin/influx-stress insert -f --stats
This time I got a kernel panic:
[  518.193477] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 6
[  518.228126] CPU: 6 PID: 28191 Comm: influx-stress Tainted: PF          O 3.10.108 #42218
[  518.267191] Hardware name: HP ProLiant MicroServer Gen8, BIOS J06 04/04/2019
[  518.301214]  ffffffff814a2759 ffffffff814a16b1 0000000000000010 ffff880409b88d60
[  518.337704]  ffff880409b88cf8 0000000000000000 0000000000000006 0000000000000001
[  518.374045]  0000000000000006 ffffffff80000001 0000000000000030 ffff8803f4c4bc00
[  518.410480] Call Trace:
[  518.422504]  <NMI>  [<ffffffff814a2759>] ? dump_stack+0xc/0x15
[  518.451063]  [<ffffffff814a16b1>] ? panic+0xbb/0x1df
[  518.475140]  [<ffffffff810a9eb8>] ? watchdog_overflow_callback+0xa8/0xb0
[  518.508166]  [<ffffffff810db7d3>] ? __perf_event_overflow+0x93/0x230
[  518.539141]  [<ffffffff810da612>] ? perf_event_update_userpage+0x12/0xf0
[  518.571601]  [<ffffffff810152a4>] ? intel_pmu_handle_irq+0x1b4/0x340
[  518.603124]  [<ffffffff814a9d06>] ? perf_event_nmi_handler+0x26/0x40
[  518.634926]  [<ffffffff814a944e>] ? do_nmi+0xfe/0x440
[  518.659908]  [<ffffffff814a8a53>] ? end_repeat_nmi+0x1e/0x7e
[  518.688056]  <<EOE>>
[  518.698239] Rebooting in 3 seconds..
So this is a *huge* improvement on where I was before, but it still crashes if I really push it, and I'm not sure why @Kouill's server *didn't* crash running the same test 🤔

One thing to note is that the NC360T card is still physically installed in my machine, but disabled in the BIOS.

He used 420Xnu repo, are there fixes in there which are not included in jumkey's repo?

zhbitmaye · October 13, 2021

I add mp2sas driver use LSI9211-8I on ESXi，I just Identify 6 HDDs，but I have 8 HDDSs，I tried this config：

1.

"DiskIdxMap" : "0002",
"SataPortMap" : "28",

"DiskIdxMap" : "00020C",
"SataPortMap" : "288",

"DiskIdxMap" : "000C02",
"SataPortMap" : "268",

vmware disk is 2

SAS is 3~8 lose latest 2 HDD，just Identify 6 HDDs

2."DiskIdxMap" : "0006",
"SataPortMap" : "58",

vmware disk is 2

SAS is 6 7 8 12 13 lose latest 2 HDD too，just Identify 6 HDDs

3.

"DiskIdxMap" : "0009",
"SataPortMap" : "88",

vmware disk is 2

SAS is 12 13 14 15 lose latest 4 HDD too，just Identify 4 HDDs

I tried more ，bug only can Identify 6 HDDs to more

is this loader can‘t Identify all LSI-SAS Disks？

cferra · October 13, 2021

Seeing all this… @ThorGroup I can’t help but hope that an easy to use GUI interface for the build process might be in the cards for the beta… the driver thing while good seems like it can get complicated. Looking at it right now I get a bit overwhelmed haha

Edited October 13, 2021 by cferra

WiteWulf · October 13, 2021

7 hours ago, Mixpower said:

He used 420Xnu repo, are there fixes in there which are not included in jumkey's repo?

Yes, I realise that. It’s the redpill-load repo, though, not redpill-lkm, so this only affects how the image is built and not the code that’s actually run on the system.

I’m using haydibe’s docker scripts to build my images (kouill appears to be building by hand) and couldn’t get them to run successfully using the 420Xnu redpill-load repo (which I’m unfamiliar with), so I used jumkey’s repo, which is a known-good alternative to the official TTG repos for unsupported DSM versions, and is what I’ve been using for all the other builds I’ve been testing.

Edited October 13, 2021 by WiteWulf

WiteWulf · October 13, 2021

58 minutes ago, cferra said:

Seeing all this… @ThorGroup I can’t help but hope that an easy to use GUI interface for the build process might be in the cards for the beta… the driver thing while good seems like it can get complicated. Looking at it right now I get a bit overwhelmed haha

Go back and read through ThorGroup's posts in this thread, there aren't many of them and you'll learn a lot. A GUI of some sort is definitely planned, just not sure if it'll be around in time for the beta. Choosing and adding drivers is always going to be necessary as storage space restrictions mean that everything can't be loaded by default, but again, there was talk of a custom image that could be loaded before building a redpill boot stick that will figure out what drivers might be necessary for a given system.

Kouill · October 13, 2021

il y a 34 minutes, WiteWulf a dit :

Yes, I realise that. It’s the redpill-load repo, though, not redpill-lkm, so this only affects how the image is built and not the code that’s actually run on the system.

I’m using haydibe’s docker scripts to build my images (kouill appears to be building by hand) and couldn’t get them to run successfully using the 420Xnu redpill-load repo (which I’m unfamiliar with), so I used jumkey’s repo, which is a known-good alternative to the official TTG repos for unsupported DSM versions, and is what I’ve been using for all the other builds I’ve been testing.

I'm using the haydibe's docker for building images

I've just modify the repo for the loader and use of tg3 driver

cferra · October 13, 2021

2 hours ago, WiteWulf said:

Go back and read through ThorGroup's posts in this thread, there aren't many of them and you'll learn a lot. A GUI of some sort is definitely planned, just not sure if it'll be around in time for the beta. Choosing and adding drivers is always going to be necessary as storage space restrictions mean that everything can't be loaded by default, but again, there was talk of a custom image that could be loaded before building a redpill boot stick that will figure out what drivers might be necessary for a given system.

No you’re right, the forum is definitely informative, I was following along up until build docker .10 then life happened haha

Yeah I think that if there were a gui or some kind of menu to select drivers and add them to the config file automatically it would be great, manually adding the extensions url to the config xml can get dicey, but you’re probably right about not hitting prime time until after the beta.

im also sure that once it gets closer to that stage documentation will be more consolidated in a more digestible way too.

Edited October 13, 2021 by cferra

psychoboi32 · October 13, 2021

420Xnu is me [emoji23], it is used and tune for my specific use case, and i was trying latest point release and pull from orphiee repo, and tune for my use case, when i tried it worked but i can’t take guarantees and responsibility if that doesn’t work for you. If you don’t know what you are doing always stick to thor group official repo.

Sent from my iPhone using Tapatalk

WiteWulf · October 13, 2021

4 minutes ago, cferra said:

manually adding the extensions url to the config xml can get dicey, but you’re probably right about not hitting prime time until after the beta.

There's hopefully a better way of managing extensions coming to haydibe's docker scripts soon....watch this space

RedwinX · October 13, 2021

16 hours ago, haydibe said:

The first red line of the error message indicates that the "no space left on device" error is raised by the target, which is the first patition of the bootloader image.

Just create a new disk :

dd if=/dev/zero of=redpill-template.img bs=1024k seek=256 count=0

+100M for the 1st partition

+75M for the 2nd

all for the rest

losetup -P /dev/loop18 redpill-template.img

mkdosfs -F32 /dev/loop18p1
mkfs.ext2 /dev/loop18p2

mv redpill-template.img boot-image-template.img

gzip redpill-template.img

cp redpill-template.img.gz to ext folder (replace the existing)

Works fine

WiteWulf · October 13, 2021

3 hours ago, Kouill said:

I'm using the haydibe's docker for building images

I've just modify the repo for the loader and use of tg3 driver

I've tried doing it exactly the same as you, but it fails whenever I try to create the image as Orphee removed some extensions from their repo that weren't working properly (they were never intended for public use, but psychoboi32 used them in their forked repo. I'm guessing that as you have the extensions cached locally it carries on building okay for you.

psychoboi32 · October 13, 2021

2 minutes ago, WiteWulf said:

I've tried doing it exactly the same as you, but it fails whenever I try to create the image as Orphee removed some extensions from their repo that weren't working properly (they were never intended for public use, but psychoboi32 used them in their forked repo. I'm guessing that as you have the extensions cached locally it carries on building okay for you.

Do you need orphee extensions ? if not then ```docker run``` command and ext-manager remove that extension

Edited October 13, 2021 by psychoboi32

WiteWulf · October 13, 2021

39 minutes ago, psychoboi32 said:

Do you need orphee extensions ? if not then ```docker run``` command and ext-manager remove that extension

Yeah, just trying to figure that out. You added them to the bundled_exts.json file, though, so I don't believe they can be removed with ext-manager

*edit*

Never mind. Kouill gave me a copy of their boot loader image and it still kernel panics my system when running the influx-test

Edited October 13, 2021 by WiteWulf

RedwinX · October 13, 2021

Hey,

Have a little pb with a baremetal. It boot correctly (918+ 7.0 redpill repo), status : not installed

Install ok, reboot and then : status not installed...

didn't have serial outpout. Any idea ?

tcs · October 13, 2021

3 hours ago, RedwinX said:

Just create a new disk :

dd if=/dev/zero of=redpill-template.img bs=1024k seek=256 count=0

+100M for the 1st partition

+75M for the 2nd

all for the rest

losetup -P /dev/loop18 redpill-template.img

mkdosfs -F32 /dev/loop18p1
mkfs.ext2 /dev/loop18p2

mv redpill-template.img boot-image-template.img

gzip redpill-template.img

cp redpill-template.img.gz to ext folder (replace the existing)

Works fine

I think the broader point being that with the size of the current drivers being put out, that volume needs to be larger by default unless there's some reason I'm unaware of that it's stuck at 40MB. Probably wouldn't be a bad idea to make it a flag or part of the config for the docker script @haydibe @ThorGroup. For instance, I can tell you with the mptsas drivers alone that pretty much blows up all the space for drivers by themselves. If you need mptsas and mpt2sas or mpt3sas and the 42218 build you're dead in the water without a larger partition.

haydibe · October 13, 2021

I am not going to implement functionality to the docker toochain that modifies the behavior of rp-lkm and rp-load.

It is already hard enough for people to distinguished if problems are located in the docker toochain or rp-load.

The only thing that might sense from my perspective is to support hooks for custom before/after scripts.

Though, even that will introduce a new vector for problems nobody wants to deal with.

On the other side, the toolchain builder script/files are all in "clear text" and not realy hard to understand..

Everyone is free to taylor them to their needs

Edited October 13, 2021 by haydibe

pocopico · October 13, 2021

1 hour ago, RedwinX said:

Hey,

Have a little pb with a baremetal. It boot correctly (918+ 7.0 redpill repo), status : not installed

Install ok, reboot and then : status not installed...

didn't have serial outpout. Any idea ?

It occured once or twice, i dont remember the solution .. maybe replace the hdd ?

tcs · October 13, 2021

2 minutes ago, haydibe said:

I am not going to implement functionality to the docker toochain that modifies the behavior of rp-lkm and rp-load.

It is already hard enough for people to distinguished if problems are located in the docker toochain or rp-load.

The only thing that might sense from my perspective is to support hooks for custom before/after scripts.

Though, even that will introduce a new vector for problems nobody wants to deal with.

On the other side, the toolchain builder script/files are all in "clear text" and not realy hard to understand..

Everyone is free to taylor them to their needs

Understand, that's why I @'d you both. I foresee endless requests for help by people adding drivers if those sizes remain the way they are. If the end goal is: usable by everyone, the manual steps listed above probably won't cut it. If the intended audience is the same as the beta: you understand enough coding and linux to unwind problems yourself, then it's not an issue.

RedPill - the new loader for 6.2.4 - Discussion

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites