docker causing kernel panics after move to 6.2.4 on redpill


Recommended Posts

Regarding soft lockups on hypervisor guests, I just found this:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_monitoring_and_updating_the_kernel/keeping-kernel-panic-parameters-disabled-in-virtualized-environments_managing-monitoring-and-updating-the-kernel

 

Redhat explicitly state that in a vm guest you should have:

nmi_watchdog=0

and 

softlockup_panic=0

...to avoid spurious soft lockups. I think everyone running on a hypervisor should be setting these parameters in their setups 🤔

Link to post
Share on other sites
il y a 27 minutes, WiteWulf a dit :

@Kouill was kind enough to give me a copy of their boot img so I could attempt to replicate their influx-test without crashes.

 

Unfortunately the influx-test kernel panic'd my system within a few seconds.

 

NB. there were the following differences:

- kouill boots their system from the internal microSD slot, whereas mine was on a USB stick on the internal port

- in the grub.cfg I changed the serial number to match mine

- in the grub.cfg I changed mac1 to match mine

- in the grub.cfg I changed I added mac2 with my mac address

 

Sorry for the kernel panic issues, we had the same hardware possible to have a difference of bios settings and cpu can explain the crashes ?

Link to post
Share on other sites
1 hour ago, Kouill said:

 

Sorry for the kernel panic issues, we had the same hardware possible to have a difference of bios settings and cpu can explain the crashes ?

No need to apologise, thank you for helping out! We've got a slightly different CPU, but when I collated a list of people's CPUs earlier it included a broad range of models.

 

I'll go through my BIOS and make a note of all settings, page by page, so that we can compare.

Link to post
Share on other sites

I've drawn up a quick spreadsheet with as much information as I could get relating to hardware, firmware, BIOS settings and general configuration. These are spread across four different tabs within the spreadsheet.

 

https://docs.google.com/spreadsheets/d/1aBrWkJHzwBl6HcYqpBZhRyeX-5CxT5yTmXZ9p3d8zAY/edit?usp=sharing

 

Feel free to comment on any of the settings you think are right or wrong.

 

@Kouill could you compare this to your setup and either fill the sheet in with your details or just reply back here with anything you think is relevant, please?

  • Like 2
Link to post
Share on other sites

They're passed to the kernel at boot time, so technically they're grub.cfg parameters, but practically on redpill they should go in the "extra_cmdline" stanza of <platform>_user>config.json, eg.

 

{
  "extra_cmdline": {
    "pid": "0xc75c",
    "vid": "0x05dc",
    "sn": "1230LWNXXXXXX",
    "mac1": "001132XXXXX",
    "mac2": "001132XXXXXX",
    "nmi_watchdog": "0"
  },
  "synoinfo": {},
  "ramdisk_copy": {}
}

 

They can also be set manually on a running system, eg:

echo 0 > /proc/sys/kernel/nmi_watchdog

 

But this won't persist a reboot, obviously.

Edited by WiteWulf
  • Thanks 1
Link to post
Share on other sites

I've gone through the spreadsheet and highlighted differences in configuration that I think may be relevant (I've ignored cosmetic stuff like power on boot logos and fn key prompts).

 

What stands out to me is that @Kouill is running slower RAM, but that matches their CPU and is configured appropriately in the "Maximum Memory Bus Frequency" section of the BIOS.

 

Also, I've got "Intel Virtualization Technology" disabled, as I'm not running a hypervisor and read that's best to have it turned off in that situation. But can this affect docker? It shouldn't, as docker is basically a glorified chroot 🤔

 

We've all got slightly different IRQ assignments, too.

Link to post
Share on other sites

On the hardware tab it shows:

 

2x 8GB 1600MHz DDR3 ECC

 

But in the BIOS setting for "Maximum Memory Bus Frequency" it's explicitly set to 1300MHz.

 

Ah, Intel says max memory speed for that CPU (E3-1260L) is 1300MHz:

https://ark.intel.com/content/www/us/en/ark/products/52275/intel-xeon-processor-e3-1260l-8m-cache-2-40-ghz.html

 

That's correct

Link to post
Share on other sites

My choices for ram speed in bios are 1066 or 1333.

I could let it in auto that's true but I've already had issues in the past with non hp ecc recommanded ram with the auto settings.

 

For the Intel Virtualization Technology enabled, i have linux/windows VM with Virtual Machine Manager.

Edited by Kouill
  • Like 1
Link to post
Share on other sites
On 10/14/2021 at 12:30 PM, WiteWulf said:

Also, I've got "Intel Virtualization Technology" disabled, as I'm not running a hypervisor and read that's best to have it turned off in that situation. But can this affect docker? It shouldn't, as docker is basically a glorified chroot 🤔

 

It does not. Docker just uses CGROUPS (=what ressources it is allowed to use), Kernel Namespaces (=comparable to fenced partition of kernel components) and capabilities (=what low level features is the processes allowed to use). From the host perspective, container are merly fenced processes that are restricted to see whatever part of the host it is allowed for them to see. The container itself has no idea that it is restricted from the outside (this is not 100% true because a container can access some stats from the outside like the host cpu count and the installed total memory... containerized java proccesses suffer from this if done wrong).

 

  • Like 1
Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.