Lets Encrypt intermediate certificate expired, affects Sabnzbd

[stereoguy]

 

 

SO I am encountering an issue with Sabnzbd not connecting to my news servers (Eweka, Thecubenet) with an untrusted certificate from eweka error. I found the following post on Sabnzbd site which seems to translate to an OS issue with incorrect certificates due to  an intermediate signing cert from Lets Encrypt that expired Sept 30 2021. Easy enough to fix on windows, but how about on XPE?

 

https://www.sslshopper.com/ssl-checker. ... news.eweka.nl shows the certificate chain of eweka is correct.

So problem is client side - which in my case is XPE 

https://scotthelme.co.uk/lets-encrypt-o  ... xpiration/ is a very long story, with this sentence "The certificate in here that is going to cause a problem is this one, the IdenTrust DST Root CA X3." ... "expiration date of 30th Sep 2021" ... which is today.

Any ideas on how to fix this? I built a bunch of xpe servers for friends, and the calls are about to start!

 

 

[dmapdm]

hi

i have the same problem.

i have a xpenology ds918+ with DSM 6.2.3-25426 Update 3.

 

We have'nt the update with update on certificat : look here (6.2.4-25556 Update 2) https://www.synology.com/en-global/releaseNote/DSM
go to 10. Updated the OpenSSL setting in response to new rules of Let's Encrypt certificates.

Is it correct ?

how to manualy update openssl setting ??????

 

too, my sabnzbd and the download station are off. impossible to connect them to newsgroup server with strict ssl check.

 

i have try to restart : synonoly, sabnzbd, box.... nothing better

Thanks to all help from Community.

[premikkoci]

Does this mean we have to update to 6.2.4 or find another solution? OpenVPN seems to be also affected.

[dmapdm]

Hummmm, it's not possible. if i upgrade i brick my Xpennology.

[dmapdm]

before test, we can backup somes files non ?

 

is it exact ?

in root, i have success backup as : 

 

cp -r -p /etc/ca-certificates.conf /etc/ca-certificates.conf-orig

cp -r -p /etc/ssl /etc/ssl-orig

 

but now .... how can i update them without change ma Xpenology version ?

Thanks to all help from Community.

Edited October 2, 2021 by dmapdm

[seagl]

Hi

Suddenly could not acces my NAS via OpenVPN and found out that cert from Lets Encrypt has expired Sept 30 2021.

Have the same problem: no response from the destination server synology

So can't update Let's Encrypt certificate via DSM control panel.

Have you found the solution?

Edited October 3, 2021 by seagl

[seagl]

As I understood we just need to upload new certs, right? So where can we get it from?

Not sure there was the same problem on 6.1.X but there was a solution here.

 

 

[humancaviar]

12 hours ago, seagl said:

As I understood we just need to upload new certs, right? So where can we get it from?

Not sure there was the same problem on 6.1.X but there was a solution here.

 

 

That tutorial was to update the certificate signing module to support some new methodologies implemented by LetsEncrypt, that wont help us in this case.  Only useful for fixing the "request new certificate from LetsEncrypt"... I may try this separately as I'm having to request with acme.sh, but wont help our issue in this thread.

 

I've tried a bunch of things, adding the new X3 certificate and ISRG root to /usr/share/ca-certificates/mozilla/

Added those same certificates in pem format to /etc/ssl/certs and hashed them

Modified /etc/ca-certificates.conf to include the newly added certificates in step 1

 

I think the issue may be the invalid root certificates still existing.  Does anyone know a good way to find which .crt/.pem files correspond to the newly invalidated certificates?

 

I'm wondering why this isnt a bigger issue, not seeing a lot of chatter about this.

[moretty]

13 hours ago, seagl said:

Hola

De repente, no pude acceder a mi NAS a través de OpenVPN y descubrí que el certificado de Lets Encrypt expiró el 30 de septiembre de 2021.

Tiene el mismo problema: no hay respuesta del servidor de destino synology

Por lo tanto, no se puede actualizar el certificado Let's Encrypt a través del panel de control de DSM.

¿Has encontrado la solución?

I am also with openvpn stopped. could you solve it?

[haydibe]

Actualy this one should do everything that is required, but it still doesn't seem to work:

 

 

CERT_DIR=/usr/share/ca-certificates/le
mkdir -p "${CERT_DIR}"

# Root Certificates
curl --insecure --silent "https://letsencrypt.org/certs/isrgrootx1.pem" --output "${CERT_DIR}/isrgrootx1.crt"
curl --insecure --silent "https://letsencrypt.org/certs/isrg-root-x2.pem" --output "${CERT_DIR}/isrg-root-x2.crt"

# Intermediate Certificates
curl --insecure --silent "https://letsencrypt.org/certs/lets-encrypt-r3.pem" --output "${CERT_DIR}/lets-encrypt-r3.crt"
curl --insecure --silent "https://letsencrypt.org/certs/lets-encrypt-e1.pem" --output "${CERT_DIR}/lets-encrypt-e1.crt"

# add certificates to ca-certifactes, not sure if required?
cat ${CERT_DIR}/*.crt >> /etc/ssl/certs/ca-certificates.crt

for cert in ${CERT_DIR}/*.crt; do
  crt=${cert##*/}
  pem=${crt/.crt/.pem}
  # create symlink from downloaded certs to /etc/ssl/certs
  ln -fs ${CERT_DIR}/${crt} /etc/ssl/certs/${pem}
  pushd /etc/ssl/certs
  # create symlink with hash of the pem certificate.
  ln -fs ${pem} `openssl x509 -hash -noout -in ${pem}`.0 
  popd
done

 

In Theory that should work. But it doesn't. Make sure to run the `cat` command not more than once, otherwise you might end up with duplicate entries in the ca-certifcates.crt file.

 

Either the server side still uses a certificate with the outdated certificate chain in one of the both cross signed paths inside the certificate, or we realy do have an issue with our openssl version. 

 

 

Edited October 3, 2021 by haydibe

[haydibe]

There is a way to check if certifcates are outdated: 

for cert in /etc/ssl/certs/*.pem; do
  openssl verify -CApath /etc/ssl/certs $cert 
done

 

According these tests the cross-signed LE certifictes are outdated?! That's why I removed them from aboves download list.

Edited October 3, 2021 by haydibe

[haydibe]

The problem does not exist with DSM7. I just copied the /etc/ssl/certs/ca-certificates.crt file from DSM7 to DSM6.2.3. It still does not work.

 

When a staticly compiled version of curl is used (see: https://github.com/moparisthebest/static-curl/releases/tag/v7.79.1).

It works, even though it uses the very same cabundle....

 

Smells like a library problem in an outdated openssl version. That's terrible!

 

It is still worth to check if the obeves action solves the issue for sabnzbd, as it uses python, which may or may not use the openssl libraries from the OS. The commands won't do anything more worse than they are now ^^

 

And then there is always the way to run things in a docker container, which won't suffer from the old library problem. 

Edited October 3, 2021 by haydibe

[humancaviar]

Thanks @haydibe, lot of good information.

In my case, chain appears broken for:

• Sabnzbd (native app, not docker) connecting to letsencrypt signed servers.
• Synology VPN server app (I use LetsEncrypt for signing that and DSM, et. al).

I'm wondering if those apps are using sandboxed openssl libraries.  Going to keep digging with some of the new stuff I gleaned from your posts.  Not in a good position to upgrade DSM, running a 24 drive bare metal rig which usually has issues updating.   

[premikkoci]

Any updates?

[humancaviar]

Havnt had the time to mess with this further.  It seems as though the CA/Chain needs to be updated somewhere I'm not aware of at the moment, or I've missed something?  Needs stating, I'm no expert only an enthusiast, hoping someone with more experience will come along and finish the job

[haydibe]

I have checked the content of the 6.2.3u3 pat, it inclides a new ca-certificates.crt and the new LE CA. 

Maybee installantion of the u3.pat might solve the issue?

 

I am not sure why, but since I moved from ESXI to Proxmox, updates fail for me as well. Previosuly I used a sata loader, now I use a usb loader and it seems like it is kind of related to why the updates fail. 

Edited October 5, 2021 by haydibe

[premikkoci]

I am already on 6.2.3. update 3. Perhaps a fresh install might do a trick?

[haydibe]

I doubt that it will.

 

If the u3 does not provide new openssl libraries then, the updated ca-certificates.crt alone are worth nothing for everything that depends on these libraries. I couldn't find evidence whether openssl libraries are part of u3 as well. 

 

Like I wrote: if a static compiled version of curl is used (with build in openssl library), accessing LE services work like a charm with the very same ca-certificates.crt.

 

I repeat: the problem is not only that ca certifcates are missing/ expired in ca-certificates.crt. 

 

Hope you guys find a solution with DSM6.2.3.

Edited October 5, 2021 by haydibe

[elmuziko]

Just throwing a bit more on this in the hope we find something.

My Synocommunity packages list is now empty, and if I try to add a new source, I get told to sod off in not so many words.

I was going to try and install nzbget as a temp work around until someone has identified and fixed the issue, though reading on the internet, nzbget is suffering the same issue.

 

NZBGet have published a healthy page on resolutions. https://nzbget.net/certificate-verification

Unfortunately a manual install of NZBGet is failing for me on my DSM so I can't put the steps into practise. But maybe some breadcrumbs for someone smarter than I?

 

Then there's this:

https://github.com/nzbget/nzbget/releases/tag/v21.2-r2333

 

 

[deep470]

My old Xpenology is running on DSM 5.2, but yesterday I found the time to build a new one on ESXI 6.7. It took me hours to realize that the error isn't in my setup but in DSM 6.3 and DSM 6.2 (I freshly installed 6.2 after 6.3 didn't work).

 

I managed to get rid of the curl error 60 by adding a curl ca-path to php.ini and setting the variable export SSL_CERT_FILE=/path/to/ca.pem

 

But that didn't work for the webinterface. So as a workaround I set my machines date to September 27th, 2021 and was finally able to add Synocommunity and  install all my needed packages. Needless to say that I still have to find a way to fix this or I'll never receive updates again 😕

 

EDIT

Okay, I've deleted the expired certificate but as haydibe wrote: we need a newer openssl version to make this work again.

"If you provide an API or have to support IoT devices, you’ll need to make sure of two things: (1) all clients of your API must trust ISRG Root X1 (not just DST Root CA X3), and (2) if clients of your API are using OpenSSL, they must use version 1.1.0 or later. In OpenSSL 1.0.x, a quirk in certificate verification means that even clients that trust ISRG Root X1 will fail when presented with the Android-compatible certificate chain we are recommending by default."

 

 

Edited October 5, 2021 by deep470

[humancaviar]

I'm thinking that installing Entware and attempting an openssl update to 1.1.1?  Anyone have experience trying this?

 

https://github.com/Entware/Entware/wiki/Install-on-Synology-NAS

[haydibe]

I doubt that it will fix the general issue within DSM...

 

Python3 application seem to use a binding that makes use of the systems openssl libraries. So all Python3 should be affected.

 

Some SynoComunity packages (like git) comme with an update openssl library under the hood - I wonder if it's possible to copy the binaries and libs into the main system... Though, please do NOT try this on your live systems! I repeat: It is a stupid idea, don't do it on you live system!

 

 

[humancaviar]

So... Funny thing happened.  Reverted my original attempts to load updated crt/pem's and then ran @haydibe's script:

Think I missed something in my original attempt. 

Restarted Sabnzbd and was able to connect to my news provider without issue.

Just checked package center, and wouldn't you believe it, there was an updated VPN Server application who's changelog listed chain of trust fixes.

Only thing remaining, for me at least, is package center can't connect to SynoCommunity...  Not the end of the world, but a bit of a bummer that I wont get updates.

 

Forgot to mention, if it helps someone else, I'm running: DSM 6.1.7-15284 Update 3

On: Bare metal, 24 drive, Gigabyte H97N-WIFI, i3-4370 based system as a DS3615xs

[haydibe]

The script just adds missing certifactes to the ca certifcates list and to the ca-certificates.crt file (which btw. IS important, as most tools use it instead of the loose single .pem and .cert files in the folder). Glad that it helped at least a single person

 

Packages usualy provider their own libraries - updated versions of packages shouldn't be affected.

curl for instance, will be affected until the openssl libraries are updated.

 

 

[moretty]

To test, I replaced in the config file *. ovpn the line and I created the VPN on the DSM client side

proto udp
Replaced by

udp6 proto
and the connection was made ... between the NAS.

 

* I did that which is quoted by user virouf and it works:

https://www.nas-forum.com/forum/topic/69472-perte-connexion-vpn-entre-2-nas-synology/

[elmuziko]

On 10/3/2021 at 5:42 PM, haydibe said:

Actualy this one should do everything that is required, but it still doesn't seem to work:

 

 

CERT_DIR=/usr/share/ca-certificates/le
mkdir -p "${CERT_DIR}"

# Root Certificates
curl --insecure --silent "https://letsencrypt.org/certs/isrgrootx1.pem" --output "${CERT_DIR}/isrgrootx1.crt"
curl --insecure --silent "https://letsencrypt.org/certs/isrg-root-x2.pem" --output "${CERT_DIR}/isrg-root-x2.crt"

# Intermediate Certificates
curl --insecure --silent "https://letsencrypt.org/certs/lets-encrypt-r3.pem" --output "${CERT_DIR}/lets-encrypt-r3.crt"
curl --insecure --silent "https://letsencrypt.org/certs/lets-encrypt-e1.pem" --output "${CERT_DIR}/lets-encrypt-e1.crt"

# add certificates to ca-certifactes, not sure if required?
cat ${CERT_DIR}/*.crt >> /etc/ssl/certs/ca-certificates.crt

for cert in ${CERT_DIR}/*.crt; do
  crt=${cert##*/}
  pem=${crt/.crt/.pem}
  # create symlink from downloaded certs to /etc/ssl/certs
  ln -fs ${CERT_DIR}/${crt} /etc/ssl/certs/${pem}
  pushd /etc/ssl/certs
  # create symlink with hash of the pem certificate.
  ln -fs ${pem} `openssl x509 -hash -noout -in ${pem}`.0 
  popd
done

 

In Theory that should work. But it doesn't. Make sure to run the `cat` command not more than once, otherwise you might end up with duplicate entries in the ca-certifcates.crt file.

 

Either the server side still uses a certificate with the outdated certificate chain in one of the both cross signed paths inside the certificate, or we realy do have an issue with our openssl version. 

 

 

 

So I'm going to show my complete novice at life.

Two things.

sudo cat ${CERT_DIR}/*.crt >> /etc/ssl/certs/ca-certificates.crt
Is giving me :

 

elmuziko@DSM:~$ sudo cat ${CERT_DIR}/*.crt >> /etc/ssl/certs/ca-certificates.crt
-sh: /etc/ssl/certs/ca-certificates.crt: Permission denied

am I doing something wrong?

And secondly, how do I run the for ; do loop from a PUTTY command loop. Will it not take each line as a single command? 

 

Thank you
 

 

Edited October 19, 2021 by elmuziko