SAB, python 2.7.9 and untrusted HTTPS (index) sites

[sander]

As FYI: there is a "problem" with SABnzbd 0.7.20 (and lower) with python 2.7.9 with untrusted HTTPS index/RSS sites (EDIT September 2015: ... and/or with incorrect SSL root certificates on your system running SABnzbd, especially NAS-devices)

I say "problem" because maybe it is not a problem, but a feature: not trusting untrusted HTTPS sites seems like a good thing.

This problem can happen with for example Ubuntu 15.04, *BSD and OSX. It won't happen if you use the prepackaged Windows version of SABnzbd

SABnzbd 0.7.20 with Python 2.7.9 (on Ubuntu 15.04-to-be) gives an error with https://www.nzbindex.nl/rss/?q=Hello&so ... esc&max=25 . (Untested, but my guess is older SAB-versions don't show an error, but can't get the RSS feed either)

The error message is:

Code: Select all

2015-01-15 20:36:35,349::DEBUG::[interface:1744] RSS READOUT = True
2015-01-15 20:36:35,351::DEBUG::[rss:332] Running feedparser on https://nzbindex.nl/rss/?q=hello&sort=agedesc&max=25
2015-01-15 20:36:35,389::DEBUG::[rss:334] Done parsing https://nzbindex.nl/rss/?q=hello&sort=agedesc&max=25
2015-01-15 20:36:35,389::INFO::[rss:353] Failed to retrieve RSS from https://nzbindex.nl/rss/?q=hello&sort=agedesc&max=25: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>

So:

Code: Select all

urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

It seems https://www.nzbindex.nl/rss/?q=Hello&so ... esc&max=25 is untrusted; Chrome says:

This server could not prove that it is http://www.nzbindex.nl; its security certificate is from http://www.nzbindex.com. This may be caused by a misconfiguration or an attacker intercepting your connection.

Note nzindex.NL uses a certificate for nzbindex.COM. That is not good, and thus untrused.

It seems python 2.7.9 is more strict with checking HTTPS

Workaround:
- use a trusted HTTPS site, like nzbindex.com instead of nzbindex.nl
- use plain HTTP instead of HTTPS
- downgrade to a python version < 2.7.9
- patch SABnzbd 0.7.x with the few lines of code below
- run SABnzbd 0.8.x

If you want the check if a HTTPS site is untrusted, you can do that via Chrome or Firefox, or via https://www.digicert.com/help/

EDIT client side problems

How to recognize a problem on your own side (your NAS for example):

Code: Select all

2015-09-22 22:48:54,268::INFO::[urlgrabber:116] Grabbing URL https://api.oznzb.com/getnzb/ed4efblabla
2015-09-22 22:48:54,574::INFO::[urlgrabber:199] Retry URL https://api.oznzb.com/getnzb/ed4efblabla

[sander]

OK, workaround created in the SABnzbd code:

Based on https://www.python.org/dev/peps/pep-0476/#opting-out and the tip on https://forums.sabnzbd.org/viewtopic.ph ... =15#p98208, I put this quick-n-dirty code into SABnzbd.py (after "import re" in line 41):

Code: Select all

opt_out_of_certificate_verification = True
if opt_out_of_certificate_verification:
    try:
        import ssl
        ssl._create_default_https_context = ssl._create_unverified_context
    except:
        pass

And now SABnzbd can read the untrusted https://nzbindex.nl/ again ...

FYI: the try/except is needed for python versions < 2.7.9 that don't know ssl._create_unverified_context

[WoLpH]

OK, workaround created in the SABnzbd code:
FYI: the try/except is needed for python versions < 2.7.9 that don't know ssl._create_unverified_context

A better solution might be something along the lines of:

Code: Select all

if hasattr(ssl, '_create_unverified_context'): ...

The clean solution would be to put this in the url grabbing function itself, or perhaps a more generic solution such as using the requests library instead. The requests library takes care of a few of these cases automatically.

[sander]

OK, workaround created in the SABnzbd code:
FYI: the try/except is needed for python versions < 2.7.9 that don't know ssl._create_unverified_context

A better solution might be something along the lines of:

Code: Select all

if hasattr(ssl, '_create_unverified_context'): ...

The clean solution would be to put this in the url grabbing function itself, or perhaps a more generic solution such as using the requests library instead. The requests library takes care of a few of these cases automatically.

Probably. The PEP advices this approach:

Code: Select all

import ssl
# This restores the same behavior as before.
context = ssl._create_unverified_context()
urllib.urlopen("https://no-valid-cert", context=context)

... but then I have to deep-dive into the SAB code ...

[WoLpH]

... but then I have to deep-dive into the SAB code ...

No, it's not that complicated

https://github.com/sabnzbd/sabnzbd/blob ... er.py#L108

[shypike]

We'll get a solution in 0.8.0
The problem isn't the disabling of the verification.
The verification is actually good, unfortunately it will cost extra effort to enable it for other platforms.
On the latest Ubuntu releases, Python 2.7 automatically uses the Ubuntu root certificate store.
For other platform we will need to carry our own copy of the root certificates (we'll probably copy the one from curl).
Of course we need to have an option that will disable verification for sites that don't have valid certificates.,
but which the user still wants to use.

Verifying the certificates of Usenet servers is just as important, but will require more effort.

[WoLpH]

Personally I'd like to see the option of storing the certificate on first usage (for unknown/self-signed ones) and just asking again if it would change. But that's probably going to be a lot of work since it also requires user-interface work.

Having a configurable list of hostnames excluded from certificate checking seems an easy alternative Perhaps with some wildcards using fnmatch (https://docs.python.org/2/library/fnmatch.html) so you can do '*.domain.com'

[shypike]

I agree that pinning the certificate after the first encounter is better.
Since RSS feeds need to be setup, SABnzbd is already interacting with the user, so that would be a good time.
I'll see what I can do.

[WoLpH]

It just occurred to me that there's one scenario you might want to keep in mind, captive portals. In some cases you will temporarily see a different certificate without it being permanent. Not sure how you are planning to implement it exactly but mapping a certificate to a domain would probably cause other problems, a whitelist of certificates is most likely the safest solution.

[shypike]

Using SABnzbd in an environment with a captive portal?
Somehow that sounds awkward to me, bandwidth-hoggers like SABnzbd in such an environment?

[WoLpH]

Using SABnzbd in an environment with a captive portal?
Somehow that sounds awkward to me, bandwidth-hoggers like SABnzbd in such an environment?

Let me explain the backstory here

Sabnzbd also runs on my laptop, while I only download when I'm at home (I automatically pause it when I'm not at home with a ControlPlane script), the nzb's still get added to the queue regardless of whether I'm at home or not. And I can imagine it going wrong if it can't download the nzb's due to certificate errors and not retrying at a later moment when I'm not behind a captive portal

Since all of the trains, train stations, coffee places and restaurants have captive portals these days... bound to give some problems

[sander]

SP, you wrote:

On the latest Ubuntu releases, Python 2.7 automatically uses the Ubuntu root certificate store.
For other platform we will need to carry our own copy of the root certificates (we'll probably copy the one from curl).

The PEP says this:

Python would use the system provided certificate database on all platforms. Failure to locate such a database would be an error, and users would need to explicitly specify a location to fix it.

Based on the PEP, I would expect that the platform, so OS/Libaries (and not SABnzbd) should provide and take care of the root certificates?

[shypike]

Based on the PEP, I would expect that the platform, so OS/Libaries (and not SABnzbd) should provide and take care of the root certificates?

Except when they're not.
Also, many platforms don't have 2.7.9 (none of the embedded platforms and not OSX)
That leaves recent Ubuntus and maybe Windows (I'll examine that).
Might as well just disable verification.

[sander]

Based on the PEP, I would expect that the platform, so OS/Libaries (and not SABnzbd) should provide and take care of the root certificates?

Except when they're not.
Also, many platforms don't have 2.7.9 (none of the embedded platforms and not OSX)
That leaves recent Ubuntus and maybe Windows (I'll examine that).
Might as well just disable verification.

How about this:

For python 2.7.9 and higher:
Rely on the OS for the root certificates. Access https://www.google.com/ if things work.
If the OS does not provide them, always give a warning telling the OS does not provide it. Offer an overrule option to disable verification.
The same overrule option can be used in case a site is untrusted.

[WoLpH]

Based on the PEP, I would expect that the platform, so OS/Libaries (and not SABnzbd) should provide and take care of the root certificates?

Except when they're not.
Also, many platforms don't have 2.7.9 (none of the embedded platforms and not OSX)
That leaves recent Ubuntus and maybe Windows (I'll examine that).
Might as well just disable verification.

Actually, people running OS X with homebrew like me do use python 2.7.9. I'm not saying that everyone is running homebrew or similar package managers but I would suspect a strong correlation between homebrew and sabnzbd users.