Page 1 of 2
SAB, python 2.7.9 and untrusted HTTPS (index) sites
Posted: January 15th, 2015, 2:53 pm
by sander
As FYI: there is a "problem" with SABnzbd 0.7.20 (and lower) with python 2.7.9 with
untrusted HTTPS index/RSS sites (EDIT September 2015: ... and/or with incorrect SSL root certificates on your system running SABnzbd, especially NAS-devices)
I say "problem" because maybe it is not a problem, but a feature: not trusting untrusted HTTPS sites seems like a good thing.
This problem can happen with for example Ubuntu 15.04, *BSD and OSX. It won't happen if you use the prepackaged Windows version of SABnzbd
SABnzbd 0.7.20 with Python 2.7.9 (on Ubuntu 15.04-to-be) gives an error with
https://www.nzbindex.nl/rss/?q=Hello&so ... esc&max=25 . (Untested, but my guess is older SAB-versions don't
show an error, but can't get the RSS feed either)
The error message is:
Code: Select all
2015-01-15 20:36:35,349::DEBUG::[interface:1744] RSS READOUT = True
2015-01-15 20:36:35,351::DEBUG::[rss:332] Running feedparser on https://nzbindex.nl/rss/?q=hello&sort=agedesc&max=25
2015-01-15 20:36:35,389::DEBUG::[rss:334] Done parsing https://nzbindex.nl/rss/?q=hello&sort=agedesc&max=25
2015-01-15 20:36:35,389::INFO::[rss:353] Failed to retrieve RSS from https://nzbindex.nl/rss/?q=hello&sort=agedesc&max=25: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>
So:
Code: Select all
urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
It seems
https://www.nzbindex.nl/rss/?q=Hello&so ... esc&max=25 is untrusted; Chrome says:
Note nzindex.NL uses a certificate for nzbindex.COM. That is not good, and thus untrused.
It seems python 2.7.9 is more strict with checking HTTPS
Workaround:
- use a trusted HTTPS site, like nzbindex.com instead of nzbindex.nl
- use plain HTTP instead of HTTPS
- downgrade to a python version < 2.7.9
- patch SABnzbd 0.7.x with the few lines of code below
- run SABnzbd 0.8.x
If you want the check if a HTTPS site is untrusted, you can do that via Chrome or Firefox, or via
https://www.digicert.com/help/
EDIT client side problems
How to recognize a problem on your own side (your NAS for example):
Code: Select all
2015-09-22 22:48:54,268::INFO::[urlgrabber:116] Grabbing URL https://api.oznzb.com/getnzb/ed4efblabla
2015-09-22 22:48:54,574::INFO::[urlgrabber:199] Retry URL https://api.oznzb.com/getnzb/ed4efblabla
Re: SAB, python 2.7.9 and untrusted HTTPS (index) sites
Posted: January 15th, 2015, 3:39 pm
by sander
OK, workaround created in the SABnzbd code:
Based on
https://www.python.org/dev/peps/pep-0476/#opting-out and the tip on
https://forums.sabnzbd.org/viewtopic.ph ... =15#p98208, I put this quick-n-dirty code into SABnzbd.py (after "import re" in line 41):
Code: Select all
opt_out_of_certificate_verification = True
if opt_out_of_certificate_verification:
try:
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
except:
pass
And now SABnzbd can read the untrusted
https://nzbindex.nl/ again ...
FYI: the try/except is needed for python versions < 2.7.9 that don't know ssl._create_unverified_context
Re: SAB, python 2.7.9 and untrusted HTTPS (index) sites
Posted: January 15th, 2015, 3:49 pm
by WoLpH
sander wrote:OK, workaround created in the SABnzbd code:
FYI: the try/except is needed for python versions < 2.7.9 that don't know ssl._create_unverified_context
A better solution might be something along the lines of:
Code: Select all
if hasattr(ssl, '_create_unverified_context'): ...
The clean solution would be to put this in the url grabbing function itself, or perhaps a more generic solution such as using the requests library instead. The requests library takes care of a few of these cases automatically.
Re: SAB, python 2.7.9 and untrusted HTTPS (index) sites
Posted: January 15th, 2015, 3:53 pm
by sander
WoLpH wrote:sander wrote:OK, workaround created in the SABnzbd code:
FYI: the try/except is needed for python versions < 2.7.9 that don't know ssl._create_unverified_context
A better solution might be something along the lines of:
Code: Select all
if hasattr(ssl, '_create_unverified_context'): ...
The clean solution would be to put this in the url grabbing function itself, or perhaps a more generic solution such as using the requests library instead. The requests library takes care of a few of these cases automatically.
Probably. The PEP advices this approach:
Code: Select all
import ssl
# This restores the same behavior as before.
context = ssl._create_unverified_context()
urllib.urlopen("https://no-valid-cert", context=context)
... but then I have to deep-dive into the SAB code ...
Re: SAB, python 2.7.9 and untrusted HTTPS (index) sites
Posted: January 15th, 2015, 5:35 pm
by WoLpH
sander wrote:... but then I have to deep-dive into the SAB code ...
No, it's not that complicated
https://github.com/sabnzbd/sabnzbd/blob ... er.py#L108
Re: SAB, python 2.7.9 and untrusted HTTPS (index) sites
Posted: January 16th, 2015, 4:54 am
by shypike
We'll get a solution in 0.8.0
The problem isn't the disabling of the verification.
The verification is actually good, unfortunately it will cost extra effort to enable it for other platforms.
On the latest Ubuntu releases, Python 2.7 automatically uses the Ubuntu root certificate store.
For other platform we will need to carry our own copy of the root certificates (we'll probably copy the one from curl).
Of course we need to have an option that will disable verification for sites that don't have valid certificates.,
but which the user still wants to use.
Verifying the certificates of Usenet servers is just as important, but will require more effort.
Re: SAB, python 2.7.9 and untrusted HTTPS (index) sites
Posted: January 16th, 2015, 5:55 am
by WoLpH
Personally I'd like to see the option of storing the certificate on first usage (for unknown/self-signed ones) and just asking again if it would change. But that's probably going to be a lot of work since it also requires user-interface work.
Having a configurable list of hostnames excluded from certificate checking seems an easy alternative
Perhaps with some wildcards using fnmatch (
https://docs.python.org/2/library/fnmatch.html) so you can do '*.domain.com'
Re: SAB, python 2.7.9 and untrusted HTTPS (index) sites
Posted: January 16th, 2015, 6:45 am
by shypike
I agree that pinning the certificate after the first encounter is better.
Since RSS feeds need to be setup, SABnzbd is already interacting with the user, so that would be a good time.
I'll see what I can do.
Re: SAB, python 2.7.9 and untrusted HTTPS (index) sites
Posted: January 16th, 2015, 8:36 am
by WoLpH
It just occurred to me that there's one scenario you might want to keep in mind, captive portals. In some cases you will temporarily see a different certificate without it being permanent. Not sure how you are planning to implement it exactly but mapping a certificate to a domain would probably cause other problems, a whitelist of certificates is most likely the safest solution.
Re: SAB, python 2.7.9 and untrusted HTTPS (index) sites
Posted: January 16th, 2015, 9:16 am
by shypike
Using SABnzbd in an environment with a captive portal?
Somehow that sounds awkward to me, bandwidth-hoggers like SABnzbd in such an environment?
Re: SAB, python 2.7.9 and untrusted HTTPS (index) sites
Posted: January 16th, 2015, 9:33 am
by WoLpH
shypike wrote:Using SABnzbd in an environment with a captive portal?
Somehow that sounds awkward to me, bandwidth-hoggers like SABnzbd in such an environment?
Let me explain the backstory here
Sabnzbd also runs on my laptop, while I only download when I'm at home (I automatically pause it when I'm not at home with a ControlPlane script), the nzb's still get added to the queue regardless of whether I'm at home or not. And I can imagine it going wrong if it can't download the nzb's due to certificate errors and not retrying at a later moment when I'm not behind a captive portal
Since all of the trains, train stations, coffee places and restaurants have captive portals these days... bound to give some problems
Re: SAB, python 2.7.9 and untrusted HTTPS (index) sites
Posted: January 16th, 2015, 10:55 am
by sander
SP, you wrote:
shypike wrote:
On the latest Ubuntu releases, Python 2.7 automatically uses the Ubuntu root certificate store.
For other platform we will need to carry our own copy of the root certificates (we'll probably copy the one from curl).
The PEP says this:
Python would use the system provided certificate database on all platforms. Failure to locate such a database would be an error, and users would need to explicitly specify a location to fix it.
Based on the PEP, I would expect that the platform, so OS/Libaries (and not SABnzbd) should provide and take care of the root certificates?
Re: SAB, python 2.7.9 and untrusted HTTPS (index) sites
Posted: January 16th, 2015, 1:07 pm
by shypike
sander wrote:
Based on the PEP, I would expect that the platform, so OS/Libaries (and not SABnzbd) should provide and take care of the root certificates?
Except when they're not.
Also, many platforms don't have 2.7.9 (none of the embedded platforms and not OSX)
That leaves recent Ubuntus and maybe Windows (I'll examine that).
Might as well just disable verification.
Re: SAB, python 2.7.9 and untrusted HTTPS (index) sites
Posted: January 16th, 2015, 3:11 pm
by sander
shypike wrote:sander wrote:
Based on the PEP, I would expect that the platform, so OS/Libaries (and not SABnzbd) should provide and take care of the root certificates?
Except when they're not.
Also, many platforms don't have 2.7.9 (none of the embedded platforms and not OSX)
That leaves recent Ubuntus and maybe Windows (I'll examine that).
Might as well just disable verification.
How about this:
For python 2.7.9 and higher:
Rely on the OS for the root certificates. Access
https://www.google.com/ if things work.
If the OS does not provide them, always give a warning telling the OS does not provide it. Offer an overrule option to disable verification.
The same overrule option can be used in case a site is untrusted.
Re: SAB, python 2.7.9 and untrusted HTTPS (index) sites
Posted: January 16th, 2015, 9:16 pm
by WoLpH
shypike wrote:sander wrote:
Based on the PEP, I would expect that the platform, so OS/Libaries (and not SABnzbd) should provide and take care of the root certificates?
Except when they're not.
Also, many platforms don't have 2.7.9 (none of the embedded platforms and not OSX)
That leaves recent Ubuntus and maybe Windows (I'll examine that).
Might as well just disable verification.
Actually, people running OS X with homebrew like me do use python 2.7.9. I'm not saying that everyone is running homebrew or similar package managers but I would suspect a strong correlation between homebrew and sabnzbd users.