SAB, python 2.7.9 and untrusted HTTPS (index) sites

[shypike]

There are problems when using third-party Pythons while using UI libraries too.
Also only the "build" version of SABnzbd can use the UI libraries.
That's why we've stuck with Apple Pythons so far.

[sander]

Weird: https://nzbindex.COM/ is now also untrusted according to python 2.7.9:

Code: Select all

sander@superstreamer:~/.sabnzbd/logs$ grep -i nzbindex sabnzbd.log 
2015-02-16 22:14:27,631::DEBUG::[rss:332] Running feedparser on https://nzbindex.com/rss/?q=part&sort=agedesc&max=250
2015-02-16 22:14:27,661::DEBUG::[rss:334] Done parsing https://nzbindex.com/rss/?q=part&sort=agedesc&max=250
2015-02-16 22:14:27,662::INFO::[rss:353] Failed to retrieve RSS from https://nzbindex.com/rss/?q=part&sort=agedesc&max=250: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>

Code: Select all

sander@superstreamer:~/.sabnzbd/logs$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu Vivid Vervet (development branch)
Release:	15.04
Codename:	vivid

sander@superstreamer:~/.sabnzbd/logs$ python --version
Python 2.7.9

https://www.digicert.com/help/ says about nzbindex.com:

Protocol Support

TLS 1.2, TLS 1.1, TLS 1.0, SSL 3.0
SSL 3.0 is an outdated protocol version with known vulnerabilities.

SSL Certificate is not trusted

The certificate is not signed by a trusted authority (checking against Mozilla's root store). If you bought the certificate from a trusted authority, you probably just need to install one or more Intermediate certificates. Contact your certificate provider for assistance doing this for your server platform.

curl seems to agree:

Code: Select all

$ curl -sS 'https://nzbindex.com/rss/?q=part&sort=agedesc&max=250'  2>&1  | grep -A1 "certificate problem"
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

and "openssl s_client" too:

Code: Select all

$ echo | openssl s_client -connect nzbindex.com:443 2>&1 | grep -i error
verify error:num=20:unable to get local issuer certificate
verify error:num=27:certificate not trusted
verify error:num=21:unable to verify the first certificate

So is this now a problem on the side of nzbindex.COM (which was working ok in december)?

[sander]

I was wondering which NZB sites have problems causde by untrusted SSL certificates. To find out, I wrote a python script ssl-checker.py to verify a HTTPS site, to be run with python 2.7.9+:

Code: Select all

#!/usr/bin/env python
import urllib2
import sys
url = sys.argv[1]
try:
	response = urllib2.urlopen(url, timeout=3)
	print url, "OK!"
except Exception as e: 
	print url, "Not good:", e

Example usage:

Code: Select all

$ ./ssl-checker.py https://www.nzbindex.com/
https://www.nzbindex.com/ Not good: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>

$ ./ssl-checker.py https://www.google.com/
https://www.google.com/ OK!

To get NZB sources, I harvested them from http://wiki.sabnzbd.org/nzb-sources, wrote a oneliner, and fed it into my ssl-checker.py, filtering "SSL: CERTIFICATE_VERIFY_FAILED". Result:

Code: Select all

$ lynx --dump http://wiki.sabnzbd.org/nzb-sources | grep -A1000 References | grep " http" |  awk -F/ '{ print "./ssl-checker.py  https://" $3 "/" }'  | sort -u | /bin/sh | grep "SSL: CERTIFICATE_VERIFY_FAILED"

https://fastnzb.info/ Not good: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>
https://forums.sabnzbd.org/ Not good: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>
https://usenetreviewz.com/ Not good: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>
https://www.gingadaddy.com/ Not good: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>
https://www.nzbindex.nl/ Not good: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>
https://www.rssnzb.com/ Not good: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>
https://yubse.com/ Not good: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>

So those are the problematic NZB sources according to python 2.7.9+

[sander]

FWIW: I'm a bit surprised about https://forums.sabnzbd.org/. Of course it's not a NZB source site, but apparantly python 2.7.9 thinks is not a valid SSL certificate, whereas Chromium is not complaining. Curl seems to agree with python 2.7.9:

Code: Select all

$ curl -m 3 -v https://forums.sabnzbd.org/ 2>&1 
* Hostname was NOT found in DNS cache
*   Trying 173.255.201.163...
* Connected to forums.sabnzbd.org (173.255.201.163) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

https://www.digicert.com/help/ says about https://forums.sabnzbd.org/:

Protocol Support

TLS 1.2, TLS 1.1, TLS 1.0, SSL 3.0

SSL 3.0 is an outdated protocol version with known vulnerabilities.

SSL Certificate is not trusted

The certificate is not signed by a trusted authority (checking against Mozilla's root store). If you bought the certificate from a trusted authority, you probably just need to install one or more Intermediate certificates. Contact your certificate provider for assistance doing this for your server platform.

Interesting execise for maintainer of https://forums.sabnzbd.org/ to solve that.

[shypike]

Each root certificate store maintainer makes his own choices.
The leading ones are the browser designers like Microsoft, Mozilla and Google.
Unfortunately other sources are usually behind.
This is especially hits the affordable certificate suppliers, for some reason.

[jwhitt12]

Thanks for all the replies!

I choose to use the solution that sander suggested. I have no idea if its working yet but I have HOPE!! <g>

This is a mispost as I was replying to another topic. My fault, sorry ............

[WoLpH]

FWIW: I'm a bit surprised about https://forums.sabnzbd.org/. Of course it's not a NZB source site, but apparantly python 2.7.9 thinks is not a valid SSL certificate, whereas Chromium is not complaining. Curl seems to agree with python 2.7.9:

Unfortunately system root certificates for Python are notoriously out of date on most systems...

One good solution might be to install certifi: https://pypi.python.org/pypi/certifi
It's a collection of known to be safe root certificates (which means the Chinese ones are filtered out as far as I know).

That's still not a 100% solution though. If your Python is linked to an old version of OpenSSL (below 1.0.2) it still won't work since it simply doesn't support modern/safe certificates.