[SOLVED] No response from indexer (Python 2.7.9 and FreeBSD)

Post by **sander** » December 30th, 2014, 4:44 pm

I still don't get it:

Yes, you can disable SSLv3 altogether on the server side, like facebook has done:

sander@flappie:~$ nmap --script ssl-enum-ciphers facebook.com | grep -i sslv3
|   SSLv3: No supported ciphers found

But gmail apparantly hasn't done that, and I'm quite sure gmail/Google is safe:

Code: Select all

sander@flappie:~$ 
sander@flappie:~$ nmap --script ssl-enum-ciphers gmail.com | grep -i sslv3
|   SSLv3:

Thus usenet-crawler might be safe too:

Code: Select all

sander@flappie:~$ nmap --script ssl-enum-ciphers http://www.usenet-crawler.com | grep -i sslv3
|   SSLv3:

And usenet-crawler.com also support TLSv1.1 and TLSv1.2, which are safe:

Code: Select all

sander@flappie:~/Downloads/nmap-6.47$ ./nmap --script ssl-enum-ciphers www.usenet-crawler.com -p 443 

Starting Nmap 6.47 ( http://nmap.org ) at 2014-12-30 22:42 CET
Nmap scan report for www.usenet-crawler.com (46.19.141.134)
Host is up (0.026s latency).
rDNS record for 46.19.141.134: westmere.scyst.com
PORT    STATE SERVICE
443/tcp open  https
| ssl-enum-ciphers: 
|   SSLv3: 
|     ciphers: 
|       TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_AES_128_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_AES_256_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA - strong
|       TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA - strong
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA - strong
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA - strong
|       TLS_RSA_WITH_3DES_EDE_CBC_SHA - strong
|       TLS_RSA_WITH_AES_128_CBC_SHA - strong
|       TLS_RSA_WITH_AES_256_CBC_SHA - strong
|       TLS_RSA_WITH_CAMELLIA_128_CBC_SHA - strong
|       TLS_RSA_WITH_CAMELLIA_256_CBC_SHA - strong
|     compressors: 
|       NULL
|   TLSv1.0: 
|     ciphers: 
|       TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_AES_128_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_AES_256_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA - strong
|       TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA - strong
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA - strong
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA - strong
|       TLS_RSA_WITH_3DES_EDE_CBC_SHA - strong
|       TLS_RSA_WITH_AES_128_CBC_SHA - strong
|       TLS_RSA_WITH_AES_256_CBC_SHA - strong
|       TLS_RSA_WITH_CAMELLIA_128_CBC_SHA - strong
|       TLS_RSA_WITH_CAMELLIA_256_CBC_SHA - strong
|     compressors: 
|       NULL
|   TLSv1.1: 
|     ciphers: 
|       TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_AES_128_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_AES_256_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA - strong
|       TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA - strong
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA - strong
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA - strong
|       TLS_RSA_WITH_3DES_EDE_CBC_SHA - strong
|       TLS_RSA_WITH_AES_128_CBC_SHA - strong
|       TLS_RSA_WITH_AES_256_CBC_SHA - strong
|       TLS_RSA_WITH_CAMELLIA_128_CBC_SHA - strong
|       TLS_RSA_WITH_CAMELLIA_256_CBC_SHA - strong
|     compressors: 
|       NULL
|   TLSv1.2: 
|     ciphers: 
|       TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_AES_128_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 - strong
|       TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 - strong
|       TLS_DHE_RSA_WITH_AES_256_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 - strong
|       TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 - strong
|       TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA - strong
|       TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA - strong
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA - strong
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 - strong
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 - strong
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA - strong
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 - strong
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 - strong
|       TLS_RSA_WITH_3DES_EDE_CBC_SHA - strong
|       TLS_RSA_WITH_AES_128_CBC_SHA - strong
|       TLS_RSA_WITH_AES_128_CBC_SHA256 - strong
|       TLS_RSA_WITH_AES_128_GCM_SHA256 - strong
|       TLS_RSA_WITH_AES_256_CBC_SHA - strong
|       TLS_RSA_WITH_AES_256_CBC_SHA256 - strong
|       TLS_RSA_WITH_AES_256_GCM_SHA384 - strong
|       TLS_RSA_WITH_CAMELLIA_128_CBC_SHA - strong
|       TLS_RSA_WITH_CAMELLIA_256_CBC_SHA - strong
|     compressors: 
|       NULL
|_  least strength: strong

Nmap done: 1 IP address (1 host up) scanned in 2.33 seconds

So ... why is your python 2.7.9 not able to make a connection using TLSv1.1 and TLSv1.2?

Post by **sander** » December 30th, 2014, 5:27 pm

Within a virtual machine (Snappy Ubuntu "Ubuntu Vivid Vervet (development branch)", so 15.04-to-be), I have python "Python 2.7.9rc1", and I have no problem getting stuff from google.com and usenet-crawler over HTTPS.

So the general statement "python 2.7.9 has a problem" is not true, IMHO.
So:
1) that is good news for SABnzbd running on python 2.7.9
2) more research is needed why it does not work on you (FreeBSD) setup

Code: Select all

ubuntu@localhost:~$ python
Python 2.7.9rc1 (default, Nov 27 2014, 23:23:59) 
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

and

Code: Select all

ubuntu@localhost:~$ python get-url-SJ.py 
url is https://www.google.com/

header is:
Date: Tue, 30 Dec 2014 22:26:54 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=84d83f7a61459f07:FF=0:TM=1419978414:LM=1419978414:S=nAnC4QXwMwdwNBNQ; expires=Thu, 29-Dec-2016 22:26:54 GMT; path=/; domain=.google.nl
Set-Cookie: NID=67=Ni1PkwglkZ4uHHt4seSzmhLrPJ0T1qrWz8rPIWGHkItx_AjNv07v-64i8AjmgI1e7Jj4EQa54vdPDUxMsSiXJFA8CmmrcGDD5cWlC2xtyEkaujTLIT3frh_YGP9OQlFy; expires=Wed, 01-Jul-2015 22:26:54 GMT; path=/; domain=.google.nl; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Alternate-Protocol: 443:quic,p=0.02


First lines from webserver response:
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="nl"><head><meta content="/images/google_favicon_128.png" itemprop="image"><title>Google</title><script>(function(){window.google={kEI:'riajVM_DI4rkasz4grgP',kEXPI:'4011559,4016824,4020347,4020562,4021597,4024207,4025181,4025254,4026005,4026558,4027317,8300111,8500394,8500572,8500819,8500863,8500923,10200083,10200716,10200818,10200833,10200834,10200838',authuser:0,kSID:'riajVM_DI4rkasz4grgP'};google.kHL='nl';})();(functi

and

Code: Select all

ubuntu@localhost:~$ python get-url-SJ.py 
url is https://www.usenet-crawler.com/rss

header is:
Server: nginx/1.6.0
Content-Type: text/html; charset=UTF-8
Connection: close
X-Powered-By: PHP/5.6.4
Set-Cookie: PHPSESSID=i9huhjpqnru0iulegbfg1ran72; path=/
Pragma: no-cache
Date: Tue, 30 Dec 2014 22:32:56 GMT
X-Page-Speed: 1.8.31.2-3973
Cache-Control: max-age=0, no-cache, no-store


First 5 lines from webserver response:
<!doctype html> 
<head>
	<meta http-equiv="content-type" content="text/html; charset=UTF-8"/>
	<meta http-equiv="X-UA-Compatible" content="IE=9"/>
	<meta name="keywords" content="Login,nzb search,nzbs,download,indexer,nzb download,nzb crawler,nzb index,usenet-crawler"/>
	<meta name="description" content="You need to be logged in to use this feature"/>	
	<meta name="application-name" content="usenet-crawler"/>
	<title>Login - usenet-crawler - A dark place to download nzbs</title>
	<link href="/vi

yodax · Post by **yodax** » December 30th, 2014, 7:43 pm

I have done a bit of research, and perhaps someone with better knowledge of the subject could verify this, but I think it doesn't matter if SSLv3 is used. The POODLE attack is based around the attacker forcing the connection to fallback to SSLv3. That is why Chrome is opting to disable the fallback option.

Both the client and the server support tls v1+ which is sufficient, but because an attacker could force them back to SSLv3 it is not making the connection. Python 2.7.9 is now enforcing this. So the fix has to be made on the serving side to drop sslv3 support. (Reading about this should only break ie6)

If found this reddit: http://www.reddit.com/r/usenet/comments ... r_privacy/

I am looking into that. I run my downloads inside a VM anyway so cloning it and doing some testing isn't a problem. (Except for time constraints of course)

For other people looking to downgrade Python on FreeBSD (AT OWN RISK) I did the following:

Code: Select all

cd /var/cache/pkg/

ll | grep -i python27

# check if you have an old version available.. IF SO

pkg delete -f python27

pkg add python27-2.7.8_6.txz # <= old version listed in the grep

I missed your posts, so my post might seem a bit out of context

However gmail.com supports: TLS_FALLBACK_SCSV. This allows the client and the server to say, we are not falling back. But does allow IE6 users to connect using SSLv3, and be vulnerable, but that is the least of their problems...

So ... why is your python 2.7.9 not able to make a connection using TLSv1.1 and TLSv1.2?

You would assume it should be able to do that to google.com:443.

My install is a fairly recent FreeBSD 10.0, nothing fancy. Everything from the port collection. I will create a clean VM and see if I can reproduce the problem.

My Python version is not the RC version like Ubuntu's, my version is the released version. Not sure if that would make a difference.

yodax · Post by **yodax** » December 31st, 2014, 2:00 am

I have a brand new VM with the latest FreeBSD and latest Python:

Code: Select all

uname -ra
FreeBSD git 10.1-RELEASE FreeBSD 10.1-RELEASE #0 r274401: Tue Nov 11 21:02:49 UTC 2014     [email protected]:/usr/obj/usr/src/sys/GENERIC  amd64

/usr/local/bin/python2 --version
Python 2.7.9

When connecting to google.com:

IOError: [Errno socket error] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

I shall start a new Google quest to see if I can find out why.

btw; judging from your name and hostname, we are probably from the same country

yodax · Post by **yodax** » December 31st, 2014, 2:09 am

This url: https://www.python.org/dev/peps/pep-0476/ shows how to disable the new feature of 2.7.9.

Code: Select all

import ssl

ssl._create_default_https_context = ssl._create_unverified_context

Applying that to the test script makes it work. (Without a proper certificate validation, basically like running 2.7.8 )

It also mentions that Python opted to not bundle their own root certificate store, but use the systems store instead. I think therein lies the problem with FreeBSD.

yodax · Post by **yodax** » December 31st, 2014, 2:24 am

I got the test script working!

First you need to make sure that the package ca_root_nss is installed. (It is by default)

Check by running:

Code: Select all

pkg info | grep ca_root_nss

Then link the installed certificates to a place where Python looks by default:

Code: Select all

ln -s /usr/local/share/certs/ca-root-nss.crt /etc/ssl/cert.pem

This solves it for the test script. Next I will check it with Sabnzbd, but I don't expect any problems.

yodax · Post by **yodax** » December 31st, 2014, 2:35 am

Sorry for the post spam, I don't know what the etiquette is for creating a new post or updating an previous one.

However, I can confirm that this solves the issue for sabnzbd as well on FreeBSD! This is a FreeBSD specific issue related to the new Python release and how it handles certificates.

So for everyone TL/DR:

On FreeBSD do:

Code: Select all

ln -s /usr/local/share/certs/ca-root-nss.crt /etc/ssl/cert.pem

This should be a huge improvement in security to your setup. An will make this error go away.

Thanks for the help Sander! Much appreciated!

Post by **sander** » December 31st, 2014, 2:55 am

Good catch!

Now I have a question: what happens if you remove the "ln" again, and use a NNTPS to your newsserver to download articles? Does that work?

(Sorry: I can't reproduce the problem on my system, so I can't test)

yodax · Post by **yodax** » December 31st, 2014, 3:05 am

I removed the link and downloaded an nzb over NNTPS, this worked.

Perhaps that doesn't use urllib and doesn't verify the SSL certificate, or verifies in a different way. Not quite sure how to determine that.

WoLpH · Post by **WoLpH** » January 15th, 2015, 6:17 am

Just a heads up, it's not just FreeBSD. I have similar issues on OS X Yosemite, I'll try to debug further but I do think the logging patch should be included in Sabnzbd to make debugging easier.

The "No response from indexer, retry after 60 sec" error is just plain wrong.

The actual CertificateError I'm getting is that the hostname doesn't match the certificate (which is correct, I'm proxying it myself). But I've never had this issue before.

Code: Select all

Traceback (most recent call last):
  File "sabnzbd/sabnzbd/urlgrabber.py", line 131, in run
    fn, header = opener.retrieve(url)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 245, in retrieve
    fp = self.open(url, data)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 213, in open
    return getattr(self, name)(url)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 443, in open_https
    h.endheaders(data)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 850, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 812, in send
    self.connect()
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1212, in connect
    server_hostname=server_hostname)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 350, in wrap_socket
    _context=self)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 566, in __init__
    self.do_handshake()
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 796, in do_handshake
    match_hostname(self.getpeercert(), self.server_hostname)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 273, in match_hostname
    % (hostname, dnsnames[0]))
CertificateError: hostname 'xxx' doesn't match u'yyy'

WoLpH · Post by **WoLpH** » January 15th, 2015, 6:33 am

After a bit of research I found out that it's even more widespread. On OS X with Python 2.7.9 it's impossible to download from indexers using self-signed (anything not in the system CA) certificates.

[edit]Just tried the latest develop branch, works without a problem now

Post by **sander** » January 15th, 2015, 7:59 am

WoLpH wrote:Just tried the latest develop branch, works without a problem now

Then I wonder what SABnzbd git develop branch has done? I don't see it in https://github.com/sabnzbd/sabnzbd/commits/develop

Because I tried the get-url.py test script (with 2.7.9rc1) against https://www.nzbindex.com/ and https://www.nzbindex.nl/, and I get an error. It's ok for https://www.google.com/

Code: Select all

$ python get-url-SJ.py 
2.7.9rc1 (default, Nov 27 2014, 23:23:59) 
[GCC 4.9.2]
url is https://www.nzbindex.com/search/?q=hello&age=&max=25
Traceback (most recent call last):
  File "get-url-SJ.py", line 22, in <module>
    fn, header = opener.retrieve(url)
  File "/usr/lib/python2.7/urllib.py", line 245, in retrieve
    fp = self.open(url, data)
  File "/usr/lib/python2.7/urllib.py", line 213, in open
    return getattr(self, name)(url)
  File "/usr/lib/python2.7/urllib.py", line 443, in open_https
    h.endheaders(data)
  File "/usr/lib/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 850, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 812, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 1219, in connect
    server_hostname=server_hostname)
  File "/usr/lib/python2.7/ssl.py", line 350, in wrap_socket
    _context=self)
  File "/usr/lib/python2.7/ssl.py", line 566, in __init__
    self.do_handshake()
  File "/usr/lib/python2.7/ssl.py", line 788, in do_handshake
    self._sslobj.do_handshake()
IOError: [Errno socket error] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
ubuntu@localhost:~$

FWIW: https://www.digicert.com/help/ can give info about a HTTPS site validity.

WoLpH · Post by **WoLpH** » January 15th, 2015, 10:14 am

Well, initially my problem was an incorrect hostname. After fixing that I got the error because of SNI (I'm guesing) which is apparently fixed in the current develop branch.

Perhaps this change fixed it? https://github.com/sabnzbd/sabnzbd/comm ... a03b72bfb6
Changes to the SSL socket context in Python can be global in some cases, perhaps that's the difference. Beyond that I'm not sure... the upgrade did clear my sabnzbd queue so I had to re-add the nzb's but I don't see how that could make a difference. The URL hasn't changed.

Post by **shypike** » January 15th, 2015, 11:26 am

I need to look into this.
I can tell you now that NNTP traffic doesn't validate certificates.
HTTPS is another thing, there SABnzbd uses urllib2 calls.
Python 2.7's urllib2 can optionally verify certificates, but only when you offer it a store of root certificates.
SABnzbd doesn't do the latter (yet).

The SSL type choice has no meaning in this problem, I think, because it only applies to NNTP.

WoLpH · Post by **WoLpH** » January 15th, 2015, 1:44 pm

If that's the case than I wonder why the development version of sabnzbd does download nzb's from this host while the master and 0.7.x branch didn't. Note that I cannot do a full test here, it did download the nzb's but it's not actually downloading through NNTP due to different SSL issues

Or at least, that appears to be the problem.

Support Forum

[SOLVED] No response from indexer (Python 2.7.9 and FreeBSD)

Re: No response from indexer

Re: No response from indexer

Re: No response from indexer

Re: No response from indexer

Re: No response from indexer

Re: No response from indexer

Re: No response from indexer

Re: No response from indexer

Re: No response from indexer

Re: [SOLVED] No response from indexer (Python 2.7.9 and Free

Re: [SOLVED] No response from indexer (Python 2.7.9 and Free

Re: [SOLVED] No response from indexer (Python 2.7.9 and Free

Re: [SOLVED] No response from indexer (Python 2.7.9 and Free

Re: [SOLVED] No response from indexer (Python 2.7.9 and Free

Re: [SOLVED] No response from indexer (Python 2.7.9 and Free