SABnzbd Forums

Posted: **December 30th, 2014, 11:59 am**

Hi,

I seem to be having a strange problem. I keep getting these messages:

2014-12-30 17:49:44,374::INFO::[urlgrabber:116] Grabbing URL https://www.usenet-crawler.com/getnzb/xxxx
2014-12-30 17:50:06,475::DEBUG::[urlgrabber:372] No response from indexer, retry after 60 sec
2014-12-30 17:50:06,475::INFO::[urlgrabber:199] Retry URL https://www.usenet-crawler.com/getnzb/xxxx

These url's contain authentication data. A manual curl -vs -o on the command line works just fine, also works from a non authenticated browser on a different machine. I tried adding the url manually and replacing the dns with a static ip but no luck.

System:

Freebsd 10.0
Sabnzbd: 0.7.20
Python: 2.7.9

Raw curl from the same machine, same account:

Code: Select all

curl -vs -o mynzb.nzb "https://www.usenet-crawler.com/getnzb/dxx" | grep Content-Length 
* Hostname was NOT found in DNS cache
*   Trying 46.19.141.134...
* Connected to http://www.usenet-crawler.com (46.19.141.134) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /usr/local/share/certs/ca-root-nss.crt
  CApath: none
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server key exchange (12):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
* 	 subject: OU=Domain Control Validated; OU=PositiveSSL; CN=www.usenet-crawler.com
* 	 start date: 2013-12-12 00:00:00 GMT
* 	 expire date: 2015-12-12 23:59:59 GMT
* 	 subjectAltName: http://www.usenet-crawler.com matched
* 	 issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=PositiveSSL CA 2
* 	 SSL certificate verify ok.
> GET /getnzb/xxx HTTP/1.1
> User-Agent: curl/7.39.0
> Host: http://www.usenet-crawler.com
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx/1.6.0
< Date: Tue, 30 Dec 2014 16:34:52 GMT
< Content-Type: application/x-nzb
< Content-Length: 240259
< Connection: keep-alive
< X-Powered-By: PHP/5.6.4
< Set-Cookie: PHPSESSID=x; path=/
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< X-DNZB-Name:x
< X-DNZB-Category: TV > HD
< X-DNZB-Details: https://www.usenet-crawler.com/details/xx
< X-DNZB-NFO: 
< X-DNZB-Failure: https://www.usenet-crawler.com/getnzb/xx
< X-DNZB-ProperName: x
< X-DNZB-EpisodeName: x
< X-DNZB-MoreInfo: http://www.tvrage.com/shows/id-33829
< Content-Disposition: attachment; filename="xx"
< 
{ [data not shown]
* Connection #0 to host http://www.usenet-crawler.com left intact

Anyone have a suggestion to help me along?

Posted: **December 30th, 2014, 12:18 pm**

Seems a bit related to (but not same as) http://forums.sabnzbd.org/viewtopic.php ... 082#p98080

I then said:

Code: Select all

So I added a debug line (logging.debug("SJ: I got in the except due to %s", sys.exc_info()[0])) to my sabnzbd/urlgrabber.py,

So can you do that in sabnzbd/urlgrabber.py around line 140 (and as you run freebsd I'm sure you can do that! ;-) ):

Original code:

Code: Select all

                try:
                    fn, header = opener.retrieve(url)
                except:
                    fn = None

change to:

Code: Select all

                try:
                    fn, header = opener.retrieve(url)
                    logging.debug("SJ: fn %s and header %s", fn, header)
                except:
                    import sys
                    logging.debug("SJ: I got in the except due to %s", sys.exc_info()[0])
                    fn = None

Run again, and post the output here.

Posted: **December 30th, 2014, 12:34 pm**

Thanks for the quick reply. I had seen that topic, thought it was a slightly different issue. I did check for the duplicate content length (wasn't there). I tried changing the url grabber. This crashes on the exception handling.

URLGRABBER CRASHED Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/sabnzbd/urlgrabber.py", line 135, in run logging.debug("SJ: I got in the except due to %s", sys.exc_info()[0]) NameError: global name 'sys' is not defined

I only dabbled in python while writing a post processing script. So not quite sure how to tackle that.

======

Spoke too soon, I do know how to tackle that

. I added import sys to the header (or what you may call it in python

).

Grabbing URL https://www.usenet-crawler.com/getnzb/xx
2014-12-30 18:32:48,429::DEBUG::[urlgrabber:136] SJ: I got in the except due to <type 'exceptions.IOError'>
2014-12-30 18:32:48,429::DEBUG::[urlgrabber:375] No response from indexer, retry after 60 sec
2014-12-30 18:32:48,429::INFO::[urlgrabber:202] Retry URL https://www.usenet-crawler.com/getnzb/xxx

IOError sort of matches with the no response.

Posted: **December 30th, 2014, 12:47 pm**

Ah, "import sys" ... sorry, I'll add that to my post.

So: "exceptions.IOError". No blocking firewall nor proxy?

Let's do it the other way round: a short piece of code. FIll out your own secret url and run this:

Code: Select all

import urllib

url = 'https://www.usenet-crawler.com/getnzb/xxx'

opener = urllib.FancyURLopener({})
fn, header = opener.retrieve(url)
print "\nheader is:\n", header
print "\nFirst 5 lines from webserver response:"
with open(fn) as myfile:
	print myfile.read(50)

What is the result?

Posted: **December 30th, 2014, 12:53 pm**

There we go:

Code: Select all

Traceback (most recent call last):
  File "test.py", line 7, in <module>
    fn, header = opener.retrieve(url)
  File "/usr/local/lib/python2.7/urllib.py", line 245, in retrieve
    fp = self.open(url, data)
  File "/usr/local/lib/python2.7/urllib.py", line 213, in open
    return getattr(self, name)(url)
  File "/usr/local/lib/python2.7/urllib.py", line 443, in open_https
    h.endheaders(data)
  File "/usr/local/lib/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python2.7/httplib.py", line 850, in _send_output
    self.send(msg)
  File "/usr/local/lib/python2.7/httplib.py", line 812, in send
    self.connect()
  File "/usr/local/lib/python2.7/httplib.py", line 1212, in connect
    server_hostname=server_hostname)
  File "/usr/local/lib/python2.7/ssl.py", line 350, in wrap_socket
    _context=self)
  File "/usr/local/lib/python2.7/ssl.py", line 566, in __init__
    self.do_handshake()
  File "/usr/local/lib/python2.7/ssl.py", line 788, in do_handshake
    self._sslobj.do_handshake()
IOError: [Errno socket error] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

[SSL: CERTIFICATE_VERIFY_FAILED] Interesting. Could this be one of those new ssl certificates?

BTW: Your help is much appreciated

Posted: **December 30th, 2014, 12:57 pm**

I tried it without the https:

Code: Select all

2014-12-30 18:55:49,756::INFO::[urlgrabber:117] Grabbing URL http://www.usenet-crawler.com/getnzb/xx
2014-12-30 18:55:49,872::DEBUG::[urlgrabber:136] SJ: I got in the except due to <type 'exceptions.IOError'>
2014-12-30 18:55:49,872::DEBUG::[urlgrabber:375] No response from indexer, retry after 60 sec
2014-12-30 18:55:49,873::INFO::[urlgrabber:202] Retry URL http://www.usenet-crawler.com/getnzb/xx

With test.py

Code: Select all

python test.py
Traceback (most recent call last):
  File "test.py", line 7, in <module>
    fn, header = opener.retrieve(url)
  File "/usr/local/lib/python2.7/urllib.py", line 245, in retrieve
    fp = self.open(url, data)
  File "/usr/local/lib/python2.7/urllib.py", line 213, in open
    return getattr(self, name)(url)
  File "/usr/local/lib/python2.7/urllib.py", line 350, in open_http
    h.endheaders(data)
  File "/usr/local/lib/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python2.7/httplib.py", line 850, in _send_output
    self.send(msg)
  File "/usr/local/lib/python2.7/httplib.py", line 812, in send
    self.connect()
  File "/usr/local/lib/python2.7/httplib.py", line 793, in connect
    self.timeout, self.source_address)
  File "/usr/local/lib/python2.7/socket.py", line 571, in create_connection
    raise err
IOError: [Errno socket error] [Errno 60] Operation timed out

This works in a browser, i suspect that redirects to https though because the download link still shows https

Posted: **December 30th, 2014, 1:01 pm**

Please put your program output in CODE blocks: easier to read. I've done it for your other posts

Posted: **December 30th, 2014, 1:07 pm**

There is something strange/wrong with your system setup (python? SSL?), but not SABnzbd:

With my own usenet-crawler account, everything works, both in SAB and the test script, both with http and https.

So if you use http://www.google.com resp https://www.google.com as url in the script (and SAB?), what do you get?

Posted: **December 30th, 2014, 1:11 pm**

I read that by default ssl shouldn't be validated using urllib. It does however try to do that. I am trying to figure out how

a) python does ssl validation
b) to temporarily disable that

Weird that it does work on your machine. Must be a FreeBSD Python specific thing.

Good suggestion about the google url. Tried that. Same error.

Code: Select all

Traceback (most recent call last):
  File "./test.py", line 8, in <module>
    fn, header = opener.retrieve(url)
  File "/usr/local/lib/python2.7/urllib.py", line 245, in retrieve
    fp = self.open(url, data)
  File "/usr/local/lib/python2.7/urllib.py", line 213, in open
    return getattr(self, name)(url)
  File "/usr/local/lib/python2.7/urllib.py", line 443, in open_https
    h.endheaders(data)
  File "/usr/local/lib/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python2.7/httplib.py", line 850, in _send_output
    self.send(msg)
  File "/usr/local/lib/python2.7/httplib.py", line 812, in send
    self.connect()
  File "/usr/local/lib/python2.7/httplib.py", line 1212, in connect
    server_hostname=server_hostname)
  File "/usr/local/lib/python2.7/ssl.py", line 350, in wrap_socket
    _context=self)
  File "/usr/local/lib/python2.7/ssl.py", line 566, in __init__
    self.do_handshake()
  File "/usr/local/lib/python2.7/ssl.py", line 788, in do_handshake
    self._sslobj.do_handshake()
IOError: [Errno socket error] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

It is even highlighted on the documentation that ssl isn't verified:

https://docs.python.org/2/library/urllib.html

Warning When opening HTTPS URLs, it does not attempt to validate the server certificate. Use at your own risk!

Posted: **December 30th, 2014, 1:27 pm**

It is probably due to openssl:

Code: Select all

openssl s_client -connect www.google.com:443
CONNECTED(00000003)
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
   i:/C=US/O=Google Inc/CN=Google Internet Authority G2
 1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
   i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
   i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEdjCCA16gAwIBAgIIMWlIfN/J8ZwwDQYJKoZIhvcNAQEFBQAwSTELMAkGA1UE
BhMCVVMxEzARBgNVBAoTCkdvb2dsZSBJbmMxJTAjBgNVBAMTHEdvb2dsZSBJbnRl
cm5ldCBBdXRob3JpdHkgRzIwHhcNMTQxMjEwMTIwNTM5WhcNMTUwMzEwMDAwMDAw
WjBoMQswCQYDVQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwN
TW91bnRhaW4gVmlldzETMBEGA1UECgwKR29vZ2xlIEluYzEXMBUGA1UEAwwOd3d3
Lmdvb2dsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCvpWri
+zfXtX8H7wt4g5ANjbeSCWJlBOzlVOvtHPRB2SjlPmUjRg4YVZHMcrkaxSZ+Nhgd
UlhjspFpcXU2ZsAM0/jeeKFX4LhcnFop61vWFfOJcJNrvDJsk8L/YTsQlDypjOxN
AmM781SK2n5TUCABJu4ASqdmIAx0GYj60ZeTkW7Ym50jvHtCKVdhcMqfYXNR3SCy
NyhGbBNzSqCje8PTlq+E7PNt+IbyMrs7PYVowHRLeUz4Rp4bzkOL9Z4HcgRDg88z
5hTeRyLKt22trbFHPe+R1lusr93kkXiUDkkcvibeXXACA+OOYI4FulaLkNXyZckw
9Vx58ATNXy0660znAgMBAAGjggFBMIIBPTAdBgNVHSUEFjAUBggrBgEFBQcDAQYI
KwYBBQUHAwIwGQYDVR0RBBIwEIIOd3d3Lmdvb2dsZS5jb20waAYIKwYBBQUHAQEE
XDBaMCsGCCsGAQUFBzAChh9odHRwOi8vcGtpLmdvb2dsZS5jb20vR0lBRzIuY3J0
MCsGCCsGAQUFBzABhh9odHRwOi8vY2xpZW50czEuZ29vZ2xlLmNvbS9vY3NwMB0G
A1UdDgQWBBSen657biCKaCLtgTbal15QKnOjwzAMBgNVHRMBAf8EAjAAMB8GA1Ud
IwQYMBaAFErdBhYbvPZotXb1gba7Yhq6WoEvMBcGA1UdIAQQMA4wDAYKKwYBBAHW
eQIFATAwBgNVHR8EKTAnMCWgI6Ahhh9odHRwOi8vcGtpLmdvb2dsZS5jb20vR0lB
RzIuY3JsMA0GCSqGSIb3DQEBBQUAA4IBAQBgW7kim/6TobVoWe4qso/ccBBOuwXh
scYejoxpMpgVB+L5zhRFWjG6WWb9mlsOQeZhV5mb/zl5lt/pgFx7WqJ8+CD+ovDz
uLltBy2OQBtHN01cw+Ue6OeUZ3nqxEFEmbDoY4KfxqIFcfWS/TFUDdij4Ndpik2Q
D/el30mOJuKli5oVGJMdLy15rMZxTyE4xTq0RTceGP/vN9e1i0NI7FEGFxD3C/Ea
r1LJ6X0XFv89auVu7v9vBtzFl5djYiRJ2Q7vVIDFVO1AY+GtLjNQrUZajlxHuFGd
YAAxjtPOaESK031SAbrSvynC8dBkkLM4MVNBDyJMgS6KWp5S/YoBMwBD
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
issuer=/C=US/O=Google Inc/CN=Google Internet Authority G2
---
No client certificate CA names sent
---
SSL handshake has read 3719 bytes and written 435 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 4A54533F8760A90BB7ABCE87ECE198673A5CF550C8C243C45EE95B574FFD249E
    Session-ID-ctx: 
    Master-Key: E4781D4B95962076FA2794230B39C7400A6017A1047E1C0C9B6F28670E3E19E0859064AA338C1A46566ADB0F2FDB5880
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 100800 (seconds)
    TLS session ticket:
    0000 - ed 3a 00 fa ae 3e e7 05-80 98 8c c4 71 d2 eb 96   .:...>......q...
    0010 - 88 ea 5e 29 37 e9 a6 a5-08 3b ea 3e 1c 15 f3 45   ..^)7....;.>...E
    0020 - 77 6f 79 ee 4c a5 4a 2b-0d e1 4d 70 da b2 0f e4   woy.L.J+..Mp....
    0030 - d2 ea 1e a6 dd bf 29 48-c2 16 2b 66 6b ee 4b a9   ......)H..+fk.K.
    0040 - b9 25 e3 7b c1 e7 37 93-35 cb e6 33 df bb 44 c0   .%.{..7.5..3..D.
    0050 - 2d d4 54 96 c0 70 0a e1-7e bd 7f f0 05 72 b4 3c   -.T..p..~....r.<
    0060 - 99 a0 c1 cc af 26 d6 6a-32 a1 1a ae ed 37 e7 35   .....&.j2....7.5
    0070 - f0 50 be ba 25 57 03 18-f9 83 ef c1 85 c3 15 95   .P..%W..........
    0080 - 84 66 83 d3 76 db 69 9e-04 c5 ee 7a 45 07 18 3f   .f..v.i....zE..?
    0090 - fa ee df d1 80 88 2d 2f-07 5b 28 92 fc 6a 0d 31   ......-/.[(..j.1
    00a0 - d3 6c a6 8e                                       .l..

    Start Time: 1419963971
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---

Posted: **December 30th, 2014, 2:22 pm**

lang/python27: Update to 2.7.9

The FreeBSD Python team welcomes Python 2.7.9 to the Ports tree!

Announcement: https://www.python.org/downloads/release/python-279/
Changelog: http://hg.python.org/cpython/raw-file/v2.7.9/Misc/NEWS

Important changes for this release:
- The entirety of Python 3.4's ssl module has been backported for Python
2.7.9. See PEP 466 for justification.
- HTTPS certificate validation using the system's certificate store is
now enabled by default. See PEP 476 for details.
- SSLv3 has been disabled by default in httplib and its reverse
dependencies due to the POODLE attack.
- The ensurepip module module has been backported, which provides the
pip package manager in every Python 2.7 installation. See PEP 477.

- While updating, try to improve the NIS detection for FreeBSD

PR: 193650
Differential Revision: https://reviews.freebsd.org/D1323
Reviewed by: koobs
With hat: python

This is the problem on FreeBSD 10.0 with Python2.7.

Code: Select all

SSLv3 has been disabled by default in httplib and its reverse
  dependencies due to the POODLE attack.

Now how to fix this... For now I will try and revert to 2.7.8

UPDATE: Actually I think this will affect everyone using Sab and Python 2.7.9.

Posted: **December 30th, 2014, 2:29 pm**

But disabling SSLv3 to avoid POODLE is good, isn't it? So circumventing that is not so good.

Via https://www.tinfoilsecurity.com/poodle I checked https://www.usenet-crawler.com/, and https://www.tinfoilsecurity.com/poodle says usenet-crawler is ... vulnerable ... :-(

It looks like your server supports SSLv3 with vulnerable ciphers. Many servers still do so today in order to support clients on Windows XP or other older systems, but you should update your cipher list to prefer safe RC4 ciphers, ideally removing any CBC ciphers. See here for a ton of information on patching your clients and servers. Learn more.

Vulnerable Ciphers Still Supported:
TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_CBC_SHA

I checked, and I have got python 2.7.6. So: still accepting SSLv3, and thus vulnerable, right?

Posted: **December 30th, 2014, 3:01 pm**

Jep still vulnerable. I don't really want to downgrade, but i do want to use sabnzbd.... Haven't figured out yet how to do that.

Posted: **December 30th, 2014, 3:23 pm**

I managed to downgrade to Python 2.7.8 and this "solves" the problem. Sabnzbd is able to download nzb's from usenet-crawler once again. However it is still vulnerable to the POODLE attack. (I even believe i didn't check ssl certs at all)

These attacks do require a bit of effort on the attacking side. So no risk of being hacked directly. (unless someone crafts a malicious nzb with a mit attack and you execute whatever they have you download)

While debugging this I also found that flexget has the same problem (which is due to the underlying python library).

What still bothers me though is that www.google.com:443 also didn't work. When checking that with openssl it supports both tls1.2 and sslv3. Google supports fallback to TLS... This suggests to me that fixing this might be a bit harder than updating an ssl certificate and setup. This however is not my area of expertise.

Tommorrow I will see if i can contact usenet-crawler about their SSL cert.

Posted: **December 30th, 2014, 4:09 pm**

Useful thread!

In the curl command you posted: can you see that SSLv3 is used or not?

SABnzbd Forums

[SOLVED] No response from indexer (Python 2.7.9 and FreeBSD)

[SOLVED] No response from indexer (Python 2.7.9 and FreeBSD)

Re: No response from indexer

Re: No response from indexer

Re: No response from indexer

Re: No response from indexer

Re: No response from indexer

Re: No response from indexer

Re: No response from indexer

Re: No response from indexer

Re: No response from indexer

Re: No response from indexer

Re: No response from indexer

Re: No response from indexer

Re: No response from indexer

Re: No response from indexer