[2870] high failure rate on rss feed from nzbs.org; "try again" doesn't work

[hjbotha]

Hi all,

I realise my subject is rather non-descriptive but I'm having trouble pinning this one down.

When I Preview my RSS feed and click on Download a bunch of times, a lot of them will fail.

My feed address is for example: http://nzbs.org/rss.php?type=2&i=myuser ... l=1&num=50

Just now, from adding 9 RSSs, 3 succeeded.

(Incidentally, the queue says "Trying to fetch NZB from http://nzbs.org/index" instead of having the full URL - is that normal?)

This wouldn't be so bad if I could hit try again, but that doesn't do any good. The "trying to fetch" bit gets readded but fails consistently thereafter.

This may be something on nzbs.org's side because it seems to be recent and I've tried earlier sabnzbd svns without any luck, but I can't figure out exactly what might cause it.

If I run sniffit against the traffic everything seems to be going perfectly smoothly - a well-formed request followed by a gzipped reply.

If anybody else wants to have a go, refer to http://nzbs.org/index.php?action=rss for the proper RSS url (should be one that includes &dl=1). I'd appreciate some feedback to let me know if it's just me or anybody else has run into the same.

Thanks

[hjbotha]

Bit more info...

Even with debug logging enabled I only get the following:
2009-10-23 17:33:02,732::INFO::[urlgrabber:152] Retry URL http://nzbs.org/index.php?action=getnzb ... &h=USERKEY
2009-10-23 17:33:02,733::INFO::[urlgrabber:128] Grabbing URL http://nzbs.org/index.php?action=getnzb ... &h=USERKEY
2009-10-23 17:33:07,974::INFO::[urlgrabber:152] Retry URL http://nzbs.org/index.php?action=getnzb ... &h=USERKEY
2009-10-23 17:33:07,975::INFO::[urlgrabber:128] Grabbing URL http://nzbs.org/index.php?action=getnzb ... &h=USERKEY
2009-10-23 17:33:13,197::INFO::[urlgrabber:152] Retry URL http://nzbs.org/index.php?action=getnzb ... &h=USERKEY
2009-10-23 17:33:13,198::INFO::[urlgrabber:128] Grabbing URL http://nzbs.org/index.php?action=getnzb ... &h=USERKEY
2009-10-23 17:33:18,404::INFO::[urlgrabber:152] Retry URL http://nzbs.org/index.php?action=getnzb ... &h=USERKEY
2009-10-23 17:33:18,404::INFO::[urlgrabber:128] Grabbing URL http://nzbs.org/index.php?action=getnzb ... &h=USERKEY

I ran wireshark on the box, saved the HTTP response to a file, hexedited it to remove the http header and gunzipped the rest. The result was a perfectly formed NZB that I could successfully import into sabnzbd manually.

Could someone add some debug logging to the sections involved in this process so we can see where it fails?

I'm running Ubuntu Linux 9.04; sabnzbd was using python 2.6 before but I've switched over to 2.5 now with no better luck.

Thanks

[hjbotha]

I commented out the following from urlgrabber.py:

opener.addheader('Accept-encoding','gzip')

And it seems to work now. I guess I'll be costing nzbs.org a little bit more bandwidth until this works again.

Still willing to test code, just let me know.

Cheers

[shypike]

I'm a bit confused.
Which SABnzbd version are you using?
The aborted URL in the queue suggests that you are using the trunk source from a few days ago.
Other than that I don't think we support nzbs.org yet, since they have their own authentication method.
(This lack of standardization is killing, each of these sites seem to think they only one.)

[hjbotha]

Hi, thanks for replying.

I'm on r2875 now, but posted the topic on 2870 (see topic)

As far as I can tell from packet sniffing, nzbs.org is returning a properly formed gzipped reply, so it's not an authentication issue. Additionally, commenting out the accept-encoding bit so we get a plaintext http response works fine.

It was working until recently, so it's entirely likely that something changed on the nzbs.org side, but I don't really know what to tell them unless I can figure out why it fails.

Cheers

[switch]

I am having trouble reproducing this issue, the failing nzb files you specify work fine on my pc. The only thing that has changed in this part of the code in recent versions is the user agent SABnzbd sends out, which required tvbinz to do some small changes however I don't believe nzbs.org have any user agent requirements. Not sure what else to add..

[shypike]

I had to add specific code for nzbs.org, because they have a very peculiar format.
You need to filter out the right http link in a large piece of HTML thay have in their "summary_detail" field.
That specific link contains the required authentication string.

I'm surprised that you get any results at all, due to the required authentication.

I will commit my changes today.

[hjbotha]

Hi again,

If you append &dl=1 to the end of your rss feed url it will have direct links to the nzbs so no need to dive into the xml.

I'm trying to reproduce it right now but nzbs.org has decided to return non-gzipped content so it's working again now. I suspect once it starts gzipping stuff again it will fail but I'll leave gzip on for now and wait and see.

Thanks for looking into this.

H

[shypike]

That's right, I had forgotten all about the &dl=1 requirement.
I just removed the regex method and instead will force &dl=1 in the feed URL, when it's missing.

Everything should work now.
BTW: it works find with the gzip header present.
Must have been a bug in the nzbs.org site.
If they cannot handle gzip, they should just ignore the header.

[switch]

As hjbotha mentioned though, the response from nzbs.org was correctly gzipped and ungzipping produced a standard nzb.

[shypike]

Does it work properly now or not?

[hjbotha]

nzbs.org has started returning plaintext content (even though "Accept-encoding: gzip" is present in the request header) so it's working at the moment.

I'll post here again if it fails and try to confirm that it's the gzipping that breaks things.

Annoying that it's inconsistent, wonder if it's to do with site load ...