RSS NZB retrieval failing

[sabuser1]

Version: 0.5.3 Final
OS: Debian Lenny i386
Install-type: Python source
Firewall Software: None
Are you using IPV6? No
Is the issue reproducible? possibly

I have a custom RSS feed that lists NZBs from nzbs.org, but the feed itself is not hosted at nzbs.org but on my own webserver.  Some NZBs fail to add, and instead display a size of 0B and an age of 14797d.  (The blurred NZB name below is the title field of the RSS feed item, and not an error message.)



I have a feeling this may be due to NZBs.org anti-hammer protection. If there are too many connections, a HTTP 503 response code is given, asking you to wait a few seconds.  If this is ignored, you are temporarily banned and all pages serve an HTTP 200 response code with a content-length of 0.  I think your code might be misinterpreting the 200 response code as a successful download, and not retrying the NZB retrieval at a later date.  The anti-hammer script uses a combination of HTTP referrer, user agent, locale, "accept" header and IP address to identify users.

I can ban myself through multiple:

Code: Select all

curl -i "http://nzbs.org/rss.php?action=search&q=web" > /dev/null

but Sabnzbd must be using different user agent, locale and accept headers since I can still add and retrieve http://nzbs.org NZBs via the web interface.  I think that this would be reproducible if you know the pertinent HTTP headers sent.  I've found out that a timeout of 5 seconds between each request to nzbs.org is a pretty much foolproof way of preventing bans -- what is the current timeout between fetching NZBs (if any)?

If I'm talking out my ass and this is caused by something more obvious, please tell me!

Thanks

[shypike]

I don't quite understand your issue.
If the feed comes from your server, where is SABnzbd supposed to get its NZB files from?

There's no mis-interpreting code 200. It means "The request was fulfilled".
So if they send that, they are fools.
They didn't bother to document their 503 code, so how should we know what it means?
BTW: there's already a 5 second delay between requests.

I'm moving this to another section, because it's not a bug.

[sabuser1]

Thanks for the quick response.

The RSS feed comes from my web server, but each 's and is to an URL to an NZB at http://nzbs.org.  It's pretty much identical to a NZBS.org RSS feed but the feed itself is just not actually hosted there.

HTTP status code 503 is a generic code for "service unavailable".  I had thought that sabnzbd might assume a 200 code explicitly implies the NZB download was successful, without checking the content length first, hence me seeing a 0B NZB in the queue when banned.

[shypike]

Code 200 with length 0 is a perfectly valid response.
Some servers do indeed send empty responses when they don't have a file.
It seems unbounded paranoia is needed to deal with all these idiosyncratic indexing sites
We'll look at this for a future release.

[sabuser1]

Thanks shypike, I appreciate it.

(FYI, nzbs.org uses http://corz.org/serv/tools/anti-hammer/ specifically.)

[lhbandit]

nzbs.org now sends the 503 header instead of just die() when you're perm banned.