test nzb before download

[paranone]

I have the same error with both my Easynews and Newsdemon accounts (both Highwinds too) using the following command (also tried with 1 or 30 threads and the us and european servers) :

Code: Select all

nzb-checker-threaded.py --nzb=<xxxxxxxx>.nzb --server=useast.newsdemon.com --thread=10 --user=<xxxxxxxxx> --pass=<xxxxxxxx> --debug=1

Error message :

Code: Select all

Wait until threads are done checking all articles, which can take a few minutes
...
response on command:
time-out talking to NNTP server, waiting for reaction, or something else wrong with reaction
command is: authinfo pass <xxxxxxxxx>

And it keeps showing time outs for the rest of the process.

It works with Astraweb though and I can use NZB Download Checker with both Easynews and Newsdemon accounts no problem.

Also login and pass is the same non complex for one of the failing accounts and Astraweb so that's not the problem (I was wondering if punctuation and other characters were supported in the login and pass).

When doing more tests with NZB's from previous downloads with my Astraweb account, I often get time outs on some articles so can't really get accurate results but I don't always get the same results as NZB Download Checker (but then it might be because of using STAT or HEAD as pobox mentioned).
I'm not familiar with Netcat but if you want me to do some tests, I could try it (I'm a Windows only guy though).

Also I don't see how I could use this with SABNZBD, the pre-queue scripts seem to only allow some kind of filtering or transformations, so would there be a way to streamline the nzb check in SABNZBD ?

Edit : I just tried connecting through Telnet with the same commands (with Windows Telnet client) : authinfo user, authinfo pass and group alt.binaries.test, it connects fine with both servers and shows "211 120000001 299324621 419324621 alt.binaries.test" for Easynews and "211 161509884 245992121 407502004 alt.binaries.test" for Newsdemon.

[shypike]

Hey guys.
Just want to let you know I'm working on this feature for SABnzbd.

[paranone]

Great news, hope to see it coming soon
I don't know if you need testers...

[sander]

@paranone:

Good that you use the debug option to get more output. Could you use one thread, and post the full output (first page or so); I want to see messages like "NNTP base connection OK"

[shypike]

I don't know if you need testers...

So far I have something that does a full article test and accept it when:
data + pars - missing > data

Unfortunately, for me it costs nearly as long as the actual download.
It looks like Giganews takes as much time to find the article as to actually send it.

It will be part of release 0.7.0.
I'll contact you when I'm confident enough about it's lack of side-effects.

[sander]

I don't know if you need testers...

So far I have something that does a full article test and accept it when:
data + pars - missing > data

Unfortunately, for me it costs nearly as long as the actual download.
It looks like Giganews takes as much time to find the article as to actually send it.

It will be part of release 0.7.0.
I'll contact you when I'm confident enough about it's lack of side-effects.

Shypike, do you use my multi-threaded nzb-checker? My single-thread tester takes a long time.

And I guess on your 100 Mbps connection the real download takes very short time.

So, a normal use case is a multi-threaded NZB-tester on a 10 - 20 Mbps line. For a 5GB download, the test will take a minute or so, the real download an hour.

[shypike]

The check uses the same code as normal downloading.
I just run the download code twice, once with 'stat' and once with 'body'.
Yes, I have a distorted view because I have a 100Mbit connection

[paranone]

Here you go Sander :

Code: Select all

trying to connect to newsserver ...
Start reading NZB file ...
* next line in NZB-file [x7]
groupname is 

* next line in NZB-file
* next line in NZB-file
* next line in NZB-file
next article line
. size is 198874
* next line in NZB-file
next article line
size is 199066
* next line in NZB-file
next article line
size is 198948
[...]
* next line in NZB-file
* next line in NZB-file
* next line in NZB-file
... reading NZB file completed
Wait until threads are done checking all articles, which can take a few minutes ...
NNTP base connection OK

trying to logon ...
command is: authinfo user [username] response on command: 
time-out talking to NNTP server, waiting for reaction, or something else wrong with reaction
command is: authinfo pass [password] response on command: 
time-out talking to NNTP server, waiting for reaction, or something else wrong with reaction
command is: group alt.binaries.test response on command: 
time-out talking to NNTP server, waiting for reaction, or something else wrong with reaction
Error, NNTP connection is not working. Username/Password needed or incorrect?

* I'm now handling 
<[articlenumber]@news.astraweb.com>
command is: group  response on command: 
time-out talking to NNTP server, waiting for reaction, or something else wrong with reaction
command is: stat <[articlenumber]@news.astraweb.com> response on command: 503 Idle timeout exceeded

I'm done handling 
<[articlenumber]@news.astraweb.com>
* I'm now handling 
<[articlenumber]@news.astraweb.com>
command is: stat <[articlenumber]@news.astraweb.com> time-out talking to NNTP server, waiting for reaction, or something else wrong with reaction
Oops, connection error
I'm done handling 
<[articlenumber]@news.astraweb.com>
[...repeating...]
... threads are done


Articles: 132
OK: 0
Not OK: 1
Connection error: 131
You have more than 5% Connection Errors, so the above information is unreliable. Try lowering your number of threads.
Completeness on server is 0.00 %

[sander]

Here you go Sander :

Code: Select all

trying to connect to newsserver ...

time-out talking to NNTP server, waiting for reaction, or something else wrong with reaction

So only time-outs? My script uses port 119. Does your newsserver respons on that port? Or maybe you're using a SSL server?

[paranone]

They offer both but I'm was using their non SSL server for this test and as I told you, I have succesfully logged into it using Windows telnet client, on port 119 and also other NZB verification programs that don't use SSL.

Here is the telnet connection log for each server, in case you can see the reason why it fails...

Astraweb (works with your script):

Code: Select all

200 news.astraweb.com NNRP Service Ready (posting ok) (yEnc enabled).
authinfo user [username]381 PASS required
authinfo pass [password]
281 Ok
group alt.binaries.test
211 285724158 121200749 406924906 alt.binaries.test
group
501 newsgroup
stat <[articlenumber]@news.astraweb.com>
223 0 <[articlenumber]@news.astraweb.com> status

Easynews (fails with your script):

Code: Select all

200 Welcome to Easynews.  Please Authenticate. (en-nntp-09.am2-A10O46)
authinfo user [username]
381 More authentication needed
authinfo pass [password]
281 Authenticated and ready
group alt.binaries.test
211 120000001 299878065 419878065 alt.binaries.test
group
411 No Such Group ""
stat <[articlenumber]@news.astraweb.com>
223 0 <[articlenumber]@news.astraweb.com>

Didn't have much time to look into it, let me know if there are other tests I can do.

[sander]

@paranone:

Found it and fixed it: for easynews "\n" is not enough for an end-of-line; it wants "\r\n".

So that's in the new code http://www.appelboor.com/sander/nzb-checker-threaded.py

Can you try it?

[paranone]

One more bug with Easynews

Code: Select all

* I'm now handling 
<[articlenumber]@news.astraweb.com>
command is: group 

response on command: 411 No Such Group "
command is: stat <[articlenumber]@news.astraweb.com>
response on command: "

I guess the response to group is misinterpreted so I get a missing article result and the more threads, the more missing articles reported.

Don't bother if you're bored with me already

@shypike
Do you do the calculation in bytes or can you actually figure how much missing parts there are from the missing articles (like if 2 consecutive articles are missing in one file it could be just one part missing but if they are at the beginning and end of the file, even if it's just 2 articles, there are 2 corrupted parts)

[sander]

@paranone:

it looks like my script has an empty 'group', which Easynews doesn't like.

Possible reason for empty 'group' variable: the group name in your NZB is not found/understood by my quick-n-dirty NZB-parser.

Can you send your NZB to surfer at telfort.nl ?
How did you generate your NZB?

FYI: I haven't got an Easynews account, so I can't verify my script against it

[paranone]

NZB is from NZBIndex and seems to be correctly formatted :

Code: Select all

<?xml version="1.0" encoding="iso-8859-1" ?>
<!DOCTYPE nzb PUBLIC "-//newzBin//DTD NZB 1.0//EN" "http://www.newzbin.com/DTD/nzb/nzb-1.0.dtd">
<!-- NZB Generated by NZBIndex -->
<nzb xmlns="http://www.newzbin.com/DTD/2003/nzb">
<file poster="[poster]" date="[date]" subject="[subject]">
<groups>
<group>alt.binaries.misc</group>
</groups>
<segments>
<segment bytes="198874" number="1">[....]

So your script didn't pick the group from the NZB but you could also prevent it from trying to enter a group if it didn't pick one and/or make sure it gets the response without splitting the line.

So what's missing to make this a full fledged newsgroup client better than SABNZBD ? I hope you'll put my name in the acknowledgments

[sander]

Ah: nzbindex creates a different format than binsearch (see below). I'll take care of that in my script.

And no, this is just a NZB-checker. My hope is that SAB will do something like itself, so that my script is not necessary anymore.


Edit: improved script on http://www.appelboor.com/sander/nzb-checker-threaded.py

Code: Select all

sander@netbook:~$ grep -i group Downloads/nzbindex.nzb 
<groups>
<group>alt.binaries.sounds.mp3.complete_cd</group>
<group>alt.binaries.sounds.mp3.easy-listening</group>
<group>alt.binaries.sounds.mp3.full_albums</group>
</groups>


sander@netbook:~$ 
sander@netbook:~$ grep -i group Downloads/binsearch.nzb | head -5
<groups><group>alt.binaries.mp3</group><group>alt.binaries.mom</group></groups>
<groups><group>alt.binaries.mp3</group><group>alt.binaries.mom</group></groups>
<groups><group>alt.binaries.mp3</group><group>alt.binaries.mom</group></groups>
<groups><group>alt.binaries.mp3</group><group>alt.binaries.mom</group></groups>
<groups><group>alt.binaries.mp3</group><group>alt.binaries.mom</group></groups>
sander@netbook:~$