test nzb before download

[sailmakered]

Version: 0.5.0 Final
OS: Vista
Install-type: Windows Installer
Skin (if applicable): Plush
Firewall Software: Vista
Are you using IPV6? no
Is the issue reproducible? n/a


Great program!!

Anyone use anything to test nzb file part completion before download?  Usenetserver.com claims 600 day retention but I seem to get 400 day files with a lot of the files missing 1 or 2 parts.  Too many for the pars to fix.  Any ideas?

[shypike]

You can try http://www.newsgroupdownloaders.com/nzbval/
I'm not familiar with it and you should be able to google some more.
We do have plans to add pre-download verification at some time in the future.

[sailmakered]

Thanks

Tried it....really ssslllloooowwwww.

Sounds like a great thing to add to sab.

[shypike]

Slowness can be caused by the server as well.
Did you check an NZB with lots of missing articles?

[sailmakered]

I did.

Just tried an nzb I knew was complete.  8.839GB (150 rar x 66 parts) in about 10 minutes.  Program didn't max out my connection or cpu.  Seemed slower than downloading headers in xnews but sometimes those headers remain but the file doesn't.

[sander]

We do have plans to add pre-download verification at some time in the future.

I've written a python script that does pre-download verification by STAT-ing all articles in a given NZB. It will tell the complete-ness of the post on your server.
It works, but can be a bit slow (about 10 minutes for the check of a 4.5 GB post), probably because the script is non/single-threaded: to check a 4.5GB post, this means around 13.000 stat-send/expect commands in sequential order. Apparently, one thread can do 20 STAT commands per second.
So I guess multi-threading can bring speed improvement.

The script is here: http://www.appelboor.com/sander/nzb-checker.py

Code: Select all

Usage:
nzb-checker.py <NZBfilename> <newsserver>
nzb-checker.py <NZBfilename> <newsserver> <newsserver-username> <newsserver-pass>

So, for example:
python nzb-checker.py toy71MB.nzb newszilla6.xs4all.nl
python nzb-checker.py toy71MB.nzb newsreader3.eweka.nl myname secretpass

Code: Select all

ubuntu@ubuntu:~/nzb-checker$ time python nzb-checker.py toy.nzb newszilla6.xs4all.nl
number of arguments: 2
toy.nzb newszilla6.xs4all.nl  
201 dreader33.news.xs4all.nl NNRP Service Ready - [email protected] (no posting).

NNTP connection OK

Let's start checking ...
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

total artciles in NZB 200 , total available on newsserver 133
Completeness on server is 66.50 %
size of (original) post is 71 Mbytes
age of post (in days): 3

real	0m7.453s
user	0m0.112s
sys	0m0.056s
ubuntu@ubuntu:~/nzb-checker$ 

[sander]

Followup:

I've now written a mutli-threaded nzb-checker: checking a NZB for a 4.5GB post takes around 25 seconds, when using 30 threads (=connections to the newsserver)

Let me know if you're interested in the python script.

Code: Select all

ubuntu@ubuntu:~/nzb-checker$ time python mijn-nzb-draadje.py  --nzb=young-sept17-4.5GB.nzb --server=reader.ipv6.xsnews.nl --thread=30 --user=ipv6-blabla --pass=blabla
Start reading NZB file ...
... reading NZB file completed
Wait until threads are done ...
... threads are done


Articles: 12874
OK: 12874
Not OK: 0
Connection error: 0
Completeness on server is 100.00 %

real	0m25.179s
user	0m5.100s
sys	0m1.164s
ubuntu@ubuntu:~/nzb-checker$

[inpheaux]

Holy crap great job Sander!

I wonder how hard this would be to implement as a pre-processing script?

[sander]

Holy crap great job Sander!

LOL!

I wonder how hard this would be to implement as a pre-processing script?

I guess not too hard; it should use the SABnzbd parameters (newsserver etc), and it could best be run when no other downloads are running as it uses the same newsserver connections.

It's here http://www.appelboor.com/sander/nzb-checker-threaded.py

The script works great for sunny day scenario's and it does some checks on successful connections, but it is not yet good at handling error situations / raindy day scenario's.

[phyber]

Here, have a small hack to make this support SSL connections.
Works fine for me in the few tests I've done. YMMV.

Code: Select all

--- nzb-checker-threaded.py	2011-09-19 21:54:22.000000000 +0100
+++ ssl-checker.py	2011-02-10 14:13:36.000000000 +0000
@@ -1,3 +1,4 @@
+#!/usr/bin/python
 import Queue
 from threading import Thread
 import time
@@ -36,7 +37,18 @@
 	# logon to Newsserver
 	if debug: print "trying to connect to newsserver ..."
 	try:
-		tn = telnetlib.Telnet(newsserver,119,5)
+		if newsssl == True:
+			import socket
+			import ssl
+			tn = telnetlib.Telnet()
+			tn.host = newsserver
+			tn.port = 563
+			tn.timeout = 5
+			tn.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+			tn.sock = ssl.wrap_socket(tn.sock)
+			tn.sock.connect((tn.host, tn.port))
+		else:
+			tn = telnetlib.Telnet(newsserver,119,5)
 		waarde = tn.read_until("\n",60)
 
 	except:
@@ -101,7 +113,7 @@
 
 
 def printUsage():
-	print "Usage: python usage-checker.py  --nzb=thisismynzb.nzb --server=newszilla6.xs4all.nl [--thread=4] [--user=newsservername] [--pass=newsserverpassword] [--debug=1]"
+	print "Usage: python usage-checker.py  --nzb=thisismynzb.nzb --server=newszilla6.xs4all.nl [--thread=4] [--user=newsservername] [--pass=newsserverpassword] [--ssl=1] [--debug=1]"
 
 ###########################################################################################################################
 ###########################################################################################################################
@@ -111,7 +123,7 @@
 
 
 try:
-        opts, args = getopt.getopt(sys.argv[1:], "n:s:u:p:t:d",['nzb=','server=','user=','pass=','threads=','debug='])
+	opts, args = getopt.getopt(sys.argv[1:], "n:s:u:p:t:dS",['nzb=','server=','user=','pass=','threads=','debug=','ssl='])
 except:
 	printUsage()
 	sys.exit()	
@@ -119,6 +131,7 @@
 
 # defaults:
 nzbfilename = newsserver = newsname = newspass = ''
+newsssl = False
 debug = False
 num_worker_threads = 1
 
@@ -133,6 +146,8 @@
 		newspass = arg
 	elif opt in ('-t', '--threads'):
 		num_worker_threads = int(arg)
+	elif opt in ('-S', '--ssl'):
+		newsssl = True
 	elif opt in ('-d', '--debug'):
 		if int(arg)>0:
 			debug = True

[sander]

@phyper: do you use my script? Does it work?

[phyber]

Yes, this is a patch to your script. It should apply cleanly to the download you have at http://www.appelboor.com/sander/nzb-checker-threaded.py
I tested it on a few NZBs I had around and they all seemed to check out just fine. There is a slight delay after logging in with SSL mode compared to the non-SSL mode, but that might just be giganews being weird. I don't have another SSL news server to test it against.

[sander]

The script is now unstoppable: CTRL-C does not work. Do you know how to get that working? I tried the stuff below, but that doesn't work :-(


try:
workqueue.join()
except KeyboardInterrupt:
raise
except:
print "this is the except"

[phyber]

It stops responding to ^C since it seems there is a deadlock somewhere, which is a shame. I couldn't work out where it was so I wrote a new script that does exactly what this script does, except it supports SSL and doesn't hang when using it. It also isn't threaded yet, so it's quite a bit slower.
It still needs a bit of cleaning up, but I'll hopefully be back to leave it here for people to use soon.

[warthog]

Regarding phyber's patch, I have some familiarity with python, but I'm now a little lost. Is there a way to apply the patch directly with python or some other utility in a *nix environment or does the code need to be added manually to the script from sander?

Figured it out. For anyone else who may be curious take phyber's code and paste into a file named yourfilename.diff (I used checker-patch.diff)

Code: Select all

patch nzb-checker-threaded.py checker-patch.diff