Page 1 of 4
test nzb before download
Posted: September 16th, 2010, 9:12 pm
by sailmakered
Version: 0.5.0 Final
OS: Vista
Install-type: Windows Installer
Skin (if applicable): Plush
Firewall Software: Vista
Are you using IPV6? no
Is the issue reproducible? n/a
Great program!!
Anyone use anything to test nzb file part completion before download? Usenetserver.com claims 600 day retention but I seem to get 400 day files with a lot of the files missing 1 or 2 parts. Too many for the pars to fix. Any ideas?
Re: test nzb before download
Posted: September 17th, 2010, 1:42 am
by shypike
You can try
http://www.newsgroupdownloaders.com/nzbval/
I'm not familiar with it and you should be able to google some more.
We do have plans to add pre-download verification at some time in the future.
Re: test nzb before download
Posted: September 17th, 2010, 12:53 pm
by sailmakered
Thanks
Tried it....really ssslllloooowwwww.
Sounds like a great thing to add to sab.
Re: test nzb before download
Posted: September 17th, 2010, 3:36 pm
by shypike
Slowness can be caused by the server as well.
Did you check an NZB with lots of missing articles?
Re: test nzb before download
Posted: September 18th, 2010, 11:35 am
by sailmakered
I did.
Just tried an nzb I knew was complete. 8.839GB (150 rar x 66 parts) in about 10 minutes. Program didn't max out my connection or cpu. Seemed slower than downloading headers in xnews but sometimes those headers remain but the file doesn't.
Re: test nzb before download
Posted: September 19th, 2010, 3:45 am
by sander
shypike wrote:
We do have plans to add pre-download verification at some time in the future.
I've written a python script that does pre-download verification by STAT-ing all articles in a given NZB. It will tell the complete-ness of the post on your server.
It works, but can be a bit slow (about 10 minutes for the check of a 4.5 GB post), probably because the script is non/single-threaded: to check a 4.5GB post, this means around 13.000 stat-send/expect commands in sequential order. Apparently, one thread can do 20 STAT commands per second.
So I guess multi-threading can bring speed improvement.
The script is here:
http://www.appelboor.com/sander/nzb-checker.py
Code: Select all
Usage:
nzb-checker.py <NZBfilename> <newsserver>
nzb-checker.py <NZBfilename> <newsserver> <newsserver-username> <newsserver-pass>
So, for example:
python nzb-checker.py toy71MB.nzb newszilla6.xs4all.nl
python nzb-checker.py toy71MB.nzb newsreader3.eweka.nl myname secretpass
Code: Select all
ubuntu@ubuntu:~/nzb-checker$ time python nzb-checker.py toy.nzb newszilla6.xs4all.nl
number of arguments: 2
toy.nzb newszilla6.xs4all.nl
201 dreader33.news.xs4all.nl NNRP Service Ready - [email protected] (no posting).
NNTP connection OK
Let's start checking ...
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
total artciles in NZB 200 , total available on newsserver 133
Completeness on server is 66.50 %
size of (original) post is 71 Mbytes
age of post (in days): 3
real 0m7.453s
user 0m0.112s
sys 0m0.056s
ubuntu@ubuntu:~/nzb-checker$
Re: test nzb before download
Posted: September 19th, 2010, 7:58 am
by sander
Followup:
I've now written a mutli-threaded nzb-checker: checking a NZB for a 4.5GB post takes around 25 seconds, when using 30 threads (=connections to the newsserver)
Let me know if you're interested in the python script.
Code: Select all
ubuntu@ubuntu:~/nzb-checker$ time python mijn-nzb-draadje.py --nzb=young-sept17-4.5GB.nzb --server=reader.ipv6.xsnews.nl --thread=30 --user=ipv6-blabla --pass=blabla
Start reading NZB file ...
... reading NZB file completed
Wait until threads are done ...
... threads are done
Articles: 12874
OK: 12874
Not OK: 0
Connection error: 0
Completeness on server is 100.00 %
real 0m25.179s
user 0m5.100s
sys 0m1.164s
ubuntu@ubuntu:~/nzb-checker$
Re: test nzb before download
Posted: September 19th, 2010, 3:03 pm
by inpheaux
Holy crap great job Sander!
I wonder how hard this would be to implement as a pre-processing script?
Re: test nzb before download
Posted: September 19th, 2010, 4:17 pm
by sander
inpheaux wrote:
Holy crap great job Sander!
LOL!
inpheaux wrote:
I wonder how hard this would be to implement as a pre-processing script?
I guess not too hard; it should use the SABnzbd parameters (newsserver etc), and it could best be run when no other downloads are running as it uses the same newsserver connections.
It's here
http://www.appelboor.com/sander/nzb-checker-threaded.py
The script works great for sunny day scenario's and it does some checks on successful connections, but it is not yet good at handling error situations / raindy day scenario's.
Re: test nzb before download
Posted: February 10th, 2011, 9:17 am
by phyber
Here, have a small hack to make this support SSL connections.
Works fine for me in the few tests I've done. YMMV.
Code: Select all
--- nzb-checker-threaded.py 2011-09-19 21:54:22.000000000 +0100
+++ ssl-checker.py 2011-02-10 14:13:36.000000000 +0000
@@ -1,3 +1,4 @@
+#!/usr/bin/python
import Queue
from threading import Thread
import time
@@ -36,7 +37,18 @@
# logon to Newsserver
if debug: print "trying to connect to newsserver ..."
try:
- tn = telnetlib.Telnet(newsserver,119,5)
+ if newsssl == True:
+ import socket
+ import ssl
+ tn = telnetlib.Telnet()
+ tn.host = newsserver
+ tn.port = 563
+ tn.timeout = 5
+ tn.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ tn.sock = ssl.wrap_socket(tn.sock)
+ tn.sock.connect((tn.host, tn.port))
+ else:
+ tn = telnetlib.Telnet(newsserver,119,5)
waarde = tn.read_until("\n",60)
except:
@@ -101,7 +113,7 @@
def printUsage():
- print "Usage: python usage-checker.py --nzb=thisismynzb.nzb --server=newszilla6.xs4all.nl [--thread=4] [--user=newsservername] [--pass=newsserverpassword] [--debug=1]"
+ print "Usage: python usage-checker.py --nzb=thisismynzb.nzb --server=newszilla6.xs4all.nl [--thread=4] [--user=newsservername] [--pass=newsserverpassword] [--ssl=1] [--debug=1]"
###########################################################################################################################
###########################################################################################################################
@@ -111,7 +123,7 @@
try:
- opts, args = getopt.getopt(sys.argv[1:], "n:s:u:p:t:d",['nzb=','server=','user=','pass=','threads=','debug='])
+ opts, args = getopt.getopt(sys.argv[1:], "n:s:u:p:t:dS",['nzb=','server=','user=','pass=','threads=','debug=','ssl='])
except:
printUsage()
sys.exit()
@@ -119,6 +131,7 @@
# defaults:
nzbfilename = newsserver = newsname = newspass = ''
+newsssl = False
debug = False
num_worker_threads = 1
@@ -133,6 +146,8 @@
newspass = arg
elif opt in ('-t', '--threads'):
num_worker_threads = int(arg)
+ elif opt in ('-S', '--ssl'):
+ newsssl = True
elif opt in ('-d', '--debug'):
if int(arg)>0:
debug = True
Re: test nzb before download
Posted: February 10th, 2011, 9:50 am
by sander
@phyper: do you use my script? Does it work?
Re: test nzb before download
Posted: February 10th, 2011, 9:59 am
by phyber
Yes, this is a patch to your script. It should apply cleanly to the download you have at
http://www.appelboor.com/sander/nzb-checker-threaded.py
I tested it on a few NZBs I had around and they all seemed to check out just fine. There is a slight delay after logging in with SSL mode compared to the non-SSL mode, but that might just be giganews being weird. I don't have another SSL news server to test it against.
Re: test nzb before download
Posted: February 10th, 2011, 5:00 pm
by sander
The script is now unstoppable: CTRL-C does not work. Do you know how to get that working? I tried the stuff below, but that doesn't work :-(
try:
workqueue.join()
except KeyboardInterrupt:
raise
except:
print "this is the except"
Re: test nzb before download
Posted: February 11th, 2011, 8:03 pm
by phyber
It stops responding to ^C since it seems there is a deadlock somewhere, which is a shame. I couldn't work out where it was so I wrote a new script that does exactly what this script does, except it supports SSL and doesn't hang when using it. It also isn't threaded yet, so it's quite a bit slower.
It still needs a bit of cleaning up, but I'll hopefully be back to leave it here for people to use soon.
Re: test nzb before download
Posted: April 12th, 2011, 12:02 pm
by warthog
Regarding phyber's patch, I have some familiarity with python, but I'm now a little lost. Is there a way to apply the patch directly with python or some other utility in a *nix environment or does the code need to be added manually to the script from sander?
Figured it out. For anyone else who may be curious take phyber's code and paste into a file named
yourfilename.diff (I used checker-patch.diff)
Code: Select all
patch nzb-checker-threaded.py checker-patch.diff