NZBMegasearch, opensource NZB search aggregator

[0byte]

Dear all,

tired of searching on 5 different NZB indices for finding your content? Well, I present you a solution to this problem: NZBMegasearch, a meta-search index software.

It aggregates results from your favorite indices, it is configurable with N#wsnb-based, Spotw#b-based, and many others search sites. Already after the install you will find support for 6 free indices.
It natively supports CP and SB, retrieves trending movies and shows, gives appropriate search suggestions. Many configuration and security options are available.

It is written in Python and it works on linux, mac, win. Windows binaries are also available.

It is completely open source.
https://github.com/pillone/usntssearch

I hope you will like it,
0

[sander]

Wow ... impressive!

EDIT: It works for me (Ubuntu)

[0byte]

hey,
thanks!

I add a screenshot to make easier to comprehend what it does:
click -> http://i.imgur.com/m4A1nP2.jpg

(trends and suggestions intentionally blanked)

[Faintuk]

Loving it cant wait to test it out, any plans for adding support for headphones? also any chance of increasing the trends from 5 to 10?

[0byte]

thanks!

> Headphones

I see what I can do


> Trends

No problem, i put this in the wishlist for the next release

[sander]

When I start a search, the page says "this may take up to 20s". I guess NZBmegasearch then searches the different search engines. Is it possible to show some progress? For example: the search engines being used, or the stuff already found?

[sander]

FWIW:

In mega2.py, I changed this line:

Code: Select all

	chost = '::'   ### was: '0.0.0.0'

... and now I can access NZBmegasearch's webinterface over IPv6 (and IPv4).

You can check NZBmegasearch is listening on IPv6:

Code: Select all

$ netstat -apon | grep 5000
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6       0      0 :::5000                 :::*                    LISTEN      31010/python     off (0.00/0/0)
$ 

[sander]

BTW: I keep getting

Code: Select all

>> Failure in suggestions time data '2013-09-00' does not match format '%Y-%m-%d'

See below. Is there a way to get rid of it?

Code: Select all

$ python mega2.py


~*~ ~*~ NZBMegasearcH ~*~ ~*~
>> This is the newest version available
>> version: 0.31
>> Bootstrapping...
>> Base domain and protocol: http://192.168.1.150:5000
>> Hash table has been initialized: 30515
>> Loading modules from: /home/sander/temp/usntssearch/NZBmegasearch/SearchModules/
>> Found 8 modules
>> Importing: Newznab, aa_NZBx, ab_Findnzb, ac_NZBcc, ad_NZBclub, ae_FTDworld, af_Fanzub, ah_WMBidx
2013-05-20 19:30 MVT 4
>> Failure in suggestions time data '2013-09-00' does not match format '%Y-%m-%d'
2013-05-20 19:30 SHT 5
>> Running on port 5000

[sander]

stdout says:

Code: Select all

http://www.nzbclub.com/nzbfeed.aspx -- HTTPConnectionPool(host='www.nzbclub.com', port=80): Request timed out. (timeout=15)

So: is the "15" the timeout in seconds? If so, I guess it causes a big delay?

However: a manual wget to that URL only takes 0.053s. So what is going on? Is nzbclub refusing based on UserAgent?

Code: Select all

sander@flappie:~$ time wget http://www.nzbclub.com/nzbfeed.aspx
--2013-05-20 19:41:44--  http://www.nzbclub.com/nzbfeed.aspx
Resolving http://www.nzbclub.com (http://www.nzbclub.com)... 94.75.212.200
Connecting to http://www.nzbclub.com (http://www.nzbclub.com)|94.75.212.200|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 476 [text/html]
Saving to: ‘nzbfeed.aspx.1’

100%[=====================================================================================================>] 476         --.-K/s   in 0s      

2013-05-20 19:41:44 (35.4 MB/s) - ‘nzbfeed.aspx.1’ saved [476/476]


real	0m0.053s
user	0m0.004s
sys	0m0.004s
sander@flappie:~$

EDIT:

Update: I did a wireshark analysis, and nzbclub does send back a HTTP/XML with useful content within a second: see screendump. So why is NZBMegasearch giving the time-out message?

[sander]

Does nzbmegasearch.log tell the wait time of the results? If so: sometimes nzbclub is within 15 sec:

Code: Select all

2013-05-20 20:13:18,639 - SearchModule - INFO - TS http://www.newshost.co.za/ 0.0395300388336
2013-05-20 20:13:18,734 - SearchModule - INFO - TS https://nzb.cc 0.135137081146
2013-05-20 20:13:18,830 - SearchModule - INFO - TS https://nzbx.co 0.240167856216
2013-05-20 20:13:23,786 - SearchModule - INFO - TS  https://findnzbs.info 5.19303393364
2013-05-20 20:13:31,933 - SearchModule - INFO - TS http://www.nzbclub.com 13.3368968964
2013-05-20 20:13:31,972 - megasearch - INFO - [ubuntu] [ubuntu ] 146 61 118

... and sometimes it's not:

Code: Select all

2013-05-20 20:15:22,708 - SearchModule - INFO - TS http://www.newshost.co.za/ 0.0404541492462
2013-05-20 20:15:22,771 - SearchModule - INFO - TS https://nzb.cc 0.108345031738
2013-05-20 20:15:22,855 - SearchModule - INFO - TS https://nzbx.co 0.195661783218
2013-05-20 20:15:31,484 - SearchModule - INFO - TS  https://findnzbs.info 8.82390189171
2013-05-20 20:15:37,687 - SearchModule - CRITICAL - http://www.nzbclub.com/nzbfeed.aspx -- HTTPConnectionPool(host='www.nzbclub.com', port=80): Request timed out. (timeout=15)
2013-05-20 20:15:37,697 - megasearch - INFO - [dwdd] [dwdd ] 26 0 105

Does this help?

[0byte]

wow,

super thanks. awesome info.
No time now, reply in few hours

[0byte]

First of all thanks for the comments

> When I start a search, the page says "this may take up to 20s". I guess NZBmegasearch then searches the different search engines. Is it possible to show some progress?

Yes, correct.
Well, it would be possible. Though it would require async communication, sorting, retreiving etc
Given that is a purely aesthetic change, I am not sure I want to spend the efforts in that direction.
Maybe something can be done without excessive code restructuring. I'll see...

regarding timeouts:
sometimes indexers take time to respond due to traffic and firewall ddos rules. For an analysis of performance you have to look at the median reply time, not just single measurements.
yes, TS is the log is the response time.
Overall, no worries it is normal, it might happen with some indices.

> Failure in suggestions time data '2013-09-00' does not match format '%Y-%m-%d'

That's just a warning, you can ignore. It just means that the thetvdb.com for one show has a record with invalid date.
No harm.

> IPv6
thanks for the ipv6 tip. It will be updated in the next version.


thanks,
0

[sander]

time-out: my point is that nzbclub sends it's reply within a second, but NZBmegasearch says 13 - 15+ seconds. So ... what is going on?

Ipv6: Oh ... be careful with that; some systems don't like/understand "::" (for example IPv4-only devices, and even some modern full OSes). Better to leave it to IPv4 as default. That's safer.

[0byte]

Wait, was your wireshark log from a NZBmegasearch timeout event?

[sander]

yes