Page 1 of 3
pre-check on contents of post for unwanted content
Posted: March 30th, 2014, 12:40 pm
by sander
Hi,
I have more and more nonsense downloads with a fake *.avi and a Codec/Setup.exe, which exe contains a virus like "Packed.Win32.TDSS.2!O, Artemis!33B01D7F9BF5, PUP.Optional.OptimumInstaller.A"
My feature request: do a pre-check on the first rar (so *01.rar) for the contents of the rar file. If there is 'forbidden' file extension (like .exe), pause or abort the download. That would save time and bandwidth.
So this looks like the current function 'Action when encrypted RAR is downloaded' in Config -> Switches.
Below is an example test on a *.part01.rar file
Code: Select all
sander@flappie:~/Downloads/complete$ unrar l someDownload.part01.rar
UNRAR 5.00 beta 8 freeware Copyright (c) 1993-2013 Alexander Roshal
Archive: someDownload.part01.rar
Details: RAR 4, volume
Attributes Size Date Time Name
----------- --------- -------- ----- ----
..A.... 220448 14-03-14 14:56 Blabla.XviD-Ecf/Codec/Setup.exe
..A.... 113 14-03-14 00:22 Blabla.XviD-Ecf/If_you_get_error.txt
..A.... 736100352 19-03-14 11:51 Blabla.XviD-Ecf/Blabla.XviD-Ecf.avi
----------- --------- -------- ----- ----
736320913 volume 1 3
sander@flappie:~/Downloads/complete$
Re: pre-check on contents of post for unwanted content
Posted: March 30th, 2014, 1:37 pm
by sander
OK, proof of concept written & working: The download pauses after the first 'offending' rar file is in.
In the SAB Warning page:
2014-03-30 20:27:17,702 WARNING: WARNING:: Paused job "blabla.XviD-Ecf.part13" because of unwanted content in RAR file SJ
I've put the code into assembler.py, and I've copy-pasted from the functionality and use of check_encrypted_rar():
Three new lines at the end of this block:
Code: Select all
if check_encrypted_rar(nzo, filepath):
if cfg.pause_on_pwrar() == 1:
logging.warning(Ta('WARNING: Paused job "%s" because of encrypted RAR file'), latin1(nzo.final_name))
nzo.pause()
else:
logging.warning(Ta('WARNING: Aborted job "%s" because of encrypted RAR file'), latin1(nzo.final_name))
nzo.fail_msg = T('Aborted, encryption detected')
import sabnzbd.nzbqueue
sabnzbd.nzbqueue.NzbQueue.do.end_job(nzo)
if rar_contains_unwanted_file(nzo, filepath):
logging.warning(Ta('WARNING: Paused job "%s" because of unwanted content in RAR file SJ'), latin1(nzo.final_name))
nzo.pause()
and a new function (with 'exe' hardcoded for the time being):
Code: Select all
def rar_contains_unwanted_file(nzo, filepath):
unwanted = False
if is_rarfile(filepath):
try:
zf = RarFile(filepath, all_names=True)
logging.debug('files in rar file SJ: %s', zf.namelist())
for somefile in zf.namelist() :
logging.debug('file in rar file SJ: %s', somefile)
if somefile.lower().endswith('exe'):
unwanted = True
zf.close()
except:
logging.debug('RAR file %s cannot be inspected SJ', filepath)
return unwanted
This will save me a lot of time & bandwidth.
Re: pre-check on contents of post for unwanted content
Posted: March 30th, 2014, 1:51 pm
by sander
PS:
Some stuff from sabnzbd.log:
Code: Select all
2014-03-30 20:55:10,222::DEBUG::[assembler:332] rar file to check SJ: /home/sander/Downloads/incomplete/Blabla.XviD-Ecf.part13/Blabla.XviD-Ecf.part01.rar
2014-03-30 20:55:10,222::DEBUG::[assembler:335] files in rar file SJ: ['Blabla.XviD-Ecf/Codec/Setup.exe', 'Blabla.XviD-Ecf/If_you_get_error.txt', 'Blabla.XviD-Ecf/Blabla.XviD-Ecf.avi']
2014-03-30 20:55:10,223::DEBUG::[assembler:337] file in rar file SJ: Blabla.XviD-Ecf/Codec/Setup.exe
2014-03-30 20:55:10,223::DEBUG::[assembler:337] file in rar file SJ: Blabla.XviD-Ecf/If_you_get_error.txt
2014-03-30 20:55:10,223::DEBUG::[assembler:337] file in rar file SJ: Blabla.XviD-Ecf/Blabla.XviD-Ecf.avi
2014-03-30 20:55:10,223::WARNING::[assembler:125] WARNING: Paused job "Blabla.XviD-Ecf.part13" because of unwanted content in RAR file SJ
EDIT:
Visible warning is now
2014-03-30 21:35:04,079 WARNING: WARNING:: Paused job "blabla" because of unwanted content in RAR file SJ. Unwanted file is Blabla.DVDRip.XviD-LEd/Codec/Setup.exe
Re: pre-check on contents of post for unwanted content
Posted: March 31st, 2014, 5:07 am
by shypike
Interesting idea, I will look into it.
Re: pre-check on contents of post for unwanted content
Posted: April 4th, 2014, 2:10 pm
by sander
I've uploaded my modded SABnzbd to my own SABnzbd github fork.
See git diff here:
https://github.com/sanderjo/sabnzbd/com ... c7defdb324
For those who want to test before I send a request to Shypike's git, follow these git instructions:
Code: Select all
git clone https://github.com/sanderjo/sabnzbd.git
cd sabnzbd/
git checkout 0.7.x
grep -i unwanted *
That last command should give a few lines.
Stop SABnzbd, open sabnzbd.ini for editing, and in the [misc]-part insert:
Code: Select all
unwanted_extensions = exe, bla
pause_on_unwanted_extensions = 1
Meaning of the value of pause_on_unwanted_extensions:
0 = just warn
1 = pause
2 = abort
Then start SAB the normal way:
Example of sabnzbd.log info:
Code: Select all
2014-04-04 20:49:51,697::DEBUG::[SABnzbd:1335] Unwanted extensions are ['exe', 'bla']
2014-04-04 20:50:40,040::DEBUG::[assembler:355] Unwanted file blablabla.LiMiTED.DVDRip.XviD-Ecf/Codec/Setup.exe
2014-04-04 20:50:40,040::WARNING::[assembler:127] WARNING: In "blablabla.LiMiTED.DVDRip.XviD-Ecf.part13" unwanted extension in RAR file. Unwanted file is blablabla.LiMiTED.DVDRip.XviD-Ecf/Codec/Setup.exe
2014-04-04 20:50:40,040::DEBUG::[assembler:129] Unwanted extension ... pausing
Please post back your feedback
Re: pre-check on contents of post for unwanted content
Posted: April 4th, 2014, 3:25 pm
by TonioRoffo
OK, I just did the git clone in a new folder to check this build.
Changed sabnzbd.ini and added the said lines:
under [misc]
unwanted_extensions = wmv
pause_on_unwanted_extensions = 2
I start SABnzbd.py, then exit after a while
I check my sabnzbd.ini file and my lines are gone. Where should they go, I'm obviously putting them in a wrong place.
Thanks
Re: pre-check on contents of post for unwanted content
Posted: April 4th, 2014, 3:46 pm
by sander
Oops: the options must indeed be put into the [misc] part. I'll change that in my post.
Now your problem: did you stop SABnzbd first, and then manually edit sabnzbd.ini, then save & exit ?
Re: pre-check on contents of post for unwanted content
Posted: April 4th, 2014, 3:55 pm
by Eejit
A very interesting idea. I search for exe files AFTER it's all downloaded and unpacked in my post processing script for movies.
How does your mod effect applications being downloaded? Is it category dependant?
Re: pre-check on contents of post for unwanted content
Posted: April 4th, 2014, 4:06 pm
by sander
Eejit wrote:A very interesting idea. I search for exe files AFTER it's all downloaded and unpacked in my post processing script for movies.
How does your mod effect applications being downloaded? Is it category dependant?
Well, I'm on Linux, so I never download applications from newsgroups, and thus I've not included any category dependency.
The good news: if SAB pauses the download (so: option 1), you can Resume it again, and the download continues and you'll get your full download anyway.
Re: pre-check on contents of post for unwanted content
Posted: April 6th, 2014, 6:53 am
by sander
FYI:
The SAB mod is working very well for me; it seems more and more posts contain fake content combined with a file "Codec/Setup.exe" (which contains malware)
With the mod, SAB will detect it very early and pause the download with a Warning. Screenshot:
And the Warning page:
Re: pre-check on contents of post for unwanted content
Posted: April 6th, 2014, 3:03 pm
by TonioRoffo
my bad, it works for me as well. I probably didn't stop the sabnzbd process first time I edited the .ini file.
I hope this makes it to the code.
Re: pre-check on contents of post for unwanted content
Posted: April 7th, 2014, 1:47 pm
by sander
Re: pre-check on contents of post for unwanted content
Posted: April 13th, 2014, 2:33 am
by sander
Update:
As of the git version as of today 2014-04-13, you need a new format in sabnzbd.ini, still in the [misc] part:
Code: Select all
unwanted_extensions = .exe, .bla
action_on_unwanted_extensions = 1
Changes:
The extensions MUST be in lower case and with a leading dot.
The option is now 'action_on...' (not 'pause_on...')
Logging has changed a little bit:
Code: Select all
2014-04-13 09:36:35,973::DEBUG::[SABnzbd:1327] Unwanted extensions are ... ['.exe', '.bla']
2014-04-13 09:41:44,717::DEBUG::[assembler:355] Unwanted file blabla.2013.iNTERNAL.DVDRip.XviD-nEo/Codec/CG_Xvid_Codecs_Setup.exe
2014-04-13 09:41:44,717::WARNING::[assembler:127] WARNING: In "blabla 2013 (3)" unwanted extension in RAR file. Unwanted file is blabla.2013.iNTERNAL.DVDRip.XviD-nEo/Codec/CG_Xvid_Codecs_Setup.exe
2014-04-13 09:41:44,718::DEBUG::[assembler:129] Unwanted extension ... pausing
HTH
Re: pre-check on contents of post for unwanted content
Posted: April 24th, 2014, 10:09 am
by andymac
This is great!
I was looking into ways to fix newznab as well as sab to take care of this same issue and got directed here. For myself, I prefer to simply have the job aborted so couch can go find something new. I'll pull down your clone later, but for now, thanks for posting the functions in the beginning. I just changed them to abort the job.
So I'll check it out when I look at the clone, but my only addition to what you've obviously added would be an option to abort rather than just pause.
Otherwise, I'll try to keep an eye to see if / when this is pulled into the main!
Great help, man! Now if only newznab / nZedb could index these and remove them right away. Harder for them as they'd have to pull each rar to do that (the file doesn't show up in the main rar). Pipe dream, probably
.
Thanks!
Re: pre-check on contents of post for unwanted content
Posted: April 24th, 2014, 10:50 am
by sander
andymac wrote: my only addition to what you've obviously added would be an option to abort rather than just pause.
Meaning of the value of action_on_unwanted_extensions:
0 = just warn
1 = pause
2 = abort
... so Abort is already there.