Unable to Rename Sample+ file

[jcfp]

@sander iirc par2 can handle subdirs these days, at least its changelog seems to suggest so. The substitution of the directory separator with a plus sign is probably sab's doing: there's some function in filesystem.py doing that, might be related to sanitizing input or safeguarding against directory traversals.

[sander]

When I set SABnzbd, to
turn off Deobfuscate (because running the par2-deobfuscated causes a Traceback)
turn on Download all par2 (because they are needed because the rar files are damaged)
set download to +Delete

... the download success without errors, and the result is a directory with rars, par2s and the sample.mkv.

Then I did the below

mv xxx.s10e13.hog.heaven.720p.web.h264-caffeine.rar.1 xxx.s10e13.hog.heaven.720p.web.h264-caffeine.rar
unrar x xxx.s10e13.hog.heaven.720p.web.h264-caffeine.rar
par2 r 478f6eb128f245010f9cbd2b88f2a2c4.par2
unrar x xxx.s10e13.hog.heaven.720p.web.h264-caffeine.rar

... and everything is there.

[sander]

@sander iirc par2 can handle subdirs these days, at least its changelog seems to suggest so. The substitution of the directory separator with a plus sign is probably sab's doing: there's some function in filesystem.py doing that, might be related to sanitizing input or safeguarding against directory traversals.

These lines ... ?

CH_ILLEGAL = "/"
CH_LEGAL = "+"
CH_ILLEGAL_WIN = '\\/<>?*|"\t:'
CH_LEGAL_WIN = "++{}!@#'+-"

EDIT

I changed to
CH_ILLEGAL = "#"
CH_LEGAL = "+"

but a double Traceback and still "sample+Sample..." ...

[timmyc]

When I set SABnzbd, to
turn off Deobfuscate (because running the par2-deobfuscated causes a Traceback)
turn on Download all par2 (because they are needed because the rar files are damaged)
set download to +Delete

... the download success without errors, and the result is a directory with rars, par2s and the sample.mkv.

Then I did the below

mv, unrar, par2, unrar etc

... and everything is there.

Yep, the fact that Sabnzbd can download the rar file fine, and a manual/simple unrar command works, that lends further credence to my theory that it's something to do with the way Sabnzbd is parsing arguments/filenames to unrar

[sander]

I've created a reference NZB: https://raw.githubusercontent.com/sande ... 7a2982.nzb

Special thing: it has a subdirectory Sample (because that seems to be an issue?)
par2 created with par2command (indeed @jcfp that works)
All stuff is available, so no repair needed

My results:
OK on Windows
Failure on Linux: "Repair failed, not enough repair blocks (1199 short)"

On Linux, I see stuff like;

2021-02-19 12:03:01,471::DEBUG::[filesystem:819] Renaming "/home/sander/.sabnzbd/Downloads/incomplete/OK-with-subdir-5a62fa7a2982/bla1" to "/home/sander/.sabnzbd/Downloads/incomplete/OK-with-subdir-5a62fa7a2982/Sample/sub1"

Why would SAB do that?


Contents of the post:

bla1
bla2
bla3
bla4
Sample
Sample/sub1
Sample/sub2
with-sub-dir.par2
with-sub-dir.vol000+02.par2
with-sub-dir.vol002+04.par2
with-sub-dir.vol006+08.par2
with-sub-dir.vol014+16.par2
with-sub-dir.vol030+32.par2
with-sub-dir.vol062+64.par2
with-sub-dir.vol126+74.par2

[safihre]

Par2cmdline seems to fail specifically if we provide the full paths to all the files, but succeed if we don't.
But we need to provide it the full paths because otherwise it doesn't scan fully obfuscated files.

What we could do is improve the quick check to also handle sub dirs, but that wouldn't work if the file in the sub directory is dameged itself..

[sander]

Another reference NZB: all files have unique bytes: https://raw.githubusercontent.com/sande ... 9604e5.nzb

Results in a Failed, with SAB saying:

/home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5.1/mysubdir+insub1 cannot be renamed to /home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5.1/mysubdir/insub1

That is the "+ instead of /" kicking in ?



myfile1
myfile2
myfile3
mysubdir
mysubdir/insub1
mysubdir/insub2
important-subdir.par2
important-subdir.vol000+02.par2
important-subdir.vol002+04.par2
important-subdir.vol006+08.par2
important-subdir.vol014+16.par2
important-subdir.vol030+32.par2
important-subdir.vol062+64.par2
important-subdir.vol126+74.par2

EDIT:

This is the download result

Code: Select all

 __ADMIN__/
 important-subdir.par2
 important-subdir.vol000+02.par2
 important-subdir.vol002+04.par2
 important-subdir.vol006+08.par2
 important-subdir.vol014+16.par2
 myfile1
 myfile2
 myfile3
 mysubdir+insub1
 mysubdir+insub2

... mysubdir+insub1 ... that happens both with "Make Windows compatible" on and off.

Code: Select all

sander@brixit:~$ cat .sabnzbd/logs/sabnzbd.log | grep insub1
2021-02-19 14:54:30,937::INFO::[nzbparser:133] File insub1 added to queue
2021-02-19 14:54:30,944::DEBUG::[nzbstuff:383] Finishing import on insub1
2021-02-19 14:54:36,515::INFO::[nzbstuff:1683] Detected filename based on par2: insub1 -> mysubdir/insub1
2021-02-19 14:54:36,578::DEBUG::[assembler:102] Decoding part of /home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5/mysubdir+insub1
2021-02-19 14:54:36,618::DEBUG::[filesystem:541] Applying permissions 0o600 (octal) to /home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5/mysubdir+insub1
2021-02-19 14:54:36,619::INFO::[assembler:127] Decoding finished /home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5/mysubdir+insub1
2021-02-19 14:54:37,306::DEBUG::[newsunpack:2079] Quick-check will rename mysubdir+insub1 to mysubdir/insub1
2021-02-19 14:54:37,306::DEBUG::[filesystem:819] Renaming "/home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5/mysubdir+insub1" to "/home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5/mysubdir/insub1"
2021-02-19 14:54:37,306::INFO::[newsunpack:2096] Cannot Quick-check missing file mysubdir/insub1!

So SAB creates mysubdir+insub1 ... and then says it renames to mysubdir/insub1 ... but it can't do that:

Code: Select all

You have 30 recovery blocks available.
Repair is possible.
You have an excess of 30 recovery blocks.
None of the recovery blocks will be used for the repair.
/home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5.2/mysubdir+insub1 cannot be renamed to /home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5.2/mysubdir/insub1

[Puzzled]

It will stop complaining about the rename if you make the directory first, at least in Windows. I'm not sure if that fixes everything, though.

Very quick and dirty code to be inserted before the rename in newsunpack:

Code: Select all

                    new_path = re.search("(.*)/", file)
                    if new_path:
                        new_path = os.path.join(nzo.download_path, new_path.group(1))
                        logging.debug("mkdir %s", new_path)
                        try:
                            os.mkdir(new_path)
                        except:
                            logging.debug("Failed to make path")
                    else:
                        logging.debug("No path found")

renamer(os.path.join...

[sander]

Cool, that works: both for my reference NZB and for the IRL NZB provided earliear on.

@puzzled I'll take the discussion to github.

[timmyc]

Awesome, so is this something that is/will be tracked in a Github issue moving forward? I had a look but couldn't see one already created.

Is this something likely to be worked through the development cycle and resolved in a subsequent release?

[sander]

Awesome, so is this something that is/will be tracked in a Github issue moving forward? I had a look but couldn't see one already created.

Is this something likely to be worked through the development cycle and resolved in a subsequent release?

See https://github.com/sabnzbd/sabnzbd/pull/1802 ... that git branch is working. If you know how to use git (and run SAB from source), you could try it yourself.

In subsequent release of SAB ... if/when main maintainer Safihre approves it.

[timmyc]

Lovely. I am very familiar with git, and will run a build from source tomorrow.

Thanks heaps for the work on this guys.

[sander]

Lovely. I am very familiar with git, and will run a build from source tomorrow.

And? Feedback?

[Luchtmacht]

The latest update of my SABnzbd docker container seems to have fixed the bug.
Thank you very much for investigating so quickly.