Multi-Part PAR2 script behavior across large archive is slow

[NickTO]

I was wondering if this has been addressed before (searched but can't find a reference, so apologies if it's already be dealt with).

Currently using 0.7.11 with the multhreaded par2 (not sure the multithread version makes a difference).

It's not unusual when I download a very large archive that the Par2 files are split into different parts across the archive.

Example:

Downloading a 50GB file "sample" across 320 parts RAR. (sample.part001.rar -> sample.part320.rar)

The associated Par2 files are across 3 'parts' :

sample_part_1.par2 covers the files part001.rar to part110.rar
sample_part_2.par2 covers the files part111.rar to part220.rar
sample_part_3.par2 covers the files part221.rar to part320.rar

The behavior I'm seeing is that par2 is executed with a wildcard argument for the [files] portion of the par2 invocation:

/usr/bin/par2 r /path/.sabnzbd/Downloads/incomplete/sample/sample_part_1.par2 /path/.sabnzbd/Downloads/incomplete/sample/*

I understand that this may due to ensure proper pathing, but the resulting behavior here is that when this runs, it causes everything to stall once files not included in the part range are being scanned. If I run the par2 command like this , I see it occurring.

If I run the par2 script without the explicit wildcard [files], ( /usr/bin/par2 r /path/.sabnzbd/Downloads/incomplete/sample/sample_part_1.par2 ) it seems to work just as well and avoids scanning non-covered files, and prevents stalling.

Edit: Forgot to specify, the slowness occurs when article is missing and the archive requires repair.

[NickTO]

I've run into this again, and documented it a bit better.

Upon finishing a download, the PAR2 verify dialog appears. In this case, it was a file in 424 parts rar.

When par2 runs to verify the first "sample_part_1.par2" , the sabnzbd dialog runs the verify from part 1 to 150 , this takes 3-4 minutes, but does not end once par2 scans part 150/150. The par2 process will continue to run. I killed it after 120minutes of cpu time.

Then tt appears to repair, but does not. It then verifies again. The whole process takes 3-4 minutes. Then it gets stuck at 150/150 again. I let the par2 process run for an hour, then kill it.

The verify dialog then moves to the "sample_part_2.par2", I can tell the par2 script is now running on this par2 file, but same scenario. It scans in 4 minutes, reaches part 150/150, but doesn't stop, and stalls. I kill par2, the process goes into "repair" mode for a few seconds (but does not repair), and then verifies again, gets stuck at 150/150 runs for an additional hour, then I have to kill par2... then onto the 3rd part, and same thing.

All in all, only 6 articles were missing. By running the par2 command and fetching the par2 files manually, it's all of 5 minutes to fix everything. But it would never get to a resolution otherwise.

[shypike]

That's the trade-off for supporting wacky naming schemes.
It may backfire under some circumstances.
Please email an example NZB that shows this behaviour to [email protected]
(Please add the URL of this post.)

[kouze]

Hi,

I may also be interesting in the feedback for this kind of issue. I'm sometimes creating par2 files for big amount of data and it is really taking lots of time. What is the naming convention then to "split" the pars and having sabnzbd handling them properly ?

If I took the example of NickTO, I split my partXXX.rar files into 3 parts:
part one: files part001.rar to part110.rar
part two: files part111.rar to part220.rar
part three: files part221.rar to part330.rar

And want to have splitted par2 files for part one, part two and part three. What should be the names for the par2 files then ?

Thanks

[shypike]

I'm not advising on using whacky naming schemes, because it only gets in the way
of running things smoothly,
What some poster are now doing is to run par2 over the RAR files
and after that rename all the files to random names.
They rely on par2 "repairing" the names again.