Howto: multicore par2 on Debian, Ubuntu & derivatives

[blindpet]

Thank you for the clarification jcfp, I wasn't sure if Sabnbzbd needed to start the threads or not. Will be posting the instructions for others to use and linking here. Thanks for your work on this.

[blindpet]

Just to clarify, are you cool with me posting this as a guide on my site to spread your work? Links to your github and this thread will be included in the post of course.

[jcfp]

No problem.

[blindpet]

About to post this and doing some tests, do you know if the repair time in Sabnbzd includes the verification time? Using the same nzb file with intentionally damaged blocks for consistency I get these times which is an improvement but not as significant as I would have hoped

Single core par2cmdline 198 seconds
Dual core par2cmdline 166 seconds

Verification time is 105 seconds which would make it
Single core 93
Dual core 61

[Skyrider]

According to: http://www.htpcguides.com/install-par2- ... ntu-linux/

The guide was made with the help of people's replies in this thread. (jcfp and sander).

I'm using a Pi2 model B, and it appears the entire compile fails:

http://paste.debian.net/hidden/ddc4d151/

After installing automake1.10, still fails:

http://paste.debian.net/hidden/579c06d5/

[blindpet]

Skyrider, try installing autoconf too

[Skyrider]

Thanks, but:

autoconf is already the newest version.

Forgot to mention that libtbb-dev is NOT available for raspbarian. Despite it being it being mentioned in the tutorial.

[jcfp]

Your automake problem is likely caused by the instructions at htpcguides.com telling you to run configure/make by hand rather than actually building the debian package, thereby skipping steps included in the latter such as autoreconf (and gcc hardening later on). Retry with the instruction in the topic start.

I'm not sure whether that will make that "compilation requires an ARMv7-a architecture" error go away though. What os and release are you using? How did you install tbb and its header files if not by installing libtbb-dev and libtbb2?

[Skyrider]

Your automake problem is likely caused by the instructions at htpcguides.com telling you to run configure/make by hand rather than actually building the debian package, thereby skipping steps included in the latter such as autoreconf (and gcc hardening later on). Retry with the instruction in the topic start.

I'm not sure whether that will make that "compilation requires an ARMv7-a architecture" error go away though. What os and release are you using? How did you install tbb and its header files if not by installing libtbb-dev and libtbb2?

Thanks for the reply. I'm using:

Distributor ID: Raspbian
Description: Raspbian GNU/Linux 8.0 (jessie)
Release: 8.0
Codename: jessie

As for libtbb-dev, I was unable to apt-get install the package file, So I installed the DEB files instead:

pool/main/t/tbb/libtbb-dev_4.2~20140122-5_armhf.deb
pool/main/t/tbb/libtbb2_4.2~20140122-5_armhf.deb

Located at: http://ftp.iinet.net.au/debian/debian/

[safihre]

I think the Wiki should be updated, since it still points to the ChuChuSoft website that disappeared.
http://wiki.sabnzbd.org/configure-switc ... multi-core

[shypike]

I think the Wiki should be updated, since it still points to the ChuChuSoft website that disappeared.
http://wiki.sabnzbd.org/configure-switc ... multi-core

Fixed.

[safihre]

Also on github in the README needs the new link!
Both for 0.8 and 0.7 probably!

[sander]

The script gets this file:
https://github.com/jcfp/par2tbb-chuchus ... 503.tar.gz

Does that means it's par2cmdline 0.4, whereas par2cmdline is already at 0.6.14 (with the "-N" option)?

[an3k]

How come SABnzbd NEVER uses the multithreaded version? I entered -t+ into the web config setting "Extra PAR2 Parameters" but when SABnzbd is verifying or repairing the executed command is just /usr/bin/par2 r /path/to/first_file.par2 /path/to/first_file.rar ...

I checked the code and found this nonsense:

newsunpack.py:1163 of 0.8.0 Beta 4

Code: Select all

 1    classic = classic or not cfg.par2_multicore()
 2    if sabnzbd.WIN32:
 3        # If filenames are UTF-8 then we must use par2-tbb, unless this is a retry with classic
 4        tbb = (sabnzbd.assembler.GetMD5Hashes(parfile, True)[1] and not classic) or not PAR2C_COMMAND
 5    else:
 6        tbb = False
 7    if tbb and cfg.par_option():
 8        command = [str(PAR2_COMMAND), cmd, str(cfg.par_option().strip()), parfile]
 9    else:
10        if classic:
11            command = [str(PAR2C_COMMAND), cmd, parfile]
12        else:
13            command = [str(PAR2_COMMAND), cmd, parfile]
14    logging.debug('Par2-classic = %s', classic)

So if we're not on Windows the if clause (line 7) which should set the extra par2 parameters (line 8) never gets executed but instead its else clause (line 9 to 13). In fact only one command is executed: line 13.

Replace line 13 with

Code: Select all

command = [str(PAR2_COMMAND), cmd, str(cfg.par_option().strip()), parfile]

and you get your extra par2 parameters passed to par2 and thus real multi-threading.

Enable debug logging and check your logfiles for Starting par2 to see what command line sabnzbd used.
Before my dirty hack it was
2016-01-10 02:52:04,437::DEBUG::[newsunpack:1195] Starting par2: ['/usr/bin/par2', 'r', '/var/tmp/some.home.videos.par2']
after my dirty hack it is
2016-01-10 02:52:04,437::DEBUG::[newsunpack:1195] Starting par2: ['/usr/bin/par2', 'r', '-t+', '/var/tmp/some.home.videos.par2']

Finally this makes sense "Processing verifications and repairs concurrently using up to 24 logical CPUs." 8)

[sander]

I posted about that yesterday: http://forums.sabnzbd.org/viewtopic.php ... 98#p103720

So did you read that and then posted about it here?

Anyway: It has been fixed in the git version of SAB: https://github.com/sabnzbd/sabnzbd/comm ... 20f3d3e1aa