Howto: multicore par2 on Debian, Ubuntu & derivatives

[an3k]

Oh, it finally got fixed? I talked about the issue with shypike and offered my help more than six month ago but see below:

Summarizing: the 64bit and Multicore options make only sense for Windows and OSX where we can distribute the binaries for par2 and unrar.
All other platforms are too much out of our control.
Maybe it would make more sense if the user could enter an alternative name (for non-Win non-OSX).

[safihre]

@shypyke, is it really not worth to do a check after the module-detection if this -t and/or -N parameters are supported and then update par2-options config accordingly?

It seems this is part of the main par2 modules we support and definitely bumps performance without much effort..
We don't have the luxury like NZBGet to compile par2 ourselves, this seems a good start

[jcfp]

The script gets this file:
https://github.com/jcfp/par2tbb-chuchus ... 503.tar.gz

Does that means it's par2cmdline 0.4, whereas par2cmdline is already at 0.6.14 (with the "-N" option)?

The chuchusoft releases are forked from the old (2004 or so) par2 0.4, primarily adding tbb support; so yes, that's 0.4. Any version of par2 greater than 0.4 is from (or based on) the github-fork-turned-official hosted at https://github.com/Parchive/par2cmdline/ Their 0.6.8 release changed the way the program tries to find misplaced blocks, 0.6.13 added the -S and -N options. The latter re-enables the old file scanning behaviour found in pre-0.6.8, including any and all versions that identify themselves as 0.4.

The ones you have to be careful with are 0.6.8-0.6.12 (inclusive), they suffer from a bug that assumes the offset of a misplaced block is always positive (see this bug report) and don't have any way to adjust or override the new bahaviour.

[TioSolid]

Im having trouble building the multicore par2 in the ARM architeture too. The thing is that the problem seems to be related to the par2 package itself (verificationhashtable.h).

Here is my output (from the last command in the tutorial in the first post of this thread):

Code: Select all

In file included from par2cmdline.h:615:0,
                 from par1repairersourcefile.cpp:20:
verificationhashtable.h:429:49: warning: suggest parentheses around '&&' within '||' [-Wparentheses]
                        checksummer.ShortBlock() && checksummer.BlockLength() != nextentry->GetDataBlock()->GetLength()
                                                 ^
g++ -DHAVE_CONFIG_H -I.   -D_FORTIFY_SOURCE=2 -Wall -I./../tbb43_20141023oss/include   -std=c++11 -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -c -o par2creator.o par2creator.cpp
In file included from par2cmdline.h:615:0,
                 from par2creator.cpp:32:
verificationhashtable.h: In member function 'bool VerificationHashEntry::operator<(const VerificationHashEntry&) const':
verificationhashtable.h:69:40: warning: suggest parentheses around '&&' within '||' [-Wparentheses]
     return crc < r.crc || crc == r.crc && hash < r.hash;
                                        ^
verificationhashtable.h: In member function 'bool VerificationHashEntry::operator>(const VerificationHashEntry&) const':
verificationhashtable.h:73:40: warning: suggest parentheses around '&&' within '||' [-Wparentheses]
     return crc > r.crc || crc == r.crc && hash > r.hash;
                                        ^
verificationhashtable.h: In static member function 'static const VerificationHashEntry* VerificationHashEntry::Search(const VerificationHashEntry*, const MD5Hash&)':
verificationhashtable.h:186:47: warning: suggest parentheses around '&&' within '||' [-Wparentheses]
     if (entry->crc < crc || entry->crc == crc && entry->hash < hash)
                                               ^
verificationhashtable.h:190:52: warning: suggest parentheses around '&&' within '||' [-Wparentheses]
     else if (entry->crc > crc || entry->crc == crc && entry->hash > hash)
                                                    ^
verificationhashtable.h: In member function 'const VerificationHashEntry* VerificationHashTable::FindMatch(const VerificationHashEntry*, const Par2RepairerSourceFile*, const FileCheckSummer&, bool&) const':
verificationhashtable.h:405:54: warning: suggest parentheses around '&&' within '||' [-Wparentheses]
                             checksummer.ShortBlock() && checksummer.BlockLength() != currententry->GetDataBlock()->GetLength()
                                                      ^
verificationhashtable.h:412:56: warning: suggest parentheses around '&&' within '||' [-Wparentheses]
                               checksummer.ShortBlock() && checksummer.BlockLength() != currententry->GetDataBlock()->GetLength()
                                                        ^
In file included from par2cmdline.h:615:0,
                 from par2creator.cpp:32:
verificationhashtable.h:429:49: warning: suggest parentheses around '&&' within '||' [-Wparentheses]
                        checksummer.ShortBlock() && checksummer.BlockLength() != nextentry->GetDataBlock()->GetLength()
                                                 ^
/tmp/cco7GNtB.s: Assembler messages:
/tmp/cco7GNtB.s:4993: Error: thumb conditional instruction should be in IT block -- `cmpeq r1,r3'
/tmp/cco7GNtB.s:4994: Error: thumb conditional instruction should be in IT block -- `strexdeq lr,r2,r3,[fp]'
/tmp/cco7GNtB.s:5098: Error: thumb conditional instruction should be in IT block -- `cmpeq r3,r5'
/tmp/cco7GNtB.s:5099: Error: thumb conditional instruction should be in IT block -- `strexdeq r1,r4,r5,[fp]'
/tmp/cco7GNtB.s:5165: Error: thumb conditional instruction should be in IT block -- `strexeq r2,r10,[r3]'
/tmp/cco7GNtB.s:26751: Error: thumb conditional instruction should be in IT block -- `cmpeq r7,r3'
/tmp/cco7GNtB.s:26752: Error: thumb conditional instruction should be in IT block -- `strexdeq lr,r0,r1,[r5]'
make[2]: *** [par2creator.o] Error 1
make[2]: Leaving directory `/root/par2tbb/debpkg-par2tbb/par2cmdline-tbb-0.4+20150503'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/root/par2tbb/debpkg-par2tbb/par2cmdline-tbb-0.4+20150503'
dh_auto_build: make -j1 returned exit code 2
make: *** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2

Is there any way to solve this?

Edit: Sorry, here is my specs:

- Ubuntu 14.04.4 LTS
- Odroid X2 (1.7GHz Quad core ARM Cortex-A9 MPCore)

[jcfp]

Looks like a compiler and/or assembler issue. The internets suggest passing -Wa,-mimplicit-it=thumb to the build process, which you could do by editing the debian/rules file just after the uscan step in the tutorial. Insert something like this above the line that starts with a percent sign:

Code: Select all

CXXFLAGS:=$(shell dpkg-buildflags --get CXXFLAGS) -Wa,-mimplicit-it=thumb
export CXXFLAGS

And with that, I'm out of suggestions....

[TioSolid]

Using this command gives me a "shell - command not found" error

I tried issuing
dpkg-buildpackage -S -us -uc -d -Wa -mimplicit-it=thumb

after the uscan command, but it gives me a "dpkg-buildflags: unknown option `-Wa'" error too

[sandersonjobs]

Hi all. Did anyone ever figure this out? I have a Raspberry Pi 3 ARMv8 Quad core and Sabnzb is really sucking in single core mode. I have been using this app a long time, but from what I hear, nzbget has this functionality built in already. Anyone have a work-around, or should I just make the switch?

[kcallis]

The script gets this file:
https://github.com/jcfp/par2tbb-chuchus ... 503.tar.gz

Does that means it's par2cmdline 0.4, whereas par2cmdline is already at 0.6.14 (with the "-N" option)?

The chuchusoft releases are forked from the old (2004 or so) par2 0.4, primarily adding tbb support; so yes, that's 0.4. Any version of par2 greater than 0.4 is from (or based on) the github-fork-turned-official hosted at https://github.com/Parchive/par2cmdline/ Their 0.6.8 release changed the way the program tries to find misplaced blocks, 0.6.13 added the -S and -N options. The latter re-enables the old file scanning behaviour found in pre-0.6.8, including any and all versions that identify themselves as 0.4.

The ones you have to be careful with are 0.6.8-0.6.12 (inclusive), they suffer from a bug that assumes the offset of a misplaced block is always positive (see this bug report) and don't have any way to adjust or override the new bahaviour.

So how does one get the update version on the Ubuntu box. I thought i could just use git and the new URL, but when I do the uscan I get the same issue:

Code: Select all

kcc@cortana:/tmp/tmp.SCY2wr3ZeE/debpkg-par2tbb$ uscan --force-downloaduscan warning: In watchfile debian/watch, reading webpage
  https://github.com/jcfp/par2tbb-chuchusoft-sources/releases failed: 500 Can't connect to 127.0.0.1:8888

[sandersonjobs]

Sorry...I s witched to nzbget and have not had a single issue. I'm now just going to recommend the switch since nzbget makes it's own par2 that works with multiple architectures.

[safihre]

EDITED: par2cmdline-mt doesn't work well with SABnzbd, because often it complains about files being 'out of basepath'.

[piSAB]

Hi Jcfp

Unfortunately this procedure no longer seems to work on a Raspberry PI 2 (Jessie build).
I posted what's listed below on http://www.htpcguides.com/install-par2- ... ntu-linux/
but was prompted to contact you here:


Unfortunately this procedure no longer seems to work on a Raspberry PI 2 (Jessie build)
Even after switching the installed dependencies to use libopencv-dev instead of libtbb-dev:

sudo apt-get install devscripts build-essential git debhelper dh-autoreconf libwww-perl libopencv-dev -y

The Install PAR2 Multicore on ARMv7 procedure fails with a fatal error at the make command:

sudo make -j2

par2cmdline.h:495:39: fatal error: tbb/task_scheduler_init.h: No such file or directory
#include "tbb/task_scheduler_init.h"
^
In file included from par2cmdline.cpp:26:0:
par2cmdline.h:495:39: fatal error: tbb/task_scheduler_init.h: No such file or directory
#include "tbb/task_scheduler_init.h"
^
compilation terminated.
compilation terminated.
Makefile:445: recipe for target 'buffer.o' failed
make[1]: *** [buffer.o] Error 1
make[1]: *** Waiting for unfinished jobs....
Makefile:445: recipe for target 'par2cmdline.o' failed
make[1]: *** [par2cmdline.o] Error 1
make[1]: Leaving directory '/home/pi/par2tbb/debpkg-par2tbb/par2cmdline-tbb-0.4+20150503'
Makefile:327: recipe for target 'all' failed
make: *** [all] Error 2

Any help here would be appreciated. Thanks in advance.

Mike Mod
It could be a temporarily buggy script Simon, you can try asking the developer of the script on the sabnzbd forum

[jcfp]

libopencv-dev

An image processing library? Why?? You must install libtbb and its development headers for this version of par2. As it says right next to the instructions: supported architectures and operating systems are probably limited to whatever has a recent version of libtbb available. Everything else leads to this:

fatal error: tbb/task_scheduler_init.h: No such file or directory

Please stick to the build instructions in the topic start: no installing libopencv as if it were a replacement for tbb, no manual configure runs, no 'sudo make' with parellel build options set. It's hard enough already to guess what's going on all these different platforms and hardware architectures.

[safihre]

I also tried many ways to get tbb on my RP3, but never succeeded.

[jcfp]

Maybe the hardware isn't compatible, maybe the os provides only a subset of debian packages (remember users having to build their own unrar?). I dunno.

[piSAB]

Thanks for your replies. I was simply following the guide on: http://www.htpcguides.com/install-par2- ... ntu-linux/

I made the replacement for libtbb-dev as suggested in the above mentioned guide as others had reported success in doing so.

libtbb-dev is not not available on the PI2 so as you suggest this is now most probably a no go for Raspberry PI.