Howto: multicore par2 on Debian, Ubuntu & derivatives

[jcfp]

The commands below build a 'par2-tbb' deb based on the multithreaded par2 source code released by chuchusoft. To start using the multicore version, just install your freshly-built deb; to go back to the original, simply (re)install the standard par2 package from the repositories. Note that when switching between par2 and par2-tbb (in either direction) you should uninstall the "other one" first, or you may get errors about conflicting packages.

While par2-tbb is a drop-in replacement for the standard par2 package as far as SABnzbd+ is concerned, the wiki page about multi-core par2 support is still required reading!

Code: Select all

sudo apt-get install devscripts build-essential git debhelper dh-autoreconf libtbb-dev
DIR="$(mktemp -d)"
cd "$DIR"
git clone https://github.com/jcfp/debpkg-par2tbb.git
cd debpkg-par2tbb
uscan --force-download
dpkg-buildpackage -S -us -uc -d
dpkg-source -x ../par2cmdline-tbb_*.dsc
cd par2cmdline-tbb-*
dpkg-buildpackage -b -us -uc
echo; echo "to install, run: sudo dpkg -i $(readlink -f ../par2-tbb_*.deb)"

As always, no warranty of any kind. Tested on Debian unstable and Ubuntu trusty, ymmv on anything else. Supported architectures and operating systems are probably limited to whatever has a recent version of libtbb available. Don't worry if you get lots of output during build: verbose logging is on by default and in addition there's plenty of compiler warnings with these par2 sources.

Latest changes:
20150723: now uses copies of the chuchusoft source releases hosted on my github account, the chuchusoft website was just too unreliable.
20151102: patch configure.ac to handle building on *-*-linux-gnu<something>
20160618: patch Makefile.am to support compiling with older libc

[sander]

TL;DR: I think debhelper and dh-autoreconf should be installed too. So:

Code: Select all

sudo apt-get install devscripts build-essential git debhelper dh-autoreconf

Long:

I put the commands into a script, and running it, I get this (on Ubuntu 15.04-to-be):

Code: Select all

+ uscan --force-download
uscan warning: In debian/watch,
  no matching hrefs for watch line
  http://www.chuchusoft.com/par2_tbb/download.html par2cmdline-([\d\.]*)-tbb-(\d*).tar.gz

On a second run, I got:

Code: Select all

dh clean --with autoreconf
make: dh: Command not found
debian/rules:7: recipe for target 'clean' failed
make: *** [clean] Error 127
dpkg-buildpackage: error: fakeroot debian/rules clean gave error exit status 2

After

Code: Select all

sudo apt-get install debhelper

rerunning the script resulted in:

Code: Select all

dh clean --with autoreconf
dh: unable to load addon autoreconf: Can't locate Debian/Debhelper/Sequence/autoreconf.pm in @INC (you may need to install the Debian::Debhelper::Sequence::autoreconf module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.20.2 /usr/local/share/perl/5.20.2 /usr/lib/x86_64-linux-gnu/perl5/5.20 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.20 /usr/share/perl/5.20 /usr/local/lib/site_perl .) at (eval 11) line 2.
BEGIN failed--compilation aborted at (eval 11) line 2.

debian/rules:7: recipe for target 'clean' failed
make: *** [clean] Error 2
dpkg-buildpackage: error: fakeroot debian/rules clean gave error exit status 2

EDIT:

Code: Select all

sudo apt-get install dh-autoreconf

solves that.

After a few minutes of building, I indeed have debpkg-par2tbb/par2-tbb_0.4+20141125-1_amd64.deb in the tmp-blabla directory.

Code: Select all

sudo apt-get remove par2
sudo dpkg -i debpkg-par2tbb/par2-tbb_0.4+20141125-1_amd64.deb 

... and it works.

Code: Select all

$ par2 | grep -i thread
Not enough command line arguments.
Concurrent processing utilises Intel Thread Building Blocks 4.3 Update 1,
  -t<+<n>|0<n>|-<n>>: Threaded processing. The options are:

Cool

[jcfp]

Thanks for testing. Just made some changes to pull the install of the build-depends forward (should solve your debhelper/dh-autoreconf issues) and add a debugging option to uscan. It appears the chuchusoft site is hosted on some free webhost that occasionally returns a "server too busy, try again later" page. Obviously, uscan cannot find the upstream source tarball when it gets that response.

[sander]

Is it possible to automatically bail out in case of problems?

[jcfp]

Add 'set -e' at the beginning, that should make it stop as soon as there's a nonzero exit status from any of the commands

[lorddark462]

I was trying to install this on my Pi2 running minibian but I'm running into issues with the libtbb is there anyway to install this?

[jcfp]

I was trying to install this on my Pi2 running minibian but I'm running into issues with the libtbb is there anyway to install this?

I don't have the time to look up what minibian is, so -given your issue- I'm going to assume it's something based on debian wheezy. If so, the libtbb package of that vintage didn't work with armhf, see here. Newer tbb releases seem to support that architecture though, so try backporting: howto here, source packages there. Do keep us informed of your progress.

PS: seems the chuchusoft website/domain with its freebie hosting is once again having issues...

[blindpet]

I am trying to build on an armv7 device (Orange Pi) with Ubuntu vivid, the dependencies install but this error is thrown during building

Code: Select all

configure: exit 1
dh_auto_configure: ./configure --build=arm-linux-gnueabihf --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=${prefix}/lib/arm-linux-gnueabihf --libexecdir=${prefix}/lib/arm-linux-gnueabihf --disable-maintainer-mode --disable-dependency-tracking returned exit code 1
debian/rules:7: recipe for target 'build' failed
make: *** [build] Error 255
dpkg-buildpackage: error: debian/rules build gave error exit status 2

However, if I go in to ~/par2tbb/debpkg-par2tbb/par2cmdline-tbb-0.4+20150503 I can build and install

Code: Select all

./configure --prefix=/usr --build=arm-linux-gnu && sudo make -j4 && make install

[jcfp]

Could it be the filesystem where the builddir is located (mktemp -d typically creates it somewhere under /tmp) happens to be mounted with the noexec option? If so, please create and change into a directory somewhere else that doesn't have that problem in lieu of the DIR="$(mktemp -d)" and cd "$DIR" commands.

If not, please provide the full output of the command that results in this error. In case it's too much for the forum just pastebin it somewhere. Vivid works fine on amd64 here, don't have anything resembling your specific hardware though.

[blindpet]

I adjusted the script prior to execution to use a regular folder before execution so it's not the /tmp permission issue.

If you look at the autogenerated config parameters they are different than the ones that worked, specifically the build parameter is different.

When running the autogenerated config parameters manually here is the failed output

Code: Select all

root@OrangePiPC:~/par2tbb/debpkg-par2tbb/par2cmdline-tbb-0.4+20150503# ./configure --build=arm-linux-gnueabihf --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=${prefix}/lib/arm-linux-gnueabihf --libexecdir=${prefix}/lib/arm-linux-gnueabihf --disable-maintainer-mode --disable-dependency-tracking
configure: WARNING: unrecognized options: --disable-maintainer-mode
checking build system type... arm-unknown-linux-gnueabihf
checking host system type... arm-unknown-linux-gnueabihf
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for g++... g++
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking for style of include used by make... GNU
checking dependency style of g++... none
checking for gcc... gcc
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... none
checking dependency style of gcc... none
checking for dirent.h that defines DIR... yes
checking for library containing opendir... none required
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for stdbool.h that conforms to C99... yes
checking for _Bool... yes
checking for ANSI C header files... (cached) yes
checking stdio.h usability... yes
checking stdio.h presence... yes
checking for stdio.h... yes
checking endian.h usability... yes
checking endian.h presence... yes
checking for endian.h... yes
checking aio.h usability... yes
checking aio.h presence... yes
checking for aio.h... yes
checking errno.h usability... yes
checking errno.h presence... yes
checking for errno.h... yes
checking getopt.h usability... yes
checking getopt.h presence... yes
checking for getopt.h... yes
checking for size_t... yes
checking whether byte ordering is bigendian... no
checking for an ANSI C-conforming const... yes
checking for inline... inline
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... 64
checking for _LARGEFILE_SOURCE value needed for large files... no
checking for working memcmp... yes
checking for stricmp... no
checking for strcasecmp... yes
checking for strchr... yes
checking for memcpy... yes
checking for getopt... yes
checking for getopt_long... yes
checking for realpath... yes
checking for std::unique_ptr<T> or std::auto_ptr<T>... found.std::unique_ptr<T>
checking that generated files are newer than configure... done
configure: error: conditional "PLATFORM_LINUX" was never defined.
Usually this means the macro was only invoked conditionally.

When I adjust it and remove the unrecognized ones this works

Code: Select all

./configure --build=arm-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=${prefix}/lib/arm-linux-gnu --libexecdir=${prefix}/lib/arm-linux-gnu

For my purposes it makes more sense to just configure manually rather than relying on the autotool.

Par2 multicore is still greyed out but I'm assuming that doesn't matter since the new par2 is a drop in? Or does Sabnzbd need to start the number of threads per cpu core for it to take advantage of all the cores?

[jcfp]

Blindpet, thank you for the extra info and buildlog showing the actual error:

configure: error: conditional "PLATFORM_LINUX" was never defined.

Looks like a minor issue with the upstream configure.ac assuming the buildsys string on linux always ends in exactly -linux-gnu, thus failing with the likes of arm-linux-gnueabihf. Patch below should fix that and has already been included in the packaging on git too:

Code: Select all

--- a/configure.ac
+++ b/configure.ac
@@ -109,7 +109,7 @@
 esac
 
 case "$host" in
-	*-*-linux-gnu)
+	*-*-linux-gnu*)
 		sys_linux=true
 		AM_CONDITIONAL(PLATFORM_LINUX, true)
 		AM_CONDITIONAL(PLATFORM_DARWIN, false)

As said before, lacking the correct hardware I cannot really test any of this, so a quick confirmation that the instructions in the topic start now successfully build on your arm platform would be much appreciated.

Par2 multicore is still greyed out but I'm assuming that doesn't matter since the new par2 is a drop in? Or does Sabnzbd need to start the number of threads per cpu core for it to take advantage of all the cores?

Multithreading is on by default in the tbb version. Sab will use whatever par2 it finds first on the PATH, whereas on windoze (and mac?) it comes with its own copies of both the normal and multicore par2 thus permitting users to choose between the two. If you want the ability to supply extra params on *nix (e.g. to manipulate the number of cores used and such), you'll have to convince shypike.

[blindpet]

Just tested and its still adding that aebif at the end in some places but it is building now. Building however is much slower compared to when I do it manually and specify make -j$(ncpu), is your automethod taking advantage of all cpu cores?

[jcfp]

Just tested and its still adding that aebif at the end in some places but it is building now. Building however is much slower compared to when I do it manually and specify make -j$(ncpu), is your automethod taking advantage of all cpu cores?

Thanks for testing, always nice when a 1 byte patch does what it's supposed to.

Just for clarity: arm-linux-gnueabihf isn't wrong, it just wasn't anticipated by the configure.ac that comes with the chuchusoft sources. Package build doesn't use parallel make afaik, adding it just isn't worth the potential for trouble given the short build time.

[blindpet]

Just tested and its still adding that aebif at the end in some places but it is building now. Building however is much slower compared to when I do it manually and specify make -j$(ncpu), is your automethod taking advantage of all cpu cores?

Thanks for testing, always nice when a 1 byte patch does what it's supposed to.

Just for clarity: arm-linux-gnueabihf isn't wrong, it just wasn't anticipated by the configure.ac that comes with the chuchusoft sources. Package build doesn't use parallel make afaik, adding it just isn't worth the potential for trouble given the short build time.

Glad to help and thanks for the fix. It will always be a short building time on modern x86/x64 architecture, arm processors however is a different story

I was under the impression that if a package doesn't support parallel threads adding -j4 doesn't negatively affect it, it will just do things with as many threads as are supported. I could of course be wrong.

Can you confirm that nothing needs to be done on the Sabnzbd side?

[jcfp]

Par2-tbb installs the executable as /usr/bin/par2, same as the standard par2 package (so sab will find and use it automatically) and by default uses all available logical CPUs to checksum and create/repair concurrently. That makes it a plug-in replacement on linux, even when sab appears unaware of these extra capabilities and Enable MultiCore Par2 in the config remains as greyed out as it always was.