Downloads are almost always corrupted (par2 is defective on linux i386)

[anton]

Hi

First of, thank you for making this tools, it looks awesome, and I really look forward to using it.

I have had sabnzbd running for a couple of days now, unfortunately it almost always (5 successfull out of 65 dl's)reports failures, typically CRC failures, but repair failures as well. The odd thing is, if I use klibido to download, using the same nzb files, everything works great.

I searched the forum for similar issues, and one suggestion was to use unrar/rar from rarlabs, but that is already part of my distro.  When I run par2 by hand, it reports errors in the parts, which I believe sabnzbd has already fixed.

I'm using:
Version: 0.4.12 from the ppa repo
on Ubuntu Karmic

any suggestions on how to fix this, is greatly appreciated.

[daniel_owen_uk]

Are you using the same servers (and backup servers) on both bit's of software.

[anton]

I use the same usenet servers, but two different machines, sabnzbd is running on a local server, but klibido on my desktop, do you want me to try to perform the repair on the server?

[shypike]

Upgrade your unrar to the original Rarlab's 3.90 release.
Do not use unrar from repositories.

http://www.rarlab.com/rar/rarlinux-3.9.0.tar.gz

There are many unrar binaries for other platforms:
http://www.rarlab.com/rar_add.htm

[anton]

running rar already reports this:
RAR 3.90 beta 2  Copyright (c) 1993-2009 Alexander Roshal  3 Jun 2009

[anton]

sorry .. my bad
I didn't notice the beta

[anton]

I doubt that's the solution, I'm still getting errorsm both "» Not enough repair blocks left (have: 0, need: 471) " and " ERROR: CRC failed". although I have only two trial runs using the updated rar. Besides, I don't think a defective rar explains the par2 issue.

If you ask me, I think there is some issue in the yyenc module, that would definately fit my symtoms

[shypike]

I did not read original message careful enough.
First: which files are downloaded, do you get all the rar-s and par2-s?
We run the installed par2 in the normal way (there's not much to choose in options).
I don't see why a subsequent par2 run would produce different results.
Unless you have memory issues.

BTW: Failing CRC's on individual articles are fairly common.

You can try to un-install the yEnc module.
If it's missing, SABnzbd will use its built-in (much slower) yEnc code.
I doubt that this is the problem, because par2 should catch any inconsistencies.

[anton]

I don't think I made the par2 issue very clear, in any case, this is how I determined that there were some par2/yenc issue:

(sorry for the delay, but I wanted to re-verify my process, and double check some configurations on my server and on my desktop)

it turns out that for some reason par2 fails to repair the files on one computer, but not on another,
it completes the repair, but when validating it the repair, it seems as if the files aren't written, because par2 reports failueres in the very same files it did before the repair.

The odd thing is, this only happens on my server, on my desktop the par2 repair worked without issues.

I found out that I'm not really the only one having problems with par2, http://forums.sabnzbd.org/index.php?act ... pic=1085.0, has some definate similarities, and this one from debian, is very similar: http://www.mail-archive.com/debian-bugs ... 18034.html

The last one mentions possible hardware problems, so based on that I did a complete check of both my nic and ram, but everything cheks out.

any suggestions on how to figure out what is going wrong?

[anton]

Some additional info...

I am starting to believe this has to do with access rights!!!

the hard disk sabnzbd is us using, is running out of space (borked dowonloads piling up), so I shifted the output to a different hard drive (I cannot move the temporary download, because the queue isn't empty, but I moved the completed download folder)

I then dug into one of the _UNPACK_ folders that sabnzbd has created and ran par2repair, and this time the repair worked, and the following unrar worked as well. The only obvious difference I can think off  between the "old" disk and the new one, is that the mount entry for the new disk has a umask=000. giving full access to the world, while the old output resides within the homefolder of the account running sabnzbd.

What is the recommmeded setup for accounts, rights, folders etc.?

[shypike]

The user account that runs SABnzbd must have full access to the "download/incomplete" folder.
That user account should also be the owner of download/incomplete.
Preferably it should also own download/complete.
You can use the "permissions" setting to have SABnzbd set proper permissions on the final result.

[anton]

Just a little progress report on my investigations so far:

I have identified at least two different issues,

1 is an issue with dealing with unicode characters inside the .par2 master file. I have reported this under bugs (http://forums.sabnzbd.org/index.php?topic=2864.0), although from what I understand it really isn't a bug, more like an incompatibility between various versions of par2 generators, based on some implementors not bothering to read the fine print inside the specs.

2 again has something to do with differences between versions of par2, although I don't fully understand this one. Some part are not being repaired correctly by par2 on one 32 bit ubuntu karmic server, but are repaired correctly on a 64 bit ubuntu karmic desktop. Currently I suspect that it is also related to file size, since I haven't observed the error on file sets less than 1 GB.

Here is my odd results so far
1) Download a NZB file from newbin, I have a few where I know it will always fail
2) use sabnzbd on the 32 bit server to download the file set
3) observe that sabnzbd fails to unpack due to CRC errors
4) run par2repair on the _FAILED_ folder, observe that par2 during the repair phase claims to have repaired the file, but when validating reports errors
5) mount the exact same folder using nsf on my 64 bit desktop
6) run par2repair on the exact same datafiles, only this time from my 64 bit desktop, and observe that par2 reports success both in repairing and validating the repair
7) running "rar e " succeeds on both machines, regardless of using the released rar 3.90 or the beta 2 rar 3.90

next, I'll try to install a viirtual 32 bit machine on my 64 bit desktop, just to be 100% certain that it is indeed due to different versions of par2

[anton]

The second part of my mystery issue has been fixed as well, it turns out there is a bug in par2. It fails to repair on certain input. I have only seen the bug in i386. My experience is very similar to the bug reported here: http://sourceforge.net/tracker/?func=de ... tid=399698 (referencing a debian bug report: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=296387)

The bug was reported 3 years ago, but has never been fixed, the package appears to be no longer actively maintained, I did however find a fork which appears to be actively maintained, the fork, besides being able to perform the par2 repairs faster, has consistently repaired all my test cases. The updated version can be downloaded from here:

http://chuchusoft.com/par2_tbb/download.html

The new version depends on Intels threading libraries, on my Ubuntu, I had to do a

sudo apt-get install libtbb2

in order to get the .so's installed, they are part of the archive, but using apt, I know that they will be installed correctly

following the installation of libtbb2, I simply replaced the par2 binary in /urs/bin with the new version, par2verify, par2repair and par2create still works as they should, since they simply link to the par2 executble.


ps. I will report back on this thread, how a prolonged burn-in test using the replacement par2 pans out