About CRC Errors

[Nemesh]

A bad .NZB will also do the trick.
At this moment a .NZB generated by Binsearch will give therse errors, while the same .NZB generated bij Yabse works fine -> no CRC errors.

[colson22]

I just wanted to post an update to this issue.

It appears that the issue was VirtualBox and EXT4.  Under high load using EXT4 in VirtualBox the files being written to the volume would get corrupted.  After installing a fresh copy of Ubuntu Server 10.04 on EXT3, the problem has disappeared.  I have also setup a new Physical server with Ubuntu Server 10.04 on EXT4 and have had NO CRC errors while using SabNzbd.  It must be a combination of Virtualbox and EXT4.

Thanks for the great software and your help,

Frank

[psycik]

While i have no issue with what's causing it, understand it will happen ever so often, what I'm finding annoying is when getting CRC errors, the par files are deleted. 

I have quickpar on my system as well, fo sorting this out, but when I see one of these CRC error files, I then have to go and redownload all of the par files.

How can I keep the par files from being deleted until AFTER a successful unpack?

[shypike]

How can I keep the par files from being deleted until AFTER a successful unpack?

See: http://wiki.sabnzbd.org/configure-switches-v2 item "Enable Par cleanup".

[psycik]

How can I keep the par files from being deleted until AFTER a successful unpack?

See: http://wiki.sabnzbd.org/configure-switches-v2 item "Enable Par cleanup".

yeah this is the weirdness.  That is turned on.  So would have expected one of these jobs - with a crc error to KEEP the par files.  Turning it off turning it off for everything, getting annoying really quick.

Does the unrar not return an error code that tells the program its been successful or not? BTW on .5.2 on windows xp (just upgraded to .5.3 today)

[shypike]

Does the unrar not return an error code that tells the program its been successful or not? BTW on .5.2 on windows xp (just upgraded to .5.3 today)

The par2 files are deleted before unrar runs.
If par2 says that the files verify properly, there's no use in keeping those files.
The only way out is to use a user script that removes the par2 files.
The script will receive a status code for the success of par2 and unrar.
See: http://wiki.sabnzbd.org/user-scripts

[tpas]

Well, after days of research I can add a new cause of crc errors! For some reason either sabnzbd or python can't deal properly with raid 5 arrays on Linux. I tried at least 10 different downloads and all failed because of crc errors. Then I downloaded the same  again on a Windows system, wihout errors.
After that I tried changing all kinds of settings on my Linux sytem, all without result.
Finally I completely reinstalled Ubuntu (10.4), again wihout result.
As a last try I installed an extra single harddisk in my Linux system and let sabnzbd dowload on that disk: it worked!!!
The same downloads again on my raid array: crc erors again....
Extra info: The raidarray consists of 3 WD 1.5TB disks and is built in mdadm.

[shypike]

For some reason either sabnzbd or python can't deal properly with raid 5 arrays on Linux.

RAID systems are handled 100% by the operating system.
It's not like applications need to do something special to support them.
I'd sooner doubt the OS and/or the disk array.
par2 and unrar are quite capable of giving the disk system a good exercise.
BTW: the influence of SABnzbd is low.
par2 is responsible for verifying the downloaded files.
If it says they're OK, SABnzbd just launches unrar and lets it do its job.

[tpas]

RAID systems are handled 100% by the operating system.
It's not like applications need to do something special to support them.
I'd sooner doubt the OS and/or the disk array.

Yeah, that was off course my first impression too.  I did very many extensive disk tests with the raid array, and it passed them all.

par2 and unrar are quite capable of giving the disk system a good exercise.
BTW: the influence of SABnzbd is low.
par2 is responsible for verifying the downloaded files.
If it says they're OK, SABnzbd just launches unrar and lets it do its job.

That's something I tested also. I downloaded a file on the raid array and let sabnzbd do the whole job, e.g. par2 and rar. par2 was necessary and failed, so rar didn't start.
Then I downloaded on the single (not raid) disk and copied all the rar and par2 files to the raidarray and did the postprocessing "by hand". Then par2 wasn't necessary and rar unpacked flawlessly. So I still suspect the download-part messing things up, and not the par2 or rar part.

[shypike]

So I still suspect the download-part messing things up, and not the par2 or rar part.

SABnzbd and the underlying Python don't know whether they operate on a simple disk or on a disk array.
So when things don't work on a disk array but do work on a single disk, how can this
be blamed on SABnzbd and/or Python?
I wouldn't know where to begin looking for such an issue.

[tpas]

I wouldn't know where to begin looking for such an issue.

I hear you.... I will try and check my logfile tonight, to see if I can find any indication. Maybe this thread helps?

http://forums.sabnzbd.org/index.php?topic=4150.0

[shypike]

Maybe this thread helps?
http://forums.sabnzbd.org/index.php?topic=4150.0

It's about the same complaint.
For me it's still an OS problem, or maybe an error in the Python run-time library.
The assumption of SABnzbd is that the OS and Python are OK.
Have you looked for similar complaints in the Ubuntu forums?
Python is supported by Canonical and is an essential tool for Ubuntu.
Of course, I cannot prove that the error is not in SABnzbd,
but like I said before applications have no need to something special for raid.
Apart from that, we simply don't have the hardware to test it.
We already have several VMWare images to do some basic compatibility tests.

About the only thing I can imagine is that the size of the disk confuses the
Python run-time. It could be that there is some magic number (2T ?) above which
things go wrong.
We've fixed SABnzbd in the past because it used an older method to request
the disk size, which had problems with anything above 1T (I forgot what is was exactly).

[tpas]

OK. Thanks for all your replies.
On the seaparte disk it works now, but I don't like the idea of possibly having an unstable raid-array (nerver noticed it though......)
I will try to change python-versions tonight, to see if THAT brings anything. Which version of python do you recommend?

[shypike]

I will try to change python-versions tonight, to see if THAT brings anything. Which version of python do you recommend?

Most testing is done on Python 2.6.x where X is the latest supported by Ubuntu and ActiveState (for Windows).
As an extra test you could try to run a large par2 repair followed by an urar,
from the command line, without SABnzbd.
par2 and unrar are notorious disk bashers.
Also you can try to download stuff without post-processing and par and unar
like described above outside of SABnzbd.
That should give you some insight in the reliability of the disk array and might give
you clues about where the problem is located.

BTW: do you use a multi-code version of par2?

[tpas]

Most testing is done on Python 2.6.x where X is the latest supported by Ubuntu and ActiveState (for Windows).

That's the one installed in my distro....

As an extra test you could try to run a large par2 repair followed by an urar,
from the command line, without SABnzbd.
par2 and unrar are notorious disk bashers.
Also you can try to download stuff without post-processing and par and unar
like described above outside of SABnzbd.
That should give you some insight in the reliability of the disk array and might give
you clues about where the problem is located.

Did that... When the download has been succesfull (without crc failures) the commandline par2 and unrar work perfectly, when there were failures par2 sometimes is able to correct, sometimes not. The disk seems to work OK.

BTW: do you use a multi-code version of par2?

I'm not sure what you mean...