About CRC Errors

[shypike]

Thanks for the tips.
One thing though. I re-downloaded the par files that were deleted and ran unpar manually and everything fixed up fine. I did this for all the files and there were no errors at all. Now I did use QuickPar instead of the built in par2 could that be the problem?

Quickpar is more permissive regarding badly named par2 files.
Just in case there are other causes, can you email the NZB to bugs at sabnzbd.org ? (Please add URL of this message.)

[JoshFink]

I'll send the nzb in. Is there a better par program that I can substitute in that might be more forgiving?

[JoshFink]

So here is something strange. I'm only experiencing CRC errors when I'm downloading movies. TV shows coming from Sickbeard work great but Movies are just not downloading properly. Always getting the CRC errors.

Another thing I noticed was that I unchecked "enable par cleanup" and the pars are still being deleted even though the downloaded rars have errors in them.

Thanks

[JoshFink]

Ok.. I just moved the par2 files back into the directory and ran par2.exe from the command line. It found that it needed a repair and fixed it up. Why wouldn't this work from inside the program but it would work from the command line?

Code: Select all

Repair is required.
6 file(s) exist but are damaged.
83 file(s) are ok.
You have 3873 out of 3879 data blocks available.
You have 194 recovery blocks available.
Repair is possible.
You have an excess of 188 recovery blocks.
6 recovery blocks will be used to repair.

Computing Reed Solomon matrix.
Constructing: done.
Solving: done.

Wrote 900000000 bytes to disk

[JoshFink]

Well, after several posts rambling about the problems. I turned off "Enable Quick Check" and everything started working...

[shypike]

Well, after several posts rambling about the problems. I turned off "Enable Quick Check" and everything started working...

In that case I would be highly interested in sample NZB files
because this would mean that there's a bug in QuickCheck.
Can you send some NZB which caused problems to bugs at sabnzbd.org ?
(Please add the URL of this message).
You already did, just didn't have the time to check it yet.

[isofan]

Hi,

I get a lot of CRC errors since I run Sabnzbd on my Windows Home Server (WHS), but I am not sure if it is WHS, or XLned, or simply "bad" usenet posts. I am currently running Prime95 to check for memory errors, but haven't found any so far. Could you imagine what might cause these errors, especially when saving the downloads to a WHS share?

[shypike]

I get a lot of CRC errors since I run Sabnzbd on my Windows Home Server (WHS), but I am not sure if it is WHS, or XLned, or simply "bad" usenet posts. I am currently running Prime95 to check for memory errors, but haven't found any so far. Could you imagine what might cause these errors, especially when saving the downloads to a WHS share?

You should be suspicious when CRC errors occur after par2 verification finds no problems.
You'll have to do some diagnostics yourself.
The easiest test is to re-download a failing job on your normal PC.
One of the previous posters said his problems were gone after disabling QuickCheck (Config->Switches).
It's worth a try, but please do report any positive or negative results.

[sacii]

Turning off Quick Check does seem to be the answer.

Since changing to a raid5 array, about 80% of my downloads failed with a CRC error. I have been able to fix the downloads by redownloading the par2 files and manually repairing and unpacking. However, sometimes par2 would say all the rar files were repaired but when unpacking I would get corrupt volume (rar) errors. Then I have been able to fix the downloads by redownloading just the corrupt rar files and manually repairing and unpacking.

Last night I downloaded 5 files, 1 completed, and 4 failed with CRC errors. Based on shypike's comments, I redownloaded the 4 failed jobs with Quick Check off. 2 of the downloads completed normally. 1 download completed with a status of failed but no error message, however the download was unpacked, renamed and moved to the correct directory.  1 download completed with a status of completed but with an error message (unable to find xxx.r14.100), however the download was unpacked, renamed and moved to the correct directory. Both of the downloads with abnormal messages did not completely clean up after themselves, each leaving behind a single repaired rar archive, but that is minor.

I do not believe for a second that SABnzbd is causing any errors, but I do believe that Quick Check overlooks errors introduced by raid5. Despite the argument that the operating system does not distinguish between a single disk and a raid5 array, writes to a raid5 array are a lot more complicated. Additionally, not all raid5 arrays are alike. I have been trying a software raid5 under FreeNAS 0.7.2 (FreeBSD 7.3) for 1 and 1/2 weeks and have had to run fsck_ufs on the array several times to fix errors and would have lost files if not for backups. Fortunately, I took heed of the truism that raid arrays are not a substitute for backing up your files.

[sacii]

Sorry for the pompousness of replying to my reply.

Some quotes from Wikipedia - Standard RAID Levels (I don't want to hear anyone's dissing of Wikipedia, their vetting process is very good.):

"The parity blocks are not read on data reads, since this would add unnecessary overhead and would diminish performance. The parity blocks are read, however, when a read of blocks in the stripe fails due to failure of any one of the disks, and the parity block in the stripe are used to reconstruct the errant sector. The CRC error is thus hidden from the main computer. Likewise, should a disk fail in the array, the parity blocks from the surviving disks are combined mathematically with the data blocks from the surviving disks to reconstruct the data from the failed drive on-the-fly.

This is sometimes called Interim Data Recovery Mode. The computer knows that a disk drive has failed, but this is only so that the operating system can notify the administrator that a drive needs replacement; applications running on the computer are unaware of the failure. Reading and writing to the drive array continues seamlessly, though with some performance degradation."

and

"RAID 5 implementations suffer from poor performance when faced with a workload which includes many writes which are smaller than the capacity of a single stripe. This is because parity must be updated on each write, requiring read-modify-write sequences for both the data block and the parity block. More complex implementations may include a non-volatile write back cache to reduce the performance impact of incremental parity updates."

The above suggests the following possible solutions may let you keep Quick Check turned on:

1. Use an article cache (Config > General) of sufficient size to hold all the downloaded articles that make up an rar so that the rar is assembled in memory and then written to the raid5 array. (Decrease multiple writes of small files to raid5).

2. Don't put the temporary and system folders (Config > Folders) on the raid5 array, put them on a single disk (such as the operating system disk) and use categories (Config > Categories) to move the unpacked files to the raid5 array. (Only write large files to raid5).

Personally, I will use the second solution.

[shypike]

Sorry for the pompousness of replying to my reply.

No problem.
Please understand that QuickCheck checks the files before the final files are written to disk.
When the RAID screws up, SABnzbd won't know until unrar starts unpacking.
The (I think reasonable) assumption of SABnzbd is that the file system can be relied upon.
I agree with 2), that using RAID for SABnzbd's temporary download folder is probably not a good idea.
You don't need the redundancy and you will only suffer the performance degradation.
But I also think it's quite worrying that SABnzbd cannot write to the RAID reliably.
I mean, all it does is "open file", "write", "write", "write", "close".
How hard is that for the OS (and RAID) to handle?
Does other software suffer from this?
Should SABnzbd do things differently when on a RAID?
Why is that? Aren't RAIDs supposed to be transparent?
You're not the first one with this issue and I would like to know the root cause.

[JoshFink]

This might be the same problem I was having. Mine was being written to a Drobo.

[sacii]

It's been 2 days and 13 downloads since I moved SABnzbd's temporary download folders off the raid5 and I have not had a single CRC error.

What this means about SABnzbd's interaction with my raid5 is not possible to intrepret without a lot more testing. I am using a software raid5 under FreeNAS 0.7.2 (FreeBSD 7.3). Perhaps my CPU just doesn't have the power to reliably run downloading software and raid5 parity calculations at the same time. I don't have the resources to test SABnzbd's interaction with hardware raid5, thats why I am using FreeNAS on old hardware.

My raid5 is again in a degraded state due to a wonky disk, amazingly I can still write video files to the raid5 and then stream them across my network to a digital media player without any glitches.

[isofan]

I've found that if I get a CRC error, it usually doesn't recur if I repeat the download.  So somewhere between the usenet host (I use newsdemon) and my PC something messes up, but it doesn't happen if I just download again.  It could be in the repair phase, in which case re-running PAR2 and unrar may be all I need to do, but that's too much work.  Much easier to just let sabnzbd take care of it for me.

I have exactly the same problem, sometimes my RARs are defective, but redownloading using the same connection fixes the problem. Can I be sure it's really a transfer- and not a PC-related problem?

[isofan]

I get a lot of CRC errors since I run Sabnzbd on my Windows Home Server (WHS), but I am not sure if it is WHS, or XLned, or simply "bad" usenet posts. I am currently running Prime95 to check for memory errors, but haven't found any so far. Could you imagine what might cause these errors, especially when saving the downloads to a WHS share?

You should be suspicious when CRC errors occur after par2 verification finds no problems.
You'll have to do some diagnostics yourself.
The easiest test is to re-download a failing job on your normal PC.
One of the previous posters said his problems were gone after disabling QuickCheck (Config->Switches).
It's worth a try, but please do report any positive or negative results.

Hi,

I unchecked this option, but it didn't really alter anything.

I have recently downloaded another set of RARs, 10 of 20 had between 1 to 4 defective RARs.

After redownloading the defective RARs, 8 of the 10 worked, but 2 of them are stilldefective, no matter how often I redownload them.

I have no idea why whis actually happens. I have checked my RAM over and over again, but cannot find anything with Memtest86 or Prime95