start verifying while downloading

[razorhail]

is is somehow possible to tell sab to already start verifying downloaded archives while still downloading the rest?

lets say Im downloading a nzb with 50 splitted archives....

as of now, sab downloads all 50 files, and after they're all finnished, it starts to verify them.


but its far more efficient to start verifying the first files, before all 50 have been downloaded.....

[shypike]

No, it's not possible.
Ideally, you should not make aggregate NZB's.
For a future release we will be looking at auto-splitting aggregate NZBs,
but this isn't without pitfalls.

[razorhail]

No, it's not possible.
Ideally, you should not make aggregate NZB's.
For a future release we will be looking at auto-splitting aggregate NZBs,
but this isn't without pitfalls.

what dyou mean by "aggregate"?
I'm not combining several downloads into a single nzb-file

I'm talking about e.g. a 4GB DVD-Image-file that is splitted into 40x100mb-archives (.r00, .r01, .r02,...)

[shypike]

OK, I see.
I'm sorry to say that what you propose it not possible.
SABnzbd does a first on-the-fly verification, called QuickCheck.
If everything comes in correctly then no further action is needed and unpacking will start.
When an error is detected, a par2 repair is needed.
This repair can only be done on the full set of files.
That's the way the par2 method works.

[razorhail]

I understand
but wouldn't it be possible to easily implement this kind of functionality (and make it at least a (non-default) option) ?

because this way, if the first downloaded files are already incomplete it would be wise to also download the repair-blocks



because as of now, sab works like this:
- downloading files (no real processor activity - wasted processing power)
- quickcheck fails
- start verifying every file (no download activity - wasted bandwidth)
- start downloading additional files (no real processor activity - wasted processing power)
- start repairing

It's just more time- and resource-efficient if downloading and verifying would be done simultaneously



but this SHOULD be generally possible right? :-P

I'm just wondering if this 'feature' is somewhere on the "product backlog" to be implemented at some point in the future

[shypike]

Maybe I don't explain it right.
The par2 tool needs to have *all* the files to do a repair run.
par2 is an external tool that we have no control over.
It doesn't work in a "streaming mode".

[razorhail]

haha, I do understand what you're saying. 

I'm not saying that the REPAIR should start before all files have been downloaded - it doesn't work, i get it

I'm saying that the par-VERIFICATION could/should start before all files have been downloaded


e.g.: the first file could be verified while the second file is downloading,
and the second file could be verified while the third file is downloading.....and so on

and by the time the last file has been downloaded, only this last file has still to be verified.
after that is done, (and enough repair-blocks for the repair are available) the repair could start 'immediately'.


verification and repair are 2 separate things.
the files have to be verified first, so it can be determined if they are incomplete - and how many blocks are missing

[shypike]

It already does that: it's called QuickCheck.
You may be confused by the fact that each repair is preceded by a verification run.
That's just the way the par2 works.

[Goetterdaemmerung]

Hello,

I had a similar question, and I don't think I understood the discussion that happened in this thread.

Let's say I download a 1GB ISO from usenet, with 200MB of par2 parity archives. sabnzbd.log shows that this is what happens:

1) part001.rar is downloaded; then part087.rar is downloaded; etc (the .nzb did not order the files alphabetically). The log shows "Decoding <file>" for each part###.rar file, and no other information.
2) eventually the .par2 index is downloaded
3) the rest of the contents of the .nzb are downloaded. Download completes.
4) sabnzbd.log shows INFO category logs: "Starting PostProcessing", then "Par2 check starting", then "Running repair", then "Scanning".
5) Five minutes pass.
6) Repair finishes. No repair is needed, and sabnzbd begins extracting.

This is not ideal. The following is entirely possible:

1) Sabnzbd identifies the .par2 index file in the nzb, and downloads it first. Sabnzbd sorts the nzb contents alphabetically (or something smarter).
2) begin downloading part###.rar
3) finish downloading part###.rar, begin par2 verification of part###.rar (finishes 20 seconds later), and simultaneously begin downloading part(###+1).rar
4) ...
5) finish downloading final part###.rar, begin par2 verification of final part###.rar
6) 20 seconds pass.
7) finish par2 verification of final rar. If no repair needed, extract. Otherwise, do the normal thing.

A further optimization (also entirely possible) would be to (at least have the toggleable option to) optimistically extract a multi-archive download as soon as archives are verified to have par2 integrity. Obviously extraction would be halted when the first corrupt archive is found -- but in the common case, at least with high-quality providers, verification and extraction would only take a few moments past the completion of the download.

The windows application NewsLeecher does the parallel par2 verification. Why doesn't Sabnzbd?

[inpheaux]

1) Sabnzbd identifies the .par2 index file in the nzb, and downloads it first. Sabnzbd sorts the nzb contents alphabetically (or something smarter).
2) begin downloading part###.rar
3) finish downloading part###.rar, begin par2 verification of part###.rar (finishes 20 seconds later), and simultaneously begin downloading part(###+1).rar
4) ...
5) finish downloading final part###.rar, begin par2 verification of final part###.rar
6) 20 seconds pass.
7) finish par2 verification of final rar. If no repair needed, extract. Otherwise, do the normal thing.

This is what should be happening, just not with actual par verification. We do our own Quick Check process using par2 checksums concurrently with the download process, and then if that fails do full par2 verification and repair after the download finishes.

Doing complete Par2 is only useful if the post isn't 100%. If a post is completely fine (which the vast majority should be), then running complete par2verify is overkill and a waste of cpu cycles, even if done concurrent with the download process.

I guess what we could do is continue to do Quick Check as we currently do, and as soon as one part fails Quick Check start the par2 verify thread *then*. Then there's still no unnecessary par2verify, and par2verify starts as soon as we know we're going to need it. We'd just have to make it tied to the "pause during post-processing" toggle, so we do the current par behavior if that setting is on.

[depl0y]

Was this ever implemented. Verifying still takes ages after a download is complete, while most downloads require no repairs. It takes a lot of time doing this when the complete job is complete.

[shypike]

When you have QuickCheck enabled AND the download contains no errors,
the external par2 verification is skipped.