Option to fill missing parts with zeroes

Want something added? Ask for it here.
Post Reply
kukushka
Newbie
Newbie
Posts: 4
Joined: January 21st, 2012, 2:51 pm

Option to fill missing parts with zeroes

Post by kukushka »

Right now missing parts are just getting skipped. The problem is that resulting filesize is different and if .rar files have recovery record - it's not possible to use it. so please-please, if it's possible, make that option so the resulting file could be restored.
Oh, btw, i saw a topic about recovery volumes - tbh, i didn't quite understood, last i checked, winrar was shareware, not payware. Anyway auto recovery using recovery record and/or volumes would be nice, but if it's not possible in auto mode - please make it possible for a future recovery in manual mode by giving the option of keeping the filesize intact (basing on the uniformity of size of posts). (probably in the case of missing par2) thank you
User avatar
shypike
Administrator
Administrator
Posts: 19774
Joined: January 18th, 2008, 12:49 pm

Re: Option to fill missing parts with zeroes

Post by shypike »

Why do you think that filling zeros has any actual use?
It isn't possible to insert zeros for missing parts, because the NZB format doesn't
specify the actual amount of useful data in an article (only the raw byte count of the article).
So only extending the file with zeros would be possible and I have serious doubt that will solve anything.
kukushka
Newbie
Newbie
Posts: 4
Joined: January 21st, 2012, 2:51 pm

Re: Option to fill missing parts with zeroes

Post by kukushka »

shypike wrote:Why do you think that filling zeros has any actual use?
It isn't possible to insert zeros for missing parts, because the NZB format doesn't
specify the actual amount of useful data in an article (only the raw byte count of the article).
So only extending the file with zeros would be possible and I have serious doubt that will solve anything.
..actually there's no need to make a fake post, it's just for pure binary part that will come out of it..
just made a field test - uploaded intentionally something with a missing part, downloaded nzb, figured out the missing part, doubled the previous string (with a changed part number), downloaded "custom" nzb with sabnzbd, it gave crc error check (of course), then - rar r badarch.rar - and voila, worked like a charm. ..before this with skipped part rar got stucked.
..if it can be useful, i can pm you links to nzb's or it can be easily found by searching for mvo in .boneless . the missing part has a #24
User avatar
shypike
Administrator
Administrator
Posts: 19774
Joined: January 18th, 2008, 12:49 pm

Re: Option to fill missing parts with zeroes

Post by shypike »

So, you're asking us to compensate for a bad design of Winrar?
It's rather lame that rar would need fillers.
Are peope actual posting self-repairing rar files instead of par2 files?
I've never seen such a post.
kukushka
Newbie
Newbie
Posts: 4
Joined: January 21st, 2012, 2:51 pm

Re: Option to fill missing parts with zeroes

Post by kukushka »

shypike wrote:So, you're asking us to compensate for a bad design of Winrar?
It's rather lame that rar would need fillers.
Are peope actual posting self-repairing rar files instead of par2 files?
I've never seen such a post.
nobody's perfect. the whole concept of putting binaries into something that was intended initially for text is far from being ideal as you may imagine too. rar recovery record scenerio works with corrupted bytes instead of skipped ones - so why not help him just a little bit to do its job if there is such a possibility. sometimes people post bad par2, bad archives, recovery record generally operates with smaller blocks than par2 does, files can come from ddl where rr is quite common and just dumped to newz..
another variant - the big set of archives with rr in every volume and par2 for the whole set. a good part of archive got lost so it's beyond repair with par2. on other parts - just a few blocks are missing. rr can deal perfectly with this situation.
so there are scenarios where par2 can't help while rr can. people use rr - jbinup, warezfaq guides recommend to use it (while, au contraire, quickpar guide discourage it).par2 should be present in normal situation - i agree on that, but if for whatever reason it fails - there is a relatively easy variant to have a second line of defense.
ps didn't experimented with it yet, but probably, even if there's no rr, rar will restore more files from a "full" broken archive than from a "short" variant.
joetilbone
Newbie
Newbie
Posts: 1
Joined: July 6th, 2012, 12:42 pm

Re: Option to fill missing parts with zeroes

Post by joetilbone »

Make use of rar recovery thanks to you are able to recover muli-part archives of more than 4GB
kenmacd
Newbie
Newbie
Posts: 1
Joined: August 12th, 2013, 5:58 pm

Re: Option to fill missing parts with zeroes

Post by kenmacd »

This is kind of an old feature request, but here's how I'd find the option useful:

Sometimes there are missing articles for some file, but there also exists a torrent file using the same set of RAR files. Currently running a Bittorrent hash-check it seems to reject the files and download a lot of data. If the file was zero-filled the data would be more usable.
funberry
Newbie
Newbie
Posts: 8
Joined: January 14th, 2017, 12:50 pm

Re: Option to fill missing parts with zeroes

Post by funberry »

I'd like to add myself to the long-ignored voices asking for this feature:

Please add the option to pad empty space when assembling downloads comprising missing articles. I've used this feature in other clients and it is essential for repairing downloads using torrents in the case that par2 files are absent or insufficient.

shypike mentions in this General Help thread https://forums.sabnzbd.org/viewtopic.ph ... 73&p=86808 where a user is trying to enable this feature:
shypike wrote:I have my doubts about whether there's a significant number of people actually repairing Usenet downloads using torrents.
I certainly have no doubt that very few users are repairing sabnzbd downloads using torrents, and that is because sabnzbd "mangles" the assembly, rendering this technique mostly useless. Please excuse my insulting terminology, I do appreciate that skipping missing articles is quicker and saves disk space, and would be the preferred option for some. But that is a reason the feature should be an optional switch, not a reason to reject the feature.

A search through these forums finds a significant number of users that want this feature. I can only guess how many more are looking for it but haven't bothered to register and make a forum post.

I've even found that one user wrote a patch implementing the feature: https://bugs.launchpad.net/sabnzbd/+bug/615535

There's an indication in 2010 that the patch will be added when the article cache is no longer needed. I don't think that ever happened. Users like myself run cache_limit = -1 to disable the cache. Perhaps that could be a prerequisite to enabling this feature, if necessary.

Anyways, I hope this gets some serious reconsideration. Thanks for reading.
User avatar
shypike
Administrator
Administrator
Posts: 19774
Joined: January 18th, 2008, 12:49 pm

Re: Option to fill missing parts with zeroes

Post by shypike »

One big issue is that we would need to guess the amount of missing bytes from the context (so, the other articles).
The NZB files only says how many raw bytes an article contains.
The amount of actual data is different and cannot be determined beforehand.
funberry
Newbie
Newbie
Posts: 8
Joined: January 14th, 2017, 12:50 pm

Re: Option to fill missing parts with zeroes

Post by funberry »

Yes, I think you've mentioned this in older threads. And the implication is that the "guess" may be wrong? If that's the case, do you have an idea of how often it might be wrong? Let's say it's right 99% of the time, surely that would be acceptable? I will note that other newsreaders (if they are using downloaded articles to guess) do make it work in my experience.

Also, the patch I linked to (https://bugs.launchpad.net/sabnzbd/+bug/615535) looks to use a different technique? Namely, writing data to the correct offset in the file based upon the =ypart directive. Would that be another solution? The author claims that it works.
User avatar
safihre
Administrator
Administrator
Posts: 5521
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Option to fill missing parts with zeroes

Post by safihre »

We have an idea how to implement it, however, we would like to combine it with more optimizations of the writing process so it might take a while!
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
Post Reply