Decoder failure: Out of memory

[Puzzled]

I notice it says segment bytes="-21730" for one of the articles, but I don't think that's used in the decoder. It might depend on which usenet server is being used.

[loopdemack]

I downloaded that .NZB with SABnzbd ... and no problem at all: a nice *mp4 in the resulting directory, no problems in GUI nor LOG

Yes as I told its a special circumstances its not that easy.

[Puzzled]

safihre: Maybe you could add a sanity check for the amount of RAM it tries to malloc and give an error that doesn't pause the downloading if it's clearly wrong (negative or > 15M)? Don't know if that's the issue, but at least you'd find out.

[sander]

I notice it says segment bytes="-21730" for one of the articles, but I don't think that's used in the decoder. It might depend on which usenet server is being used.

Ah ... good catch. I can now reproduce with a NZB with all "segment bytes" parts set to negative value, for example 'segment bytes="-728914"'

Crafted NZB here: https://raw.githubusercontent.com/sande ... estnzb.nzb

Code: Select all

2019-06-21 07:22:37,230::WARNING::[decoder:139] Decoder failure: Out of memory
2019-06-21 07:22:37,232::DEBUG::[nzbstuff:293] Finishing import on reftestnzb 100MB auto 2151de41 2019-06-15 [01/19] - "mytestpost.part01.rar" yEnc (1/15) 10485760
2019-06-21 07:22:37,233::INFO::[decoder:141] Decoder-Queue: 0, Cache: 0, 0, 1003487232
2019-06-21 07:22:37,235::DEBUG::[__init__:924] [sabnzbd.nzbstuff.finish_import] Loading data for SABnzbd_nzf_XX_hp5 from /home/sander/Downloads/incomplete/test-negative-segment-bytes-reftestnzb/__ADMIN__/SABnzbd_nzf_XX_hp5
2019-06-21 07:22:37,235::INFO::[decoder:142] Traceback: 
Traceback (most recent call last):
  File "/usr/share/sabnzbdplus/sabnzbd/decoder.py", line 124, in run
    data = self.decode(article, lines, raw_data)
  File "/usr/share/sabnzbdplus/sabnzbd/decoder.py", line 222, in decode
    decoded_data, output_filename, crc, crc_expected, crc_correct = sabyenc.decode_usenet_chunks(raw_data, article.bytes)
MemoryError
2019-06-21 07:22:37,239::INFO::[downloader:279] Pausing

And Sab says "Decoder failure: Out of memory"

Furthermore, SAB GUI says it has downloaded "0 MB / -113 MB" (so: negative), so also SAB itself (not just sabeync) reads & uses those negative values.

[sander]

I've created a github issue: https://github.com/sabnzbd/sabyenc/issues/13

[safihre]

Very strange.
We do check for negative numbers.. Why doesn't that work? :O

https://github.com/sabnzbd/sabyenc/blob ... enc.c#L564

[Puzzled]

Are they using a sabyenc version compiled before this commit? https://github.com/sabnzbd/sabyenc/comm ... 3bbbb0e04b

[sander]

Safihre and I found out the cause in the C code of sabyenc. and I've sent a Pull Request https://github.com/sabnzbd/sabyenc/pull/15
Afther that PR, SAB will not give an OoM anymore, and the download will succeed.
If you run from source and know how to compile and install sabyenc (on Linux juist one command, or three command including the git commands), you can use it now. Otherwise wait for a new SAB release.

The real root cause is of course a wrong NZB: a negative article size in the NZB is impossible and illegal. So SABnzbd should give a warning for that, and/or refuse the NZB completely.

[loopdemack]

Safihre and I found out the cause in the C code of sabyenc. and I've sent a Pull Request https://github.com/sabnzbd/sabyenc/pull/15
Afther that PR, SAB will not give an OoM anymore, and the download will succeed.
If you run from source and know how to compile and install sabyenc (on Linux juist one command, or three command including the git commands), you can use it now. Otherwise wait for a new SAB release.

The real root cause is of course a wrong NZB: a negative article size in the NZB is impossible and illegal. So SABnzbd should give a warning for that, and/or refuse the NZB completely.

Could you speculate which tool did they use to create those crazy nzb files. I blame myself for not backing up erroneous nzb files which made out of memory bug, but I stumble on almost 450 on nzbs.org which is gone now. I will try to ask poster which tool did he used to create such nzb, we should warn indexing sites.

[safihre]

My fault guys, I just made a plain old coding bug... Thanks for reporting and pushing that it was really a bug and not your system! You were right!

[safihre]

I released a new version of SABYenc, 3.3.6.
How do you use SABnzbd, on which platform? I can create a new release for you.

[loopdemack]

I'm using ubuntu.

[sander]

I'm using ubuntu.

The commands to get the fixed sabyenc on your Ubuntu:

Code: Select all

cd
git clone https://github.com/sabnzbd/sabyenc.git
cd sabyenc
git checkout sabyenc-python2 
sudo python setup.py install

Done

[safihre]

(if all the prerequisites are installed, info here how to get those: https://sabnzbd.org/wiki/installation/sabyenc.html )

[sander]

It looks like @jcfp has already updated his ppa for python-sabyenc with the newest, fixed sabyenc. See https://launchpad.net/~jcfp/+archive/ubuntu/sab-addons ... version 3.3.6-0ubuntu1~jcfp1

Cool! Thank you, @jcfp

So ... Ubuntu users ... just run the sudo apt-get update / upgrade, and your SABnzbd on Ubuntu is protected against Out-of-Memory.