Wrong error msg in specific situations involving lower than expected free space

[guest999]

I recently download a video using a nzb which downloaded successfully but during unpacking the process ended up with failure & an error msg "password required for archive". When I checked manually I found out the archive in question did not have any password to begin with(I even extracted the video file within it manually to another drive & it played fine). After carefully checking & again downloading these are my findings.

I downloaded nzb for a video file sized 30gb. My final download folder drive had 40gb free space while my Temp folder drive had 140gb free space. The video file was within a regular non-password archive file which itself though was split between multiple encrypted archives the password for which was recognized successfully by sabnzbd using header info. When the download is finished sabnzbd first recreates/extracts the single 30gb archive file using the password info. At this step the available drive space left was 10gb which clearly isn't sufficient to extract the 30gb video file from the single archive file. At this point sabnzbd gave password required for the archive in question error msg. After I created enough free space in download folder drive this does not happen with same setup/conditions.

I think sabnzbd should consider above scenarios while calculating space required in the final download location or there should be some sort of preliminary check for nested archives to correspondingly calculate the free space required in final download location. There may also be a failsafe for such cases where sabnzbd should try extracting the final content to Temp location if enough free space is available there & then delete the intermediary content from Download folder & move back the final extracted content from temp folder to download folder.

[sander]

Nice report!

My thoughts

1) SABnzbd is not perfect in calculating needed space on Temp/Incomplete and Complete and possible category assigned directories: sometimes too alarming, sometimes too permissive (as you experience). Perfecting all the use cases is difficult. IMHO if you're in the business of downloading 30GB images, keep at least 60GB free. Disk space is cheaper than writing, maintaining and supporting difficult code.

2) SABnzbd should not give the wrong error message. So not "password required for archive" if that is not the case. In your sabnzbd.log, can you find the output of unrar? Maybe you have to set logging to +Debug. That info could help to let SABnzbd report the real problem.

[guest999]

I understand the point about free space, it is just that sometimes due to one reason or the other it slips out of mind to free up some space.

At the time of error msg the logging was set to info & after that it downloaded successfully after making enough free space in download folder drive. I will see if I can recreate this same scenario again to get this same error msg in next 1-2 days. Btw my main suggestion regarding this is to at least give user some option/msg to check free drive space in download folder drive & if there is some big file then user can at least extract the final content without having to re-download the same thing again(& also getting same error msg again if still not enough free space). I only have a basic knowledge of programming so only assuming that it should be easy to write a few lines of code to check free drive space in such failed extraction scenarios & display a msg to user saying "please check free drive space before trying download again".

[sander]

With unrar, we depend on what unrar says. That triggers what we report in SABnzbd.

So hopefully you can provide that DEBUG logging from your sabnzbd.log

From my sabnzbd.log:

unrar output when password on rar file, but not provided, leading to SABnzbd saying "Unpacking failed, archive requires a password (...)"

Code: Select all

2021-12-15 15:21:47,283::DEBUG::[misc:1098] Popen arguments: {'stdin': -1, 'stdout': -1, 'stderr': -2, 'startupinfo': None, 'creationflags': 0}
2021-12-15 15:21:47,350::DEBUG::[newsunpack:828] UNRAR output
UNRAR 6.10 beta 3 freeware      Copyright (c) 1993-2021 Alexander Roshal


Extracting from /media/zeegat/incomplete/test_password_is_geheim/my_file_10GB.part01.rar

Incorrect password for my_file_100MB.bin

Another unrar output ... leading to "repair not possible"

Code: Select all

Repair is required.
5 file(s) are missing.
1 file(s) are ok.
You have 769 out of 1998 data blocks available.
You have 30 recovery blocks available.
Repair is not possible.

[guest999]

I will have to recreate the entire scenario which means shuffling of data so may take some time, will try to post it(if able to reproduce error) this weekend.

[safihre]

The error depends on your version of unrar, some versions of unrar report password failures similar as disk full.
What OS are you on?

[guest999]

Win 10 21H2, winrar not installed. This is the relevant portion from the debug log:

Code: Select all

7-Zip (a) 19.00 (x86) : Copyright (c) 1999-2018 Igor Pavlov : 2019-02-21

Scanning the drive for archives:

1 file, 209715200 bytes (200 MiB)

Extracting archive: \\?\X:\temp7z\Dune 2021 Remux njc\KIXHCUasLnSCIfcLk3X6isXXNsmVpUoI.7z.001

--

Path = \\?\X:\temp7z\Dune 2021 Remux njc\KIXHCUasLnSCIfcLk3X6isXXNsmVpUoI.7z.001

Type = Split

Physical Size = 209715200

Volumes = 147

Total Physical Size = 30726095852

ERROR: Can not set length for output file : There is not enough space on the disk. : \\?\Q:\Newshosting\_UNPACK_Dune 2021 Remux njc\KIXHCUasLnSCIfcLk3X6isXXNsmVpUoI.7z

ERROR: There is not enough space on the disk.

Archives with Errors: 1

Sub items Errors: 1

2021-12-18 23:39:52,578::INFO::[misc:1098] [sabnzbd\newsunpack.py.seven_extract_core] Running external command: ['X:\\Temp\\SABnzbd-3.4.2-win64-bin\\SABnzbd-3.4.2\\win\\7zip\\7za.exe', 'x', '-y', '-aou', '-tsplit', '-ssc-', '-p', '-o\\\\?\\Q:\\Newshosting\\_UNPACK_Dune 2021 Remux njc', '\\\\?\\X:\\temp7z\\Dune 2021 Remux njc\\KIXHCUasLnSCIfcLk3X6isXXNsmVpUoI.7z.001']

2021-12-18 23:39:53,616::DEBUG::[newsunpack:1042] 7za output: 

7-Zip (a) 19.00 (x86) : Copyright (c) 1999-2018 Igor Pavlov : 2019-02-21

Scanning the drive for archives:

1 file, 209715200 bytes (200 MiB)

Extracting archive: \\?\X:\temp7z\Dune 2021 Remux njc\KIXHCUasLnSCIfcLk3X6isXXNsmVpUoI.7z.001

--

Path = \\?\X:\temp7z\Dune 2021 Remux njc\KIXHCUasLnSCIfcLk3X6isXXNsmVpUoI.7z.001

Type = Split

Physical Size = 209715200

Volumes = 147

Total Physical Size = 30726095852

ERROR: Can not set length for output file : There is not enough space on the disk. : \\?\Q:\Newshosting\_UNPACK_Dune 2021 Remux njc\KIXHCUasLnSCIfcLk3X6isXXNsmVpUoI_1.7z

ERROR: There is not enough space on the disk.

Archives with Errors: 1

Sub items Errors: 1

2021-12-18 23:39:53,641::INFO::[notifier:122] Sending notification: Error - Unpacking failed, archive requires a password (KIXHCUasLnSCIfcLk3X6isXXNsmVpUoI.7z) (type=error, job_cat=None)
2021-12-18 23:39:53,641::ERROR::[newsunpack:1000] Unpacking failed, archive requires a password (KIXHCUasLnSCIfcLk3X6isXXNsmVpUoI.7z)
2021-12-18 23:39:53,641::INFO::[newsunpack:271] 7za finished on \\?\X:\temp7z\Dune 2021 Remux njc
2021-12-18 23:39:53,641::INFO::[postproc:432] Unpacked files ['\\\\?\\Q:\\Newshosting\\_UNPACK_Dune 2021 Remux njc\\KIXHCUasLnSCIfcLk3X6isXXNsmVpUoI.7z', '\\\\?\\Q:\\Newshosting\\_UNPACK_Dune 2021 Remux njc\\KIXHCUasLnSCIfcLk3X6isXXNsmVpUoI_1.7z']
2021-12-18 23:39:53,641::INFO::[filesystem:319] Checking if any resulting filenames need to be sanitized
2021-12-18 23:39:53,641::INFO::[postproc:436] Finished unpack_magic on Dune 2021 Remux njc
2021-12-18 23:39:53,642::DEBUG::[filesystem:932] Renaming "\\?\Q:\Newshosting\_UNPACK_Dune 2021 Remux njc" to "\\?\Q:\Newshosting\_FAILED_Dune 2021 Remux njc.1"
2021-12-18 23:39:53,642::DEBUG::[filesystem:975] [sabnzbd\postproc.py.rename_and_collapse_folder] Removing dir \\?\Q:\Newshosting\_UNPACK_Dune 2021 Remux njc
2021-12-18 23:39:53,642::INFO::[nzbstuff:1835] [sabnzbd\postproc.py.process_job] Purging data for job Dune 2021 Remux njc (delete_all_data=False)
2021-12-18 23:39:53,642::DEBUG::[articlecache:149] Purging 0 articles from the cache/disk
2021-12-18 23:39:53,642::INFO::[notifier:122] Sending notification: Download Failed - Dune 2021 Remux njc (type=failed, job_cat=*)
2021-12-18 23:39:53,693::DEBUG::[sorting:610] Initial guess for Dune 2021 Remux njc is MatchesDict([('title', 'Dune'), ('year', 2021), ('other', 'Remux'), ('release_group', 'njc'), ('type', 'episode')])
2021-12-18 23:39:53,693::DEBUG::[sorting:364] Using tv sorter for Dune 2021 Remux njc
2021-12-18 23:39:53,709::INFO::[database:293] Added job Dune 2021 Remux njc to history
2021-12-18 23:39:53,709::INFO::[postproc:130] Saving postproc queue
2021-12-18 23:39:53,710::DEBUG::[__init__:962] [sabnzbd\postproc.py.save] Saving data for postproc2.sab
2021-12-18 23:39:53,710::DEBUG::[__init__:894] [sabnzbd\__init__.py.save_admin] Saving data for postproc2.sab in \\?\C:\Users\nitin\AppData\Local\sabnzbd\admin
2021-12-18 23:39:53,712::INFO::[downloader:395] Post-processing finished, resuming download

These are the pics of sabnzbd error & relevant folders:






Trying to open archives in above two folders manually by 7-zip gives error unexpected end of archive meaning these are not complete archives.

This is the pic of Temp folder containing the 29.5gb archive in question & which extracts correctly if done manually:

[sander]

Oh, it's 7zip unpacking "Archive: \\?\X:\temp7z\Dune 2021 Remux njc\KIXHCUasLnSCIfcLk3X6isXXNsmVpUoI.7z.001", which then says "ERROR: There is not enough space on the disk.".

So, this is not about a rar set being unrarred?

[safihre]

Right, so we parse the output of 7zip incorrectly. I'll see if I can fix that.

Your other comment about Sab calculating the required space for nested archives is just way too much work. We only know the space the second stage needs after we unpack the first stage.
You can also just disable Recurivse Unpacking in the settings to prevent this.

[guest999]

Well if it is too much work then I guess it is not worth implementing but if possible can you please do something about my other suggestion of check free drive space in such failed download scenarios & display a msg to user saying "please check free drive space before trying download again" before removing the failed/incomplete files from downloads folder.

[safihre]

But, we don't remove the files, the user does manually.
Or did I miss something?
If the correct error is shown, doesn't the user know disk size is the problem?

[guest999]

For me the files in download & working/temp folder were removed after I restarted the download in sabnzbd by adding the same nzb file again thinking it might be some temporary glitch. If the correct error(of insufficient free space) is shown then it should be fine though.

[safihre]

Yeah of course, if you re-add it will remove them..

[sander]

I tried with different versions of 7z executables, with a disk with too little space:

7z version 16.04 (default on Ubuntu 20.04):

Code: Select all

$ 7z.ORG x  /home/sander/post_this/zip_2GB/random_1GB.7z.001

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
...
ERROR: E_FAIL

Archives with Errors: 1

... nothing about the cause.

7z version 21.06

Code: Select all

$ 7z x  /home/sander/post_this/zip_2GB/random_1GB.7z.001
7-Zip (z) 21.06 (x64) : Copyright (c) 1999-2021 Igor Pavlov : 2021-11-24

Archives with Errors: 1
System ERROR:
errno=28 : No space left on device

So ... clear.

But different wording than the OP's 7z "There is not enough space on the disk."

PS: the "errno=28 : No space left on device" seems to be a Linux/Unixs error message.


PS2:

the Windows-7za (from Wine) gives yet another message

Code: Select all

$ wine ~/git/sabnzbd/win/7zip/7za.exe  x 'C:\zippie\random_1GB.7z.001'

7-Zip (a) 19.00 (x86) : Copyright (c) 1999-2018 Igor Pavlov : 2019-02-21


ERROR: Can not set length for output file : Disk full. : .\my_random_file
ERROR: Disk full.
Archives with Errors: 1
Sub items Errors: 1