CRITICAL: Corrupted files because of full drive

[Gfy]

Configuration:
Version: SABnzbd Version: 0.5.4
OS: openSUSE 11.3
Install-type: python source
Skin (if applicable): smpl Version: 1.3
Firewall Software: None
Are you using IPV6? Yes
Is the issue reproducible? Yes

I have around 3 gigabytes of free space on my root drive. Everything I download is to an external disk.
Temporary Download Folder: /home/username/link_to_map_on_external_drive/incomplete
Minimum Free Space for Temporary Download Folder: 100M
Completed Download Folder: /home/username/link_to_map_on_external_drive/complete
Post-Processing Scripts Folder: I use awescript

Article Cache Limit: 400M

Only Get Articles for Top of Queue: disabled
Post-Process Only Verified Jobs: enabled
Default Post-Processing: +Repair

Pause Downloading During Post-Processing: disabled
If you enable this option, the bug will not occur. I use this as a work around to be sure no corruption will find place.

System Folders
In "/home/username/.sabnzbd"
Cache Folder: cache

Problem description:
When the downloads reaches speeds of 10MiB/s, having a large queue and on 5+ year old hardware, then the root partition where the cache folder is located, will get full. Stuff will get discarded and later queued items will not be successful after par2 repairing because too much is thrown away.

Expected solution:
Pause the queue when the drive with the /home/username/.sabnzbd/cache folder is (almost) full. Keep the downloaded articles in memory instead of discarding them because the drive is full. First process completed files (and memory) before resuming download.

Similar problem:
An other (related?) problem I have: When I click pause, the downloads do not pause and just keeps downloading. The following does appear at the same time:
It says the complete path does not exist, when it actually does exist.
I start sabnzbd when the external drive isn't coupled. The queue will not be empty because of the folder it watches, so I change the complete path to the external drive. The incomplete folder is still on the root partition.
Result: I have complete downloads in the incomplete folder of the root partition, because I get errors the complete path does not exist. However, _UNPACK_name.download folders do appear in on the external drive! These folders contain exactly 2 random rar archives from the download with 0 byte size. There is plenty of space on that drive.
But it doesn't stop there, because it keeps downloading, some files of the later queued nzbs start to be incomplete. After that, all the next files will be 0 bytes and it just keeps going to the next nzb. This is on the incomplete folder at the root partition: /home/username/download/incomplete (path when the configured directory is not available at startup)

I expect sabnzbd to pause because there isn't enough space for the non processed downloads.
I think I couldn't manually pause before the drive was full. The label in the gui did show the queue is paused, but my traffic monitor and the label beneath the pause, showed traffic.

[shypike]

You should keep cache and incomplete on the same drive and
set a minimum free disk space in Config->Folders.
That way downloading will be halted before the disk is really full.
You can set a separate path for cache.

BTW: the issue is moot for the coming 0.6.0 release, because that one will
cache in the job folders in incomplete.

Downloading after pause:
All outstanding requests will still be handled, so pause means that no
new requests are sent to the servers.
The alternative is to discard incoming articles, which will not please pay-per-MB users.

[Gfy]

You should keep cache and incomplete on the same drive and
set a minimum free disk space in Config->Folders.

I have set the disk limit and the second time both were on the same drive!
It's a huge bug. I hope I won't have the same problem again with the next release.

Downloading after pause:
All outstanding requests will still be handled, so pause means that no
new requests are sent to the servers.
The alternative is to discard incoming articles, which will not please pay-per-MB users.

Yes, I know how that works. But it was still downloading 10min after pause at full speed!

[shypike]

Did you set the limit above 100M?
When writing to the cache, the limit isn't checked.
It's only checked when writing the completed file.
So you should set the limit above the size of two typical files.
It's not ideal, but checking disk size can be a costly operation on some OSses,
so we don't do it when writing to cache.

Did the UI actually show "paused"?
I say that, because sometimes the UI can be a bit unresponsive.
Do you have many connections and many servers?
For one server and  a few connections, 10 minute is too much delay.

[Gfy]

Did you set the limit above 100M?

See first post. The cache limit (ram) is set higher than the free disk space. Can this cause problems?

When writing to the cache, the limit isn't checked.
It's only checked when writing the completed file.
So you should set the limit above the size of two typical files.
It's not ideal, but checking disk size can be a costly operation on some OSses,
so we don't do it when writing to cache.

I'm getting confused here. There is the cache limit: the stuff in ram; the cache folder on the disk (where it its never checked and causes this bug?) and the incomplete/complete folders on disk where you can set a limit.

I remember I have had this same bug in a somewhat older version of NewsLeecher too: cache folder on c drive while downloads on another. Result: incomplete files.

You should keep cache and incomplete on the same drive

This is a bad assumption, especially since users are allowed to set absolute paths for the download locations, but to set the cache folder on another drive, the user needs to know about the existence of links. Not every user is an IT guru! (I have not tested absolute paths work there too, but the UI makes me strongly believe it is not possible.)

and set a minimum free disk space in Config->Folders.

I am not sure, but isn't this set by default like it should be? Then there is a very small chance of data loss because you are pretty sure the last files from cache can be written on disk(?) and less chance to conflict with other programs that need the space.

BTW: the issue is moot for the coming 0.6.0 release, because that one will cache in the job folders in incomplete.

I don't know how you are going to implement it/implemented it and I don't know the current situation, but can't this cause a huge performance drop if using the same disk? This is with my assumption that the files in /home/user/.sabnzbd/cache are the not yEnc decoded articles.
And if they are already decoded, do you combine them without reading all the parts and write the file? If it isn't, I think there is a noticeable performance loss. I guess a lot of heavy users aren't downloading on their home partition now, so it's a moot issue for most of them now. Writing/reading/writing/deleting at the same time, and for some over usb, can cause a notable performance loss: not that much of a problem for the downloads I think, but if you want to access other files it can be a problem. I sometimes do have a skip/small hang in a video I'm watching while other programs access the disk.

Anyway, I just try to give you guys some thoughts and be constructive. I really do appreciate all the work done for this really nice app. I also hope I'm wrong at some points because otherwise there are additional problems to take care of

Oh, I have another technical question: aren't there errors thrown when it's not possible to write all the bytes to disk? If so, then it wouldn't be that hard to solve the cache problem without checking for free disk space on the root file system. Linux/Mac will still function fine with no space left. I don't know for Windows. (I got a friend with a mac and he installed a microsoft office update. The update didn't check if the disk was full/didn't check that it could perform the update and crashed the install because there was no space left and left a mess.)

Did the UI actually show "paused"?
I say that, because sometimes the UI can be a bit unresponsive.
Do you have many connections and many servers?
For one server and  a few connections, 10 minute is too much delay.

Yes, the gui did show paused! While just beneath it I saw the speed and the queue getting smaller. I know it can be a bit unresponsive some times, but the change was pretty instant at a given moment. (not always though) I was not using ipv6, so max 3 servers and max 10 connections I guess. There wasn't a delay like it aught to be, it kept going!

[shypike]

The article cache (memory) and the cache folder have little relationship.
However, if your article cache is large enough, no articles need to be written to the cache folder.

The cache folder will be obsolete in the next major release, so we're not going to change things for 0.5.x anymore.

When you don't set a limit SABnzbd will write until the gets disk errors.
It's your decision because there's no safe minimum which is also very small.
Due to the parallelism it's hard to estimate the required space at any given moment.

If you want performance, you need to use a proper article cache.
Later we are going to write articles to a pre-allocated file, removing the write/read/write altogether.

There are few systems that don't screw up when their main disk is full.
Windows will start shouting at you when the disk runs lowish.

These things are going to be addressed at some time.
I'm not worrying too much about them now.
I mean, if you want to do seriuous downloading: make sure you have ample disk space.