Jobs 100% Finished (but not finishing) and strange behavior

[SlinkingFerret]

Issues:
100% Finished
Jobs will say 100% finished, but won't leave the queue. I can change their Priority to anything other than Stop.
I verified the par files are downloaded in the temp directory.

The log shows:

Code: Select all

2018-07-08 15:00:25,723::INFO::[nzbqueue:893] Found idle job <nzbname>
2018-07-08 15:00:25,724::INFO::[nzbqueue:907] Resetting bad trylist for job <nzbname>
2018-07-08 15:00:25,760::INFO::[nzbqueue:799] [sabnzbd.nzbqueue.stop_idle_jobs] Ending job <nzbname>

For each file. This seems to be logged about once a minute.

It'll happen with one item, then start downloading the next item until that's 100%... all the way until I have a full queue of 100%.

The CPU for sabnzbd varies from 1-30%.

I noticed two items getting to 98% and they were stuck there with the notice about missing too many parts.
But even these weren't moved to the next stage (history)

cannot perform graceful shutdown from gui
using the ui, it just won't. A box will show up saying "Shutting down", but it won't shut down.
I can wait 3 hours, then just refresh the page and keep using it.

to "shutdown", I have to kill the process (kill -9 pid)

cannot start sabnzbd a second time

After having killed the process, sabnzbd is showing as a "<defunt>" process using the same PID.
Starting as a daemon (sabnzbd -d) will have a 30 second wait, then I regain control - but it hasn't started. No errors in the logs.
Starting without daemon will give me a longer wait (60 seconds?) and tell me that it cannot bind to port 8080.
I'll check the port usage (lsof -i :8080) and see "mono" having taken it. I can kill mono, then nothing is using 8080. Then trying to start sabnzbd again gives it back to mono and gives me the error.

To start sabnzbd, I have to restart ubuntu. Then it starts fine.

on restart the queue is orphaned
I see all of the 100% items on the orphaned jobs tab.
I can add a job back into the queue and it'll start the download over again (it looses the files it had already downloaded) and it completes.

at some point the 100% Finished issue returns

So I have a sort of solution... reboot server and re-download everything.
I'm not sure what condition causes the "100% Finished" to start occurring, other than when the server gets busy.

If these are 2-3 issues, then sorry for loading them all up in a single post. I figured that this might show a pattern someone is familiar with.

OS: Ubuntu 16.04
- running on a virtualization station on qnap
- this is not the qnap version of sabnzbd, but vanilla sab running on vanilla linux
- all variable folders in the settings are pointing to a local path (a symlink of a share from the nas with about 800gb free)
SABnzbd version: 2.3.4
Other software on server:
- Sonarr
- Radarr
both start fine and have no issues

[safihre]

Can you check the servers that you use and disable any un-reliable servers? How many servers do you have?
All other problems are related to force-killing it, making it unable to do the proper cleanup.

[SlinkingFerret]

Can you check the servers that you use and disable any un-reliable servers? How many servers do you have?

I only use giganews for a server.

You think something with the server is causing the jobs to not complete (proceed past downloading to 100%) and prevent a proper shutdown/restart?

I don't believe my provider has a usenet server for me to access. Is there another server I should try to add?

All other problems are related to force-killing it, making it unable to do the proper cleanup.

Seems likely. So how do I not force kill it? Is there a command line graceful shutdown?
I'm not sure if the frontend shutdown/restart request made it to the backend. If not, then I could manually trigger it.

[SlinkingFerret]

ooh, just got it to happen again.

Moved a few orphaned jobs back to the queue and that worked fine. They completed, processed and post-processed.

Then I told sabnzbd to add the whole orphaned list back to the queue. It was working fine for a little while, but started to get backed up.

History has one item in "direct unpack" status (seems to be stuck) and another was in "unpacking" for 25 minutes (usually takes about 1 minute) and finally went to post processing.

I have 22 other items in history stuck on "waiting"
I have 3 items in the queue at 100% and it's downloading a fourth (12%)

used cache is going from back 'n forth between 500mb to 499mb with 690 articles.
The other items in "status and interface options" were saying "loading" for a while, now they say "connection failed!"

It seems like the two items unpacking at once filed up the cache and broke the queue for loading completed items into history for processing.
The server is still running and sending out information, but doesn't seem to be responsive to taking in any new calls. That would explain the shutdown/restart not working.

I'm going to see if I can kill the direct unpack and see if that helps at all... and see about increasing the cache from 500mb

[SlinkingFerret]

Changed cache limit from 500.1 M to 5000.1 M.
cache is constantly building up to about 50MB as 64 articles are loaded, then it starts the cycle again.
The Status and interface options became responsive again and I was able to perform a restart.

It's only working on one item in the history queue at once, but at a rate of 1-2min per item.
All of the 100% Finished dropped down into history from the queue.

Seems like the cache was filled up and locked the workers mid process so they couldn't grab completed items from the queue... or respond to a shutdown response.
It didn't show more than 500mb being used for cache. I think setting it to 5000MB in settings just cleared it out and got it unstuck.

[safihre]

No that didn't do it, just coincidence.
It has to do with some occurrence of events when servers fail and cause a Retry loop of the failed articles.
The cache up and down is just a file being assembled while it downloads and then written to disk.
These corner cases when something goes wrong with article fetching ads really hard to diagnose :/

[CyberMew]

Interestingly I also had this problem. Was using Radarr to grab a bunch of old movies, the first one went from the Queue to the History and completed no issues. Then, the rest of the downloads in the queue even though when 100% wouldn't shift to the History. I went to change the Article Cache Limit from 256M to 512M and went back to the dashboard to find that the next 100% completed job went from Queue to History. Not sure if this is a coincidence or not. Interestingly, the other 100% jobs remained in Queue, and the job that went into the History has been in Moving ... status for the past 10 minutes already when it's only a 5 mins job to move a 8GB file.

[safihre]

Restart sabnzbd. Disable any less reliable servers or reduce the number of connections to the minimum to saturate the connection.
The Moving thing might be something else though.
You can force jobs to the post processing by setting their priority to Stop.