SAB slows to a halt during post-processing

[SSelmer]

I tried searching for this issue because I know I've seen it mentioned before.

It would seem that the larger the queue, the longer it takes to move files in and out of the queue (which is every time you finish a job).

I've had as little as appx. 100GB in the queue and as much as 2TB in queue and the bigger the queue, the longer it takes. Here's a snippet from top:

Code: Select all

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
 3669 *****    20   0 7266m 6.7g 3180 S   99 86.2  82:57.96 sabnzbdplus

This is from running SABnzbd for appx. 3 hrs!

I've tried to find if there's anything that makes this worse or not, but so far I've struck out. I thought it might be related to "Sort by age" but I don't use that atm.

Any idas how I can get SABnzbd under control? Do I really have to feed it 1-2 seasons of series at a time (one DVD at a time)?

(PS: This behavior has been observed both on OSX 10.6 and Ubuntu 9.10 Server)

[shypike]

How is your "Only get articles for top of the queue" setting?
"On" should give better performance.

I'm not aware of any delay when moving a job from the download queue
to the post-processor queue. That doesn't mean there isn't one.
In general we don't test often with very large queues.
It's on the checklist for later releases.

[SSelmer]

How is your "Only get articles for top of the queue" setting?
"On" should give better performance.

It's on.

Here's the stat just now:

Code: Select all

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
 3669 *****    20   0 7330m 6.7g 3444 S    5 86.8 179:04.65 sabnzbdplus        

And it STILL hasn't finished one DVD (even though it normally takes appx. 1h15m per DVD).

[shypike]

Not sure what I'm supposed to conclude from the "stat".
Could you try another experiment?
Set "Pause downloading during post-processing" on.

[SSelmer]

Set "Pause downloading during post-processing" on.

Done, no effect.

The problem lies somewhere in the processing of jobs in the download-queue itself.
If I pause a single download it happens without a pause.
If I add a new download-job to the queue then SABnzbd is stuck for a while doing something.
If I delete a download-job the queue then SABnzbd is stuck for a while doing something.

When a download-job is done and the "stuck-ness" has passed, then post-processing starts and goes at full speed (until I need to add a job to get more par2-files or the next download-job is ready to be post-processed). With "Pause downloads..." off then download-speed drops to 0kB during the "stuck-ness" and as soon as the post-processing starts, downloading resumes at full speed.
(OK, so I guess the original title of this post is a bit misleading... My bad. )

[shypike]

Tested on Windows 7 with about 180 jobs of 7 G each in the queue.
No problem whatsoever.
OK, Classic is not really usable because of the 180 jobs in the queue,
but Plush and smpl are OK.

What I did see once is that smpl doesn't always keep the display up to date.
After refreshing, I saw that 2 jobs jumped to history at the same time.
(Despite the fact that I had refresh set to 2 seconds).
Logging showed the expected timing.

I haven't had time to test on Linux.
OSX needs to be tested by other team members.

[SSelmer]

Ok, I'll test how this is with smpl and Plush. Maybe it's an issue with Classic.

EDIT: Tried adding jobs in Plush until it started to slow down (when I added a zip-file with eight 8GB jobs and the queue jumped from ~105GB to 168GB) I realized the problem doesn't seem to be in just Classic. I changed to smpl and added another job (another 8x8GB) and it took even longer. So it seems it's not down to the skin. Maybe it's an OS issue? Right now I can't install FreeBSD, but I'm willing to test this under FreeBSD 8.0 at a later time.

[shypike]

Classic is not suitable at all for large queues.
It has no pagination and will simply choke on the huge amount of HTML code
(due to the number jobs each of which also has all files included, though hidden).

[SSelmer]

I've converted to Plush for now.. smpl just looks ugly in my eyes.
I've tried to keep my queue down to 150GB (appx. 24hrs worth of downloading) at a time. That makes SABnzbd+ nice and responsive.

[shypike]

Maybe you should download your queue until it's empty
and then restart once with the --clear option.
Possibly your cache folder is so polluted that it slows down disk access.
The current queue design leaks cached article files.
(We're working on that).

Other than that: for me Plush is just as responsive as smpl.
Only Classic is a pain.

[SSelmer]

Tried the --clean option, it removed ~20 orphan articles, no improvement as far as I can see. Downloading halts while adding jobs, halting increases with the size of the queue.

[shypike]

NZB parsing competes with downloading, so it makes sense
that it has influence on downloading.
Again, I cannot reproduce that it is a significant pause.