RC6 using a lot of ram and cpu

[billern]

I've been running RC6 since it came out. It just started doing this thing where it starts using about a gig of ram (or more!) and 100% of one of my CPU cores.

I'm running a fresh install of Windows server 2008 r2 64 bit on a quad core athlon CPU with 4 gigs of ram.

The two times this has happened seemed to start when I have tried to add a bunch of nzb files (like 20+ at a time) to the directory sabnzbd is set to load files from. The software becomes more and more unresponsive, uses one of the cores, and starts sucking up ram. It gets to a certain point and then the web-server stops responding. Some of the nzb files are left in the loading directory so it seems like it may be freezing up in the process.

I'd prefer to avoid deleting my queue/cache/config file but I'll do that if necessary.

Does anyone have any suggestions of what I can do?
thanks!

edit: here is my info
Version: 0.5.0 RC6
OS: Windows Server 2008 R2 x64
Install-type: Windows Installer
Skin (if applicable): Default
Firewall Software: None.
Are you using IPV6? No
Is the issue reproducible? It seems to have happened on my a couple times. Restarting the computer and/or SABNZBD doesn't seem to clear it up. Last time, deleting the queue seemed to have temporarily fixed it.

Also, it finally started up after a long delay. I got this error message:
2010-02-08 21:02:32,157 ERROR: Loading C:\Users\Administrator\AppData\Local\sabnzbd\cache\SABnzbd_nzo_myb00f failed

[shypike]

Did you use 0.5.0 Beta/RC releases before?
RC6 is not much different from other releases in this area.

How much data does the average NZB download?
Do you have some third-party utility communicating with SABnzbd
(problematic when it uses a very high query rate).
Do you have a very high refresh rate on the skin.
Which skin do you use (smpl, Plush or Classic)?

We have also seen crazy behavior caused by some virus scanners
and software firewalls.

[billern]

This is my first beta release I've tried so there is nothing to indicate that this is an issue that started with RC6.

I haven't configured the skin. I believe that means I'm using classic. No software firewall or antivirus running on the server and no third party utilities communicating with sabnzbd.

Some of the nzbs are pretty large. At times I may have more than 1,000GB of downloads queued. Right now I have about 500GB queued from a total of 90 separate nzb jobs.

note: I hope this isn't relevant but I haven't cleaned out some of the nzb directories for a long time. My archive of downloaded nzb files is huge (for duplicate checking).

[inpheaux]

I haven't configured the skin. I believe that means I'm using classic. No software firewall or antivirus running on the server and no third party utilities communicating with sabnzbd.

Some of the nzbs are pretty large. At times I may have more than 1,000GB of downloads queued. Right now I have about 500GB queued from a total of 90 separate nzb jobs.

Try switching to a theme that has queue / history pagination, like smpl or plush. Your high ram usage may be due to SABnzbd having to generate all that html every time you hit the queue, and then not releasing it because it will probably just need to use it again the next time you hit the page. This is one of the reasons why smpl is the new default theme.

[billern]

I tried smpl and it is still using about 500 megs of ram. It is running a little better but it is seizing up.

I tried to delete a couple nzbs in the end of the list and then I got this:
Unrecoverable error in the server.
Traceback (most recent call last):
  File "cherrypy\_cpwsgi.pyo", line 79, in setapp
  File "cherrypy\_cpwsgi.pyo", line 219, in get_response
  File "cherrypy\_cprequest.pyo", line 579, in run
TimeoutError

[shypike]

This is probably just a symptom of the heavy load.
As such it does no harm, the Web UI is quite crash resistant.

To be honest we haven't tested under such loads.
And indeed "Classic" is the worst skin to use in these circumstances.
500 MB is not unexpected with this queue size,
also since the Python garbage collector is reluctant to return memory to the OS.

We will investigate further.
I assume downloading does run properly?

[billern]

I'm using the smpl skin now and the symptoms are still there.

Downloading runs okay. Whenever I try to do something that slows it down (like delete an nzb file or two or add new nzb files to the directory that automatically gets loaded), the web interface becomes unresponsive and the download speed temporarily plummets from my normal speed of about 1,000 KB/sec to less than 100 KB/sec. While this is happening, sabnzbd is using 25% of my cpu processing power (1 of my cores on my 4 core cpu) so it looks like that it the bottleneck.

[nunofgs]

I can confirm this issue. Please see:


This only started the other day that I upgraded to RC6. This is a virtual machine with ubuntu server that has 512MB assigned to it. Never had any RAM issues, even while downloading 30+ GB nzbs. Suddenly, while downloading NOTHING, sabznbd eats up memory to the point that simply ssh-ing to the machine is unbearable.

I do have a theory. See all those apache processes on the screenshot? That's probably because I am using apache with a ProxyPass rule to be able to access sabnzbd through apache. My guess is that apache isn't killing the proxy and somehow sabnzbd doesn't release memory. Even if that is the case, this is still a fault with sabnzbd, as this did not happen in earlier betas.

I am using the plush skin. I will try using the smpl skin and post the results.

[shypike]

Doesn't ring a bell.
Can you backtrack to when this started?
Why are you so sure it's an SABnzbd problem and not an Apache issue?

[stester]

I'm a different person than the OP, but I would just mention that
I'm using 3x as much RAM with the beta RCs than I was with 0.4.12 stable.
50MB vs 150MB for my small queues of around 5 items. It's not a complaint,
but I don't get what is filling up all that RAM.

[shypike]

SABnzbd's 0.5.0 code is much larger (more features),
it uses more libraries and also the QuickCheck feature
uses extra memory (no use turning it off  ;))

Everything comes at a price.

BTW we did manage to reduce the excessive peaks
that 0.4.12 has when reading large NZBs.
Also stability has been a bigger concern, we'll look
at memory use later again.

[billern]

Maybe through the quickcheck into another process or make it multi-threaded so the different components stay responsive.

Part of my problem seems to be that that sabnzbd is doing something inefficiently (I swear some of them seem to be relating to adding and deleting nzb files) which is slowing down the responsiveness of the web interface and some of the other components while the CPU is busy doing other things.

[elmer91]

I have noticed similar behavior while upgrading from 0.4.x series to 0.5.x (RC6 and previous since Beta1)

SAB is running on a small embedded server (500 MHz ARM).
It was enough to maintain a 1 Mb download/decoding smoothly with 0.4.x.

Since 0.5.x upgrade, I notice periodic peaks and slow downs on download speed.

I have investigated a little bit more and narrowed my searches around the decoding thread.
Looking at the logs, I can see messages like "delaying downloader" / "undelaying downloader" (can't remember the exact term).
Looking at the cache directory, I can see LOTS of article...

At download end, the decoding thread still processes for a long time before checks will be started...
It seems the decoder thread cannot decode quickly enough (relatively to the download speed).

I never noticed such delay with 0.4.x

Global performance comparison:
0.4.x 1000 KB/s (just under 100% CPU)
0.5.x 560 KB/s (100% CPU, system overloaded)

I had a look on decoder.py and compared 0.4.x and 0.5.x versions...
Nothing particular except uuencode detection changes (may eat a little bit more CPU)

My python installation is still the same, yenc correctly installed.

I notice also that 0.5.x needs a little more RAM that 0.4.x (lets say 20% more for a single 5Mb job).

With 0.5.x system is completely overloaded -> SSH is unresponsive.
I changed my startup script to 'nice +10' SAB and then to be able to use again my system.
Never had this problem with 0.4.x and lots of jobs (same provider/same download speed).

[shypike]

I have investigated a little bit more and narrowed my searches around the decoding thread.
Looking at the logs, I can see messages like "delaying downloader" / "undelaying downloader" (can't remember the exact term).
Looking at the cache directory, I can see LOTS of article...

At download end, the decoding thread still processes for a long time before checks will be started...
It seems the decoder thread cannot decode quickly enough (relatively to the download speed).

The "delaying" messages are a symptom that the system cannot keep up.
It temporarily delays the downloading. Can happen in 0.4.12 too.
The decoder hasn't changed but it's more likely that other parts of the system are taking more CPU.
One thing is QuickCheck (on the fly verification), it takes some time, this is payed back in skipping
par2 verification in most cases.

Increased RAM is just a consequence of having more features.

SABnzbd was never very good at supporting minimal systems.

Focus for 0.5.0 has been features and stability (especially in combination with lousy servers),
not performance.

[shypike]

Global performance comparison:
0.4.x 1000 KB/s (just under 100% CPU)
0.5.x 560 KB/s (100% CPU, system overloaded)

Have you tried with QuickCheck off? Does that bring performance
to the old level?

(I don't have a system slow enough to test this.)