High Memory Usage / Potential Memory Leaks Investigation Thread

[shypike]

All angles worth of investigation.
These problems will be addressed, but it will take a while.

Regarding the file handle usage.
Any idea if there is a relationship with intensive web UI usage?
The CherryPy web server and the Cheetah web template handler
are both heavy users of temporary files.
Any clue would try help direct the search.

In the mean time, there is a crude work around to prevent excessive growth.
You can schedule a SABnzbd restart in SABnzbd's scheduler.
Just restart it once of twice day...

[Polydwarf]

In my unscientific and not-rigorous testing, I notice that frequently (however, not all the time) when an nzb goes in via the API (I use sickbeard), 4-5 MB will get used (It seems that the memory gets grabbed after sab culls out par2 files and actually begins the download).

Deleting the download job and sending the same nzb to my watch dir doesn't show the memory usage.

Re-sending it via a sickbeard search will still often cause the memory to go up, so I don't think it's a cached nzb thing.

[shypike]

when an nzb goes in via the API (I use sickbeard), 4-5 MB will get used (It seems that the memory gets grabbed after sab culls out par2 files and actually begins the download).

Reading an NZB costs memory because data structures need to be built up.
Deleting will not immediately return the memory to the OS.
Adding the NZB again will reuse the memory that wasn't returned.

[mdegat01]

So I realize this is somewhat of an old thread but I'm hoping I can bump it up again because I have begun encountering this issue too. I'm having serious problems with my computer locking up. It doesn't blue screen or error in any visible way but nothing works, it hangs. My suspicion was naturally a memory leak so after investigating the programs I had installed since it started showing up I narrowed it down to either SAB or sickbeard (I run both too). I couldn't get any conclusive answer on either but when looking into Sickbeard someone suggests showing how many handles the programs were holding and sure enough my SAB installation is grabbing and holding on to handles as well. It was at 6500 or so when I noticed it, after restarting it dropped to around 300. 2 handles get added every 5 to 10 seconds on my system and they are never released, I assume eventually they hit the point where it clogs my system and I need to reboot.

I've set up a schedule to reboot twice a day (normally I would have to reboot my system at about 9am and my clock would be stopped at 4amish when I woke up indicating it hung then, so twice is necessary), so hopefully that will be a temporary fix. I've also had the debug log printing out for a couple of days now so I can send that to anyone that wants to take a look, I'm also going to try and get a logman print out overnight to get a more accurate read on what is leaking. If it helps my basic systems specs are:
Windows 7 64-bit
6gb RAM
SABnzbd Version: 0.6.5
Antivirus is Norton Internet Security 2011

And for the record I saw the known issues memory leak with windows 7 and antivirus post and I tried to install the hotfix but windows informed me the hotfix was not applicable to my system.

[shypike]

Personally I distrust any system with Norton stuff on it.
Of course that's a personal opinion, based on past experiences.
I test all the time on Windows 7 64bit, but with MS Security Essentials.
I cannot recall ever having seen the handles running up so high.

[mdegat01]

Yea I understand, new norton is not bad though. It's actually considered one of the better ones, even for being low bloat (new norton not old), the only problem I've really had with it is it being tricky to configure which is definitely causing me to worry about it right now. After my subscription runs out I'll re-evaluate and probably end up with MS SE.

In terms of the issues though I found the handles problem. Every time something connects to sab over the network a couple handles are added that never get released. So the reason it runs so high is I have a chrome extension for 2 computers as well as an app on my phone that just connect and refresh to let me know the status of my downloads. So if all 3 of these happen to be running the numbers can build up quickly. Obviously I disabled them now that I know but it appears that any time a network connection is made no matter where it came from anywhere from 1-3 handles are permanently added. That is not limited to those apps, the same thing can be seen if I sit on the sabnzbd page in my browser, even the if I am on localhost and not trying to connect remotely, every time the page is refreshed or changed a couple more handles are added. I believe sickbeard adds some every time it connects as well although that is tougher to see since it usually connects to add a download and each new download creates a lot of new handles that work and release as normal. It is just web connections that cause the problem. This also happens even if my antivirus/firewall is completely shut off so I don't think I can attribute it to Norton.

Still not 100% certain this is causing the computer hangup I see periodically but it is something all the same. Any ideas why this might be happening?

[shypike]

Anything connecting to SABnzbd's API talks to its internal webserver (CherryPy).
Possibly there's some handle leak there.
I'll check CherryPy's release notes and bugtracker.
What complicates things is that we are on a patched interim release, because their
official 3.2 release is both incompatible and has a glaring error in https handling.

BTW: what do you use to track handles?

[mdegat01]

I was going simple and watching it with the taskmanager as I refreshed the page. I have also been logging the sabnzbd process using logman.exe most of the time for the past couple days, it has been saving a full report of the process (everything the task manager could show you if you unhid every column) every 20 seconds for a little while now. That's how I noticed although the handles are going up they aren't continuously going up. I am still printing that out so if they go up again even without the extensions I will take a look at the debug log and see what other activities overlap.

[mdegat01]

I do have Process Explorer and the SysInternalsSuite available if you want me to use those tools to look into it or get something about the process that they can show you.

Thanks for your help with this btw.

[shypike]

It's CherryPy that's leaking handles.
Each time the Plush skin does a refresh the handle count increases by two,
exactly in sync with the refresh rate.
I guess it's because each time the browser opens a new socket to the webserver.
These lost sockets should time-out after some time.
I'll do some more digging.
On my little home server, SABnzbd runs sometimes for weeks but I never noticed
any trouble because of a handle exhaustion...

It's also possible that handles are being consumed by accessing the sqlite database.
In that case, it's the SABnzbd code itself that does something wrong.

[loopdemack]

Maybe by fixing this problem maybe you will fix the main issue from the beginning of this thread.

Also I noticed something more.
"orphaned jobs in the download folder" sometimes could made refresh or browsing almost crippling slow, sometimes by deleting the orphaned jobs from the list would free sabnzbd webgui to normal loading speed.

[shypike]

I think I'm close to a solution for the handles.
The CherryPy session manager has a leak when using in-memory session storage.
The leak is gone when using file-based sessions.
Now testing for side-effects.

Never seen issues with orphaned jobs, but I'll check anyway.
Which amount of orphans are you talking about? Which skin do you use?

[loopdemack]

SABnzbd Version: 0.6.5 | smpl Version: 1.3
Sometimes its two sometimes its more. But sometimes even if I had 6 the loading speed would be ok.

[mdegat01]

Ah, ok and I do use the plush skin so that makes sense. I'm at work right now and can't seem to access it so I'll change up the skin when I get home and let you know if that stops the handle leaking.

[shypike]

and let you know if that stops the handle leaking.

It won't, each http request will leak a handle.
There really is a bug in the CherryPy module.
I'm now testing if session support can be turned off because I strongly suspect that it's not needed at all.