Page 3 of 6
Re: High Memory Usage / Potential Memory Leaks Investigation Thread
Posted: July 25th, 2010, 12:13 pm
by sander
In SABnzbd, I've created a thread that each minute will print out the memory usage as seen by heapy. The strange thing: during a download, the memory usage reported by heapy only goes from 8.5 MB tot 12.8 MB. FYI: top -bn1 reports "248m 42m" as VIRT and RES memory usage.
I also tried a "print dir()" in the separate thread to see all variables, but in the thread only the local variables are seen, and not the variables used in other parts of SAB.
So not very useful results so far.
Are the SAB devs still interested in this case, or am I the only one ... ?
Code: Select all
dumpthread active!
Partition of a set of 122643 objects. Total size = 8592560 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 58991 48 3375316 39 3375316 39 str
1 29165 24 1146728 13 4522044 53 tuple
2 320 0 645248 8 5167292 60 dict of module
3 8362 7 568616 7 5735908 67 types.CodeType
4 838 1 528112 6 6264020 73 dict (no owner)
5 8151 7 456456 5 6720476 78 function
6 593 0 326408 4 7046884 82 dict of class
7 604 0 270064 3 7316948 85 type
8 604 0 236320 3 7553268 88 dict of type
9 1897 2 202272 2 7755540 90 unicode
<352 more rows. Type e.g. '_.more' to view.>
Gelukt, na x pogingen: 0
... and dumpthread is sleeping again
and
Code: Select all
dumpthread active!
Partition of a set of 149395 objects. Total size = 12281196 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 71747 48 5273024 43 5273024 43 str
1 29199 20 1150964 9 6423988 52 tuple
2 2345 2 1113800 9 7537788 61 dict of sabnzbd.nzbstuff.Article
3 945 1 658440 5 8196228 67 dict (no owner)
4 326 0 651824 5 8848052 72 dict of module
5 8385 6 570180 5 9418232 77 types.CodeType
6 8173 5 457688 4 9875920 80 function
7 602 0 327632 3 10203552 83 dict of class
8 611 0 273200 2 10476752 85 type
9 611 0 238424 2 10715176 87 dict of type
<363 more rows. Type e.g. '_.more' to view.>
Gelukt, na x pogingen: 0
... and dumpthread is sleeping again
Re: High Memory Usage / Potential Memory Leaks Investigation Thread
Posted: July 25th, 2010, 12:14 pm
by shypike
The "guardian" runs every 30 seconds (although not exactly reproducible).
See sabnzbd/__init__.py, function
check_all_tasks (near the bottom of the file).
Or you just use the scheduler.
Open sabnzbd/scheduler and add this call near the end of function
init (look for string
cfg.RSS_RATE.callback(schedule_guard).
__SCHED.add_daytime_task(action, 'heapy', range(1,
, None, (h, m), kronos.method.sequential, None, None)
Where action = name of the parameterless function to be called, h = hour, m = minute
The first method will likely give the best results, because it's the main thread.
Re: High Memory Usage / Potential Memory Leaks Investigation Thread
Posted: July 25th, 2010, 12:17 pm
by shypike
sander wrote:
Are the SAB devs still interested in this case, or am I the only one ... ?
I'm interested and it's great that you are giving it so much attention.
However, in the coming three weeks I have 0 time for this.
Re: High Memory Usage / Potential Memory Leaks Investigation Thread
Posted: July 25th, 2010, 1:14 pm
by sander
shypike wrote:
The "guardian" runs every 30 seconds (although not exactly reproducible).
See sabnzbd/__init__.py, function check_all_tasks (near the bottom of the file).
I did this:
Code: Select all
def check_all_tasks():
""" Check every task and restart safe ones, else restart program
Return True when everything is under control
"""
if __SHUTTING_DOWN__ or not __INITIALIZED__:
return True
f = open('/tmp/sabnzbd.dumpheap.output2.txt', 'a')
print >>f,"check_all_tasks is active!"
print >>f,dir()
print >>f,"... and check_all_tasks is done again"
f.close()
# Non-restartable threads, require program restart
However, the result of dir() is local again (as also stated in
http://docs.python.org/library/functions.html):
Code: Select all
check_all_tasks is active!
['f']
... and check_all_tasks is done again
check_all_tasks is active!
['f']
... and check_all_tasks is done again
So only the file descriptor f is known and reported. :-(
Re: High Memory Usage / Potential Memory Leaks Investigation Thread
Posted: July 25th, 2010, 2:05 pm
by shypike
I thought you were using heapy to get info.
dir() is indeed not going to tell you anything.
Alas, memory profiling with Python sucks, big time.
I have looked at this before and I cannot find any true memory leaks.
It looks as though a) the garbage collector behaves different between platforms and Python versions
and b) on some platforms/versions Python refuses to return memory to the OS.
After my holidays I'll pick this up again.
Re: High Memory Usage / Potential Memory Leaks Investigation Thread
Posted: July 25th, 2010, 2:24 pm
by sander
shypike wrote:
I thought you were using heapy to get info.
As reported an hour ago or so, heapy will only show 8 - 12 MB memory usage. Strange.
Maybe it's python itself (not SAB) using a lot of memory ...
Re: High Memory Usage / Potential Memory Leaks Investigation
Posted: July 25th, 2010, 4:08 pm
by sander
OK, I have found a way to limit SAB's/python's memory usage: open a terminal, type
Code: Select all
ulimit -d 500111 -m 500111 -v 500111
and then in that terminal, start SABnzbd.py
FYI: setting the limits too low (like 100111 thus 100 MB) will cause problems with SABnzbd.
The memory usage will stay limited to "422m 138m". See included screendump: on the right ulimit is at work. ;-)
Code: Select all
sander@athlon64:~$ ulimit -d 500111 -m 500111 -v 5001111
sander@athlon64:~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) 500111
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) 500111
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) 5001111
file locks (-x) unlimited
sander@athlon64:~$
Re: High Memory Usage / Potential Memory Leaks Investigation Thread
Posted: July 27th, 2010, 12:04 am
by loopdemack
@Sander tell me one thing, is the 500111 the lowest value which gave you the stability or did you tried to test the lower values like 400111 or 300111 or even 200111?
Re: High Memory Usage / Potential Memory Leaks Investigation Thread
Posted: July 27th, 2010, 12:52 am
by sander
loopdemack wrote:
@Sander tell me one thing, is the 500111 the lowest value which gave you the stability or did you tried to test the lower values like 400111 or 300111 or even 200111?
I indeed tried lower values (I believe 100111 and 200111), which resulted in strange (startup) errors. You should try yourself. And try which parameter (-d, -m and/or -v) is really needed...
Remember: it seems that after setting a value, you can't reset it. So close that terminal, and open another terminal and set another value.
Re: High Memory Usage / Potential Memory Leaks Investigation Thread
Posted: October 31st, 2010, 7:37 am
by lennardw
I have the same issue.
I did find that this only occurs for if my rar-files, that I'm downloading (not unpacking or par-checking) are larger than 128MB (arround this at least). Smaller rar-files give no issues whatsoever.
I have 2GB available RAM in a virtual machine.
My findings:
Downloading files smaller than roughly 128MB:
Downloading 20 threads
no cache-limit
postprocessing going on...
Top memory usage seen: around 450-500MB (and dropping some times)
RAR files larger than roughly 128MB:
Downloading 6 threads
no other post-processing going on,
memory usage will suddenly rise very quickly, and par-checking and/or unrarring will become unpossible due to lack of available memory.
(it will rise even faster if it starts unrarring after download)
Memory is hardly ever returned to the OS
This is reproducable on my system. I hope this helps.
I'm using python 2.6 on a Virtual machine running on Centos 5.5
Re: High Memory Usage / Potential Memory Leaks Investigation Thread
Posted: November 2nd, 2010, 7:54 am
by elmer91
May be this link could explain the SAB/Python memory behavior:
http://pushingtheweb.com/2010/06/python-and-tcmalloc/
Re: High Memory Usage / Potential Memory Leaks Investigation Thread
Posted: November 2nd, 2010, 10:33 am
by elmer91
I have made following things on my system (Synology NAS DS710+)
1) Compiled/installed tcmalloc buy my own
2) modified my SAB startup script to force python using tcmalloc:
export TCMALLOC_SKIP_SBRK=1
export LDPRELOAD="/opt/lib/libtcmalloc.so"
python ... SABnzbd.py
I will report later results of testing
Re: High Memory Usage / Potential Memory Leaks Investigation Thread
Posted: November 2nd, 2010, 11:36 am
by loopdemack
Great I am eager to hear the results.
Re: High Memory Usage / Potential Memory Leaks Investigation Thread
Posted: November 2nd, 2010, 12:17 pm
by elmer91
Still under testing, but it seems the reported behavior remains the same !
Test conditions: SAB 0.5.4 running on Linux 2.6.32 (python 2.5.5 + yenc)
Cache: 128M, Quickcheck on
One 4 GB nzb (91 rar files 50M each) downloaded several times (2 bad articles on this NZB)
Only one job in queue at a time.
First download (ram used VIRT/RES reported by htop, measurements taken at different download stages)
rar50: 138M/94M
rar65: 141M/98M
rar76: 142M/99M
rar86: 144M/99M
verify/repair/unrar: 144M/101M
completed: 144M/101M
Second download without restarting SAB (same nzb moved from backup folder to watched folder):
rar00: 153M/108M
rar30: 157M/114M
rar35: 157M/106M (first resident memory decrease seen)
rar40: 157M/109M
rar50: 157M/108M
rar57: 157M/114M
rar74: 157M/114M
Still downloading, but it seems the behavior is the same:
Few MB lost for every download.
When downloading huge amount of data without restarting SAB, used memory is climbing high.
As my new system has 1 GB memory, I don't have to restart SAB too often.
But on my previous system with only 128 MB memory (cache set to 0), I had to restart SAB every day.
I will continue downloading same NZB several times again...
(this 4 GB NZB take 1 hour to download on my DSL line)
Re: High Memory Usage / Potential Memory Leaks Investigation Thread
Posted: November 3rd, 2010, 4:06 am
by elmer91
After few runs, memory used by SAB once downloading is completed:
run #1: 144M/101M
run #2: 160M/117M
run #3: 162M/119M
run #4: 181M/124M
There is definitely memory not released back to the system !