Re: High Memory Usage / Potential Memory Leaks Investigation Thread
Posted: July 18th, 2010, 4:41 pm
BTW: should I close the other apps on my 4GB system to see if SAB is going to eat all the memory?
https://237678.vipkidinterhk.tech/
Like I posted: nothing filled out. It's empty, both in the GUI and the .iniloopdemack wrote: Please just take a look at sabnzbd.ini what do you got for "cache_limit="
I'm sure its leaking and not usage, also in 4 of 13 sab will broke or freeze the process due to very unstable process because of his internal environment which didnt predicted how to work in leaked state,sander wrote: SAB's mem usage is still going up (now at 1370 MB, see attachment), but alas one parameters (REAL mem) is missing (or: at 1 MB). I haven't got remote access to my machine at home, so I can't check right now what's going on.
Code: Select all
SABnzbd has downloaded "blabla"
Finished at 2010-07-19 13:35:59
Downloaded 44.33 GB
Results of the job:
Stage Download
Downloaded in 12 hours 46 minutes 40 seconds at an average of 1010.45 KB/s
Stage Repair
[HDC-blabla] Quick Check OK
Stage Unpack
[HDC-blabla] Unpacked 850 files/folders in 2 hours 57 mins 42 seconds
Code: Select all
sander@athlon64:~$ ./sabnzbd-memory-usage.sh
1348
1.0g
sander@athlon64:~$ top -bn1 | grep -i sab
7804 sander 20 0 1348m 1.0g 3196 S 0 26.4 206:25.05 SABnzbd.py
sander@athlon64:~$
Code: Select all
sander@athlon64:~$ ./sabnzbd-memory-usage.sh
1348
544
sander@athlon64:~$
Good question! ;-)pair of dimes wrote: why is it using 544MB memory though, shouldn't it be more like 20MB
OK, limit is now on 50M and SAB has been restarted.loopdemack wrote: @sander, could you test the same nzb but with caching enabled in the sabnzbd.ini, it would help a lot in finding the reasons behind the leakage, would you experience the same leak or will it be much faster, higher and more extreme.
Use cache_limit=150M or cache_limit=50M
Also it would be good that we found how to examine with python memory profiler whats going on.
Code: Select all
sander@athlon64:~$ grep -i cache_limit .sabnzbd/sabnzbd.ini
cache_limit = 50M
sander@athlon64:~$
sander@athlon64:~$ ./sabnzbd-memory-usage.sh
298
57
sander@athlon64:~$
Code: Select all
# with some help from the Python Phrasebook: Essential Code and Commands
import random
# On Ubuntu: sudo apt-get install python-guppy
from guppy import hpy; hp=hpy()
def dumpheap() :
mislukt = 0
gelukt = False
while not gelukt:
try:
print hp.heap()
gelukt = True
except:
print "niet gelukt"
mislukt += 1
print "Gelukt, na x pogingen:",mislukt
return
dumpheap()
list = []
for i in range(1, 100000):
list.append(random.randint(100,10000000000000))
dumpheap()
Code: Select all
sander@quirinius:~/heapy-exp$
sander@quirinius:~/heapy-exp$ time python testje.py
Partition of a set of 24474 objects. Total size = 1796772 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 10881 44 707552 39 707552 39 str
1 5599 23 203984 11 911536 51 tuple
2 344 1 109376 6 1020912 57 dict (no owner)
3 1583 6 107644 6 1128556 63 types.CodeType
4 70 0 107056 6 1235612 69 dict of module
5 180 1 95904 5 1331516 74 dict of type
6 200 1 88716 5 1420232 79 type
7 1507 6 84392 5 1504624 84 function
8 124 1 67552 4 1572176 88 dict of class
9 1027 4 36972 2 1609148 90 __builtin__.wrapper_descriptor
<89 more rows. Type e.g. '_.more' to view.>
Gelukt, na x pogingen: 0
niet gelukt
Partition of a set of 124483 objects. Total size = 4207376 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 99998 80 1999936 48 1999936 48 long
1 10883 9 707648 17 2707584 64 str
2 131 0 420064 10 3127648 74 list
3 5598 4 203956 5 3331604 79 tuple
4 350 0 110192 3 3441796 82 dict (no owner)
5 1583 1 107644 3 3549440 84 types.CodeType
6 70 0 107056 3 3656496 87 dict of module
7 180 0 95904 2 3752400 89 dict of type
8 200 0 88716 2 3841116 91 type
9 1506 1 84336 2 3925452 93 function
<89 more rows. Type e.g. '_.more' to view.>
Gelukt, na x pogingen: 1
real 0m4.482s
user 0m4.264s
sys 0m0.060s
sander@quirinius:~/heapy-exp$