Synology DS1512+ DLs start fast then slow to a crawl

[celestian02]

To reply to Sander's question regarding extraction, it feels like SABnzbd extraction and post-processing is faster than NZBGet after using both for a week or so.

[sander]

Both the 1512 and the 1812 are dual core. See http://www.synology.com/products/spec.p ... #p_submenu and http://www.synology.com/products/spec.p ... #p_submenu . So SABnzbd can't get the CPU usage above 50%.
Both DS system have an Intel Atom

A MacMini has an Intel Core i5 or i7.

The benchmark difference between an Atom and an i5 are about a factor 4 to 10.

So ... blame it on the Atom.

If you want a NAS with a fast processor, find a NAS with a Core i5 or i7.

[sander]

PS: some CPU benchmarks based on Blowfish (found in 'hardinfo', reported with "hardinfo -r -f text'). And apparently a smaller number is a better performance:

i3 (my laptop): 5.524

i7 (my VPS): 3.223
Atom (my Netbook NC10): 32.705
ARM (Raspberry Pi): 97.990

In other words, based on these Blowfish benchmark (which is compute intensive):
my i3 laptop is 18 times faster than my ARM Raspi.
my i3 laptop is 6 times faster than my Atom netbook.
my VPS is 30 times faster than my ARM Raspi

That will have a serious impact on SABnzbd's performance. If you want fast SABnzbd performance, get a fast processor.

[sander]

This is fun: The python script below gives a benchmark. Results:

i3 laptop: python can do 50371 os.listdir('/') per second
i7 VPS: python can do 45759 os.listdir('/') per second
Atom Netbook (NC10, about 2009): python can do 14012 os.listdir('/') per second
ARM Raspberry: python can do 4696 os.listdir('/') per second


So that is a factor 10 difference between my i3 laptop and my ARM Raspi!

So ... posters ... can you run the script on your Synology and your MacMini?

Code: Select all

import os

def timereps(reps, func):
    from time import time
    start = time()
    for i in range(0, reps):
        func()
    end = time()
    return (end - start) / reps

listdir_time = timereps(10000, lambda: os.listdir('/'))
print "python can do %d os.listdir('/') per second" % (1 / listdir_time)

EDIT / PS:

I've put the python benchmark into my SABnzbd.py, and the output is now (note the bold line):


2012-09-06 15:14:35,128::INFO::[SABnzbd:489] nice binary... found (/usr/bin/nice)
2012-09-06 15:14:35,129::INFO::[SABnzbd:493] ionice binary... found (/usr/bin/ionice)
2012-09-06 15:14:35,129::INFO::[SABnzbd:498] pyOpenSSL... found (True)
2012-09-06 15:14:35,130::INFO::[SABnzbd:1468] Starting web-interface on 0.0.0.0:8080
2012-09-06 15:14:35,344::INFO::[SABnzbd:1482] python performance: 46725 listdir operations per second (higher is better)
2012-09-06 15:14:35,345::INFO::[_cplogging:55] [06/Sep/2012:15:14:35] ENGINE Bus STARTING

HTH

[celestian02]

The Synology 1512+ is displaying an average of 27500.
As for the Mac Mini, I will try later at home as I have not setup remote ssh.

[sander]

The Synology 1512+ is displaying an average of 27500.
As for the Mac Mini, I will try later at home as I have not setup remote ssh.

Synology: twice as good as my Atom netbook. That's nice.
I'm looking forward to your MacMini results.

[celestian02]

Will do in a couple of hours. In the meantime, I looked up a bit regarding multi core use for Python and didn't know it was one of its limitation. They apparently introduced a multiprocessing module but that is not really what one would expect as it's only spawning multiple python process in a very similar way to threads.

This way, they rely on the OS ability to dispatch spawn processes on cores. Interresting enough although I doubt SAB developers would dive into that as it would mean rethinking the whole application architecture; I guess.

[shypike]

Interresting enough although I doubt SAB developers would dive into that as it would mean rethinking the whole application architecture; I guess.

We wouldn't do it again in Python...

[sander]

Interresting enough although I doubt SAB developers would dive into that as it would mean rethinking the whole application architecture; I guess.

We wouldn't do it again in Python...

... but in ... ?

[celestian02]

Yeah, Python feels quite powerful and enjoyable to work with. I'd be interested to know as well which portable programming language you would go towards to!

Sander: MacMini is displaying 38060 against 27518 for the Synology

[sander]

Yeah, Python feels quite powerful and enjoyable to work with. I'd be interested to know as well which portable programming language you would go towards to!

Sander: MacMini is displaying 38060 against 27518 for the Synology

And speeds are 17MB/s resp 7MB/s? So there is no lineair (y=ax) correlation. Pity.

BTW: what kind of Internet connection do you have that can achieve 17 MB/s = 170 Mbps? Is that FttH?

[celestian02]

Yup. Feels like Python/SABnzbd is reaching a bottleneck and can't get past that value. 17MB/s is indeed 136Mbps. I have a FttH connection and it should offer "only" 100Mbps but I can reach some speeds in upload/downloads that are beyond this limit. I think this is because we must be only a few in my area to have FttH with my ISP. This is sweet though! The day it'll drop, I will be sad

The other day, I was actually not able to reach this kind of speed on my Mac mini and I contacted Usenetserver support that did not believe me at first and thought I didn't know what I was talking about, ahah:
- He presumed I had made the common confusion between between Mbps and MB/s,
- He presumed my HDD could not support that kind of speed (hell, if HDDs couldn't reach a 17MB/s transfer rate nowadays, that would be problematic )
- He presumed the software I used was not calculating correctly the transfer speed: so he's right about SABnzbd because it does seem to miscalculate a bit the actual current bandwith (when I am comparing what is written on my HDDs or even what is indicating my router) but, come on... Miscalculating the transfer speed by 9/10MB/s He can't be serious...

In the end, I sent some actual screenshots... And solved this issue by using the european server, the general hostname was presumably redirecting me most of the time to their US servers.

[powertowerpro]

I just upgraded to 0.7.4 from 0.6.x and now I can get speeds up to 5.5MB/sec up from 4.0MB/sec on my DS. I would kill to get back up to the 10MB/sec I see on my desktop machines. I see Celestian can get the 7MB/sec so not sure what the disconnect is. As I mentioned earlier I am on a DS1812+. Here is to hoping the 0.8.x release gets me a few more MB/sec:)