SABnzbd Forums


https://237678.vipkidinterhk.tech/

Slow download speeds: RAID 6 [Qnap TS410]

https://237678.vipkidinterhk.tech/viewtopic.php?t=17383

Page 1 of 1

Slow download speeds: RAID 6 [Qnap TS410]

Posted: April 12th, 2014, 11:12 am
by TH3 M0L3
I am running SABnzbdplus 07.17 on Qnap TS410.

I was running raid 1 with 2 x 3tb drives, but added another 2 x 3tb drives and migrated to raid 6 after running out of space.

I've now noticed that SAB is running on python 2.6, but sickbeard on 2.7.
Where I was getting decent download speeds before the change, I now only average 5-600kbs with the CPU maxing on 100%

What I need advice on is if there is an advantage if I changed SAB to run on python 2.7 to minimise cpu usage, or if the raid 6 write speed is influencing the d/load speed.

I'm looking for the log files of both SAb and Qnap, but dont really know where to find them once ssh'd in to the qnap.

Any and all help is appreciated.

Re: Slow download speeds: RAID 6 [Qnap TS410]

Posted: April 12th, 2014, 7:49 pm
by Dipstick
Hi
Not sure about the python question - but could it be the increased CPU requirement of calculating the additional parity data of raid 6 ?

when you transfer a large file with Sab shutdown do you still get 100% CPU and slow transfers?

Interestingly the website lists the raid levels under the "Software" section rather than the "Hardware" section, read into that as you may - but that could be an option to explore.
http://www.qnap.com/en/index.php?lang=e ... n=3388&g=1

Re: Slow download speeds: RAID 6 [Qnap TS410]

Posted: April 13th, 2014, 4:11 pm
by shypike
The Python version doesn't matter much, we support 2.5, 2.6 and 2.7
We generally don't recommend placing the temporary download folder on a RAID array,
because of the overhead.

Re: Slow download speeds: RAID 6 [Qnap TS410]

Posted: April 14th, 2014, 4:21 pm
by TH3 M0L3
Dipstick wrote:Hi
Not sure about the python question - but could it be the increased CPU requirement of calculating the additional parity data of raid 6 ?
That is what my initial concern was, but am just seeing how it goes at the moment.
when you transfer a large file with Sab shutdown do you still get 100% CPU and slow transfers?
not tried that yet. Will give it a go and report back
Interestingly the website lists the raid levels under the "Software" section rather than the "Hardware" section, read into that as you may - but that could be an option to explore.
http://www.qnap.com/en/index.php?lang=e ... n=3388&g=1
i'll check it out, thanks
shypike wrote: The Python version doesn't matter much, we support 2.5, 2.6 and 2.7
We generally don't recommend placing the temporary download folder on a RAID array,
because of the overhead.
Thanks shypike. I have a small usb plugged into the qnap to house the watch folder. I've just ordered a 32gb usb drive to replace it so I can put the temp d/load folder onto as well. Hopefully that'll sort it out.

Re: Slow download speeds: RAID 6 [Qnap TS410]

Posted: April 14th, 2014, 5:06 pm
by TH3 M0L3
shypike wrote:The Python version doesn't matter much, we support 2.5, 2.6 and 2.7
We generally don't recommend placing the temporary download folder on a RAID array,
because of the overhead.
OK, So I tested this out and placed the temp d/load folder on the usb drive. Restarted SAB and speeds were improved at 3.5MB/s. My normal speed.
This didn't last long.

In fact they returned to the slow speeds of before, and even dropped down to single digit KB/s. The bandwidth history is like a sawtooth profile, not nice and steady at a good speed.

CPU usage of python goes from 7%, right up to 99%, while memory averages at 50-60M

it just seems like since i migrated to raid 6, my problems started.

I'm beginning to thnk a fresh install of SAb....

Any ideas / tips / pointers?

Re: Slow download speeds: RAID 6 [Qnap TS410]

Posted: April 19th, 2014, 5:25 pm
by TH3 M0L3
So, I re-installed SAB, but kept the original ini file.

I changed news providers to news demon. I've tried different ports. CPU usage is down, but memory is up.

Watch folder is on USB drive, as is temp d/load folder.

speed will burst in at top speed for a few seconds/minute, then tapers off to a few KB/s for a while before increasing again to about mid-speed, then drops....etc, etc.

It's so frustrating.

ANy ideas would be helpful. Thanks in advance

Re: Slow download speeds: RAID 6 [Qnap TS410]

Posted: April 20th, 2014, 12:44 am
by TH3 M0L3
have a load of these errors

[20/Apr/2014:04:32:55] Traceback (most recent call last): File "/share/MD0_DATA/.qpkg/SABnzbdplus/cherrypy/_cpwsgi.py", line 79, in setapp s, h, b = self.get_response() File "/share/MD0_DATA/.qpkg/SABnzbdplus/cherrypy/_cpwsgi.py", line 219, in get_response response = self.request.run(meth, path, qs, rproto, headers, rfile) File "/share/MD0_DATA/.qpkg/SABnzbdplus/cherrypy/_cprequest.py", line 579, in run raise cherrypy.TimeoutError()TimeoutErrore

and here is the ini file [passwords etc removed]

Code: Select all

__version__ = 19
[misc]
queue_complete = ""
https_port = 9090
folder_rename = 1
password_file = ""
rss_rate = 60
https_key = server.key
cleanup_list = sfv,
cache_limit = 128m
ignore_unrar_dates = 0
par_option = ""
web_color = gold
replace_spaces = 0
web_color2 = ""
dirscan_speed = 3600
enable_filejoin = 1
create_group_folders = 0
never_repair = 0
no_penalties = 0
bandwidth_limit = 0
dirscan_dir = /share/USBDisk1/SAB - NZB watched
disable_api_key = 0
email_to = [email protected],
password = ""
permissions = 777
auto_disconnect = 1
enable_date_sorting = 0
replace_dots = 1
movie_sort_extra = " CD%1"
ionice = ""
enable_unzip = 1
download_free = ""
port = 8800
email_dir = ""
pre_script = None
email_full = 0
log_dir = /share/MD0_DATA/Download/sabnzbd/logs
admin_dir = admin
nzb_backup_dir = /share/MD0_DATA/Download/sabnzbd/nzb/backup
tv_sort_countries = 1
date_categories = tv,
ssl_type = v23
sfv_check = 1
date_sort_string = ""
schedlines = "0 23 * rss_scan ", "0 0 7 remove_failed ", "0 22 1234567 scan_folder "
host = 0.0.0.0
ignore_samples = 2
enable_tsjoin = 1
auto_browser = 0
pause_on_post_processing = 1
enable_unrar = 1
email_rss = 1
email_account = [email protected]
email_server = smtp.mail.xxxxxxx.com
allow_streaming = 0
config_lock = 0
api_warnings = 1
tv_sort_string = %sn/Season %s/%sn - S%0sE%0e - %en.%ext
username = ""
unpack_check = 0
download_dir = /share/USBDisk1/SAB - Temp download folder
size_limit = ""
enable_par_cleanup = 1
safe_postproc = 1
cache_dir = /share/MD0_DATA/Download/sabnzbd/cache
complete_dir = /share/MD0_DATA/Download/sabnzbd/complete
replace_illegal = 1
par2_multicore = 0
warned_old_queue = 0
language = en
check_new_rel = 1
enable_https = 0
no_dupes = 0
top_only = 1
quick_check = 1
movie_sort_string = %title (%y)/%title (%y).%ext
email_from = [email protected]
movie_extra_folder = 0
email_endjob = 1
tv_categories = ,
pause_on_pwrar = 1
login_realm = SABnzbd
send_group = 0
refresh_rate = 0
movie_categories = movies,
script_dir = /share/MD0_DATA/.qpkg/SickBeard/autoProcessTV
email_pwd = xxxxxxxxxxxx
ignore_wrong_unrar = 0
nice = ""
enable_movie_sorting = 1
https_cert = server.cert
web_dir2 = ""
fail_on_crc = 1
auto_sort = 0
ampm = 0
web_dir = Plush
queue_complete_pers = 0
enable_tv_sorting = 1
folder_max_length = 256
api_key = 
rss_filenames = 0
nzb_key = 
win_menu = 1
allow_64bit_tools = 1
local_range = ""
movie_rename_limit = 100M
rating_enable = 0
enable_recursive = 1
pre_check = 0
rating_feedback = 1
randomize_server_ip = 0
osx_speed = 1
quota_period = m
quota_day = ""
use_pickle = 0
https_chain = ""
warn_empty_nzb = 1
history_limit = 50
quota_size = ""
wait_ext_drive = 5
ipv6_servers = 1
show_sysload = 2
sanitize_safe = 0
req_completion_rate = 100.2
fsys_type = 0
allow_incomplete_nzb = 0
uniconfig = 1
enable_meta = 1
quota_resume = 0
rating_api_key = ""
wait_for_dfolder = 0
warn_dupl_jobs = 1
fail_hopeless = 1
no_ipv6 = 0
nomedia_marker = ""
max_art_opt = 0
keep_awake = 1
max_art_tries = 3
osx_menu = 1
rss_odd_titles = nzbindex.nl/, nzbindex.com/, nzbclub.com/
prio_sort_list = ,
empty_postproc = 0
rating_host = http://www.oznzb.com
web_watchdog = 0
flat_unpack = 0
start_paused = 0
overwrite_files = 0
[logging]
log_new = 0
max_log_size = 5242880
log_level = 2
log_backups = 5
enable_cherrypy_logging = 0
[newzbin]
username = ""
bookmark_rate = 60
bookmarks = 0
password = ""
unbookmark = 1
url = http://www.newzbin2.es
[nzbmatrix]
username = ""
apikey = ""
del_bookmark = 1
[categories]
[[*]]
priority = 0
pp = 3
name = *
script = sabToSickBeard.py
newzbin = ""
dir = ""
[[music]]
priority = -100
pp = 3
name = music
script = None
newzbin = ""
dir = /share/Download/to_distribute/Music
[[movies]]
priority = -100
pp = 3
name = movies
script = None
newzbin = ""
dir = /share/Download/to_distribute/MC - to scrape
[servers]
[[news.supernews.com]]
username = [email protected]
enable = 1
name = news.supernews.com
fillserver = 1
connections = 30
ssl = 0
host = news.supernews.com
timeout = 30
password = xxxxxxx
optional = 0
port = 23
retention = 1871
[[news.newsdemon.com]]
username = xxxxxxxxx
enable = 1
name = news.newsdemon.com
fillserver = 0
connections = 50
ssl = 0
host = europe.newsdemon.com
timeout = 120
password = xxxxxxxx
optional = 0
port = 443
retention = 2073
[rss]
[[Usenet-Crawler -- Movies]]
pp = ""
enable = 1
name = Usenet-Crawler -- Movies
script = None
uri = http://www.usenet-crawler.com/rss....................................................................
cat = movies
priority = -100
filter0 = movies, "", None, A, *, -100, 1
[[Usenet - Crawler -- My Movies]]
filter5 = movies, 3, None, A, 720p*x264, -100, 1
pp = 3
enable = 1
name = Usenet - Crawler -- My Movies
script = None
uri = http://www.usenet-crawler.com/rss..................................................................................
cat = movies
priority = -100
filter4 = movies, 3, None, M, 720p, -100, 1
filter6 = movies, 3, None, A, HD, -100, 1
filter1 = "", "", None, R, Cam, -100, 1
filter0 = movies, "", None, R, Web, -100, 1
filter3 = movies, 3, None, R, R6, -100, 1
filter2 = movies, 3, None, R, TS, -100, 1
[[MyNZBgeek]]
pp = 3
enable = 1
name = MyNZBgeek
script = None
uri = https://01100111011001010110010101101011.info/rss..................................................................
priority = -100
filter0 = "", "", "", A, *, -100, 1
[[MYNZBGEEK - My movies]]
pp = ""
enable = 0
name = MYNZBGEEK - My movies
script = None
uri = https://01100111011001010110010101101011.info/rss..................................................

cat = movies
priority = 0
filter1 = "", "", "", R, TS, -100, 1
filter0 = "", "", "", R, Cam, -100, 1
filter3 = "", "", "", A, HD, -100, 1
filter2 = "", "", "", A, 720p*x264, -100, 1
[[Oznzb]]
pp = 3
enable = 1
name = Oznzb
script = None
uri = https://www.oznzb.com/rss?t...............................................................................
cat = movies
priority = -100
filter0 = "", "", "", A, *, -100, 1
[[NZBNDX]]
pp = ""
enable = 1
name = NZBNDX
script = Default
uri = http://www.nzbndx.com/rss?......................................................................................
cat = Default
priority = -100
filter0 = "", "", "", A, *, -100, 1
[growl]
ntfosd_enable = 0
growl_enable = 1
growl_server = woodlands
notify_classes = startup, download, pp, complete, other
ncenter_enable = 0
growl_password = ""
[nzbxxx]
username = ""
apikey = ""
here is netstat

Code: Select all

[~] # netstat [-laenrtuwx] -e :8080 -e :9090
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 QNAP:45760              QNAP:49152              CLOSE_WAIT
tcp        0      0 QNAP:8800               192.168.1.85:15109      ESTABLISHED
tcp    67616      0 QNAP:60955              81.171.118.148:https    ESTABLISHED
tcp        0      0 QNAP:37510              192.168.1.84:55000      CLOSE_WAIT
tcp        0      0 QNAP:60707              192.168.1.73:49153      CLOSE_WAIT
tcp    67168      0 QNAP:54506              81.171.118.147:https    ESTABLISHED
tcp    66816      0 QNAP:60965              81.171.118.148:https    ESTABLISHED
tcp        0      0 QNAP:60693              192.168.1.73:49153      CLOSE_WAIT
tcp    67168      0 QNAP:36467              81.171.118.146:https    ESTABLISHED
tcp    67200      0 QNAP:36427              81.171.118.146:https    ESTABLISHED
tcp        0      0 QNAP:53584              192.168.1.85:2869       FIN_WAIT2
tcp    67200      0 QNAP:60921              81.171.118.148:https    ESTABLISHED
tcp    67200      0 QNAP:60935              81.171.118.148:https    ESTABLISHED
tcp        1      0 QNAP:46006              192.168.1.254:2555      CLOSE_WAIT
tcp        0      0 QNAP:8800               192.168.1.85:15107      ESTABLISHED
tcp    67200      0 QNAP:36460              81.171.118.146:https    ESTABLISHED
tcp        1      0 QNAP:53583              192.168.1.85:2869       CLOSE_WAIT
tcp    57312      0 QNAP:54463              81.171.118.147:https    ESTABLISHED
tcp        0      0 QNAP:53586              192.168.1.85:2869       ESTABLISHED
tcp        0      0 QNAP:37522              192.168.1.84:55000      CLOSE_WAIT
tcp    67200      0 QNAP:54470              81.171.118.147:https    ESTABLISHED
tcp    69344      0 QNAP:60972              81.171.118.148:https    ESTABLISHED
tcp    66816      0 QNAP:60970              81.171.118.148:https    ESTABLISHED
tcp    67168      0 QNAP:54508              81.171.118.147:https    ESTABLISHED
tcp    67392      0 QNAP:54462              81.171.118.147:https    ESTABLISHED
tcp    66528      0 QNAP:36441              81.171.118.146:https    ESTABLISHED
tcp        0      0 QNAP:60694              192.168.1.73:49153      CLOSE_WAIT
tcp        0      0 QNAP:445                192.168.1.85:15934      ESTABLISHED
tcp    67200      0 QNAP:36448              81.171.118.146:https    ESTABLISHED
tcp        1      0 QNAP:53571              192.168.1.85:2869       CLOSE_WAIT
tcp        0      0 QNAP:60706              192.168.1.73:49153      CLOSE_WAIT
tcp    67200      0 QNAP:36446              81.171.118.146:https    ESTABLISHED
tcp     8011      0 QNAP:33771              192.168.1.89:1400       CLOSE_WAIT
tcp    66528      0 QNAP:60942              81.171.118.148:https    ESTABLISHED
tcp    66528      0 QNAP:54501              81.171.118.147:https    ESTABLISHED
tcp        0      0 QNAP:8800               192.168.1.85:14613      ESTABLISHED
tcp        1      0 QNAP:53557              192.168.1.85:2869       CLOSE_WAIT
tcp        0      0 QNAP:45747              QNAP:49152              CLOSE_WAIT
tcp    66608      0 QNAP:54514              81.171.118.147:https    ESTABLISHED
tcp        0      0 QNAP:37523              192.168.1.84:55000      CLOSE_WAIT
tcp    67376      0 QNAP:60934              81.171.118.148:https    ESTABLISHED
tcp        0      0 QNAP:8800               192.168.1.85:15111      ESTABLISHED
tcp        0      0 QNAP:60692              192.168.1.73:49153      CLOSE_WAIT
tcp    67200      0 QNAP:60947              81.171.118.148:https    ESTABLISHED
tcp     1902      0 QNAP:50782              192.168.1.88:1400       CLOSE_WAIT
tcp        1      0 QNAP:37930              192.168.1.65:51439      CLOSE_WAIT
tcp    67424      0 QNAP:54466              81.171.118.147:https    ESTABLISHED
tcp        0      0 QNAP:49152              QNAP:45760              FIN_WAIT2
tcp    67200      0 QNAP:36458              81.171.118.146:https    ESTABLISHED
tcp        1      0 QNAP:42151              192.168.1.65:53531      CLOSE_WAIT
tcp        0      0 QNAP:webcache           192.168.1.85:17319      ESTABLISHED
tcp    66608      0 QNAP:54516              81.171.118.147:https    ESTABLISHED
tcp    67824      0 QNAP:60913              81.171.118.148:https    ESTABLISHED
tcp    67424      0 QNAP:36425              81.171.118.146:https    ESTABLISHED
tcp        1      0 QNAP:53558              192.168.1.85:2869       CLOSE_WAIT
tcp    67168      0 QNAP:36470              81.171.118.146:https    ESTABLISHED
tcp    67200      0 QNAP:54493              81.171.118.147:https    ESTABLISHED
tcp    66816      0 QNAP:54512              81.171.118.147:https    ESTABLISHED
tcp    66816      0 QNAP:60971              81.171.118.148:https    ESTABLISHED
tcp        1      0 QNAP:53570              192.168.1.85:2869       CLOSE_WAIT
tcp        0      0 QNAP:49152              QNAP:45747              FIN_WAIT2
tcp        0      0 QNAP:8800               192.168.1.85:15108      ESTABLISHED
tcp    67200      0 QNAP:36444              81.171.118.146:https    ESTABLISHED
tcp    65664      0 QNAP:54468              81.171.118.147:https    ESTABLISHED
tcp     1902      0 QNAP:50795              192.168.1.88:1400       CLOSE_WAIT
tcp    67616      0 QNAP:36465              81.171.118.146:https    ESTABLISHED
tcp    67200      0 QNAP:60941              81.171.118.148:https    ESTABLISHED
tcp    67200      0 QNAP:36457              81.171.118.146:https    ESTABLISHED
tcp    67200      0 QNAP:54502              81.171.118.147:https    ESTABLISHED
tcp        0    708 QNAP:122                192.168.1.85:12257      ESTABLISHED
tcp        0      0 QNAP:60705              192.168.1.73:49153      CLOSE_WAIT
tcp        0      0 QNAP:37509              192.168.1.84:55000      CLOSE_WAIT
tcp    67200      0 QNAP:60946              81.171.118.148:https    ESTABLISHED
tcp    67200      0 QNAP:60956              81.171.118.148:https    ESTABLISHED
tcp    67424      0 QNAP:36456              81.171.118.146:https    ESTABLISHED
tcp        0      0 QNAP:8800               192.168.1.85:15110      ESTABLISHED
tcp    66608      0 QNAP:36477              81.171.118.146:https    ESTABLISHED
tcp        0      0 QNAP:53585              192.168.1.85:2869       FIN_WAIT2
tcp    67200      0 QNAP:60951              81.171.118.148:https    ESTABLISHED
tcp    67200      0 QNAP:36451              81.171.118.146:https    ESTABLISHED
tcp        1      0 QNAP:45993              192.168.1.254:2555      CLOSE_WAIT
tcp        0      0 QNAP:122                192.168.1.85:11956      ESTABLISHED
tcp    67200      0 QNAP:60939              81.171.118.148:https    ESTABLISHED
tcp        1      0 QNAP:37943              192.168.1.65:51439      CLOSE_WAIT
tcp    67168      0 QNAP:60961              81.171.118.148:https    ESTABLISHED
tcp    55200      0 QNAP:60916              81.171.118.148:https    ESTABLISHED
tcp    66816      0 QNAP:60963              81.171.118.148:https    ESTABLISHED
tcp    67200      0 QNAP:36445              81.171.118.146:https    ESTABLISHED
tcp        1      0 QNAP:42164              192.168.1.65:53531      CLOSE_WAIT
tcp    65280      0 QNAP:60967              81.171.118.148:https    ESTABLISHED
tcp    67200      0 QNAP:54492              81.171.118.147:https    ESTABLISHED
tcp     8011      0 QNAP:33758              192.168.1.89:1400       CLOSE_WAIT
netstat: no support for `AF INET6 (tcp)' on this system.
udp        0      0 localhost:35614         localhost:syslog        ESTABLISHED
netstat: no support for `AF INET6 (udp)' on this system.
netstat: no support for `AF INET6 (raw)' on this system.
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags       Type       State         I-Node Path
here is top

Code: Select all

Mem: 231708K used, 23968K free, 0K shrd, 1720K buff, 7656K cached
Load average: 4.08, 4.00, 3.75    (State: S=sleeping R=running, W=waiting)

  PID USER     STATUS   RSS  PPID %CPU %MEM COMMAND
 2395 admin    S        26M     1  1.3 10.5 python2.6
  151 admin    SW         0     2  1.3  0.0 kswapd0
 9528 admin    Z          0  2828  0.9  0.0 manaRequest.cgi
 9532 admin    Z          0  2828  0.9  0.0 manaRequest.cgi
 8888 admin    D        17M  8849  0.7  6.8 freshclam
 9467 admin    R        924  4770  0.7  0.3 top
  984 admin    S        332     1  0.7  0.1 daemon_mgr.nvr
 2828 admin    S       1044     1  0.3  0.4 _thttpd_
 5517 admin    S       1084     1  0.1  0.4 djmount
 1672 admin    S        608     1  0.1  0.2 hotswap
 4620 httpdusr S        416     1  0.1  0.1 lpb_scheduler
 1571 admin    SW         0     2  0.1  0.0 md4_raid1
  632 admin    SW         0     2  0.1  0.0 md9_raid1
 9533 admin    Z          0     1  0.1  0.0 _thttpd_
 5683 admin    S <     2200     1  0.0  0.8 iscsid
 9657 admin    S       2032     1  0.0  0.7 python2.7
 3314 admin    S       1556  3310  0.0  0.6 centerim
 3215 admin    S       1500  2991  0.0  0.5 mysqld
 5098 admin    S        976     1  0.0  0.3 smbd

Re: Slow download speeds: RAID 6 [Qnap TS410]

Posted: April 21st, 2014, 12:46 am
by shypike
These are time-outs in the internal web server.
A sign that SABnzbd needs all available CPU time.
The system might be just too slow for SABnzbd or you don't have enough memory.
What do you have as hardware? (CPU, memory)

Re: Slow download speeds: RAID 6 [Qnap TS410]

Posted: April 21st, 2014, 6:41 pm
by TH3 M0L3
Thanks for the reply shypike

I am running it on a QNAP TS412, so not able to upgrade memory etc.

CPU Marvell 6281 1.2GHz
DRAM 256MB DDRII RAM
Flash Memory 16MB

I was running SAB on the same server but only had 2 disks on RAID1. Since I added the two extra disks and migrated to RAID6, the speed plummeted.
I'm in the process of doing a fresh hard backup of all my data, then going to format the raid and set up as RAID 10.

As said before, the watch, and temp folders are on the usb drive and not the RAID, so am pretty lost

SAB is the only service running overnight, and it only achieves an av of 215kb/s for a d/load.

Re: Slow download speeds: RAID 6 [Qnap TS410]

Posted: April 22nd, 2014, 6:10 am
by shypike
I'd advice you to look at nzbget.
It doesn't do as much as SABnzbd, but it does need less resources.
All times are UTC-05:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited