Hashed downloading speed

[stomp]

Hi everyone,

For a few days now, the Web interface of SABnzbd+ is quite slow. It takes up to one minute to refresh the queue or to save settings. In addition I noticed the executable to be more CPU demanding. Today I saw it had an impact on speed. The speed is now "hashed": it is at 100% for 2 minutes then goes to zero for 10 seconds then goes up again to 100%. I made a test today to see if the size of the queue/files was the problem (generally I'm downloading large files, e.g. 100-200Mb). With a moderate queue composed of 50Mb's files, the speed was constant at 100%. Now back to my large queue, the speed is once again hashed.

I was previously using a 2-HDD configuration (one for the cache folder, one for the temp and complete folders). The problem still remains with a 1-HDD config. I am now back with 0.4.3 (0.4.4 seemed to me slower at first). Enabling cache writing made no change.

Is this a HDD problem ? Thanks in advance.

[shypike]

You're a bit low on details, do you expect us to be paranormal gifted people?

What's your download speed?
What type of system do you have?
What is a "large queue" in your opinion?

Having  a high Queue page refresh rate certainly has an effect on the overall speed.
Saving settings is slow because it requires a soft-reset, meaning that the memory
cache is flushed to disk first. So it's slow.
SABnzbd isn't designed for people to keep fiddling with the settings during heavy downloads.

[stomp]

I thought I gave a lot of details  So here are more: I am using Vista x86 on a E4300 with 2Gb DDR2. My download speed is 2500KB/s. I was using a RAID0 of 3 disks, I am now with one Samsung Spinpoint F1 1Tb. Large queue: well, something like hundreds of 200Mb's files. I checked once more and it seems like speed is decreasing to zero when a file is being assembled (number of articles in cache goes from around 300 to none, speed decrease/increase is very regular). Queue refresh rate is currently set to zero. Ok for settings saves.

[shypike]

I have no experience with speeds like 2500 Kbps, but I can imagine the program to be quite challenged under these conditions.
I'll have a look at some possibilities, once option is to increase the queue between the downloader and the assembler.
However, it could well be that the overall speed of the system cannot cope with a constant stream of 2500 Kbps.
SABnzbd has some overhead too.
If you use SSL, that certainly adds overhead.

[switch]

Shypike, 2500KB/s isn't that fast, my four year old system handles it fine at a constant speed with little cpu usage.

Also download speed shouldn't be slowing on assembly if the downloading articles are being cached to RAM and not hard drive.

I suspect the problem is in the very large queue, SABnzbd saves it's state in many places such as when saving the settings, closing down sabnzbd, and I think after assembling a file. This saving of state can be very slow when there are over 200 items in the queue.

I recommend trying to keep your queue under 100 items (that is 100 different nzb's, not 100 different files inside an nzb). If you wish to start from a clean queue, launch sabnzbd with --clean on the end (either add it to a shortcut or from the RUN window assuming you are using windows). And this will start you with a clean queue and history.

[stomp]

I started a download this morning. Unfortunately, the Task Manager closed so I can't say if speed was constant. Anyway, looking at average speed in history gives me a nice 2500KB/s instead of the 2350KB/s I had with the hashed downloads so I suspect the problem is gone. Interesting is that I changed the HDD for my RAID0 (much faster in writing/reading). But it makes no sense as the articles are being cached...

[shypike]

But it makes no sense as the articles are being cached...

Yes it does make a difference.
Without SABnzbd's cache all articles are first written to disk. When all articles of a file are in, the articles are read back from disk and are written to disk as a complete data-file. The memory cache prevents the articles from being written and read as files.

I agree with the diagnose of switch that having a very large queue can cause problems.
We are still looking at ways to optimize the queue design.

[shypike]

Now that I finally have my 100Mbit fibre connection  , I can confirm the problem  .
Downloading is halted during assembly.

This is probably due to the combination of the high download speed and a
modest PC (a Pentium III running Linux).

I'll look into the matter.