Download slowdowns limited by CPU on fast Linux PC

[jsbchakji]

I googled and saw many similar posts but the conclusion was that the CPU was too slow. But this is not my case and I haven't seen anything applicable to my case. I installed sabnzbdplus on Ubuntu 22.04.1 LTS via the ppa:jcfp/nobetas repository according to the instructions. I run it as a service. My PC is fairly fast. I get this in Status:

Code: Select all

System performance (Pystone)  256010 Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz AVX2
Download folder speed  625.6 MB/s /data/data/usenet/incomplete
Complete folder speed  591.2 MB/s /data/data/usenet/complete 

I am on gigabit internet connection. When downloads start they saturate the connection and go to about 104 MB/s (I set up speed limiters on my router). After a while (maybe a minute or two) this starts to happen:

Code: Select all

Used cache  1.1 GB (1498 articles)
System load 5.17 | 2.24 | 0.98 | V=3470M R=1316M
Download speed limited by  CPU (470x)

Code: Select all

Used cache  1.1 GB (1498 articles)
System load  6.05 | 3.17 | 1.40 | V=3470M R=1293M
Download speed limited by  CPU (921x)

And interestingly, further down the road I get a disk speed limitation:

Code: Select all

Used cache  1.0 GB (1497 articles)
System load  4.74 | 3.50 | 1.69 | V=3474M R=1310M
Download speed limited by  CPU (1088x) Disk speed (27x)

The disk speed limit starts to pop up in about 5 minutes after the download starts. But first the article cache fills up (my limit is set to 1G).

I use a DRAM-less NVMe SSD but I don't think it can't sustain 100 MB/s writes. The main question - why CPU limit? It's a fairly fast CPU and Pystone is 256010. I used to run SABnzb on Windows 10 on a i7-4790K CPU at 4 GHz and there were no slowdowns at all. The gigabit connection was saturated to the end. Now I get saw-tooth-like waves in speed. Can an i5-7500 CPU @ 3.40GHz be really a limit? I wouldn't think so.

I was looking for what I could try and nothing helped. I found various posts and changed num_simd_decoders from 2 to 4 in Special; checked Pause Downloading During Post-Processing; added -n10 in Nice Parameters; -c2 -n4 in IONice Parameters. I found a way to install par2-tbb instead of the default par2. Nothing changed anything.

What could be a problem here? I thought this PC should be fast enough to sustain gigabit downloads. Speed tests on it in browser saturate the link. I was hoping it would sustain 3 Gb downloads as fiber is coming to the area.

One last thought. Downloads are saved on a standalone SSD (the system is on a separate). And the SSD has a ZFS file system. Total RAM on the PC is 8 GB. Could this be a problem? I haven't seen any reports about such.

[sander]

Your system should be good enough for sustained gigabit.

Is this your own system? Or something shared?
Because: why is the system load high? Check (with and without SAB downloading) with htop

And what is the effect of ZFS? Try with both temp/incomplete and complete on your non-ZFS disk.

[jsbchakji]

It is my own system. It has a Plex server, Radarr, Sonarr etc. on it. But nothing was actively running at the time when I was doing tests. The load is high from downloading with sabnzbdplus. At idle it looks like this:

Code: Select all

Used cache  0 B (0 articles)
System load 0.02 | 0.15 | 0.17 | V=3368M R=215M

Initially I discovered using top that it was sabnzbdplus that was taking a lot of CPU. I will try htop next time (not sure about the difference with top). When unrar was in use it didn't take much CPU. So, I wonder why the CPU load is so high when I am downloading.

I thought someone would know better than me the effect of ZFS. I remember ZFS needed RAM to work well but I don't know more about it except that it is more robust than ext4. All my disk are on ZFS, including the system. Everything else works as expected.

Where is temp by the way? How is it used? Is is Linux's /tmp or it's a separate directory for SABnzb? I didn't set it (nothing is in the instructions). So, it is in the default location and may be on another SSD with the system. Maybe I misunderstood what you meant by "temp/incomplete". I didn't think that SABnzb used a temporary directory.

What else can I check? Before posting I tried everything I could find about this by googling. Thanks.

[sander]

I meant the SAB folders. See config -> folders. But if all disks are ZFS, it's not testable.

If you do the 1GB test download (via Wrench), is all good? So: 100 MB/s download speed?
And with the 10GB test download?

[safihre]

Have you tried limiting the speed to 90MB/s?

[jsbchakji]

The slowdowns don't start immediately, it takes a few minutes of steady download at max speed. So, with the 1GB test it's fine, with the 10 one the slowdowns start towards the end. Yesterday I downloaded about 50GB and the slowdowns started as usual, in a couple of minutes, and the saw-tooth-like speed pattern continued until the end. When I started the download I watched the parameters in the wrench by refreshing it. The CPU load was slowly increasing and cache was low for a while than it started to approach 1G (my limit) and the CPU load was high at that time. That's when the slowdowns start. First, I get about a 1000x CPU "speed limited by" but only later the disk speed limit appears. At the end of that ~50GB download I got this:

Code: Select all

Download speed limited by CPU (3164x) Disk speed (215x)

This was after a fresh restart of sabnzbdplus, i.e., from zero. I am not sure why the disk speed limit starts to take place. But before that the major limit was the CPU. Before installing Ubuntu I tested this PC with prime95 for hours and the hardware held.

[jsbchakji]

Have you tried limiting the speed to 90MB/s?

No. I will try that. While on Windows I never had to limit the download speed in SABnzb and it maxed out.

[jsbchakji]

Hmm, interesting. I limited the speed to 90 MB/s in SABnzb and there was no slowdown on the 10GB test. At the end the status was:

Code: Select all

Used cache  51.3 MB (74 articles)
System load 0.65 | 0.29 | 0.13 | V=2304M R=366M

So, the CPU load didn't rise high and cache load was mostly below 100 MB.

What's the reasoning behind the suggestion to limit the speed to 90 MB/s? I will try longer downloads later.

[jsbchakji]

I limited the speed to 100 MB/s and no slowdown either:

Code: Select all

System load 0.94 | 1.09 | 0.61 | V=2566M R=627M

The CPU load was slightly higher towards the end. I need to test this with larger/longer downloads. It takes a while for them to start.

P.S. Towards the end of unpacking I see this CPU load:

Code: Select all

System load 2.57 | 1.69 | 0.91 | V=2567M R=628M

[jcfp]

The disk speed limit starts to pop up in about 5 minutes after the download starts. But first the article cache fills up (my limit is set to 1G).

That could be the internal workings of the ssd, something to do with zfs, or the combination of both. Many low end nvme drives (no dram cache with qlc flash memory) cannot sustain high write speeds once their internal cache is full, actually dropping off in performance so badly they can end up slower than old fashioned hdds. You could test by copying random data to the drive for a while, say 50GB or so and see what happens?

[safihre]

Note that the Disk one might give the false impression that it's always related to disk performance, however, during file assembly we also use CPU due to the md5sum calculations. So whatever is causing your decoder problems, might also be causing that one.

[sander]

@safihre : why does limiting the download speed work for @jsbchakji ? Does it avoid that SAB chokes on too much data or processing?

[jsbchakji]

The disk speed limit starts to pop up in about 5 minutes after the download starts. But first the article cache fills up (my limit is set to 1G).

That could be the internal workings of the ssd, something to do with zfs, or the combination of both. Many low end nvme drives (no dram cache with qlc flash memory) cannot sustain high write speeds once their internal cache is full, actually dropping off in performance so badly they can end up slower than old fashioned hdds. You could test by copying random data to the drive for a while, say 50GB or so and see what happens?

I thought about the SSD slow down due to cache exhaustion but rejected that idea because:

1. Why would CPU utilization be so high first? It takes about 2 minutes to get the "speed limited by CPU" notification and then about another 3 minutes to get the "speed limited by Disk speed".
2. If the SSD slows down the speed should go down and stay at some value more or less constant. At least I would think so. In my case the speed graph has a saw-tooth-like shape. The speed goes down sometimes to almost zero then up to almost max.

I will do a large chunk of data copy test and other tests and post the results.

[jcfp]

If the SSD slows down the speed should go down and stay at some value more or less constant. At least I would think so. In my case the speed graph has a saw-tooth-like shape. The speed goes down sometimes to almost zero then up to almost max.

Sab itself can throttle download speed if it notices internal queues (such as article decoding) growing too big, which can induce a pattern like that even if the underlying limiting factor has a (more) constant performance.

What made me suspect the disk is the time it takes before you see speeds drop, with article cache use staying low in the meantime. While the cpu is taxed quite a lot by yenc decoding and crc calculations, that would affect things right from the start. ZFS also takes some cycles for io ops and checksumming, which may also have an effect. Curious to see those i/o test results