Slow Par2 speeds on Ubuntu vs Windows

[L33tMasta]

I just set up a Ubuntu 21.10 desktop with the following specs:

Core i5 10400k
25GB RAM
Silicon Power P34A60 256GB NVMe (OS, software, VMs)
Samsung HD204UI 2TB HDD (media)

I set up Sabnzbd on it and downloaded an 80GB 4K Remux file, and while the download went without a slowdown, however the verification and repair process is going at a snails pace vs doing it on my Windows machine (i9 10850k, 32GB RAM, 7200RPM 4TB media HDD). I know there is a difference in specs but there’s no way that it should be THIS slow, is there? It took 18min to verify the files and 80GB repaired in 3 hours 14 min 38 seconds.

Under system monitor I see the CPU usage for Par2 is about 90% but the read times are usually only 11-14MBps and the write speed 3-5MBps. The drive is old but checked out during an extended SMART test and I am able to network transfer files to it at almost 1Gbps.

Am I missing some setting or have I set it up improperly?

[sander]

On the Ubuntu machine:

In SABnzbd's upper right corner, click on the wrench symbol ("Status and interface options"), then click on first tab Status, and there click on the Refresh Arrow. Post the values of the lower part here.

[L33tMasta]

On the Ubuntu machine:

In SABnzbd's upper right corner, click on the wrench symbol ("Status and interface options"), then click on first tab Status, and there click on the Refresh Arrow. Post the values of the lower part here.

For the i9 10850k (7200RPM HDD):

System performance (Pystone) - 346066
Download folder speed - 109.1 MB/s
Complete folder speed - 113.3 MB/s

For the i5 10400 (5400RPM HDD):

System performance (Pystone) - 306666
Download folder speed - 61.6 MB/s
Complete folder speed - 62.9 MB/s

Am I being limited by the HDD? I can transfer files via LAN much faster than 60 MBps, around 113MBps. And my download speeds reach 113MBps.

[sander]

Both disk speeds are low.

With a NVMe typical values are 600 - 1200 MB/s.

And 60 MB/s is even low for a spinning HDD.

[L33tMasta]

Both disk speeds are low.

With a NVMe typical values are 600 - 1200 MB/s.

And 60 MB/s is even low for a spinning HDD.

Just checking and it looks like the first drive is an SMR (I bought it before the whole debacle of them bait and switching came to light and it looks like I've been had - not happy) and that could impact the speed result.

I have a brand new confirmed CMR 5400RPM 6TB WD Red Plus arriving later today that I will test and see if it makes any difference.

[safihre]

The big difference is also that on Windows we use Multipar which is a much much faster version of par2. But only available on Windows. It's also much smarter in the way it analyzes files.
On Ubuntu, make sure you have the multicore version of par2cmdline, if you don't a message will be shown on the first page of the Config. So if you don't see a message, you're good.

[L33tMasta]

The big difference is also that on Windows we use Multipar which is a much much faster version of par2. But only available on Windows. It's also much smarter in the way it analyzes files.
On Ubuntu, make sure you have the multicore version of par2cmdline, if you don't a message will be shown on the first page of the Config. So if you don't see a message, you're good.

I have a checkmark for multicore par2 so that must be good. If what you're saying about the Windows version of Par2 is true I might just move the whole server to Windows 7 and run Linux VMs. I was told Linux was supposed to be a better option for a home server but it looks like that might not be the case, especially with these analyze and repair speeds.

[safihre]

Indeed Multipar is a very specific case, usually Windows itself has more overhead indeed..
I tried to convince the Multipar guy to convert to Linux, but he didn't want to!

[L33tMasta]

Indeed Multipar is a very specific case, usually Windows itself has more overhead indeed..
I tried to convince the Multipar guy to convert to Linux, but he didn't want to!

I appreciate your insight on this matter. It's too bad that they were unwilling to even support Linux.

[jcfp]

I have a brand new confirmed CMR 5400RPM 6TB WD Red Plus arriving later today that I will test and see if it makes any difference.

It should, smr disks are really bad at sustained writing of a large amount of data - which is exactly what happens in your scenario. For smaller amounts of data, these disks keep up appearances by using an internal cache (similar to a ramdrive); once that's full and a process such as sabnzbd/par2/unrar keeps sending data it's game over performance-wise.

[L33tMasta]

I have a brand new confirmed CMR 5400RPM 6TB WD Red Plus arriving later today that I will test and see if it makes any difference.

It should, smr disks are really bad at sustained writing of a large amount of data - which is exactly what happens in your scenario. For smaller amounts of data, these disks keep up appearances by using an internal cache (similar to a ramdrive); once that's full and a process such as sabnzbd/par2/unrar keeps sending data it's game over performance-wise.

For sure. I haven’t had time to test the speeds on the new 6TB CMR but I was comparing the speeds I was getting on the Linux machine to my windows machine that did have an SMR drive in it. Even with an SMR drive it was significantly faster on Windows to verify and repair the same 80GB file vs the Ubuntu box. I think the fast that MultiPAR is so much faster comes into play more than the fact the drive is SMR.

[sander]

I think the fast that MultiPAR is so much faster comes into play more than the fact the drive is SMR.

You can run multipar's par2j...exe with wine.
So I tested that on my Ubuntu (SSD) with a 1GB set with one corrupted rar file ... and native par2repair is faster than "wine par2j.exe"

Code: Select all

cd
mkdir multipar
cd multipar/
wget https://github.com/Yutaka-Sawada/MultiPar/releases/download/v1.3.1.9/MultiPar131.zip
unzip MultiPar131.zip 
wine par2j.exe 
wine par2j64.exe

[L33tMasta]

I think the fast that MultiPAR is so much faster comes into play more than the fact the drive is SMR.

You can run multipar's par2j...exe with wine.
So I tested that on my Ubuntu (SSD) with a 1GB set with one corrupted rar file ... and native par2repair is faster than "wine par2j.exe"

Code: Select all

cd
mkdir multipar
cd multipar/
wget https://github.com/Yutaka-Sawada/MultiPar/releases/download/v1.3.1.9/MultiPar131.zip
unzip MultiPar131.zip 
wine par2j.exe 
wine par2j64.exe

Interesting to know, for sure. I’ve switched the box over to a Windows 10 Home install that I’ve heavily modified to remove a lot of the bloat, and the difference in speed using the same 2TB drive I was under Ubuntu is vast. Verification was faster, repair is MUCH faster, and overall it looks as if the process is taking full advantage of the available memory, the CPU and the HDD. I’m noticing that the CPU doesn’t hit 100%, but stays around 85-89%, HDD usage is much faster at 100% with faster read/write speeds and memory utilization is way up at 14.5GB used.

I think in this instance native Windows running MultiPAR takes the crown. As much as I want to run a Linux box these results speak for themselves.