Sabnzbd + SSL = CPU murder?

[zsh]

Hi all,

First of all, thanks for great application. Works like a charm.

However, I do have a question regarding CPU usage and SSL. I know some other people have posted regarding this as well.

My setup:

Connection: 100mbit/s (verified pumping 95mbit +)
Hardware: Asrock ION N330 1.6 ghz

I use Astraweb as provider and I am using SSL.

Now with SSL enabled, throughput is arround 5500-6000KB/s. At this time, poor 1.6ghz cpu is at 100% and then some.
Without SSL enabled, throughput is at arround 11000-12000 KB/S.

Now, you don't have to be rocket scientist to realize that SSL part takes it fare share of CPU.

I overclocked the CPU to 2.1GHz and this increased my SSL throughput to arround 8000-8100 KB/s.

Since I "can't" overclock it more I realize that this is show stopper at the moment.
So my question, to you developers/community is if there is any SSL improvement / python SSL improvement on the way which could possibly improve the CPU usage / throughput? Or is it "game over" and live with reality moment?

Is there any point jumping on to Beta release  (if any improvements related to my inquiry exist in those)?


Thanks
zsh

[inpheaux]

As you've found, SSL involves CPU overhead. This only really becomes a noticable on low-power devices. Your only options are to either not use SSL or run SABnzbd on hardware that's beefy enough so that your download speed doesn't end up CPU-bound.

[zsh]

Thanks for reply inpheaux.

I have created workarround concerning this "issue"

I am using stunnel now (which seems to be better optimized for this when it comes to resources).
So, basically I'm running stunnel to my ssl usenet provider and sabnzbd server definition points to localhost (listening port) and no SSL.
This gives me arround 11000KB/s which i think can be considered decent enough for SSL on an N330 box

[jackmetal]

I am using stunnel now (which seems to be better optimized for this when it comes to resources).
So, basically I'm running stunnel to my ssl usenet provider and sabnzbd server definition points to localhost (listening port) and no SSL.

@zsh,  that is a great idea!  I don't know why I haven't thought of that before (since I've been heavily securing my network recently).  I haven't used stunnel (I've started reading through it now); do you have any pointers on configuring this?

Thanks for the excellent idea!!

[zsh]

I am using stunnel now (which seems to be better optimized for this when it comes to resources).
So, basically I'm running stunnel to my ssl usenet provider and sabnzbd server definition points to localhost (listening port) and no SSL.

@zsh,  that is a great idea!  I don't know why I haven't thought of that before (since I've been heavily securing my network recently).  I haven't used stunnel (I've started reading through it now); do you have any pointers on configuring this?

Thanks for the excellent idea!!

this is for debian (i take no responsibility for potentiall syntax errors since i am bit tipsy atm ;-)


apt-get install stunnel
sed -i -e 's/ENABLED=0/ENABLED=1/g' /etc/default/stunnel4
sed -i -e 's/ENABLED=0/ENABLED=1/g' /etc/init.d/stunnel4
echo "[usenet]" >> /etc/stunnel/stunnel.conf
echo "accept = 9999" >> /etc/stunnel/stunnel.conf
echo "connect = my.ssl.usenet.provider:443" >> /etc/stunnel/stunnel.conf
/etc/init.d/stunnel4 start


Basically change "9999" to port you want your local system to listen on (one you should point your sabnzbd server definition to) and "my.ssl.usenet.provider:443" to the hostname/ip of your SSL usenet provider together with the correct port they use for ssl.

Good luck

[shypike]

I'm surprised that stunnel delivers so much better performance.

Unfortunately there's not much we can do to make SABnzbd better in this area
because we have to rely on the standard SSL support for our programming language Python.
(And we have no intention on re-writing such libraries).

[zsh]

I'm surprised that stunnel delivers so much better performance.

Unfortunately there's not much we can do to make SABnzbd better in this area
because we have to rely on the standard SSL support for our programming language Python.
(And we have no intention on re-writing such libraries).

No idea to be honest how or why. I just felt I had to test it since, obviously, CPU was hammered when sabnzbd was handling both the ssl and downloading.
Ofcourse, optimal solution would be  to handle SSL on a separate unit, but wife acceptance factor only approved one unit that handles it all

Without any programing background or python itself, my guess would be (and this maybe hell of a lie)  that python just isnt that optimal, performance wise, compared to C code or perhaps other libs used...what do i know :p

[sander]

I tried the lazy methode below, but that does not work: SABnzbd reports a time-out / reset by peer 20 seconds after trying to connect over SSL to localhost on port 8888

Tips? Or is this just plain wrong?

Code: Select all

sander@quirinius:~$ sudo stunnel -d localhost:8888 -r sslreader.eweka.nl:563
sander@quirinius:~$ telnet localhost 8888
Trying ::1...
Connected to localhost.
Escape character is '^]'.
help


^CConnection closed by foreign host.
sander@quirinius:~$

[zsh]

I tried the lazy methode below, but that does not work: SABnzbd reports a time-out / reset by peer 20 seconds after trying to connect over SSL to localhost on port 8888

Tips? Or is this just plain wrong?

Code: Select all

sander@quirinius:~$ sudo stunnel -d localhost:8888 -r sslreader.eweka.nl:563
sander@quirinius:~$ telnet localhost 8888
Trying ::1...
Connected to localhost.
Escape character is '^]'.
help


^CConnection closed by foreign host.
sander@quirinius:~$

Try pointing to the IP of the actual interface instead of localhost (127.0.0.1).  For example 192.168.1.111 or whatever you have on the interface itself

[sander]

I tried, but still get "(104, 'Connection reset by peer')"

Can the cause be that I created a .pem via the website? A mismatch?

Is there another test service I can try out? Or is there a s-telnet I can use?

Code: Select all

sander@quirinius:~$ sudo stunnel -d 192.168.2.163:8888 -r sslreader.eweka.nl:563
[sudo] password for sander: 
sander@quirinius:~$ telnet 192.168.2.163 8887
Trying 192.168.2.163...
telnet: Unable to connect to remote host: Connection refused
sander@quirinius:~$ telnet 192.168.2.163 8888
Trying 192.168.2.163...
Connected to 192.168.2.163.
Escape character is '^]'.
ehlp
help
Connection closed by foreign host.
sander@quirinius:~$ telnet 192.168.2.163 8888
Trying 192.168.2.163...
Connected to 192.168.2.163.
Escape character is '^]'.
help
help
Connection closed by foreign host.
sander@quirinius:~$ 

Code: Select all

syslog:Nov 16 21:17:00 quirinius stunnel: LOG5[4935:3077639024]: stunnel3 accepted connection from ::1:36945
syslog:Nov 16 21:17:00 quirinius stunnel: LOG5[4935:3077639024]: connect_blocking: connected 81.171.88.252:563
syslog:Nov 16 21:17:00 quirinius stunnel: LOG5[4935:3077639024]: stunnel3 connected remote server from 192.168.2.163:48262
syslog:Nov 16 21:18:00 quirinius stunnel: LOG3[4935:3077639024]: readsocket: Connection reset by peer (104)
syslog:Nov 16 21:18:00 quirinius stunnel: LOG5[4935:3077639024]: Connection reset: 0 bytes sent to SSL, 0 bytes sent to socket
syslog:Nov 16 21:18:44 quirinius stunnel: LOG5[7232:3077667696]: stunnel3 accepted connection from 192.168.2.163:39772
syslog:Nov 16 21:18:44 quirinius stunnel: LOG5[7232:3077667696]: connect_blocking: connected 81.171.88.252:563
syslog:Nov 16 21:18:44 quirinius stunnel: LOG5[7232:3077667696]: stunnel3 connected remote server from 192.168.2.163:48271
syslog:Nov 16 21:19:45 quirinius stunnel: LOG3[7232:3077667696]: readsocket: Connection reset by peer (104)
syslog:Nov 16 21:19:45 quirinius stunnel: LOG5[7232:3077667696]: Connection reset: 0 bytes sent to SSL, 0 bytes sent to socket
syslog:Nov 16 21:20:56 quirinius stunnel: LOG5[7232:3077667696]: stunnel3 accepted connection from 192.168.2.163:39795
syslog:Nov 16 21:21:01 quirinius stunnel: LOG3[7232:3077667696]: SSL_accept: 140760FC: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
syslog:Nov 16 21:21:01 quirinius stunnel: LOG5[7232:3077667696]: Connection reset: 0 bytes sent to SSL, 0 bytes sent to socket
syslog:Nov 16 21:21:07 quirinius stunnel: LOG5[7232:3077667696]: stunnel3 accepted connection from 192.168.2.163:39796
syslog:Nov 16 21:21:11 quirinius stunnel: LOG3[7232:3077667696]: SSL_accept: 140760FC: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
syslog:Nov 16 21:21:11 quirinius stunnel: LOG5[7232:3077667696]: Connection reset: 0 bytes sent to SSL, 0 bytes sent to socket

[zsh]

Try running in client mode.

/etc/stunnel/stunnel.conf

comment out following (as shown):
;cert = /etc/stunnel/mail.pem
;key = /etc/stunnel/mail.pem

and make sure client mode is enabled as below:

; Use it for client mode
client = yes

[sander]

With that config, it seems stunnel does not create a listening socket at all ... :-(


Wait: I have SSL checked in SABnzbd. Is that correct? Or should SABnzbd use a plain NNTP connection, which stunnel then en-ssl-s?

Code: Select all

sander@quirinius:~$ sudo stunnel -c -d 192.168.2.163:8888 -r sslreader.eweka.nl:563
sander@quirinius:~$ telnet localhost 8888
Trying ::1...
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
sander@quirinius:~$ !diff
diff /etc/stunnel/stunnel.conf /etc/stunnel/stunnel.conf.org
6c6
< ;cert = /etc/ssl/certs/stunnel.pem
---
> cert = /etc/ssl/certs/stunnel.pem
45c45
< client = yes
---
> ;client = yes
sander@quirinius:~$ 

[sander]

I tried some more, based on https://agora.cs.illinois.edu/display/t ... structions

Still no good result. Maybe I should just skip this method; I tried this as a faster method ;-(

Code: Select all

sander@quirinius:~$ cat nntps.conf
;foreground = yes 
;output = /dev/stdout 
pid = /tmp/stunnel.pid 
 
[nntps] 
 
client = yes 
 
accept = localhost:9999 
connect = news.cs.illinois.edu:563 
;connect = sslreader.eweka.nl:563



sander@quirinius:~$ sudo stunnel ~/nntps.conf
2009.11.16 23:03:32 LOG7[8212:3079456976]: RAND_status claims sufficient entropy for the PRNG
2009.11.16 23:03:32 LOG7[8212:3079456976]: PRNG seeded successfully
2009.11.16 23:03:32 LOG4[8212:3079456976]: Wrong permissions on /etc/stunnel/stunnel.pem
2009.11.16 23:03:32 LOG7[8212:3079456976]: Certificate: /etc/stunnel/stunnel.pem
2009.11.16 23:03:32 LOG7[8212:3079456976]: Certificate loaded
2009.11.16 23:03:32 LOG7[8212:3079456976]: Key file: /etc/stunnel/stunnel.pem
2009.11.16 23:03:32 LOG7[8212:3079456976]: Private key loaded
2009.11.16 23:03:32 LOG7[8212:3079456976]: SSL context initialized for service stunnel
inetd mode must define a remote host or an executable
sander@quirinius:~$ 

sander@quirinius:~$ 
sander@quirinius:~$ 
sander@quirinius:~$ telnet localhost 119
Trying ::1...
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
sander@quirinius:~$ 

[shypike]

Without any programing background or python itself, my guess would be (and this maybe hell of a lie)  that python just isnt that optimal, performance wise, compared to C code or perhaps other libs used...what do i know :p

Nah, all SSL code is handled by C-coded libraries.
There's hardly any Python code involved.
Nevertheless, it looks like the stunnel writers did a better job.

[zsh]

With that config, it seems stunnel does not create a listening socket at all ... :-(


Wait: I have SSL checked in SABnzbd. Is that correct? Or should SABnzbd use a plain NNTP connection, which stunnel then en-ssl-s?

Hi,
No SSL should not be enabled in Sabnzbd.

I dont have access to my htpc atm, but as soon as i get home I will post my exact configuration files for you to make it easier.