ENGINE Error in HTTP server

typhoe · Post by **typhoe** » August 10th, 2011, 9:23 am

Hi,

I'm using SABnzbd Version: 0.6.7 with Python 2.6.6 on a Synology NAS.

Linux nas 2.6.32.12 #1742 SMP Tue Jun 7 14:02:32 CST 2011 x86_64 GNU/Linux synology_x86_411+II

This is the Zebulon spk package for Synology nas.
http://synoblog.superzebulon.org/2011/0 ... 6-7-1-spk/

I'm using the default skin Plush.

There is no specific firewall rules (but I'm using HTTPS to connect to the web gui).

There was no active connexion at the crash times (the only thing connecting might be the chromium extension SABconnect++ v0.5.5 and its refresh rate is set to 1mn).

I'm getting seemingly random HTTP server crashs (and have seen these since version 0.5.x I think):

Code: Select all

2011-08-09 06:14:13,464::INFO::[postproc:605] Queue has finished, launching: None (None)
2011-08-09 13:53:33,559::ERROR::[_cplogging:55] [09/Aug/2011:13:53:33] ENGINE Error in HTTP server: shutting down
Traceback (most recent call last):
  File "/usr/local/sabnzbd/share/SABnzbd/cherrypy/process/servers.py", line 75, in _start_http_thread
  File "/usr/local/sabnzbd/share/SABnzbd/cherrypy/wsgiserver/__init__.py", line 1655, in start
    self.tick()
  File "/usr/local/sabnzbd/share/SABnzbd/cherrypy/wsgiserver/__init__.py", line 1703, in tick
    s, addr = self.socket.accept()
  File "<string>", line 4, in accept
  File "/usr/local/python26/lib/python2.6/socket.py", line 197, in accept
error: [Errno 24] Too many open files

2011-08-09 13:53:33,560::INFO::[_cplogging:55] [09/Aug/2011:13:53:33] ENGINE Bus STOPPING
2011-08-09 13:53:33,562::INFO::[_cplogging:55] [09/Aug/2011:13:53:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('0.0.0.0', 9090)) shut down
2011-08-09 13:53:33,564::ERROR::[_cplogging:55] [09/Aug/2011:13:53:33] ENGINE Error in HTTP server: shutting down
Traceback (most recent call last):
  File "/usr/local/sabnzbd/share/SABnzbd/cherrypy/process/servers.py", line 75, in _start_http_thread
  File "/usr/local/sabnzbd/share/SABnzbd/cherrypy/wsgiserver/__init__.py", line 1655, in start
    self.tick()
  File "/usr/local/sabnzbd/share/SABnzbd/cherrypy/wsgiserver/__init__.py", line 1703, in tick
    s, addr = self.socket.accept()
  File "/usr/local/python26/lib/python2.6/socket.py", line 197, in accept
error: [Errno 24] Too many open files

2011-08-09 13:53:33,566::INFO::[_cplogging:55] [09/Aug/2011:13:53:33] ENGINE Bus STOPPING
2011-08-09 13:53:33,571::INFO::[_cplogging:55] [09/Aug/2011:13:53:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('0.0.0.0', 9090)) already shut down
2011-08-09 13:53:33,577::INFO::[_cplogging:55] [09/Aug/2011:13:53:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('0.0.0.0', 8080)) shut down
2011-08-09 13:53:33,581::INFO::[_cplogging:55] [09/Aug/2011:13:53:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('0.0.0.0', 8080)) shut down
2011-08-09 13:53:33,590::INFO::[_cplogging:55] [09/Aug/2011:13:53:33] ENGINE Stopped thread '_TimeoutMonitor'.
2011-08-09 13:53:33,590::INFO::[_cplogging:55] [09/Aug/2011:13:53:33] ENGINE Stopped thread '_TimeoutMonitor'.
2011-08-09 13:53:33,591::INFO::[_cplogging:55] [09/Aug/2011:13:53:33] ENGINE Bus STOPPED
2011-08-09 13:53:33,592::INFO::[_cplogging:55] [09/Aug/2011:13:53:33] ENGINE Bus STOPPED
2011-08-09 13:53:33,592::INFO::[_cplogging:55] [09/Aug/2011:13:53:33] ENGINE Bus EXITING
2011-08-09 13:53:33,593::INFO::[_cplogging:55] [09/Aug/2011:13:53:33] ENGINE Bus EXITING
2011-08-09 13:53:33,594::INFO::[_cplogging:55] [09/Aug/2011:13:53:33] ENGINE Bus EXITED
2011-08-09 13:53:33,595::INFO::[_cplogging:55] [09/Aug/2011:13:53:33] ENGINE Bus EXITED

So far, I have not identified anything special running at the time of each crash...

I see the message "error: [Errno 24] Too many open files" but there was no download in progress...

Any idea?

Thank you!

Post by **shypike** » August 10th, 2011, 9:45 am

Possibly there some limit on the amount of open sockets in either Synology's OS or its Python port.
You should check on the Synology forums or ask the package creator.

typhoe · Post by **typhoe** » August 10th, 2011, 9:48 am

Thanks for the quick answer, I'll check these ideas.

K3nnyfr · Post by **K3nnyfr** » August 25th, 2011, 12:53 pm

Hi, i've got exactly the same error but i'm on a laptop (DELL D620)

Code: Select all

2011-08-23 22:06:45,742::INFO::[urlgrabber:75] URLGrabber starting up
2011-08-24 12:11:52,903::ERROR::[_cplogging:55] [24/Aug/2011:12:11:52] ENGINE Error in HTTP server: shutting down
Traceback (most recent call last):
  File "/usr/share/sabnzbdplus/cherrypy/process/servers.py", line 75, in _start_http_thread
  File "/usr/share/sabnzbdplus/cherrypy/wsgiserver/__init__.py", line 1655, in start
    self.tick()
  File "/usr/share/sabnzbdplus/cherrypy/wsgiserver/__init__.py", line 1703, in tick
    s, addr = self.socket.accept()
  File "<string>", line 4, in accept
  File "/usr/lib/python2.6/socket.py", line 197, in accept
error: [Errno 24] Too many open files

2011-08-24 12:11:52,904::INFO::[_cplogging:55] [24/Aug/2011:12:11:52] ENGINE Bus STOPPING
2011-08-24 12:11:52,905::INFO::[_cplogging:55] [24/Aug/2011:12:11:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('172.16.1.11', 9090)) shut down
2011-08-24 12:11:52,910::INFO::[_cplogging:55] [24/Aug/2011:12:11:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('172.16.1.11', 8080)) shut down
2011-08-24 12:11:52,959::INFO::[_cplogging:55] [24/Aug/2011:12:11:52] ENGINE Stopped thread '_TimeoutMonitor'.
2011-08-24 12:11:52,960::INFO::[_cplogging:55] [24/Aug/2011:12:11:52] ENGINE Bus STOPPED
2011-08-24 12:11:52,960::INFO::[_cplogging:55] [24/Aug/2011:12:11:52] ENGINE Bus EXITING
2011-08-24 12:11:52,961::INFO::[_cplogging:55] [24/Aug/2011:12:11:52] ENGINE Bus EXITED
2011-08-25 19:43:57,499::WARNING::[__init__:159] Signal 15 interceptÃ©, enregistrement et fermeture en cours...
2011-08-25 19:43:57,500::INFO::[nzbqueue:167] Saving queue
2011-08-25 19:43:57,500::INFO::[__init__:833] Saving data for queue9.sab in /root/.sabnzbd/admin/queue9.sab
2011-08-25 19:43:57,500::INFO::[__init__:833] Saving data for totals9.sab in /root/.sabnzbd/admin/totals9.sab
2011-08-25 19:43:57,501::INFO::[__init__:833] Saving data for rss_data.sab in /root/.sabnzbd/admin/rss_data.sab
2011-08-25 19:43:57,501::INFO::[__init__:833] Saving data for bookmarks.sab in /root/.sabnzbd/admin/bookmarks.sab
2011-08-25 19:43:57,501::INFO::[__init__:833] Saving data for watched_data.sab in /root/.sabnzbd/admin/watched_data.sab
2011-08-25 19:43:57,502::INFO::[postproc:83] Saving postproc queue
2011-08-25 19:43:57,502::INFO::[__init__:833] Saving data for postproc1.sab in /root/.sabnzbd/admin/postproc1.sab
2011-08-25 19:44:12,969::INFO::[sabnzbdplus:1163] --------------------------------
2011-08-25 19:44:12,969::INFO::[sabnzbdplus:1164] sabnzbdplus-0.6.8 (rev=2993)
2011-08-25 19:44:12,969::INFO::[sabnzbdplus:1177] Platform = posix
2011-08-25 19:44:12,969::INFO::[sabnzbdplus:1178] Python-version = 2.6.6 (r266:84292, Sep 15 2010, 15:52:39)
[GCC 4.4.5]

I have Sabconnect++ also, refresh rate 1min.

Any suggestion ?

Post by **shypike** » August 25th, 2011, 3:48 pm

K3nnyfr wrote:Hi, i've got exactly the same error but i'm on a laptop (DELL D620)
Any suggestion ?

Yes, that you have an OS that's configured with a very low amount of allowed open ports.
Only you're not telling us which OS it is.

K3nnyfr · Post by **K3nnyfr** » August 26th, 2011, 5:10 am

I'm on Ubuntu 10.04 LTS ...

Post by **shypike** » August 26th, 2011, 6:14 am

I suggest that you try without SABconnect first.

typhoe · Post by **typhoe** » September 2nd, 2011, 3:48 am

Hi,

I can confirm that disabling SABconnect++ let SABNZBD work perfectly fine without crash anymore... So I now only enable it when I need it to send links to SAB...

Regards

void.pointer · Post by **void.pointer** » September 2nd, 2011, 8:53 am

I'm one of the developers for SABconnect++ and I'm not really convinced yet that this is an issue with SABconnect.

The issue could be either related to the SABnzbd API *or* a bug in the Chrome extension API. As far as which would be the problem, I can't say for sure right now.

SABconnect++ simply uses the HTTP protocol to communicate with the SABnzbd API. It is not possible to open a connection to SABnzbd and keep it open, that logic is handled by the protocol itself (by Chrome). If there are orphaned connections, that is probably an issue with SABnzbd not closing its sockets properly when API requests come in.

I don't know how SABnzbd works but I'd like to see a SABnzbd developer respond here to collaborate with me on possibly finding the source of the issue. There is such a lack of evidence and debug information right now to pin the problem on any particular side.

Also, I have never encountered this problem. Where in SABnzbd do you see this error? How can I check for this?

typhoe · Post by **typhoe** » September 2nd, 2011, 9:09 am

Hi,

I would gladly help debug the real cause of this problem.

I have not seen any "proof" in any log pointing the sabnzbd web engine crashes to sabconnect++ but all I know is that I noticed that disabling it stops the crashes from happening.

The informations I can provide you to help debug are that I use sabnzbd on https (internal https engine from sabnzbd, not from a reverse proxy), that I use sabconnect++ on several linux chromium (all using the ubuntu ppa chromium-daily/beta) but that I usually have only one browser open (maximum is two computer running both at the same time) and that I set sabconnect++ to use a refresh rate of 1 minute and of course, connect using https (I have to open the sabnzbd page once before sabconnect++ see anything du to the unknow certificate in chromium).

regards

void.pointer · Post by **void.pointer** » September 2nd, 2011, 9:19 am

Without having investigated the SABnzbd source code, I think I can make a safe assumption by stating that normal SABnzbd operation directly through the web interface does not use the public API (I don't see why it should or would). If this is indeed the case, then eliminating SABconnect from the picture indeed would hide the issue because it is not going through the API anymore.

I really don't want to have to debug SABnzbd code to resolve this issue so I'll need the expertise of someone who understands the intrinsics of the SABnzbd API.

There was an issue filed on the SABconnect++ Google Code project page about this, but no steps to reproduce were provided. I'll need steps to reproduce this issue. I need to know what was being done in SABconnect++, as well as how you determine when this failure happens (I assume you just look at the SABnzbd logs through the web interface). As I said I have never noticed this issue.

I do my SABconnect++ development and testing on Chrome Beta channel using Windows 7. I don't have Python installed directly, I believe that the SABnzbd installer already has Python bundled with it. I'll have to double check this.

Post by **shypike** » September 2nd, 2011, 9:39 am

The Plush skin does use the API (for some functions at least).
All web traffic is handled by the third-party CherryPy module which is bloody complicated and a nightmare to debug (for me at least).
We're on an intermediate patched version, mainly because the latest official release has a glaring https bug, making it unusable.

There was a change in our CherryPy configuration in SABnzbd 0.6.6,
which solved a memory leak in Windows (don't know if the issue occurred on Unix too).
You can revert to the old situation by using the --sessions parameter.
So, typhoe could you try this to see if it changes anything?

I'll try to find some time reproduce this.
So it was: Ubuntu 10.04, Chrome (which one) and SABconnect++ ?

typhoe · Post by **typhoe** » September 2nd, 2011, 10:17 am

I don't do anything special in sabconnect++.

Just enabling it and letting it poll sabnzbd seems enough. The crashes don't occur at a fixed time... it simply happens, sometime it takes 2/3 days before I notice a crash.

And to do so, well, I simply can't connect anymore to the web interface. The main process are still running but the web engine is dead.

My sabnzbd is running on a synology nas (linux) and the problem is older than 0.6.6.

void.pointer · Post by **void.pointer** » September 2nd, 2011, 11:18 am

Looks like you are using SABconnect++ and SABnzbd on the same machine. I think we can rule out SABconnect entirely if you test this on a separate machine. In other words, run SABconnect++ on a separate physical machine or VM and connect to another machine on your LAN that runs SABnzbd. If the problem still happens on the SABnzbd machine, then it can't possibly be SABconnect.

Post by **sander** » September 2nd, 2011, 11:55 am

void.pointer wrote:Looks like you are using SABconnect++ and SABnzbd on the same machine. I think we can rule out SABconnect entirely if you test this on a separate machine. In other words, run SABconnect++ on a separate physical machine or VM and connect to another machine on your LAN that runs SABnzbd. If the problem still happens on the SABnzbd machine, then it can't possibly be SABconnect.

Can you explain that? I don't understand your reasoning. For example: if there is a tool A opening more and more connections (and leaving them unattended) on a remote machine, and the server B on that remote machine fills up and stops answering, I would not say it's not tool A.

I would say there's a protocol problem between A and B, or SABconnect and SABnzbd, and we have to find out what's going on. Maybe a wireshark logging is needed.

Support Forum

ENGINE Error in HTTP server

ENGINE Error in HTTP server

Re: ENGINE Error in HTTP server

Re: ENGINE Error in HTTP server

Re: ENGINE Error in HTTP server

Re: ENGINE Error in HTTP server

Re: ENGINE Error in HTTP server

Re: ENGINE Error in HTTP server

Re: ENGINE Error in HTTP server

Re: ENGINE Error in HTTP server

Re: ENGINE Error in HTTP server

Re: ENGINE Error in HTTP server

Re: ENGINE Error in HTTP server

Re: ENGINE Error in HTTP server

Re: ENGINE Error in HTTP server

Re: ENGINE Error in HTTP server