sabnzbd_monitor.py - Detects and prevents downloads stalling and keeps everything running 100%.

[tagwolf]

I created a script that ensures sabnzbd is always stable and running to fix any stalls. This works great on my ubuntu system. Adjust as needed.

Update the Config options for host, port, protocol, apikey, state_file, and restart_command (if needed). The state_file config option isn't anything special. It is just where you want to store a small file that holds the last mbleft value. Make sure it's a path that exists and somewhere you have read and write permission. The script will create the file for you if so.

sabnzbd_monitor.py

Code: Select all

#!/usr/bin/env python3

import requests
import os
import time

# Config
host="127.0.0.1"
port="8081"
protocol="http"
apikey="b235eaa5232345c88edfd804c4094768"
state_file="/home/fetch/scripts/sabnzbd.state"
restart_command="sudo service sabnzbdplus restart"

# Should not need to be changed
apipath = '/sabnzbd/api?mode=queue&output=json&apikey='
url = protocol + "://" + host + ":" + port + apipath

def restart_sabnzbd():
    # Ensure the following is in sudo and the user running this script is in the sudo group:
    # %sudo   ALL=(ALL:ALL) NOPASSWD: ALL
    print("Restarting sabnzbd. Ctrl-C to abort...")
    print("Restarting in 3...", end="\r")
    time.sleep(1)
    print("Restarting in 2...", end="\r")
    time.sleep(1)
    print("Restarting in 1...", end="\r")
    time.sleep(1)
    print("Restarting sabnzbd...")
    os.system(restart_command)


def get_sabnzbd_data():
    response = requests.get(url + apikey)
    return response


if not os.path.exists(state_file):
    open(state_file, 'w').close()

try:
    print("Attempting connection to sabnzbd API at " + "\"" + url + ("*" * len(apikey)) + "\"...")
    get_sabnzbd_data()
except:
    print("First attempt to connect failed.")
    restart_sabnzbd()
    time.sleep(10)
    try:
        get_sabnzbd_data()
    except:
        print("Could not connect! Check URL, API key, and sabnzbd running status.")
        exit(1)

print("Retrieving data from sabnzbd...")
response = get_sabnzbd_data()
parsed_response = response.json()
state = open(state_file, 'r')

if parsed_response['queue']['paused'] == True:
    print("sabnzbd paused.")
    restart_sabnzbd()
else:
    mbleft = float(parsed_response['queue']['mbleft'])
    noofslots_total = int(parsed_response['queue']['noofslots_total'])
    last_mbleft = state.readline().split('\n')[0]
    state.close()
    if mbleft == 0 and noofslots_total == 0:
        print("No downloads queued, no action taken.")
        exit(0)
    else:
        try:
            last_mbleft = float(last_mbleft)
        except:
            last_mbleft = 0.0
        diff_mbleft = last_mbleft - mbleft

        print("mbleft difference since last monitor check:", f'{diff_mbleft:.2f}' + "Mb")
        if last_mbleft == mbleft:
            print("Potential stall. No download changes since last run.")
            restart_sabnzbd()
        else:
            print("Stable sabnzbd detected, no action taken.")
            state = open(state_file, 'w')
            state.write(str(mbleft))
        state.close()
exit(0)

My crontab to manage diskspace (deletes files once an hour that are older than a day, which (for me) means stalled / dead / trash) and run the sabnzbd_monitor.py script:

Code: Select all

# m h  dom mon dow   command
0 * * * * rm -r /downloads/complete/movies/_FAILED_*
0 * * * * find /downloads/incomplete -mindepth 1 -atime 1 -delete
0 * * * * find /downloads/complete -atime 1 -not \( -name 'complete' -or -name 'movies' -or -name 'series' \) -delete
#0 * * * * find /downloads/complete/movies -mindepth 1 -atime 1 -delete
#0 * * * * find /downloads/complete/series -mindepth 1 -atime 1 -delete
0 * * * * find /downloads -atime 1 -type f -delete
*/5 * * * * /home/fetch/scripts/sabnzbd_monitor.py

Please note, be very careful with the delete paths above. For me, all my active downloads are in /downloads. I keep all my media in /media so the crontab above (for me) will never delete any of my library files.

If you only want to run the monitor script and not cleanup stale downloads, only use the last line in my crontab. Again, adjust for your system. Also don't forget to change the Config section of the script below for your setup and add your API key.

Also note, if you intentionally pause sabnzbd to process downloads, etc., you will probably want to adjust the crontab to not run as aggressively as I do (every 5 minutes). Maybe once an hour or every day is good enough for you. Up to you! For reference, once an hour would be the following:

Code: Select all

0 * * * * /home/fetch/scripts/sabnzbd_monitor.py

Expected sabnzbd_monitor.py output (interactive run)

Code: Select all

Attempting connection to sabnzbd API at "http://127.0.0.1:8081/sabnzbd/api?mode=queue&output=json&apikey=********************************"...
Retrieving data from sabnzbd...
mbleft difference since last monitor check: 6328.27Mb
Stable sabnzbd detected, no action taken.

Output if sabnzbd is paused

Code: Select all

Attempting connection to sabnzbd API at "http://127.0.0.1:8081/sabnzbd/api?mode=queue&output=json&apikey=********************************"...
Retrieving data from sabnzbd...
sabnzbd paused.
Restarting sabnzbd. Ctrl-C to abort...
Restarting sabnzbd...

Output if sabnzbd hasn't downloaded anything since last run and there are downloads in queue

Code: Select all

Attempting connection to sabnzbd API at "http://127.0.0.1:8081/sabnzbd/api?mode=queue&output=json&apikey=********************************"...
Retrieving data from sabnzbd...
Potential stall. No download changes since last run.
Restarting sabnzbd. Ctrl-C to abort...
Restarting sabnzbd...

Output if no downloads in queue

Code: Select all

Attempting connection to sabnzbd API at "http://127.0.0.1:8081/sabnzbd/api?mode=queue&output=json&apikey=********************************"...
Retrieving data from sabnzbd...
No downloads queued, no action taken.

Output if sabnzbd is not running, can't be started, and/or your config is wrong

Code: Select all

Attempting connection to sabnzbd API at "http://127.0.0.1:8081/sabnzbd/api?mode=queue&output=json&apikey=********************************"...
First attempt to connect failed.
Restarting sabnzbd. Ctrl-C to abort...
Restarting sabnzbd...
Could not connect! Check URL, API key, and sabnzbd running status.

If you find this useful or have any issues let me know. The only thing this doesn't deal with is orphaned jobs as I personally would rather those get deleted out. If you want you can always add an API call for mode=status&name=add_all_orphan and run that at some interval.

There may also be certain stuck conditions I haven't experienced myself. If you find any let me know and I'll happily add to this script.

[safihre]

Wouldn't it be more useful to help us fix why it's stalls?

[tagwolf]

Wouldn't it be more useful to help us fix why it's stalls?

This is an interesting idea . I haven't looked much at the sabnzbd source code? What language is it in? I'm primarily a python developer but know a few other languages.

[tagwolf]

Just looked at the sabnzbd source code. I'm primarily a python developer so lucky all around that it's python!

Is there any developer documentation on files, libraries, etc.

Also a list of known problems and bugs?

[Puzzled]

There isn't much documentation other than what's in the code. The known issues are mostly what's on Github: https://github.com/sabnzbd/sabnzbd/issues

I fixed a bug related to stalled downloads a while ago, maybe it can point you in the right direction: https://github.com/sabnzbd/sabnzbd/comm ... dea87df3d4

You probably need a good understanding of the queue structure to fix these kinds of bugs. It has several layers: NZB-files (nzo) -> individual files from the nzb (nzf) -> articles. When it fails it's usually because something prevents sabnzbd.NzbQueue.get_articles() from finding more articles from any servers for the nzo even though not all nzfs have been registered as finished.

[tagwolf]

Yea, there's got to be a way to implement some stall detection. Even if it's something as simple as a tunable timeout vs the size of the file for sab to just mark it as failed and move on.

Or better yet, an amount of time where the download hasn't progressed after X time.