Is something like this possible to do?

[kl1k]

I've got the following scenario. There is  local nntp server with some content cached and a international news server with all content cached. Traffic to the local server is not throttled, but traffic to the international news server is throttled until after hours.

If I download from the local news server and the content is not cached, the server retrieves the content from a upstream server but at a reduced rate.

Is something like the following possible.

I start downloading nntp from the local news server  - is it programmitcally possible to check if the content is cached. If not then schedule the download to happen after hours but from the international server.

[sander]

Question 1: *why* do want to schedule the non-cached, slow download until hours? Is it because you don't want to slow down downloads later in the queue? Or is it because the download is forbidden / unwanted?


Question 2: if you try to download non-cached content during 'business' hours, will the download be slow? So: much slower than downloading cached content?
If so, how about this method: measure the download speed via the SABnzbd API. If it's low, the current download is non-cached, and you can pause it and re-activate it later on (or, easier: move it to the end of the queue)

[kl1k]

To explain a bit further.

I've got SAB setup to a local nntp server and a international nntp server. During the day the ISP throttles the data limit to the international nntp server and opens it up after hours. The nntp server at the ISP is not thottled (the traffic is local so its not throttled)

If I had this in the queue:
1) ABC
2) XYZ

ABC is not cached on the local server, but cached on the international server.
XYZ is cached on the local server.

I would like ABC to start off downloading and then use this "measure the download speed via the SABnzbd API" to check that the speed is low and them move the job to the end of the queue so that the next job can start and come in at full speed.

I think the API thing might do the trick - will have to do some reading up on how it works. 

If I can monitor the speed and then if a job falls below the certain speed move it to the end of the queue.

[kl1k]

Unfortunately not - unable to tell the difference.  There's no way to tell if its cached or not without downloading.

Just reading up on the API now and xmls and was thinking of using the API to get the speed - if its below a threshhold then make the priority low.  I should be able to put together a perl script or something to do this.

[bluenote]

SAB has a setting somewhere that prevents slow downloads from blocking the queue.  If it operates how I think, it should solve your problem.

Its an exercise up to you to find the setting / documentation though

[shypike]

Config->Switches->"Only get articles for top of the queue"
When set off it generally speeds up things (at the expense of using a bit more memory).
It primarily enables the beginning of an new job to overlap the end of an old job.

If a large job happens to come in slow (e.g.. because it's very old and the server
needs to get it from another place), it won't help.

[kl1k]

I've put together  a bash script to do what I want - might be useful to someone else.

Might not be the most efficient code but it seems to work. Just include it in a cron job.

Code: Select all

#!/bin/bash

#Configs
Server="1.2.3.4"
Port="8080"
API="xxxxxxxxxxxxxxxxxxxxxxx"
Threshold="100"         #Kbps

#Get xml file
curl --silent --output api.xml "http://$Server:$Port/api?apikey=$API&mode=queue&start=START&limit=LIMIT&output=xml"

#Get Current Speed
speed=`grep kbpersec api.xml | awk -F'[<|>]' '{print $3}'`
rounded=`echo $speed | awk '{printf("%d\n",$1 + 0.5)}'`

#Get ID of current download
id=`grep -i -m 1 '<nzo_id>' api.xml | awk -F'[<|>]' '{print $3}'`


#Get priority of current download
priority=`grep -i -m 1 '<priority>' api.xml | awk -F'[<|>]' '{print $3}'`

echo $priority

#If priority is set to low then exit as we cant make it any lower.
if [ $priority == "Low" ]
then
        exit 0
fi


# Check for low speed download
echo "Speed" $rounded
echo "Theshold" $Threshold

if  [ $rounded -lt $Threshold ]

then

        # Sleep for 30 secs and then check speed again
        sleep 30
        #Call API again and get speed - if its still low then change priority
        curl --silent --output api.xml "http://$Server:$Port/api?apikey=$API&mode=queue&start=START&limit=LIMIT&output=xml"
        speed=`grep kbpersec api.xml | awk -F'[<|>]' '{print $3}'`
        rounded=`echo $speed | awk '{printf("%d\n",$1 + 0.5)}'`

        if  [ $rounded -lt $Threshold ]
        then
        #Guess its really slow
                echo "Low Speed - set priority to Low"
                curl --silent --output queue.txt "http://$Server:$Port/api?apikey=$API&mode=queue&name=priority&value=$id&value2=-1"
        fi

fi