Manipulating Powerboost

[Snatch]

I hate to disappoint you, but it isn't high on our todo list.
We already have a large backlog of other requests that have a larger potential audience.

We do welcome working patches 

So, just to verify my understanding. If I write a patch for you that does this feature request, you'll incorporate it in ongoing developement?

I don't want to rebuild your source upon every release, therefore I haven't tried doing this yet. But if it would be fully adopted, I will try taking a stab at it.

Cheers!

[shypike]

Sure. The best way is to checkout the trunk release.
Then make your changes and (if you want) do regular
sync with the trunk. Often the Tortoise client will do automatic merges,
sometimes you have to help it.

The URL to use for the subversion archive is: https://svn2.assembla.com/svn/SABnzbd/trunk/main

If you prefer you can checkout the latest tagged release
and work from a non-changing source set.
If you do a nice job, we can merge in into the trunk ourselves.

Maybe the last option is the best, because we plan lots of refactoring
and other cleanups for after 0.5.0, that may make life difficult if you want
to keep up with the trunk.

[phunkysai]

Charter has now implemented a Boost feature like this as well...

[Snatch]

It took me a while to get some free time, but I wrote a quick and dirty patch: http://pastebin.com/m4c9834b5

This was patched against the current SVN 3079 and I wrote the Plush code and English configuration settings.

Fundamentally it works as expected. After X number of kilobytes SABnzbd+ does a forced reset of the connections and then lets it restart.

I'm about to do some before and after tests as the act of disconnecting and reconnected obviously takes some time.  I think it would be better to actually restart individual connections after each stream reaches the specified byte limit instead. Over time, I think the average utilization will stay higher than cutting everything periodically.

Please feel free to test and comment/update/etc.

Cheers!

[mbruni]

+1 Id like to see this implemented in a further release.

[Snatch]

Yeah, I never reported the speed, but it wasn't any faster, sadly.

I did try stepping through the connections with a delay between them. However, as this is my first attempt with Python, I wasn't calling it right and it just bombs out.

I'll be off again next week through the end of the year, I think I should be able to squeeze it in!

Cheers!

[Snatch]

Here's another patch that actually cycles each thread as they cap out: http://pastebin.com/f3a7a7aa1

Unfortunately, it isn't any faster though.

I think we may need to look at the actual TCP socket call for "__reset_nw" and make sure it actually closes the socket and opens a new socket.

Any code guru's maybe able to offer some advise at this point?

Thanks & Cheers,
Snatch

[ziddey]

Powerboost or whatever different providers call it is becoming standard practice. Really actually not a bad idea if you think about it. The only real negative criticism for it is that you can't get proper speedtest results anymore since it'll just show your inflated powerboost speeds.

[Omikron]

Was this ever implemented?

[shypike]

No it wasn't.
It may though, when I find the time to look into it.
One handicap is that none of the team is exposed to
an ISP that actually has this Powerboost nonsense...
So the incentive to implement it is low and the opportunity to test is even lower.

[Cladmonitor]

I hope everyone realizes that the implementation of Powerboost/Speedboost varies by each provider. Some are done on the CMTS others done on the local node or UBR. The Timing in which you can re-boost is entirely up-to the provider to set.

Cox For instance in SD;
Powerboost is 30Mbps
Standard QOS is 20Mbps
After your connection has gone idle which is >2Mbps for 60 seconds+.
Duration of Speed Boost is really dependent on the time of day and available bandwidth at said device. But I get 15seconds at 3am (lowest usage for my area in a 24hour day)

Basic math here..
If we want to dl a 1GB file (1,048,576,000 bytes)
With Speed boost in one shot : 44.928 sec
Without Speed boost : 52.4288
I mean how would adding 60 seconds here be of any benefit?

With all things given, how is this ever going to be a plus, I mean ISP's are not dumb enough to allow powerboost ever to be used for anything but its intended (and almost worthless) boost of speed.

[shypike]

One  more argument that supports the hesitation of the team...

[Snatch]

In response to the hesitation over the ISP variance; I offered in reply #5 that the issue could be mediated with a user modifiable option for either time or byte limits. Given the vast majority of users would hit byte limits far before any arbitrary time threshold, I'd vote for the byte limit myself. However, nothing is stopping from offering both time and byte thresholds for users to modify to match their ISP. I can assert that when I start my downloads with SABnzbd, I get over 5MBps and they quickly drop to 1.5MBps (or 12mbps that I subscribe to). Done effectively, we should be able to sustain the the full Powerboost speeds (minus the overhead required to log in the expired threads).

Furthermore, I did my best to dissect the inner-working of SABnzbd, and I offered a working (albeit subjectively, but working nonetheless) patch on reply #41. As I mentioned, I thought we'd have to tackle the "__reset_nw" call and make sure it was actually closing the existing socket connection and not trying to re-use an established concurrent session.

I appreciate the continued consideration of this feature. It was initially requested back on July 2008. I offered some working example code in December 2009. Can we just have someone more experience look at the patch I offered and see if I possibly missed something hopefully obvious? As for functional testing, please feel free to PM or just drop a test build here for users to test for you. I'm sure you won't have to work too long to see verifiable results.

Thanks again for your support!

[Snatch]

I took my last patch and updated it again the current trunk build (3388). Here's another attempt: http://pastebin.com/raw.php?i=ZWNVKnVZ

By watching the logs, I'm fairly certain that this is actually killing the thread and starting a new one after the configured byte limit is reached. Unfortunately, I'm still not seeing much benefit on the overall throughput.

I really need someone that knows what they are doing with the actual sockets to review my code and see if there is a better way to kill the established thread and start a new one in its place. I am not confident that I'm actually doing this correctly on an actual socket level.

The meat of the code is under 'downloader.py':

Code: Select all

                if self.byte_limit > 0:
                    BPSMeter.do.update(bytes)
                    try:
                        bytecount = bytecount + bytes
                        if bytecount > self.byte_limit:
                            logging.info("Byte Limit Reached: %s, Byte Count: %s, Bytes: %s, NW: %s", self.byte_limit, bytecount, bytes, nw)
                            quit = nw.connected and server.active
                            self.__reset_nw(nw, "forcing disconnect", warn=False, wait=False, quit=quit)
                            nw.soft_reset()
                            #server.busy_threads.remove(nw)  <- crashed downloader assuming reset threads aren't busy so I moved it to idle_threads.remove instead
                            server.idle_threads.remove(nw)
                            server.idle_threads.append(nw)
                            bytecount = 0
                            continue
                    except NameError:
                        bytecount = 0

I'd really appreciate it if anyone else can help look at this more intelligently with me.

Cheers!

[shypike]

I will have a look at it, but it may take some time.
The truth is that the downloader/newswrapper is a horrible piece of code that we prefer not to touch.
Python's socket library is not it's strongest point and a lot of trickery is used to get some performance.