[Windows] MCDF: monitor completed download folder(s)

[Ludwig]

Sorry it took me so long, but that array bug was a pesky one that took hours to show or, well, not show.

Download MCDF from here: http://www.multiupload.nl/SF6VQJNGHL (AutoIt source code included; feel free to modify as you see fit)

Changelog:
- changed MCDF so it checks every download that isn't already paused

• This was necessary because SABnzbd may be downloading more than one download at a time, making it possible for a download further down the list being completed before a download closer to the top.

- added IPv6 support, thanks to Sander

Why this is necessary:

It pauses downloads when the drive(s) where your completed downloads go to is full.

What happens when the "complete" disk is full while unpacking an archive is unpacking is aborted and the file that was being written to deleted, freeing up a bit of space. So you end up with too little disk space to fully unpack a download but still have some free space left. Sabnzbd then goes and moves files from the temp folder to the complete download folder until that folder can't store any more of the parts, but then you have your download sitting in two places. 1st part still in the temp dir on one partition, 2nd part in the complete dir on the other partition.

Sabnzbd then tries to finish the rest of the download queue. It downloads and tries to extract. Some smaller downloads might fit into the gap left by the previous bigger download. But most medium sized downloads are just split again. Sabnzbd downloads them, tries to unpack, notices there isn't enough disk space, deletes any partially written files, tries to move over parts, aborts moving as the disk becomes too full to store another part, then continues with the next download. So after one night of leaving it running you end up with 15 to 20 downloads that didn't unpack and are scattered between two locations waiting for you to piece them back together.

How MCDF does what it does:

MCDF assembles a look up table consisting of categories and their paths and puts drive letters next to the categories if these categories don't already contain drive letters. It then parses the topmost active download in your download queue and checks whether or not there's enough free space for it to unpack. If there isn't, that download is paused and the next one being checked.

Of course guessing the free space requirement a download needs for unpacking is pretty much impossible to do, so you have to specify how much more space than the download's size should be free on the completed downloads partition. E.g. your download is 3GB in size and you've set the free space multiplier to 2. That means the download is paused if there are less than 6GB of free space available for that download. Of course smaller downloads need less free space, so these are continued while bigger downloads are paused.



Quick howto:

- extract the downloaded archive to any place you like and can find easily
- copy the unzipped file(s) and folder(s) and paste them into your sabnzbd folder, the one containing the sabnzbd.exe file
- run mcdf.exe
- set free space multiplier in mcdf.ini as you like; changes are applied without having to restart the script
- you can display some status information by pressing Ctrl+Alt+D while MCDF is running; it should be working when none of the windows displayed are empty



Other important things:

MCDF makes use of GNU Wget, which is included in the archive. This was necessary to work around an issue AutoIt currently has in that it displays a "busy" mouse pointer when it shouldn't. This can be distractive if the system SABnzbd is running on is also being used by someone. GNU Wget is distributed under the GPL licence which can be found at http://www.gnu.org/copyleft/gpl.html

MCDF accesses SABnzbd via the host ip and http port as set up in SABnzbd. It currently does not support the use of secure http connections. For MCDF to work you have to set up SABnzbd accordingly or modify MCDF's source code.

Please report any issues or suggestions you might have in this thread.

[sander]

Some questions / remarks:

- it's a command line tool? So no GUI?
- Output/feedback is to mcdqueue.html? Or is that a temp file?
- It seems MCDF does not understand "::" as host name in sabnzbd.ini. See below. After replacing with 'localhost' it looks like MCDF can make a connection. "0.0.0.0" is correctly understood.
- it runs every second. Can that be lowered?
- info on the .ini file?

Disclaimer: I'm running this with wine / wineconsole under Ubuntu Linux. To get it running, I copied sabnzbd.ini into MCDF's directory, and I created a dummy sabnzbd.exe (with 'touch').



Not working with :: in sabnzbd.ini:

Code: Select all

SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
syswgetrc = Z:\home\sander\Downloads\MCDF\wget/etc/wgetrc
http://:::8080/api?mode=queue&start=START&limit=LIMIT&output=xml&apikey=2e298d010e8: Invalid host name.
SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc

Seems to be working:

Code: Select all

SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
syswgetrc = Z:\home\sander\Downloads\MCDF\wget/etc/wgetrc
--2012-04-22 12:12:29--  http://0.0.0.0:8080/api?mode=queue&start=START&limit=LIMIT&output=xml&apikey=2e2983d010e8
Connecting to 0.0.0.0:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1829 (1.8K) [text/xml]
Saving to: `C:/users/sander/Temp/mcdqueue.html'

100%[=====================================> ] 1,829       --.-K/s   in 0s      

2012-04-22 12:12:29 (16.5 MB/s) - `C:/users/sander/Temp/mcdqueue.html' saved [1829/1829]

[Ludwig]

It's a rather minimalistic GUI tool. The only GUI interaction possible is pausing / closing the script via its tray icon. It doesn't handle any command line parameters passed on to it.

mcdqueue.html and mcdpause.html are temporary files created by Wget. The latter should be deleted shortly after it's been created, the former is read by the script, deleted and its contents interpreted.

"::" is read by the script and passed on to Wget. If there are no temporary files created after calling Wget when having "::" in sabnzbd.ini, this means Wget can't interpret what the script passes on to it. I frankly have neglected the fact an address can not be a four segment IPv4 address.

Run times can be lowered of course (by changing the source code in one place), but even on a 100mbit line one second should be short enough to pause a download before it can finish.

Of course guessing the free space requirement a download needs for unpacking is pretty much impossible to do, so you have to specify how much more space than the download's size should be free on the completed downloads partition. E.g. your download is 3GB in size and you've set the free space multiplier to 2. That means the download is paused if there are less than 6GB of free space available for that download. Of course smaller downloads need less free space, so these are continued while bigger downloads are paused.

- set free space multiplier in mcdf.ini as you like; changes are applied without having to restart the script

[sander]

"::" (and any IPv6 address) is OK if you put [ ] around it to get the URL for wget. So if MCDF would check for a ":" in the host name and if found put [ ] around it, it would work.

Good:

Code: Select all

sander@R540:~$ wget http://[::]:8080/
--2012-04-22 14:02:12--  http://[::]:8080/
Connecting to :::8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 21049 (21K) [text/html]
Saving to: `index.html.10'

100%[============================================================================================================>] 21,049      --.-K/s   in 0s      

2012-04-22 14:02:12 (139 MB/s) - `index.html.10' saved [21049/21049]

Not good:

Code: Select all

sander@R540:~$ wget http://:::8080/
http://:::8080/: Invalid host name.
sander@R540:~$

[Ludwig]

Thank you for helping me patching this up. I've implemented your suggestion concerning IPv6 but can't test it since my network is still IPv4 based. Can you give it a quick try please and see if it works?

http://www.multiupload.nl/AN0DHSRQGT

[sander]

Thank you for helping me patching this up. I've implemented your suggestion concerning IPv6 but can't test it since my network is still IPv4 based. Can you give it a quick try please and see if it works?

http://www.multiupload.nl/AN0DHSRQGT

Yes, it works:

Code: Select all

SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
syswgetrc = Z:\home\sander\Downloads\MCDF\wget/etc/wgetrc
--2012-04-22 14:25:35--  http://[::]:8080/api?mode=queue&start=START&limit=LIMIT&output=xml&apikey=2e2983d77b010e8
Connecting to :::8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1829 (1.8K) [text/xml]
Saving to: `C:/users/sander/Temp/mcdqueue.html'

100%[=====================================> ] 1,829       --.-K/s   in 0s      

FYI: Since Windows Vista, Windows has IPv6 enabled. You can see that with the "ipconfig" command: you should see FE80:: addresses. So you can fill out :: as listening host in SABnzbd, restart SABnzbd, and then connect via http://[::]:8080/ and/or http://[::1]:8080/ . Can you confirm that?

[Ludwig]

Works like a charm, just like you said. Thanks. I think I really should start looking into IPv6, something I've been skipping whenever possible.

[sander]

Works like a charm, just like you said. Thanks. I think I really should start looking into IPv6, something I've been skipping whenever possible.

IPv6 can be quite easy: what does http://test-ipv6.com/ say about your IPv6 address? If you have a standard Windows setup with no brain-dead virusscanner and firewall, http://test-ipv6.com/ should show your public IPv6 address. If you have a standard Windows setup with Microsoft virusscanner and firewall ("MS Security Essentials"), IPv6 should work. So ... let me know.

[Ludwig]

Your IPv6 connection appears to be using Teredo, a type of IPv4/IPv6 gateway; currently it connects only to direct IP's. Your browser will not be able to go to IPv6 sites by name. This means the current configuration is not useful for browsing IPv6 web sites.

To my understanding this comes as no surprise since my ISP doesn't yet do IPv6. Other than that the site says I'm good.

[sander]

Your IPv6 connection appears to be using Teredo, a type of IPv4/IPv6 gateway; currently it connects only to direct IP's. Your browser will not be able to go to IPv6 sites by name. This means the current configuration is not useful for browsing IPv6 web sites.

To my understanding this comes as no surprise since my ISP doesn't yet do IPv6. Other than that the site says I'm good.

Microsoft has invented and introduced Teredo, which can give you IPv6 connectivity without the need for your ISP supporting it. The site http://test-ipv6.com/ apparently reports it working for you. That's good.

With only Teredo-IPv6-connectivity, Microsoft has turned off IPv6 DNS lookups on Vista and Win7. That means you can not IPv6-only websites in your webbrowser via their names, but you can via their IPv6 address like http://[2a00:1450:400c:c01::67]/ (= ipv6.google.com).

Funny enough, SABnzbd / python seems to do it's own IPv6 name resolution. This means you can fill out newszilla6.xs4all.nl and weathergirl-ipv6.tele2.net as newsservers in SABnzbd (free access, no account needed), and download from them. I can reach speeds of 30 - 70 Mbps with them ... !
See http://www.appelboor.com/newsserver.html for their status (quite good in my experience). On top of that, http://www.appelboor.com/newsserver.html will also show your IPv6 address (if you have one).

So could you fill out newszilla6.xs4all.nl and weathergirl-ipv6.tele2.net as newsserver in SABnzbd and let me know if it works? Thanks.

[Ludwig]

Since there have been no issues reported and it's been running fine for over a week now on my computer pausing downloads exactly as intended, I declare MCDF a beta no longer