renicing par2/unpack/yenc processes for nix/osx

[PacoBell]

However, "nice" is only nice as far as CPU loading is concerned.

Actually, that's incorrect.

When setting a thread into background state the scheduling priority is set to lowest value, disk and network IO are throttled.  Network IO will be throttled for any sockets the thread opens after going into background state.  Any previously opened sockets are not affected.  Only the super-user may lower priorities, but any thread can set itself into background state.

[shypike]

It is correct in practice.
Why do you think that ionice was created?
I can only assume that nice doesn't do a very job at reducing disk I/O.
ionice uses rather recent kernel features to give a more reliable disk I/O reduction.

[PacoBell]

Well, ionice is effectively only a Linux feature and I'm talking about OSX. It hasn't been ported to BSD, OSX, or Windows, AFAIK. Apparently, Apple thought they'd just roll ionice's features into nice. See here for more details:

IOPOL_DEFAULT    This is the default I/O policy for the first process and every new created thread.

IOPOL_NORMAL     I/Os with NORMAL policy are called NORMAL I/Os.  They are handled by the system using best-effort.

IOPOL_THROTTLE   I/Os with THROTTLE policy are called THROTTLE I/Os.  If a THROTTLE I/O request occurs within a small time window (usually a fraction of a second) of another NORMAL I/O request, the thread that issues the THROTTLE I/O is forced to sleep for a certain interval. This slows down the thread that issues the THROTTLE I/O so that NORMAL I/Os can utilize most of the disk I/O bandwidth.

IOPOL_PASSIVE    The PASSIVE I/Os are a special type of NORMAL I/O that are processed the same as NORMAL I/Os but are ignored by the THROTTLE I/Os so that the threads issuing THROTTLE I/Os are not slowed down by PASSIVE I/Os.  The PASSIVE I/O policy is useful for server type applications.  The I/Os generated by these applications are called passive I/Os because these I/Os are caused directly or indirectly by the I/O requests they receive from client applications.  For example, when an image file is mounted by DiskImages, DiskImages generate passive I/Os.  DiskImages should mark these I/Os using the PASSIVE I/O policy so that when client applications that issue THROTTLE I/Os access the volume managed by DiskImages, these client applications will not be slowed down by the I/Os generated by DiskImages.

That sounds like a decent I/O scheduler from where I'm sitting.

P.S. Is there a python wrapper for darwin's flavor of libc? Maybe PyObjC?

[shypike]

I missed the OSX part.
This is something for or our OSX porter rAf.

[rAf]

I've been working on this since 2 weeks... :

So far, I've found how to load libc library from python and call the functions.

Here the code :

Code: Select all

#Constants from sys/resources.h
#I/O type
#IOPOL_TYPE_DISK 0

#scope
#IOPOL_SCOPE_PROCESS   0
#IOPOL_SCOPE_THREAD    1

#I/O Priority
#IOPOL_DEFAULT   0
#IOPOL_NORMAL    1
#IOPOL_PASSIVE   2
#IOPOL_THROTTLE  3

#int getiopolicy_np(int iotype, int scope);
#int setiopolicy_np(int iotype, int scope, int policy);

#load libc
from ctypes import cdll
libc=cdll.LoadLibrary('libc.dylib')

print "Get IO Priority Before"
intBefore=libc.getiopolicy_np(0,1)
print intBefore

print "Set IO Priority Result"
boolSetResult=libc.setiopolicy_np(0,1,3)
print boolSetResult

print "Get IO Priority After"
intAfter=libc.getiopolicy_np(0,1)
print intAfter

Seems to works now I have to test in SABnzbd.

[stampedes]

the above code is not far from a quick-and-dirty standin for ionice:

Code: Select all

#!/usr/bin/env python

from ctypes import cdll
import sys, os, getopt

# chomp all the options that ionice takes.
# ignore them and simply set throttle priority

optlist,args = getopt.getopt(sys.argv[1:], 'tc:n:p:')

# load libc and set io priority for process
libc = cdll.LoadLibrary('libc.dylib')
boolSetResult=libc.setiopolicy_np(0,1,3)

# exec new process with set io priority
try:
	os.execvp(args[0], args)
except IndexError:
	print "missing process to execute"

this works as a standin for linux ionice. when placed in the path, the ionice options will appear in sab. fill in options that ionice would accept, and par2/unrar will run with throttle priority. worth note is that this will throttle reads only- osx seems not to be able to throttle writes. either way, this should make sure that a par2 verification doesn't slow down IO to the point where other IOs (i.e. for playback) can't be serviced.

.. or at least, i'd think it would. this wrapper works fine outside of sabnzbd, but when executed by sabnzbd, i can't get it to work. it silently and instantly fails, and the logs don't show why or how it was called. next hurdle to tackle..