Incorrect free space

[jcfp]

Seems the values from the apple python overflow at 4*1024^4. Once you know that (and compensate for that finder thing defining a TB using decimal rather than binary) the values from sab start making sense.

[safihre]

@jcfp, so how do we correct the calculation then?

Also users report it below 1TB:
https://forums.sabnzbd.org/viewtopic.ph ... 98#p112391

I am starting to also suspect an macOS update to be the culprit, since users never reported this before.

[jcfp]

Now that part I don't know. Get apple to fix their python release? Use some alternative to get the free space (if any)?

[sander]

I have no problem on a MacOS with a 5588 GB (so: > 4TB) partition:

df output:

Code: Select all

server:~ sander$ df -g /Volumes/BigDataZenato
Filesystem 1G-blocks Used Available Capacity iused      ifree %iused  Mounted on
/dev/disk5      5588  852      4736    16%  892090 4294075189    0%   /Volumes/BigDataZenato

So: >4TB partition

python script output:

Code: Select all

server:~ sander$ ./disk_space.py /Volumes/BigDataZenato
2.7.10 (default, Feb  7 2017, 00:08:15)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)]
Directory /Volumes/BigDataZenato
statvfs posix.statvfs_result(f_bsize=2097152, f_frsize=8192, f_blocks=732482664, f_bfree=620759077, f_bavail=620759077, f_files=4294967279, f_ffree=4294075189, f_favail=4294075189, f_flag=0, f_namemax=255)
disk_size 5588.39923096
available 4736.01590729

So: correctly reported

python script:

Code: Select all

#!/usr/bin/env python
import os
import sys

print sys.version

_dir = '/Users/'
_dir = sys.argv[1]
print "Directory", _dir

s = os.statvfs(_dir)
print "statvfs" , s

GB = float(1024**3)
disk_size = float(s.f_blocks) * float(s.f_frsize) / GB
available = float(s.f_bavail) * float(s.f_frsize) / GB

print "disk_size", disk_size
print "available", available

[.maxx]

Im using an other Python Version with SAB 2.1.0 [443efb5]

Code: Select all

2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 12:39:47) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] [US-ASCII]

[sander]

OK, Mac-users, here we go: is the problem in python-level statvfs() or in c-level statvfs()? Let's determine:


Do you have a C-compiler on your Mac? If so, compile and run the C-program below against your NAD drive; it does a statvfs() a the C/system level.

Compile:

Code: Select all

server:~ sander$ gcc -o statvfs_in_c statvfs_in_c.c 

Run:

Code: Select all

server:~ sander$ ./statvfs_in_c /Users
Available [MB]: 306153
server:~ sander$

Tested on a Mac (thanks Dr B):

Code: Select all

server:~ sander$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.12.5
BuildVersion:	16F73

Code: Select all

#include <sys/statvfs.h>
#include <stdio.h>

int main (int argc, char **argv) {

    if(argc!=2) {
        printf("Error: Missing required parameter: directory.\n");
        return -1;
    }

    struct statvfs info;
    statvfs (argv[1], &info);

    // Python SAB source says: available = float(s.f_bavail) * float(s.f_frsize)
    long available = info.f_bavail * info.f_frsize / (1024*1024);
    printf("Available [MB]: %ld\n", available);

    return 0;
}

[sander]

For those who can't compile: here's the compiled version: https://www.appelboor.com/dump/MacOS/

Direct link: https://www.appelboor.com/dump/MacOS/st ... MacOS64bit

Code: Select all

$ ll 
-rwxr-xr-x 1 sander sander 8496 jun 27 21:22 statvfs_in_c_for_MacOS64bit*

Code: Select all

$ file statvfs_in_c_for_MacOS64bit 
statvfs_in_c_for_MacOS64bit: Mach-O 64-bit x86_64 executable

Code: Select all

$ md5sum statvfs_in_c_for_MacOS64bit 
dc24d3de514a2242a57aca4bb86bba27  statvfs_in_c_for_MacOS64bit

[.maxx]

@sander

can you please help me to run statvfs_in_c_for_MacOS64bit
I can't figure out how to use this file

[sander]

@sander

can you please help me to run statvfs_in_c_for_MacOS64bit
I can't figure out how to use this file

Certainly. Glad you can test.

Download the file with your favorite webbrowser, or with curl:

Code: Select all

curl https://www.appelboor.com/dump/MacOS/statvfs_in_c_for_MacOS64bit > statvfs_in_c_for_MacOS64bit

Make it's there OK:

Code: Select all

$ ls -al statvfs_in_c_for_MacOS64bit
-rw-r-xr--+ 1 sander  staff  8496 Jun 28 13:50 statvfs_in_c_for_MacOS64bit

Make it executable:

Code: Select all

chmod a+x statvfs_in_c_for_MacOS64bit

Execute it with a directory name:

Code: Select all

$ ./statvfs_in_c_for_MacOS64bit /Users/
Available [MB]: 305052

If that works, do it for your NAS mount directories and post it here.

[.maxx]

thanks sander

here are the results:

Correct

Code: Select all

$ /statvfs_in_c_for_MacOS64bit /Users/
Available [MB]: 435223

And I thinks here's the problem.
Wrong

Code: Select all

$/statvfs_in_c_for_MacOS64bit /Volumes/Medien
Available [MB]: 226943

df -h /Volumes/Medien/

Code: Select all

$ df -h /Volumes/Medien/
Filesystem                              Size   Used  Avail Capacity     iused      ifree %iused  Mounted on
//admin@SERVER._smb._tcp.local/Medien   26Ti   22Ti  4.2Ti    84% 23575294178 4527357888   84%   /Volumes/Medien

[safihre]

So the underlying C function is wrong..
While before it was correct.
MacOS update wrongly made by Apple?

[.maxx]

the user from the initial post got a raspi

[sander]

Make sure no other program (like SAB) is saving / deleting stuff on disk.

Then do this is a terminal window. Replace '/Directory/Of/Interest' with your disk / directory, like '/Users/' or '/mnt/nas/bla':

Print disk usage via different methods:

Code: Select all

df -m
python -c "import os; s = os.statvfs('/Directory/Of/Interest'); print s; print 'free:', float(s.f_bavail) * s.f_frsize / (1024**2) "

Download & save a 1400MB file

Code: Select all

curl http://nl.cdimage.ubuntu.com/ubuntu-gnome/releases/17.04/release/ubuntu-gnome-17.04-desktop-amd64.iso > my1400MB.iso

Again print disk usage:

Code: Select all

df -m
python -c "import os; s = os.statvfs('/Directory/Of/Interest''); print s; print 'free:', float(s.f_bavail) * s.f_frsize / (1024**2) "

Post all output here.

... hopefully we can compare the outputs

[sander]

https://stackoverflow.com/questions/209 ... -statvfs64 tells there is a statvfs64() ... and it seems you get that via a gcc compile option. On linux I see the difference in the executable, but not on MacOS, but can you try it anyway:

https://www.appelboor.com/dump/MacOS/ now also contains statvfs_with_LARGEFILE64_SOURCE_for_MacOS64bit
Direct link https://www.appelboor.com/dump/MacOS/st ... MacOS64bit

Can you download & execute just like you did for the first one? And post the result here?

[.maxx]

Here the LARGEFILE64 results

Correct

Code: Select all

$ /statvfs_with_LARGEFILE64_SOURCE_for_MacOS64bit /Users/
Available [MB]: 434190

Wrong

Code: Select all

$/statvfs_with_LARGEFILE64_SOURCE_for_MacOS64bit /Volumes/Medien
Available [MB]: 223562