SABnzbd v4 compatibility with v3?

[OneCD]

Hi guys, I just wanted to get a head-start on preparing for the new SABnzbd 4.

Will the upgrade from 3.7.x be fairly straightforward?

Any new or removed external dependencies? New or removed Python modules?

And most importantly: will there be a new application icon?

[sander]

> Will the upgrade from 3.7.x be fairly straightforward?

In my experience: yes. And downgrade works too (see release notes)

> Any new or removed external dependencies? New or removed Python modules?

YES: sabctools (instead of sabyenc)
Check requirements.txt


> And most importantly: will there be a new application icon

No

[OneCD]

Thanks @sander.

I only asked about the icon because the current SVG has a very faint (but still visible) “3” in it.

[safihre]

It does? We removed that a long time ago!

[OneCD]

Ah, that would explain why I've been unable to locate the original online.

No worries, I'll ensure I use a current one instead.

[OneCD]

Can someone please confirm: is this the current icon?

[sander]

may the source be with you ... ;-)

https://github.com/sabnzbd/sabnzbd/blob ... -small.svg

[OneCD]

Awesome!

Thanks again @sander.

[OneCD]

I've been testing the v4 alpha 2 release.

Looks fine, except I still see the same issue with the 'sabctools' PyPI module I get with the 'sabyenc3' module: i.e. it needs to be force-reinstalled before SABnzbd will see it.

viewtopic.php?p=128567#p128567

[Puzzled]

Sounds like there's something peculiar with your setup. If it happens again then I suggest locating the files before and after force install to see if it's the same files in the same path.

[OneCD]

Sounds like there's something peculiar with your setup.

Ain't that the truth.

It's a QNAP NAS running QNAP's heavily-modified Ubuntu (known as QTS) as an embedded OS.

If it happens again then I suggest locating the files before and after force install to see if it's the same files in the same path.

It happens reliably everytime. Hence the workaround to reinstall.

Good idea, though. I'll see if I can provide any useful info about the files to help.

Personally, I suspect this is a Python configuration issue. It's like the sabyenc3/sabctools module isn't correctly "registered" when installed via 'pip' (it is shown as an installed module when checked via 'pip', yet Python can't import it). But, I don't know enough about Python to be able to check why.

Cheers!

[sander]

Maybe this helps in finding what is going on?

Code: Select all

$ python3
Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for path in sys.path: print(path)
... 

/usr/lib/python310.zip
/usr/lib/python3.10
/usr/lib/python3.10/lib-dynload
/home/sander/.local/lib/python3.10/site-packages
/home/sander/.local/lib/python3.10/site-packages/UNKNOWN-0.0.0-py3.10.egg
/home/sander/.local/lib/python3.10/site-packages/sabctools-6.1.0-py3.10-linux-x86_64.egg
/usr/local/lib/python3.10/dist-packages
/usr/local/lib/python3.10/dist-packages/sabyenc3-5.4.4-py3.10-linux-x86_64.egg
/usr/local/lib/python3.10/dist-packages/sabctools-6.1.1-py3.10-linux-x86_64.egg
/usr/lib/python3/dist-packages
>>> 
>>> import sabctools
>>> sabctools.__file__
'/home/sander/.local/lib/python3.10/site-packages/sabctools-6.1.0-py3.10-linux-x86_64.egg/sabctools.cpython-310-x86_64-linux-gnu.so'
>>>

Oh, that is ugly: a 6.1.0 and a 6.1.1. Brrr. Time to clean up.

Ah, I now see I still have an old sabyenc3 installed ... time to remove.

[OneCD]

Code: Select all

>>> import sys
>>> for path in sys.path: print(path)

Cheers mate!

I've done some checking after installing 'sabctools' the usual way:

Code: Select all

pip install sabctools

... and after reinstalling it the workaround way:

Code: Select all

pip install --force-reinstall --no-binary :all: sabctools

... and here what I have. This a fresh setup (almost) from scratch, and is created within a virtualenv:

Code: Select all

[/share/Public] # export PIP_CACHE_DIR=/share/Public/pip-cache VIRTUALENV_OVERRIDE_APP_DATA=/share/Public/pip-cache

Code: Select all

[/share/Public] # python3 -m virtualenv venv
created virtual environment CPython3.10.7.final.0-64 in 2527ms
  creator CPython3Posix(dest=/share/CACHEDEV2_DATA/Public/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/share/CACHEDEV2_DATA/Public/pip-cache)
    added seed packages: pip==23.0, setuptools==67.1.0, wheel==0.38.4
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

Code: Select all

[/share/Public] # . venv/bin/activate

Code: Select all

(venv) [/share/Public] # pip list
Package    Version
---------- -------
pip        23.0
setuptools 67.1.0
wheel      0.38.4

[notice] A new release of pip is available: 23.0 -> 23.0.1
[notice] To update, run: pip install --upgrade pip

Code: Select all

(venv) [/share/Public] # pip install sabctools
Collecting sabctools
  Downloading sabctools-6.1.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (174 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 174.2/174.2 kB 1.5 MB/s eta 0:00:00
Installing collected packages: sabctools
Successfully installed sabctools-6.1.1

[notice] A new release of pip is available: 23.0 -> 23.0.1
[notice] To update, run: pip install --upgrade pip

Code: Select all

(venv) [/share/Public] # python
Python 3.10.7 (main, Dec  4 2022, 09:29:05) [GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sabctools
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/share/CACHEDEV2_DATA/Public/venv/lib/python3.10/site-packages/sabctools/__init__.py", line 2, in <module>
    from sabctools.sabctools import *
ModuleNotFoundError: No module named 'sabctools.sabctools'

Code: Select all

>>> import sys
>>> for path in sys.path: print(path);
... 

/opt/lib/python310.zip
/opt/lib/python3.10
/opt/lib/python3.10/lib-dynload
/share/CACHEDEV2_DATA/Public/venv/lib/python3.10/site-packages
>>> quit()

Code: Select all

(venv) [/share/Public] # pip install --force-reinstall --no-binary :all: sabctools
DEPRECATION: --no-binary currently disables reading from the cache of locally built wheels. In the future --no-binary will not influence the wheel cache. pip 23.1 will enforce this behaviour change. A possible replacement is to use the --no-cache-dir option. You can use the flag --use-feature=no-binary-enable-wheel-cache to test the upcoming behaviour. Discussion can be found at https://github.com/pypa/pip/issues/11453
Collecting sabctools
  Downloading sabctools-6.1.1.tar.gz (228 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 228.7/228.7 kB 1.8 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: sabctools
  Building wheel for sabctools (pyproject.toml) ... done
  Created wheel for sabctools: filename=sabctools-6.1.1-cp310-cp310-linux_aarch64.whl size=35493 sha256=d9d6be1d4c906b348e16486c9b995ecb647603e658eb1a5fcc94ab8d4ea2ab25
  Stored in directory: /share/CACHEDEV2_DATA/Public/pip-cache/wheels/af/7e/24/6987efea469e00564a548e783b96a246c97b391c7837ce7fe8
Successfully built sabctools
Installing collected packages: sabctools
  Attempting uninstall: sabctools
    Found existing installation: sabctools 6.1.1
    Uninstalling sabctools-6.1.1:
      Successfully uninstalled sabctools-6.1.1
Successfully installed sabctools-6.1.1

[notice] A new release of pip is available: 23.0 -> 23.0.1
[notice] To update, run: pip install --upgrade pip

Code: Select all

(venv) [/share/Public] # python
Python 3.10.7 (main, Dec  4 2022, 09:29:05) [GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sabctools
>>> import sys
>>> for path in sys.path: print(path);
... 

/opt/lib/python310.zip
/opt/lib/python3.10
/opt/lib/python3.10/lib-dynload
/share/CACHEDEV2_DATA/Public/venv/lib/python3.10/site-packages
>>> 

... and as you can see, importing 'sabctools' now works fine, but the path list hasn't changed.

Also, if it helps, last time we were checking this, we narrowed it down to the 'manylinux' compiler. I can see 'sabctools' is originally installed with a filename ref to 'manylinux2014'.

This morning, I've been working on creating another QPKG for pyLoad, and encountered the same issue with one of its PyPI modules (brotli). It also refers to 'manylinux2014', and must also be force-reinstalled before it is recognised by pyLoad.

[OneCD]

I've now established 2 separate environments:

• #1 = 'sabctools' not recognised (but was installed the right way)
• #2 = 'sabctools' works fine (because it's been force-reinstalled)

After comparing the site-packages directories for both environments, about the only thing I can see that's different is the shared object file for 'sabctools'. I'm assuming this is some sort of compiled binary used by Python.

On the original failed setup, this file is called "sabctools.cpython-310-aarch64-linux-gnu.so" and is 591kB in size.

Code: Select all

# file sabctools.cpython-310-aarch64-linux-gnu.so 
sabctools.cpython-310-aarch64-linux-gnu.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=9578059cde79019bce57d05e3d28a2d4b47b77e3, with debug_info, not stripped

On the working setup (where 'sabctools' has been force-reinstalled), this file is called "sabctools.cpython-310.so" and is 70kB in size.

Code: Select all

# file sabctools.cpython-310.so 
sabctools.cpython-310.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, not stripped

... then I compared the __pycache__ directories.

The working one has a compiled bytecode filename " __init__.cpython-310.pyc"

... and so does the failed one.

Please excuse my Python noobness, but should the .pyc file in the failed one be generated as "__init__.cpython-310-aarch64-linux-gnu.pyc" ?

[safihre]

The pycache is controlled by Python, not the extension.
Only problem I can imagine is that the filename isn't right for the importer. Strange that pip installs this version..