How to use Deobfuscate feature in 3.1.0Beta1 ?

[sj50]

I have installed 3.1.0Beta1 [f0d31e0] and enabled "Deobfuscate final filenames" but I still see obfuscate file name.
Is there anything else I have to do to make it work?

[sander]

Can you give examples of obfuscated file names that are still there?

[sj50]

this is what I got from one of the nzb after unpack.
/sabnzbd/completed/Movies_UHD/Sunshine (dot) 2007 (dot) PROPER (dot) BluRay (dot) 1080p (dot) DTS-HD (dot) MA (dot) 5 (dot) 1 (dot) AVC (dot) REMUX-FraMeSToR /9y5gNEsQaxB6H (dot) mkv

edit - replace " (dot) " with "." as I am not allowed to post link

[sander]

So: 9y5gNEsQaxB6H.mkv is the file name.

You and I clearly see that that is obfuscated ... but how? Because there are no spaces and dots? What would you say about that? (see https://github.com/sabnzbd/sabnzbd/blob ... y#L99-L130 for the current criteria)

But let's what the function says:

Reference:

Code: Select all

sander@witte2004:~/git/sabnzbd$ python3 -c "from sabnzbd.deobfuscate_filenames import *; print(is_probably_obfuscated('599c1c9e2bdfb5114044bf25152b7eaa.mkv')) " 
True

Your filename:

Code: Select all

sander@witte2004:~/git/sabnzbd$ python3 -c "from sabnzbd.deobfuscate_filenames import *; print(is_probably_obfuscated('9y5gNEsQaxB6H.mkv')) " 
False

Let's turn on logging:

Code: Select all

sander@witte2004:~/git/sabnzbd$ python3 -c "import logging, sys; logging.basicConfig(stream=sys.stdout, level=logging.DEBUG); from sabnzbd.deobfuscate_filenames import * ; print(is_probably_obfuscated('9y5gNEsQaxB6H.mkv'))" 
DEBUG:root:Not obfuscated (default)
False

"Not obfuscated (default)" ... so not any trigger kicks in to say it's obfuscated

I find this one hard to detect. Capitals and small letters and numbers are a good sign: "My Movie 2020.mkv" and "MyMovie2020,mkv" are not obfuscated ... so how to detect your filename 9y5gNEsQaxB6H.mkv is obfuscated?

[sander]

Code: Select all

>>> filebasename = "9y5gNEsQaxB6H"

>>> sum(1 for c in filebasename if c.isupper())
5
>>> sum(1 for c in filebasename if c.islower())
5
>>> sum(1 for c in filebasename if c.isnumeric())
3
>>> sum(1 for c in filebasename if c == " " or c == ".")
0
>>> len(filebasename)
13

So ... no spaces/dots, and number of capitals is the same as number of small letters ... that is a sign of obfuscation ... ?

Or number of capitals and number of smaller letters and number of numerics are all about the same ... ?

[sj50]

"If there is no "rename.par2" available, it will rename large, not-excluded files to the job-name in the queue if the filename looks obfuscated"
There is no rename.par2 it should rename it job name, But here script does not consider file name as obfuscated right?
May be add a comparing file name length with job name length as sign of obfuscation?
I have noticed text file which contains some random string, is this string key to deobfuscation ?

[sander]

Hi sj50,

Solved in the upcoming 3.1.0 Beta2 ... thanks to your report!!

[Epedemic]

I have a different problem with the deobfuscate function, which was not a problem using the old deobfuscate py :

When downloading a blu-ray, it will take some, but not all, of the numbered 000** *dot* m2ts files from the bdmv stream folder and move them to the root "job name" folder and rename them "job name *dot* 1 *dot* m2ts", 2 , 3 etc seemingly randomly numbered, so they are very difficult to restore unless you compare original names and sizes etc.... This is present in Beta2 as well.

Only the Rars are obfuscated, but the folders inside look good when i look at the source.

Sorry about the *dot* things, but as a new user i am not allowed to post links... and apparently the forum equates file names with dots to links. Hope the post is understandable anyway

[sander]

When downloading a blu-ray, it will take some, but not all, of the numbered 000** *dot* m2ts files from the bdmv stream folder and move them to the root "job name" folder and rename them "job name *dot* 1 *dot* m2ts", 2 , 3 etc seemingly randomly numbered, so they are very difficult to restore unless you compare original names and sizes etc.... This is present in Beta2 as well.

So ... those file have extension ".m2ts"? And one blu-ray download has multiple files with that extension?
It's easy for me to exclude them in the deobfuscate function. Would that be better, so not rename them at all?

If possible, can you share such a download with me (via PM and/or pastebin)?

[safihre]

Deobfuscate.py would only rename files if there was just 1 large file. If there were multiple, it wouldn't do anything. To indeed prevent this or prevent for example a job that has multiple episodes of a series inside to be rename all the same.

[Epedemic]

So ... those file have extension ".m2ts"? And one blu-ray download has multiple files with that extension?
It's easy for me to exclude them in the deobfuscate function. Would that be better, so not rename them at all?

If possible, can you share such a download with me (via PM and/or pastebin)?

PM sent... I hope... Still a bit challenged about the "new user" thing, so i hope it is correctly sent even if it is in "outbox" and not in "sent messages". Otherwise let me know, and we'll figure out something else

[sander]

Hi Epedemic: I received your PM, and replied to it ... did you receive it?

I'm curious about the multiple file .m2ts ...

[sander]

OK, I downloaded it (65 GB download) with sabnzbdplus-3.1.0Beta2:

In between result (near the end but not yet finished, so before renaming):

Code: Select all

-rwxrwxrwx 1 root root 58156695552 sep 11 19:45 00000.m2ts
-rwxrwxrwx 1 root root  1162223616 sep 11 19:42 00001.m2ts
-rwxrwxrwx 1 root root     2850816 sep 11 19:31 00002.m2ts
-rwxrwxrwx 1 root root   200871936 sep 11 19:36 00003.m2ts
-rwxrwxrwx 1 root root  2231027712 sep 11 19:32 00004.m2ts
-rwxrwxrwx 1 root root   355411968 sep 11 19:17 00005.m2ts
-rwxrwxrwx 1 root root  1393446912 sep 11 19:21 00006.m2ts
-rwxrwxrwx 1 root root   145752064 sep 18 11:18 00007.m2ts
-rwxrwxrwx 1 root root       36864 sep 11 18:18 00012.m2ts
-rwxrwxrwx 1 root root    89751552 sep 11 18:18 00013.m2ts
-rwxrwxrwx 1 root root    17418240 sep 11 18:18 00014.m2ts
-rwxrwxrwx 1 root root   114726912 sep 11 18:18 00015.m2ts
-rwxrwxrwx 1 root root     2912256 sep 11 18:18 00016.m2ts
-rwxrwxrwx 1 root root  1097736192 sep 11 19:23 00017.m2ts

Finally:

Code: Select all

-rwxrwxrwx 1 root root 58156695552 sep 11 19:45 The.Something.1981-(USA-UHD).m2ts*
-rwxrwxrwx 1 root root  1162223616 sep 11 19:42 The.Something.1981-(USA-UHD).1.m2ts*
-rwxrwxrwx 1 root root   200871936 sep 11 19:36 The.Something.1981-(USA-UHD).2.m2ts*
-rwxrwxrwx 1 root root  2231027712 sep 11 19:32 The.Something.1981-(USA-UHD).3.m2ts*
-rwxrwxrwx 1 root root   355411968 sep 11 19:17 The.Something.1981-(USA-UHD).4.m2ts*
-rwxrwxrwx 1 root root  1393446912 sep 11 19:21 The.Something.1981-(USA-UHD).5.m2ts*
-rwxrwxrwx 1 root root   396656640 sep 11 18:18 The.Something.1981-(USA-UHD).6.m2ts*
-rwxrwxrwx 1 root root    89751552 sep 11 18:18 The.Something.1981-(USA-UHD).7.m2ts*
-rwxrwxrwx 1 root root    17418240 sep 11 18:18 The.Something.1981-(USA-UHD).8.m2ts*
-rwxrwxrwx 1 root root   114726912 sep 11 18:18 The.Something.1981-(USA-UHD).9.m2ts*
-rwxrwxrwx 1 root root  1097736192 sep 11 19:23 The.Something.1981-(USA-UHD).10.m2ts*

I do understand that result: only big files (>10MB) are renamed, and the renaming is OK ... files still in order. Question @epedemic:
1) is that OK for you,
2) or should SAB just exclude .m2ts extensions,
3) or rename with the long number in place (so: The.Something.1981-(USA-UHD).00001.m2ts


BUT ... completely missing are the smaller files, like the 00002.m2ts! That is bad. I have to find out how that can happen.

[sander]

Based on your report, I found a bug in 3.1.0 Beta2 (causing the disappearing file), which was already solved in the git version, so in 3.1.0 RC1

And in the git version, I now excluded the .m2ts (and other BR) files: no more renaming. Also in the upcoming 3.1.0 RC1.

[Epedemic]

I do understand that result: only big files (>10MB) are renamed, and the renaming is OK ... files still in order. Question @epedemic:
1) is that OK for you,
2) or should SAB just exclude .m2ts extensions,
3) or rename with the long number in place (so: The.Something.1981-(USA-UHD).00001.m2ts


BUT ... completely missing are the smaller files, like the 00002.m2ts! That is bad. I have to find out how that can happen.

I did some testing with several downloads on RC1, and can confirm everything looks good now. Thanks!

However, that seems to be in conflict with what you wrote. So to sum it up: m2ts files in a blu-ray/UHD blu-ray file structure (in bdmv\stream should NEVER be renamed). I can think of situations though where a m2ts should be renamed (eg in a remux to a single m2ts instead of mkv, but that is another case, and not really anything i dabble in)