Issue with X-DNZB-ProperName encoding

[kouze]

Hi,

I'm using my own newnab service on my server and facing encoding issue using DirectNZB feature.

For example I got this nzb with these headers:


In sabnzbd I got a directory rule to store all categories movies* in a directory, let's say C:\Movies

With this example, sabnzbd will create a directory name using X-DNZB-ProperName to save the content of the .nzb into:

C:\Movies\Team hot wheels La légende (2014)

é instead of é

If a take now another example of query but here without X-DNZB-ProperName set:


Here, as the X-DNZB-ProperName does not exist, I think sabnzbd uses the field X-DNZB-Name and it stores correcly the movie into:
C:\Movies\Un million d'années avant JC avi 1

So I guess there is an issue somewhere but I was not able to solve it on my side. Is there any issue on sabnzbd on how it handles the encoding of X-DNZB-ProperName?

Thanks,
kouze

[sander]

An ASCII versus Unicode/UTF thing? AFAIK SAB 0.7.x is ASCII oriented, and only parts can handle non-ASCII characters.

I run SAB 'develop' (which will be 0.8.x), so if you give/PM an URL I can test for you.

[kouze]

Note sure exactly why I got only the issue when X-DNZB-ProperName is set, as if you look at the second test the directory is correctly created.

I can remove the special chars/ accented chars on my side as well, not a big deal, but wanted first to check with you guys

I'll send you links in PM, thanks !

[sander]

Result on my system with SABnzbd 0.8.x:

Code: Select all

drwxrwxr-x  2 sander sander     4096 jan  6 19:44 Team.Hot.Wheels.The.Origin.Of.Awesome.2014.FRENCH.BDRiP.x264-CONSTANT.mkv.1/
drwxrwxr-x  2 sander sander     4096 jan  6 19:52 Un million d'années avant JC.avi.1/

So first download without any accent. That's different than on your system isn't it? But still different than what you expected?
Second download with accent aiguu

Maybe this helps:

Code: Select all

$ grep -irn -e dnzb-propername -e DNZB-Name  *
sabnzbd/urlgrabber.py:131:                        elif item in ('x-dnzb-name',):
sabnzbd/urlgrabber.py:135:                        elif item == 'x-dnzb-propername':

Source says this:

Code: Select all

                        elif item in ('x-dnzb-name',):
                            filename = value
                            if not filename.endswith('.nzb'):
                                filename += '.nzb'
                        elif item == 'x-dnzb-propername':
                            nzo_info['propername'] = value

So this looks like this to me:
'x-dnzb-name' sets the value of the filename?
'x-dnzb-propername' sets an info field?

Does the above help?


Full source:

Code: Select all

                    for hdr in fn.headers:
                        try:
                            item = hdr.lower()
                            value = fn.headers[hdr]
                        except:
                            continue
                        if item in ('content-encoding',) and value == 'gzip':
                            gzipped = True
                        if item in ('category_id', 'x-dnzb-category'):
                            category = value
                        elif item in ('x-dnzb-moreinfo',):
                            nzo_info['more_info'] = value
                        elif item in ('x-dnzb-name',):
                            filename = value
                            if not filename.endswith('.nzb'):
                                filename += '.nzb'
                        elif item == 'x-dnzb-propername':
                            nzo_info['propername'] = value
                        elif item == 'x-dnzb-episodename':
                            nzo_info['episodename'] = value
                        elif item == 'x-dnzb-year':
                            nzo_info['year'] = value
                        elif item == 'x-dnzb-failure':
                            nzo_info['failure'] = value
                        elif item == 'x-dnzb-details':
                            nzo_info['details'] = value
                        elif item in ('content-length',):
                            length = misc.int_conv(value)
                        elif item == 'retry-after':
                            # For NZBFinder
                            wait = misc.int_conv(value)

[sander]

FWIW:

I'm testing with curl, and at first that gave lines with X-DNZB, but now all gets show no X-DNZB info at all. There is a line

< Retry-After: 84679

... am I doing too fast sequences?

[kouze]

FWIW:

I'm testing with curl, and at first that gave lines with X-DNZB, but now all gets show no X-DNZB info at all. There is a line

< Retry-After: 84679

... am I doing too fast sequences?

Hmmm don't know. I changed the useraccount linked to admin, and I did try multiple times the links I provided to you, seems good with curl for me:

Code: Select all

kouze@mybox:~/test$ curl -vs -o test.nzb 'http://URL'
* Hostname was NOT found in DNS cache
*   Trying XXX.XXX.XXX.XXX...
* Connected to blabla.blibli (XXX.XXX.XXX.XXX) port 80 (#0)
> GET /getnzb/XXXXXX HTTP/1.1
> User-Agent: curl/7.35.0
> Host: blablabla.blibli
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Powered-By: PHP/5.5.9-1ubuntu4.5
< Set-Cookie: PHPSESSID=XXX; path=/
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< Content-type: application/x-nzb
< Content-Length: 106588
< X-DNZB-Name: Team.Hot.Wheels.The.Origin.Of.Awesome.2014.FRENCH.BDRiP.x264-CONSTANT.mkv.1
< X-DNZB-Category: Movies > Foreign
< X-DNZB-Details: http://blablabla.blibli
< X-DNZB-NFO:
< X-DNZB-ProperName: Team hot wheels: La légende
< X-DNZB-MovieYear: 2014
< X-DNZB-MoreInfo: http://www.imdb.com/title/tt3823912
< Content-Disposition: attachment; filename="Team.Hot.Wheels.The.Origin.Of.Awesome.2014.FRENCH.BDRiP.x264-CONSTANT.mkv.1_.nzb"
< Date: Tue, 06 Jan 2015 19:29:56 GMT
* Server lighttpd/1.4.33 is not blacklisted
< Server: lighttpd/1.4.33
<
{ [data not shown]
* Connection #0 to host blablabla.blibli left intact

Well I have also activated the Movie sorting (didn't tell you before sorry) on my "movies" category (=films), like that:


And the category is setup like that:


maybe this is causing the movie folder to be renamed using the X-DNZB properties ?

With the code you provided:

elif item in ('x-dnzb-name',):
filename = value
if not filename.endswith('.nzb'):
filename += '.nzb'
elif item == 'x-dnzb-propername':
nzo_info['propername'] = value
elif item == 'x-dnzb-year':
nzo_info['year'] = value

I would think that nzo_info['propername'] is used somewhere else, probably linked to the movie sorting process (which would make sense then), where nzo_info['propername'] is linked to %title and nzo_info['year'] linked to %y. And probably there there is an encoding issue.

Thanks for your tests

[shypike]

The title/episode et. attributes are only used by the Sorting functions.
Not when you're doing a basic download.

[kouze]

Could the issue come from that:
title = latin1(nzo.nzo_info.get('propername') or nzo.meta.get('propername', (None,))[0])
?

[shypike]

The assumption is that the text is in UTF-8 Unicode.
0.7.x translates that into 8bit-ASCII notation.
As long as your text is within the LATIN-1 (or ISO/IEC 8859-1) range, that works OK.
0.8.x will support the full range of Unicode in UTF-8.

[kouze]

Thanks for your response shypike. I'll remove the accented chars / special chars for the moment on my side to avoid the issue and will revert once 0.8.x will be available.