Page 1 of 1
Issue with X-DNZB-ProperName encoding
Posted: January 6th, 2015, 7:40 am
by kouze
Hi,
I'm using my own newnab service on my server and facing encoding issue using DirectNZB feature.
For example I got this nzb with these headers:
In sabnzbd I got a directory rule to store all categories movies* in a directory, let's say C:\Movies
With this example, sabnzbd will create a directory name using X-DNZB-ProperName to save the content of the .nzb into:
C:\Movies\Team hot wheels La légende (2014)
é instead of
é 
If a take now another example of query but here without X-DNZB-ProperName set:
Here, as the X-DNZB-ProperName does not exist, I think sabnzbd uses the field X-DNZB-Name and it stores correcly the movie into:
C:\Movies\Un million d'ann
ées avant JC avi 1
So I guess there is an issue somewhere but I was not able to solve it on my side. Is there any issue on sabnzbd on how it handles the encoding of X-DNZB-ProperName?
Thanks,
kouze
Re: Issue with X-DNZB-ProperName encoding
Posted: January 6th, 2015, 9:28 am
by sander
An ASCII versus Unicode/UTF thing? AFAIK SAB 0.7.x is ASCII oriented, and only parts can handle non-ASCII characters.
I run SAB 'develop' (which will be 0.8.x), so if you give/PM an URL I can test for you.
Re: Issue with X-DNZB-ProperName encoding
Posted: January 6th, 2015, 10:18 am
by kouze
Note sure exactly why I got only the issue when X-DNZB-ProperName is set, as if you look at the second test the directory is correctly created.
I can remove the special chars/ accented chars on my side as well, not a big deal, but wanted first to check with you guys
I'll send you links in PM, thanks !
Re: Issue with X-DNZB-ProperName encoding
Posted: January 6th, 2015, 2:00 pm
by sander
Result on my system with SABnzbd 0.8.x:
Code: Select all
drwxrwxr-x 2 sander sander 4096 jan 6 19:44 Team.Hot.Wheels.The.Origin.Of.Awesome.2014.FRENCH.BDRiP.x264-CONSTANT.mkv.1/
drwxrwxr-x 2 sander sander 4096 jan 6 19:52 Un million d'années avant JC.avi.1/
So first download without any accent. That's different than on your system isn't it? But still different than what you expected?
Second download with accent aiguu
Maybe this helps:
Code: Select all
$ grep -irn -e dnzb-propername -e DNZB-Name *
sabnzbd/urlgrabber.py:131: elif item in ('x-dnzb-name',):
sabnzbd/urlgrabber.py:135: elif item == 'x-dnzb-propername':
Source says this:
Code: Select all
elif item in ('x-dnzb-name',):
filename = value
if not filename.endswith('.nzb'):
filename += '.nzb'
elif item == 'x-dnzb-propername':
nzo_info['propername'] = value
So this looks like this to me:
'x-dnzb-name' sets the value of the filename?
'x-dnzb-propername' sets an info field?
Does the above help?
Full source:
Code: Select all
for hdr in fn.headers:
try:
item = hdr.lower()
value = fn.headers[hdr]
except:
continue
if item in ('content-encoding',) and value == 'gzip':
gzipped = True
if item in ('category_id', 'x-dnzb-category'):
category = value
elif item in ('x-dnzb-moreinfo',):
nzo_info['more_info'] = value
elif item in ('x-dnzb-name',):
filename = value
if not filename.endswith('.nzb'):
filename += '.nzb'
elif item == 'x-dnzb-propername':
nzo_info['propername'] = value
elif item == 'x-dnzb-episodename':
nzo_info['episodename'] = value
elif item == 'x-dnzb-year':
nzo_info['year'] = value
elif item == 'x-dnzb-failure':
nzo_info['failure'] = value
elif item == 'x-dnzb-details':
nzo_info['details'] = value
elif item in ('content-length',):
length = misc.int_conv(value)
elif item == 'retry-after':
# For NZBFinder
wait = misc.int_conv(value)
Re: Issue with X-DNZB-ProperName encoding
Posted: January 6th, 2015, 2:11 pm
by sander
FWIW:
I'm testing with curl, and at first that gave lines with X-DNZB, but now all gets show no X-DNZB info at all. There is a line
< Retry-After: 84679
... am I doing too fast sequences?
Re: Issue with X-DNZB-ProperName encoding
Posted: January 6th, 2015, 2:43 pm
by kouze
sander wrote:FWIW:
I'm testing with curl, and at first that gave lines with X-DNZB, but now all gets show no X-DNZB info at all. There is a line
< Retry-After: 84679
... am I doing too fast sequences?
Hmmm don't know. I changed the useraccount linked to admin, and I did try multiple times the links I provided to you, seems good with curl for me:
Code: Select all
kouze@mybox:~/test$ curl -vs -o test.nzb 'http://URL'
* Hostname was NOT found in DNS cache
* Trying XXX.XXX.XXX.XXX...
* Connected to blabla.blibli (XXX.XXX.XXX.XXX) port 80 (#0)
> GET /getnzb/XXXXXX HTTP/1.1
> User-Agent: curl/7.35.0
> Host: blablabla.blibli
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Powered-By: PHP/5.5.9-1ubuntu4.5
< Set-Cookie: PHPSESSID=XXX; path=/
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< Content-type: application/x-nzb
< Content-Length: 106588
< X-DNZB-Name: Team.Hot.Wheels.The.Origin.Of.Awesome.2014.FRENCH.BDRiP.x264-CONSTANT.mkv.1
< X-DNZB-Category: Movies > Foreign
< X-DNZB-Details: http://blablabla.blibli
< X-DNZB-NFO:
< X-DNZB-ProperName: Team hot wheels: La légende
< X-DNZB-MovieYear: 2014
< X-DNZB-MoreInfo: http://www.imdb.com/title/tt3823912
< Content-Disposition: attachment; filename="Team.Hot.Wheels.The.Origin.Of.Awesome.2014.FRENCH.BDRiP.x264-CONSTANT.mkv.1_.nzb"
< Date: Tue, 06 Jan 2015 19:29:56 GMT
* Server lighttpd/1.4.33 is not blacklisted
< Server: lighttpd/1.4.33
<
{ [data not shown]
* Connection #0 to host blablabla.blibli left intact
Well I have also activated the Movie sorting (didn't tell you before sorry) on my "movies" category (=films), like that:
And the category is setup like that:
maybe this is causing the movie folder to be renamed using the X-DNZB properties ?
With the code you provided:
elif item in ('x-dnzb-name',):
filename = value
if not filename.endswith('.nzb'):
filename += '.nzb'
elif item == 'x-dnzb-propername':
nzo_info['propername'] = value
elif item == 'x-dnzb-year':
nzo_info['year'] = value
I would think that nzo_info['propername'] is used somewhere else, probably linked to the movie sorting process (which would make sense then), where nzo_info['propername'] is linked to %title and nzo_info['year'] linked to %y. And probably there there is an encoding issue.
Thanks for your tests
Re: Issue with X-DNZB-ProperName encoding
Posted: January 6th, 2015, 3:07 pm
by shypike
The title/episode et. attributes are only used by the Sorting functions.
Not when you're doing a basic download.
Re: Issue with X-DNZB-ProperName encoding
Posted: January 6th, 2015, 5:27 pm
by kouze
Could the issue come from that:
title = latin1(nzo.nzo_info.get('propername') or nzo.meta.get('propername', (None,))[0])
?
Re: Issue with X-DNZB-ProperName encoding
Posted: January 7th, 2015, 6:53 am
by shypike
The assumption is that the text is in UTF-8 Unicode.
0.7.x translates that into 8bit-ASCII notation.
As long as your text is within the LATIN-1 (or ISO/IEC 8859-1) range, that works OK.
0.8.x will support the full range of Unicode in UTF-8.
Re: Issue with X-DNZB-ProperName encoding
Posted: January 7th, 2015, 7:11 am
by kouze
Thanks for your response shypike. I'll remove the accented chars / special chars for the moment on my side to avoid the issue and will revert once 0.8.x will be available.