After clarification on the new duplicate detection method

Get help with all aspects of SABnzbd
Forum rules
Help us help you:
  • Are you using the latest stable version of SABnzbd? Downloads page.
  • Tell us what system you run SABnzbd on.
  • Adhere to the forum rules.
  • Do you experience problems during downloading?
    Check your connection in Status and Interface settings window.
    Use Test Server in Config > Servers.
    We will probably ask you to do a test using only basic settings.
  • Do you experience problems during repair or unpacking?
    Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
Post Reply
augur
Newbie
Newbie
Posts: 17
Joined: September 11th, 2011, 4:54 pm

After clarification on the new duplicate detection method

Post by augur »

I had a search but couln't find anything beyond the few words on the Introducing 1.0.0 page.

I have just updated from 0.7x to 1.0.3 in preperation to move the install to a new microserver running Win7 as the previous WinXP box is very overdue for retirement. I read all the 1.0.0 page and only got caught out for a few minutes by the movement of the sabnzbd.ini file from C:\Documents and Settings\user\Local Data\Application Data\sabnzbd\sabnzbd.ini to C:\Documents and Settings\user\sabnzbd\sabnzbd.ini

I'm planning to install and configure sabnzbd like a portable installation but running as a windows service, and I think I have that figured out.

I was dreading the movement of the several thousand archived nzbs but if I am understanding the wiki correctly that is no longer necessary.
I had a look into copies of the history1.db sqlite database before and after the upgrade and see the new column 'md5sum' which I believe must be the nzb md5 hash.

Now the crux, how does the new 1.0.0 method differ from the 0.7.x method?
In the 0.7.x versions I think duplicate detection just looked for matching filenames in the nzb backup folder, am I right?

In the 1.0.3 version does sabnzbd compare both name and md5 hash with both those columns in the history1.db ?

For those older nzbs which do not have a md5sum value does it just use nzb filename in the history1.db or does it also use the contents of the nzb backup folder?

Is there any point at all to migrating the contents of the nzb backup folder over to the new server?

Is it ok to delete the nzb backup location from the sabnzbd.ini via the config folders screen?


And lastly is there any way to take those thousands of nzb backup files and add their md5sum values into the sqlite database? I reckon the answer is no or not easily but I thought I should at least ask.
User avatar
shypike
Administrator
Administrator
Posts: 19774
Joined: January 18th, 2008, 12:49 pm

Re: After clarification on the new duplicate detection metho

Post by shypike »

First of all, the INI file locations are a bit different than what you describe:
https://sabnzbd.org/wiki/configuration/1.0/configure

The duplicate detection has two modes.
Comparison against archived NZB files still works, it stops if you have no archive folder.
The second method is comparison of the checksum of the NZB file against those in History.
However, the latter only works for History entries created after you upgraded to 1.0.x

So if you want the duplicate check to work on all your previous downloads,
do copy the archived NZB files.
augur
Newbie
Newbie
Posts: 17
Joined: September 11th, 2011, 4:54 pm

Re: After clarification on the new duplicate detection metho

Post by augur »

Thanks for the clarification shypike.
It's an XP box and I can't explain why the ini locations don't match those in the wiki, one of the several reasons the install is being moved to new hardware and OS(x64).

As comparison against archived nzb files appears to be by name only (I've occasionally had to delete an nzb then shorten the filename and readd it due to windows path length limits and this doesn't trigger a duplicate warning). I was hoping that the new duplication method would use the checksum and if that isn't present fall back to looking for a match on nzb filename in the history1.db, that way the archived nzb folder would no longer be needed at all. Even though that would require two searches instead of one I would prefer it if it allowed me to dump the nzb archive folder.

I think i'll just copy a subset of my archived nzbs, say the last years worth. Then in a year when the nzb hashes have been populating the db for a year remove the backup nzb folder entirely.
Post Reply