Sabnzbdplus segfaults on Ubuntu 9.10 x64

[sabnzbd_ivo]

Hi all,

My Sabnzbdplus seems to segfault when downloading larger files (DVD size). It happens while downloading a file after couple GB's in the file.
From my /var/log/syslog:
sabnzbdplus[1369]: segfault at 2 ip 0000000000000002 sp 00007fec0cdb6ef8 error 14 in python2.5[400000+11f000]
And a snippet from sabnzbd.error.log:

Code: Select all

*** glibc detected *** /usr/bin/python2.5: corrupted double-linked list: 0x00000000046dd030 ***
======= Backtrace: =========
/lib/libc.so.6[0x7fc53d337dd6]
/lib/libc.so.6[0x7fc53d3398a2]
/lib/libc.so.6(cfree+0x6c)[0x7fc53d33c74c]
/usr/bin/python2.5[0x444eaf]
/usr/bin/python2.5[0x45b1bd]
/usr/bin/python2.5[0x444e93]
/usr/lib/python2.5/lib-dynload/cPickle.so[0x7fc53bc160b3]
/usr/lib/python2.5/lib-dynload/cPickle.so[0x7fc53bc1a554]
/usr/bin/python2.5(PyEval_EvalFrameEx+0x54d3)[0x48ebd3]
/usr/bin/python2.5(PyEval_EvalCodeEx+0x811)[0x490131]
/usr/bin/python2.5[0x4d7efd]
/usr/bin/python2.5(PyObject_Call+0x13)[0x418843]
/usr/bin/python2.5(PyEval_EvalFrameEx+0x3b81)[0x48d281]
/usr/bin/python2.5(PyEval_EvalCodeEx+0x811)[0x490131]
/usr/bin/python2.5(PyEval_EvalFrameEx+0x5459)[0x48eb59]
/usr/bin/python2.5(PyEval_EvalCodeEx+0x811)[0x490131]
/usr/bin/python2.5[0x4d7efd]
/usr/bin/python2.5(PyObject_Call+0x13)[0x418843]

This error occurs about 90% of the times I attempt to download a larger file.

I thought that maybe I have a hardware memory problem, but memtest shows nothing odd.

Tried both the ubuntu package (0.4.12) and version 0.5.0, but the same segfault occurs

Any ideas?

Version: 0.4.12 Final and 0.5.0
OS: Ubuntu server 9.10 x64
Install-type:  linux repository

[jcfp]

sabnzbdplus[1369]: segfault at 2 ip 0000000000000002 sp 00007fec0cdb6ef8 error 14 in python2.5[400000+11f000]

*** glibc detected *** /usr/bin/python2.5: corrupted double-linked list: 0x00000000046dd030 ***

This crash happens in python. Stuff like "glibc [...] corrupted double-linked list" and the "error 14" (invalid pointer?) seems to suggest there's a problem with code written in c; sabnzbd is entirely python. Causes could be hardware failure (esp. memory; how long did you let memtest run?), or bugs in various programs including (but not limited to) python, python modules with extensions used by sab, or even glibc (unlikely though).

If you used 0.5 from my ppa that would have used python 2.6, while 0.4.12 from the standard karmic repositories uses python 2.5. I've been using 0.5.x on a 64 bit karmic install ever since that came out and never seen any crash like this. If you have excluded hardware failure as a cause, you could file a bug against python in Ubuntu's bug tracker (at launchpad.net), but be aware that hunting this type of bug can be quite alot of work.

[sabnzbd_ivo]

Causes could be hardware failure (esp. memory; how long did you let memtest run?), or bugs in various programs including (but not limited to) python, python modules with extensions used by sab, or even glibc (unlikely though).

I had remtest run for 2 passes. Would you suggest to run it longer? I.e. is there reasonable chance that errors only start to show after 2 passes?

If you used 0.5 from my ppa that would have used python 2.6, while 0.4.12 from the standard karmic repositories uses python 2.5. I've been using 0.5.x on a 64 bit karmic install ever since that came out and never seen any crash like this

I have spoken with other people who also had a similar configuration and did not encounter this error (or anything close to it). That indicates that it is not a python/sabnzbd/glibc failure...

[jcfp]

I had remtest run for 2 passes. Would you suggest to run it longer? I.e. is there reasonable chance that errors only start to show after 2 passes?

Just asking, some people stop it after a single test. Two full passes seems enough to me.

I have spoken with other people who also had a similar configuration and did not encounter this error (or anything close to it). That indicates that it is not a python/sabnzbd/glibc failure...

Your system isn't a laptop or so that easily overheats? Esp. since you say it happens after a couple of GB. Usenet downloading tends to grill processor, memory and disks at the same time. If not, I'm out of suggestions...

[sabnzbd_ivo]

I am running a standalone ubuntu server (AMD phenom II 64 bit, 4 GB RAM) and the CPU, disk and memory don't seem to busy.

Allright, I think i know what is causing the issue. I have an md raid configuration for data storage:
FYI, my /proc/mdstat:

Code: Select all

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sdb1[0] sdc1[1] sdd1[2]
      976558976 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

I configured sabnzbd to write the downloaded data directly to a directory on this mounted md raid device: I get the mentioned python errors.
Now, I changed the configuration to write the downloaded data to the /home directory of the sabnzbd process that runs it. This directory resides on a different disk not part of the raid assembly.
I downloaded 15 GB of data sofar without any segfaults now.This is more than I managed to download before it crashed. I think this has solved my problem.

Guessing the md0 disk is causing python to segfault in some way...
If it occurs again, I will let you know.

[scorpius]

This is very weird... I use Ubuntu 10.04LTS and have a RAID0 array (mdadm) as download and processing volume. When I download some biig stuff, the server dies on me. It freezes up, and I have to reboot it the hard way (even ctrl-alt-del doesn't work). I tried a lot of different things, but when I read your experience, I wanted to give it a shot and I changed my download dir to my os disk, which is a single SATA disk with an EXT4 filesystem. And it seems to work ok for a couple of hours now. While this morning, when I was still at the RAID0 array, it froze 2 times allready.

The worst thing about these freezes is that I don't get any error message anywhere: syslog, kernel log, dmesg, sabnzb log, nowhere a single indication of what's going wrong. I ruled out the hardware with various benchmarking and burn-in tests to put everything through it's paces: bonnie++ hammered my disks for 6hours straight, I had the cpu at 100% (AMD X4) for some time, memtest86, nothing borked, so it has to be something that Sabnzb (or Python for that matter) triggers.

I'm running some tests now, and when I think it's safe, I'll break down the array and make it 2 single volumes and see what happens. Sofar it seems I'm onto something.

[v3gard]

I can confirm this bug on my server as well. I have an mdadm setup (10TB+) with LUKS and LVM, where the LVM is spread across three logical partitions. One for /, one for /home and one I call /mnt/shared where I keep all the data not relevant for the OS. Until I stumbled across this forum post, I used to keep the SABnzbd cache, temporary dirs and download folder on /mnt/big - and I *often* encountered problems with SABnzbd segfaulting. However, once I moved the cache and temporary download dir to the /home partiton (where the home directory of the user running SABnzbd is located), the problems disappeared.

If any developers are interested in working on this bug, feel free to contact me, and I'll try to assist with any debug logs.

edit: I'm currently running SABnzbd-0.6.9 on Slackware64 13.1

[CheeseIsNotYeOnlyTea]

I run SBANZB+ on 64bit Ubuntu with mdadm array.

I actually run SBANZB+ on a virtual machine and all downloads (temp files etc) go into a VDI (virtual disk) that lives on the array, and the final files are kept directly on the array.

I've been running this system hard for over a week now, and for the last 3 days none stop, downloading anything from 400mb to 5GB per download without the kind of problem described above.