How to debug a Python segmentation fault?

How can I debug a Python segmentation fault?

We are trying to run our python code on SuSE 12.3. We get reproducible segmentation faults. The python code has been working on other platforms without segmentation faults, for years.

We only code Python, no C extension ....

What is the best way to debug this? I know a bit ansi c, but that was ten years ago ....

Python 2.7.5

Update

The segmentation fault happens on interpreter shutdown.

I can run the script several times:

python -m pdb myscript.py arg1 arg1
continue
run
continue
run

But the segmentation faults happen, if I leave the pdb with ctrl-d.

Update 2

I now try to debug it with gdb:

gdb 
> file python
> run myscript.py arg1 arg2
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffefbe2700 (LWP 15483)]
0x00007ffff7aef93c in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0
(gdb) bt
#0  0x00007ffff7aef93c in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0
#1  0x00007ffff7af5303 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.7.so.1.0
#2  0x00007ffff7adc858 in ?? () from /usr/lib64/libpython2.7.so.1.0
#3  0x00007ffff7ad840d in PyObject_Call () from /usr/lib64/libpython2.7.so.1.0
#4  0x00007ffff7af1082 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0
#5  0x00007ffff7af233d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0
#6  0x00007ffff7af233d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0
#7  0x00007ffff7af5303 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.7.so.1.0
#8  0x00007ffff7adc5b6 in ?? () from /usr/lib64/libpython2.7.so.1.0
#9  0x00007ffff7ad840d in PyObject_Call () from /usr/lib64/libpython2.7.so.1.0
#10 0x00007ffff7ad9171 in ?? () from /usr/lib64/libpython2.7.so.1.0
#11 0x00007ffff7ad840d in PyObject_Call () from /usr/lib64/libpython2.7.so.1.0
#12 0x00007ffff7aeeb62 in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.7.so.1.0
#13 0x00007ffff7acc757 in ?? () from /usr/lib64/libpython2.7.so.1.0
#14 0x00007ffff7828e0f in start_thread () from /lib64/libpthread.so.0
#15 0x00007ffff755c7dd in clone () from /lib64/libc.so.6

Update 3

I installed gdbinit from http://hg.python.org/cpython/file/default/Misc/gdbinit and the debugging symbols from http://download.opensuse.org/debug/distribution/12.3/repo/oss/suse/x86_64/

(gdb) pystack
No symbol "_PyUnicode_AsString" in current context.

What now?

Update 4 We installed the a new RPM (python-2.7.5-3.1.x86_64). We get less segfaults, but they still happen. Here is the link to repository:

http://download.opensuse.org/repositories/devel:/languages:/python:/Factory/openSUSE_12.3/x86_64/

Update 5 Solved my initial problem:

It was http://bugs.python.org/issue1856 (shutdown (exit) can hang or segfault with daemon threads running)

Related: Detect Interpreter shut down in daemon thread

I got to this question because of the Segmentation fault, but not on exit, just in general, and I found that nothing else helped as effectively as faulthandler. It's part of Python 3.3, and you can install in 2.7 using pip.

tl;dr for python3 users.

Firstly, from the docs:

faulthandler is a builtin module since Python 3.3

Code usage:

import faulthandler

faulthandler.enable()
// bad code goes here

Shell usage:

$ python3 -q -X faulthandler
>>> /// bad cod goes here

Maybe there is a daemon thread running? There is a reproduceable bug, which was fixed only for 3.x, but not for 2.x:

http://bugs.python.org/issue1856:

shutdown (exit) can hang or segfault with daemon threads running

This is the answer to my own question. It took some time to find the root of the problem.

Here is the next question: How to code around this bug: Detect Interpreter shut down in daemon thread