1. Introduction
Detecting segfaults
Devel::Trace
gdb
Devel::bt
The End
Hunting segfaults
for beginners
Uwe V¨lker
o
XING AG
August 2012
Uwe V¨lker
o Hunting segfaults

2. Introduction
Detecting segfaults
Devel::Trace
gdb
Devel::bt
The End
1 Introduction
2 Detecting segfaults
3 Devel::Trace
4 gdb
5 Devel::bt
6 The End
Uwe V¨lker
o Hunting segfaults

3. Introduction
Detecting segfaults
What is a segfault?
Devel::Trace
Examples - C
gdb
Examples - Perl
Devel::bt
The End
1 Introduction
What is a segfault?
Examples - C
Examples - Perl
2 Detecting segfaults
3 Devel::Trace
4 gdb
5 Devel::bt
6 The End
Uwe V¨lker
o Hunting segfaults

4. Introduction
Detecting segfaults
What is a segfault?
Devel::Trace
Examples - C
gdb
Examples - Perl
Devel::bt
The End
What is a segfault?
segfault = segmentation fault
Uwe V¨lker
o Hunting segfaults

5. Introduction
Detecting segfaults
What is a segfault?
Devel::Trace
Examples - C
gdb
Examples - Perl
Devel::bt
The End
What is a segfault?
segfault = segmentation fault
every process has memory pages
these pages are mapped to physical memory
Uwe V¨lker
o Hunting segfaults

6. Introduction
Detecting segfaults
What is a segfault?
Devel::Trace
Examples - C
gdb
Examples - Perl
Devel::bt
The End
What is a segfault?
segfault = segmentation fault
every process has memory pages
these pages are mapped to physical memory
if you try to access an invalid address
(or write to a protected address)
Uwe V¨lker
o Hunting segfaults

7. Introduction
Detecting segfaults
What is a segfault?
Devel::Trace
Examples - C
gdb
Examples - Perl
Devel::bt
The End
What is a segfault?
segfault = segmentation fault
every process has memory pages
these pages are mapped to physical memory
if you try to access an invalid address
(or write to a protected address)
BOOOM!
Uwe V¨lker
o Hunting segfaults

8. Introduction
Detecting segfaults
What is a segfault?
Devel::Trace
Examples - C
gdb
Examples - Perl
Devel::bt
The End
Examples - C
using uninitialized pointers
dereferencing NULL pointers
using ”freed” pointers
Uwe V¨lker
o Hunting segfaults

9. Introduction
Detecting segfaults
What is a segfault?
Devel::Trace
Examples - C
gdb
Examples - Perl
Devel::bt
The End
Examples - Perl
bug in a XS extension
bug in Perl itself (rare)
Uwe V¨lker
o Hunting segfaults

10. Introduction
Detecting segfaults
What is a segfault?
Devel::Trace
Examples - C
gdb
Examples - Perl
Devel::bt
The End
Examples - Perl
bug in a XS extension
bug in Perl itself (rare)
Perl 5.6.1:
perl -e ’undef a’
perl -e ’*::=%::=0’
Uwe V¨lker
o Hunting segfaults

11. Introduction
Detecting segfaults
What is a segfault?
Devel::Trace
Examples - C
gdb
Examples - Perl
Devel::bt
The End
Examples - Perl
bug in a XS extension
bug in Perl itself (rare)
Perl 5.6.1:
perl -e ’undef a’
perl -e ’*::=%::=0’
Perlmonks thread: (Golf) Segfault Perl
http://perlmonks.org/?node_id=156461
Uwe V¨lker
o Hunting segfaults

12. Introduction
Detecting segfaults
On the shell
Devel::Trace
Core dump file
gdb
CGI script
Devel::bt
The End
1 Introduction
2 Detecting segfaults
On the shell
Core dump file
CGI script
3 Devel::Trace
4 gdb
5 Devel::bt
6 The End
Uwe V¨lker
o Hunting segfaults

13. Introduction
Detecting segfaults
On the shell
Devel::Trace
Core dump file
gdb
CGI script
Devel::bt
The End
On the shell
perl segfault . pl
S e g m e n t a t i o n f a u l t ( c o r e dumped )
Uwe V¨lker
o Hunting segfaults

14. Introduction
Detecting segfaults
On the shell
Devel::Trace
Core dump file
gdb
CGI script
Devel::bt
The End
On the shell
perl segfault . pl
S e g m e n t a t i o n f a u l t ( c o r e dumped )
#! / u s r / b i n / p e r l
use Debug : : DumpCore ;
Debug : : DumpCore : : s e g v ;
Uwe V¨lker
o Hunting segfaults

15. Introduction
Detecting segfaults
On the shell
Devel::Trace
Core dump file
gdb
CGI script
Devel::bt
The End
Core dump file
$ u l i m i t −c u n l i m i t e d
$ perl segfault . pl
S e g m e n t a t i o n f a u l t ( c o r e dumped )
$ l l core
−rw−r−−−−− 1 uwe uwe 1695744 J u l 26 1 4 : 0 8 c o r e
Uwe V¨lker
o Hunting segfaults

16. Introduction
Detecting segfaults
On the shell
Devel::Trace
Core dump file
gdb
CGI script
Devel::bt
The End
CGI script
personal story: CGI script in Apache
no output, no entry in logfiles (access.log and error.log)
Uwe V¨lker
o Hunting segfaults

17. Introduction
Detecting segfaults
On the shell
Devel::Trace
Core dump file
gdb
CGI script
Devel::bt
The End
CGI script
personal story: CGI script in Apache
no output, no entry in logfiles (access.log and error.log)
but when I wrote to some file, the content was there
so the script was getting executed...
Uwe V¨lker
o Hunting segfaults

18. Introduction
Detecting segfaults
Usage
Devel::Trace
How do I spot a segfault?
gdb
Other uses for Devel::Trace
Devel::bt
The End
1 Introduction
2 Detecting segfaults
3 Devel::Trace
Usage
How do I spot a segfault?
Other uses for Devel::Trace
4 gdb
5 Devel::bt
6 The End
Uwe V¨lker
o Hunting segfaults

19. Introduction
Detecting segfaults
Usage
Devel::Trace
How do I spot a segfault?
gdb
Other uses for Devel::Trace
Devel::bt
The End
Usage
”Print out each line before it is executed (like sh -x)”
Uwe V¨lker
o Hunting segfaults

20. Introduction
Detecting segfaults
Usage
Devel::Trace
How do I spot a segfault?
gdb
Other uses for Devel::Trace
Devel::bt
The End
Usage
”Print out each line before it is executed (like sh -x)”
perl -d:Trace program
for CGI: put it in your shebang line
Uwe V¨lker
o Hunting segfaults

21. Introduction
Detecting segfaults
Usage
Devel::Trace
How do I spot a segfault?
gdb
Other uses for Devel::Trace
Devel::bt
The End
Usage
”Print out each line before it is executed (like sh -x)”
perl -d:Trace program
for CGI: put it in your shebang line
>> ./ test :4: p r i n t ” Statement 1 a t l i n e 4 n” ;
>> ./ test :5: p r i n t ” Statement 2 a t l i n e 5 n” ;
>> ./ test :6: p r i n t ” C a l l t o sub x r e t u r n s ” , &x ( ) ,
>> ./ test :12: p r i n t ” I n sub x a t l i n e 1 2 . n” ;
>> ./ test :13: return 13;
>> ./ test :8: exit 0;
Uwe V¨lker
o Hunting segfaults

22. Introduction
Detecting segfaults
Usage
Devel::Trace
How do I spot a segfault?
gdb
Other uses for Devel::Trace
Devel::bt
The End
How do I spot a segfault?
look at the last few lines
if it stops immediately, it might be a segfault
Uwe V¨lker
o Hunting segfaults

23. Introduction
Detecting segfaults
Usage
Devel::Trace
How do I spot a segfault?
gdb
Other uses for Devel::Trace
Devel::bt
The End
How do I spot a segfault?
look at the last few lines
if it stops immediately, it might be a segfault
grep for your script name
output can be very large, with long lines
grep -v site perl
Uwe V¨lker
o Hunting segfaults

24. Introduction
Detecting segfaults
Usage
Devel::Trace
How do I spot a segfault?
gdb
Other uses for Devel::Trace
Devel::bt
The End
How do I spot a segfault?
look at the last few lines
if it stops immediately, it might be a segfault
grep for your script name
output can be very large, with long lines
grep -v site perl
in my case: buggy MSSQL driver (easysoft)
Uwe V¨lker
o Hunting segfaults

25. Introduction
Detecting segfaults
Usage
Devel::Trace
How do I spot a segfault?
gdb
Other uses for Devel::Trace
Devel::bt
The End
Other uses for Devel::Trace
your program is behaving strange and you have no debugger
at hand
(use grep and grep -v to filter the output)
Uwe V¨lker
o Hunting segfaults

26. Introduction
Detecting segfaults
Usage
Devel::Trace
How do I spot a segfault?
gdb
Other uses for Devel::Trace
Devel::bt
The End
Other uses for Devel::Trace
your program is behaving strange and you have no debugger
at hand
(use grep and grep -v to filter the output)
Does this code get executed?
Which part of the conditional was taken?
Uwe V¨lker
o Hunting segfaults

27. Introduction
Detecting segfaults
Introduction
Devel::Trace
Usage
gdb
Core dump file - reloaded
Devel::bt
The End
1 Introduction
2 Detecting segfaults
3 Devel::Trace
4 gdb
Introduction
Usage
Core dump file - reloaded
5 Devel::bt
6 The End
Uwe V¨lker
o Hunting segfaults

28. Introduction
Detecting segfaults
Introduction
Devel::Trace
Usage
gdb
Core dump file - reloaded
Devel::bt
The End
Introduction
GNU debugger
command line debugger
we use it to extract the stacktrace from the core dump file
Uwe V¨lker
o Hunting segfaults

29. Introduction
Detecting segfaults
Introduction
Devel::Trace
Usage
gdb
Core dump file - reloaded
Devel::bt
The End
Usage
$ gdb p e r l c o r e
Core was g e n e r a t e d by ‘ p e r l p e r l / s e g f a u l t . p l ’ .
Program t e r m i n a t e d w i t h s i g n a l 1 1 , S e g m e n t a t i o n f a u
#0 0 x 0 0 0 0 7 f 2 f 5 d 0 8 6 7 5 4 i n c r a s h n o w f o r r e a l ( s u i c i d
10 p r i n t f (”%d ” , ∗p ) ; /∗ c a u s e a s e g f a u l t ∗/
( gdb )
Uwe V¨lker
o Hunting segfaults

30. Introduction
Detecting segfaults
Introduction
Devel::Trace
Usage
gdb
Core dump file - reloaded
Devel::bt
The End
Usage
$ gdb p e r l c o r e
Core was g e n e r a t e d by ‘ p e r l p e r l / s e g f a u l t . p l ’ .
Program t e r m i n a t e d w i t h s i g n a l 1 1 , S e g m e n t a t i o n f a u
#0 0 x 0 0 0 0 7 f 2 f 5 d 0 8 6 7 5 4 i n c r a s h n o w f o r r e a l ( s u i c i d
10 p r i n t f (”%d ” , ∗p ) ; /∗ c a u s e a s e g f a u l t ∗/
( gdb ) where
#0 0 x 0 0 0 0 7 f 2 f 5 d 0 8 6 7 5 4 i n c r a s h n o w f o r r e a l ( s u i c i d
#1 0 x 0 0 0 0 7 f 2 f 5 d 0 8 6 7 8 9 i n c r a s h n o w ( s u i c i d e m e s s a g e
#2 0 x 0 0 0 0 7 f 2 f 5 d 0 8 6 8 2 0 i n XS Debug DumpCore segv (
#3 0 x0000000000488db3 i n P e r l p p e n t e r s u b ( )
#4 0 x0000000000480a7d i n P e r l r u n o p s s t a n d a r d ( )
#5 0 x00000000004336b4 i n p e r l r u n ( )
#6 0 x000000000041bddc i n main ( )
( gdb )
Uwe V¨lker
o Hunting segfaults

31. Introduction
Detecting segfaults
Introduction
Devel::Trace
Usage
gdb
Core dump file - reloaded
Devel::bt
The End
Core dump file - reloaded
ulimit -c unlimited
Uwe V¨lker
o Hunting segfaults

32. Introduction
Detecting segfaults
Introduction
Devel::Trace
Usage
gdb
Core dump file - reloaded
Devel::bt
The End
Core dump file - reloaded
ulimit -c unlimited
current directory has to be writable
(can be tricky with Apache)
Uwe V¨lker
o Hunting segfaults

33. Introduction
Detecting segfaults
Introduction
Devel::Trace
Usage
gdb
Core dump file - reloaded
Devel::bt
The End
Core dump file - reloaded
ulimit -c unlimited
current directory has to be writable
(can be tricky with Apache)
ps auxww|grep apache
ls -l /proc/1234/cwd
Uwe V¨lker
o Hunting segfaults

34. Introduction
Detecting segfaults
Devel::Trace Usage
gdb How does it work?
Devel::bt
The End
1 Introduction
2 Detecting segfaults
3 Devel::Trace
4 gdb
5 Devel::bt
Usage
How does it work?
6 The End
Uwe V¨lker
o Hunting segfaults

35. Introduction
Detecting segfaults
Devel::Trace Usage
gdb How does it work?
Devel::bt
The End
Usage
”Automatic gdb backtraces on errors”
Uwe V¨lker
o Hunting segfaults

36. Introduction
Detecting segfaults
Devel::Trace Usage
gdb How does it work?
Devel::bt
The End
Usage
”Automatic gdb backtraces on errors”
just use the module
it registers signal handlers for SIGSEGV (and a few more)
Uwe V¨lker
o Hunting segfaults

37. Introduction
Detecting segfaults
Devel::Trace Usage
gdb How does it work?
Devel::bt
The End
How does it work?
the signal handler forks off a process which runs gdb
gdb attaches to the parent and outputs the stacktrace
Uwe V¨lker
o Hunting segfaults

38. Introduction
Detecting segfaults
Devel::Trace Sources
gdb Questions?
Devel::bt
The End
1 Introduction
2 Detecting segfaults
3 Devel::Trace
4 gdb
5 Devel::bt
6 The End
Sources
Questions?
Uwe V¨lker
o Hunting segfaults

39. Introduction
Detecting segfaults
Devel::Trace Sources
gdb Questions?
Devel::bt
The End
Sources
http://en.wikipedia.org/wiki/Segmentation_fault
http://modperlbook.org/html/
21-6-Analyzing-Dumped-core-Files.html
http://www.linux-magazin.de/Heft-Abo/Ausgaben/
2007/01/Getriebeschaden
Uwe V¨lker
o Hunting segfaults

40. Introduction
Detecting segfaults
Devel::Trace Sources
gdb Questions?
Devel::bt
The End
Questions?
Uwe V¨lker
o Hunting segfaults