The document discusses buffer overflows and static buffer overflow detection. It first explains what a buffer overflow vulnerability is and how it can occur when a program writes data to a buffer that overruns the boundary and overwrites adjacent memory locations. It then shows an example of how a buffer overflow can happen in the strcpy function. The document goes on to discuss the history of buffer overflow exploits and defenses. It describes how static analysis can be used to detect buffer overflows by analyzing the program control flow graph to find loops, and then performing inner loop data dependency analysis.

1. Finding Buffer Overflows
Generating Loops
Magistère Informatique de Grenoble 2015,
Claude Goubet
VERIMAG (supervisor L. Mounier)

2. Vulnerabilities and buffer overflow
A vulnerability is a particular case of a
bug, when it can be used in order to lead
the program to have an unexpected
behavior.
A buffer overflow is a vulnerability which
appears when a program, writing data in a
buffer, overruns its boundary and
overwrites the adjacent memory locations.
2

3. How can a buffer overflow appear?
3
strcpy(char *s1, const char
*s2)
{
char *s = s1;
while ((*s++ = *s2++) !=
0)
;
return (s1);
}

4. How can a buffer overflow appear?
strcpy(char *s1, const char
*s2)
{
char *s = s1;
while ((*s++ = *s2++) !=
0)
;
return (s1);
} .While:
movl -4(%ebp),
%eax
movzbl (%eax), %edx
movl -8(%ebp),
%eax
movb %dl, (%eax)
movl -8(%ebp),
%eax
movzbl (%eax), %eax
testb %al, %al
setne %al
addl $1, -8(%ebp)
addl $1, -4(%ebp)
testb %al, %al
top
S2’s
pointer
S1’s
pointer
Return
address
s2
s1
Return
address
High addresses

5. 5
How can a buffer overflow appear?
strcpy(char *s1, const char
*s2)
{
char *s = s1;
while ((*s++ = *s2++) !=
0)
;
return (s1);
} .While:
movl -4(%ebp),
%eax
movzbl (%eax), %edx
movl -8(%ebp),
%eax
movb %dl, (%eax)
movl -8(%ebp),
%eax
movzbl (%eax), %eax
testb %al, %al
setne %al
addl $1, -8(%ebp)
addl $1, -4(%ebp)
testb %al, %al
top
S2’s
pointer
S1’s
pointer
Return
address
s2
s1
Return
address
High addresses
top
S2’s
pointer
S1’s
pointer
Return
address
s2
High addresses
s2

6. A litle history
1996: « Smashing the stack for fun and
profit », Aleph One
1997: non-executable (NX) stack
countermeasure, Alexander Peslyak on the
linux Kernel
1997: non-executable stack attacks using
library calls
2000: NX improvement: randomisation of the
mmap base, PAX team
2001: attacks on PAX
… 6

7. Static buffer overflow
detection
Static analysis :
No program execution
Assambly code
Detection principle :
Loop detection
Inner loop data-dependency analysis
7

8. Loop detection
8
int main () {
int x = 0;
while (x != 1)
{
if (x < 1)
x++;
else
x--;
}
while (x <=
15){
int i = 0 ;
while (i < 3)
{
x++;
i++;
}
}
return 0;
}

9. Loop detection
9
int main () {
int x = 0;
while (x != 1)
{
if (x < 1)
x++;
else
x--;
}
while (x <=
15){
int i = 0 ;
while (i < 3)
{
x++;
i++;
}
}
return 0;
}
Control flow
graph

10. Loop detection
10
int main () {
int x = 0;
while (x != 1)
{
if (x < 1)
x++;
else
x--;
}
while (x <=
15){
int i = 0 ;
while (i < 3)
{
x++;
i++;
}
}
return 0;
}
Control flow
graph
Back edges

11. Loop detection
11
int main () {
int x = 0;
while (x != 1)
{
if (x < 1)
x++;
else
x--;
}
while (x <=
15){
int i = 0 ;
while (i < 3)
{
x++;
i++;
}
}
return 0;
}
Control flow
graph
Inner loops

12. Loop detection
12
int main () {
int x = 0;
while (x != 1)
{
if (x < 1)
x++;
else
x--;
}
while (x <=
15){
int i = 0 ;
while (i < 3)
{
x++;
i++;
}
}
return 0;
}
Control flow
graph
Nesting loop

13. Loop detection
13
int main () {
int x = 0;
while (x != 1)
{
if (x < 1)
x++;
else
x--;
}
while (x <=
15){
int i = 0 ;
while (i < 3)
{
x++;
i++;
}
}
return 0;
}
Control flow
graph
Nested loop

14. Dataflow analysis
14
.While:
movl -4(%ebp),
%eax
movzbl (%eax), %edx
movl -8(%ebp),
%eax
movb %dl, (%eax)
movl -8(%ebp),
%eax
movzbl (%eax), %eax
testb %al, %al
setne %al
addl $1, -8(%ebp)
addl $1, -4(%ebp)
testb %al, %al
jne .While