Overview of Fuzz Testing and the latest advances in the field are discussed. Fuzz testing is a popular method to find security vulnerabilities in software systems.

1. Fuzz Testing
Abhik
Roychoudhury
National
University of
Singapore

2. APR-talk,
2022
Working in
Program
Analysis and
Software
Security (2001-)
Singapore
Cybersecurity
Consortium
(2016)
National
of Excellence in
Trustworthy
Software
Systems (2019)
2
Comprehensive university with
Science, Arts, Engineering, medical, Law, Business, Public Policy, Music, Computing, …
Public university 30K undergraduate students, 10K+ graduate students overall.
2500+ faculty members overall, 100+ in Computing (two departments CS and IS).
http://www.nus.edu.sg/about#corporate-information

3. Fuzz Testing
3
Springfield Project - Fuzzing as a service
OSS-Fuzz - Continuous fuzzing for open-source projects
Term coined by Barton Miller at University of Wisconsin in 1988
Biased random search over inputs to find crashes / hangs, tools existed until 2016.
And then, since 2016 … research in looking inside the tools

4. Who cares?
4
A team of hackers won $2 million by
building a machine that could hack better
than they could
Read more at
DARPA Cyber Grand
Challenge
Automation of Security
[detecting and fixing
vulnerabilities in binaries
automatically]
Fuzzing
Talk,
2022

5. Presented by Thuan Pham
Black-box Fuzzing
5

6. Lack of specifications
• Many programs take in structured inputs
 PDF Reader, library for manipulating TIFF, PNG images
 Compilers which take in programs as input
 Web-browsers, ...
• Generating a completely random input will likely crash the application with little
insight gained about the underlying vulnerability.
• Instead take a legal well-formed PDF file and mutate it!
 Does not depend on program at all [nature of BB fuzzing]
 Does not even depend on input structure.
 Yet can leverage complex input structure by starting with a well-formed seed and
minimally modifying it.
Fuzzing
Talk,
2022
6

7. White-box Fuzzing
7

8. Grey-box Fuzzing
8
Mutators
Test suite
Mutated files
Input Queue
Enqueue
Dequeue
Fuzzing
Talk,
2022

9. Grey-box Fuzzing Algorithm
9
• Input: Seed Inputs S
• 1: T✗ = ∅
• 2: T = S
• 3: if T = ∅ then
• 4: add empty file to T
• 5: end if
• 6: repeat
• 7: t = chooseNext(T)
• 8: p = assignEnergy(t)
• 9: for i from 1 to p do
• 10: t0 = mutate_input(t)
• 11: if t0 crashes then
• 12: add t0 to T✗
• 13: else if isInteresting(t0 ) then
• 14: add t0 to T
• 15: end if
• 16: end for
• 17: until timeout reached or abort-signal
• Output: Crashing Inputs T✗
Fuzzing
Talk,
2022

10. Space of Problems in SW Reliability
• Fuzz Testing
 Feed semi-random inputs to find hangs and crashes
• Continuous fuzzing
 Incrementally find new “problems” in software
• Crash reproduction
 Re-construct a reported crash, crashing input not included due to privacy
• Reaching nooks and corners
• Localizing reported observable errors
• Patching reported errors from input-output examples
10
Fuzzing
Talk,
2022

11. Space of Techniques
Search (Fuzzing)
• Random
• Biased-random …
• … with mutations (AFL Fuzzer)
• …
• Low set-up overhead
• Fast, less accurate
• Use objective function to steer
Symbolic Execution
• Dynamic Symbolic execution
• Concolic Execution
• Cluster paths based on symbolic
expressions of variables
• ....
• High set-up overhead
• Slow, more accurate
• Use logical formula to steer
11
Fuzzing
Talk,
2022

12. In this(?) talk …
Search
• Enhance the effectiveness of
search techniques, with symbolic
execution and model checking as
inspiration
• Think: Symbolic (Formal)
• Act: Fuzzing (Informal)
12
Fuzzing
Talk,
2022

13. Grey-box Fuzzing Algorithm
13
• Input: Seed Inputs S
• 1: T✗ = ∅
• 2: T = S
• 3: if T = ∅ then
• 4: add empty file to T
• 5: end if
• 6: repeat
• 7: t = chooseNext(T)
• 8: p = assignEnergy(t)
• 9: for i from 1 to p do
• 10: t0 = mutate_input(t)
• 11: if t0 crashes then
• 12: add t0 to T✗
• 13: else if isInteresting(t0 ) then
• 14: add t0 to T
• 15: end if
• 16: end for
• 17: until timeout reached or abort-signal
• Output: Crashing Inputs T✗
Fuzzing
Talk,
2022

14. Programming by
experienced
people
Schematic
if (condition1)
return // short path, frequented by many inputs
else if (condition2)
exit // short paths, frequented by many inputs
else ….
Fuzzing
Talk,
2022
14

15. Prioritize low probability paths
15
 Use grey-box fuzzer which keeps track of path id for a test.
 Find probabilities that fuzzing a test t which exercises π leads to an
input which exercises π’
 Higher weightage to low probability paths discovered, to gravitate to
those -> discover new paths with minimal effort.
π π'
1 void crashme (char* s) {
2 if (s[0] == ’b’)
3 if (s[1] == ’a’)
4 if (s[2] == ’d’)
5 if (s[3] == ’!’)
6 abort ();
7 }
p
Fuzzing
Talk,
2022

16. Power-Schedules
16
Constant:
AFL uses this schedule (fuzzing ~1 minute)
 (i) .. how AFL judges fuzzing time for the test exercising path i
Cut-off Exponential:
p(i) = (i)
p(i) = 0, if f(i) > µ
min( ((i)/β)*2s(i), M) otherwise
β is a constant
s(i) #times the input exercising path i has been chosen from queue
f(i) # generated inputs exercising path i (path-frequency)
µ mean #fuzz exercising a discovered path (avg. path-frequency)
M maximum energy expendable on a state
Fuzzing
Talk,
2022

17. Independent Evaluation and Deployment
• An independent evaluation by team Codejitsu found that AFLFast exposes errors in the benchmark
binaries of the DARPA Cyber Grand Challenge 19x faster than AFL.
• Picked up by Zalewski@AFL, with following observations, paraphrased
 AFLFAST assigns substantially less energy in the beginning of the fuzzing campaign.
 Most of the cycles that AFLFAST carries out, are in fact very short. This causes the queue to be cycled very
rapidly, which in turn causes new retained inputs to be fuzzed almost immediately. In other words, because
AFLFAST assigns less energy, it can process the complete queue substantially faster. We say it starts by
exploration rather than by exploitation
• Implemented inside AFL (version 2.33b, FidgetyAFL), and distributed approximately within one year
of publication
17
There remain differences between
the two in terms of path discovered.
More experiments may be needed.

18. In this talk …
• Greybox Fuzzing is frequently used, daily in corporations
 State-of-the-art in automated vulnerability detection
 Extremely efficient coverage-based input generation
 All program analysis before/at instrumentation time.
 Start with a seed corpus, choose a seed file, fuzz it.
 Add to corpus only if new input increases coverage.
 Cannot be directed, unlike symbolic execution!
• Enhance the effectiveness of search
techniques, with symbolic execution &
model checking as inspiration
– Enhance coverage, how to make it
directed?
Fuzzing
Talk,
2022
18

19. (Earlier) View-point
19
 Directed Fuzzing: classical constraint satisfaction prob.
 Program analysis to identify program paths
that reach given program locations.
 Symbolic Execution to derive path conditions
for any of the identified paths.
 Constraint Solving to find an input that
 satisfies the path condition and thus
 reaches a program location that was given.
φ1 = (x>y)∧(x+y>10)
φ2 = ¬(x>y)∧(x+y>10)
x > y
a = x a = y
x+y>10
b = a
return b
Fuzzing
Talk,
2022

20. (Later) View-point
20
 Directed Fuzzing as optimization problem!
1. Instrumentation Time:
• Instrument program to aggregate distance values.
2. Runtime, for each input
• decide how long to be fuzzed based on distance.
• If input is closer to the targets, it is fuzzed for longer.
• If input is further away from the targets, it is fuzzed for shorter.
Fuzzing
Talk,
2022

21. Power Schedules - Recap
21
• Input: Seed Inputs S
• 1: T✗ = ∅
• 2: T = S
• 3: if T = ∅ then
• 4: add empty file to T
• 5: end if
• 6: repeat
• 7: t = chooseNext(T)
• 8: p = assignEnergy(t)
• 9: for i from 1 to p do
• 10: t0 = mutate_input(t)
• 11: if t0 crashes then
• 12: add t0 to T✗
• 13: else if isInteresting(t0 ) then
• 14: add t0 to T
• 15: end if
• 16: end for
• 17: until timeout reached or abort-signal
• Output: Crashing Inputs T✗
Fuzzing
Talk,
2022

22. Instrumentation
22
 Function-level target distance using call graph (CG)
 BB-level target distance using control-flow graph (CFG)
1. Identify target BBs and
assign distance 0
2. Identify BBs that
call functions and
assign 10*FLTD
3. For each BB, compute harmonic
mean of (length of shortest path to
any function-calling BB + 10*FLTD).
CFG for function b
8.7
11
10
30
13
12
N/A
Fuzzing
Talk,
2022

23. Directed fuzzing as optimization
23
 Integrating Simulated Annealing as power schedule
 In the beginning (t = 0min),
assign the same energy
to all seeds.
 Later (t=10min), assign
a bit more energy to
seeds that are closer.
 At exploitation (t=80min),
assign maximal energy to
seeds that are closest.
Fuzzing
Talk,
2022

24. Outcomes
24
• Directed greybox fuzzer (AFLGo) outperforms
symbolic execution-based directed fuzzers (KATCH & BugRedux)
• in terms of reaching more target locations and
• in terms of detecting more vulnerabilities,
• on their own, original benchmark sets.
• Integrated as OSS-Fuzz fork (AFLGo for Continuous Fuzzing)
• Tool AFLGo publicly available, follow-up works, survey by community.
Fuzzing
Talk,
2022

25. Beyond Symb. Exec. … Verification?
Search
• Enhance the effectiveness of search
techniques, with symbolic execution as
inspiration
 Enhance coverage
 Achieve directed search
 More advanced properties than
crashes! Get close to the effect of
verification as in model checking
25
84 139 59
AFLGo KLEE
Fuzzing
Talk,
2022
Synergy

26. M |= ⏀
Model Checking
via Dir. Fuzzing
26
Bug finding search in model
checking via directed fuzzing
Cover the whole specification
language of properties for a well-
known and popular temporal logic –
LTL
No state explosion problem as in
model checking.
Fuzzing for more advanced oracles
than simply crashes and hangs!
Most uses of Software Model
Checking are for bug finding
Restricted set of
properties for software
model checking
Mostly restricted to
proving / disproving of
invariants due to nature
of state abstractions
Unnecessary state
savings and state
explosion problem.
Fuzzing
Talk,
2022

27. Software Model Checking
Fuzzing Talk, 2022 27
Consider . None of the
exec. traces of M should
satisfy .
Construct a finite-state
automata A  such that
Language(A )
= Traces
satisfying 
Construct the synch
product M  A 
Check whether any exec
trace  of M is an exec
trace of the product M  A
 i.e. check Language(M
 A  ) = empty-set?
Yes: Violation of
 found, report
counterexample

No: Property 
holds for all
exec traces of
M.
Used in SPIN Model checker

28. Linear-time Temporal Logic
28
….. …..
…..
Satisfies j1
Satisfies j1
Satisfies j1
Satisfies j1 U j2
Satisfies j2
Satisfies j1 R j2
Satisfies j2
Satisfies j2
Satisfies j2
…..
…..
Satisfies j1 R j2
…..
…..
Satisfies j2
Satisfies j2
Satisfies j2
Satisfies j1 Ùj2
Fuzzing
Talk,
2022

29. Product Automata
(i) System Model M
(iii) Product Automata M  A
p
s1 s2
p
p
true
q1 q2
(s1,q1)
(s1,q2)
(s2,q1)
(s2,q2)
true
p
p
true
true
p
(ii) Property Automata A
M |= GF p
29
Fuzzing
Talk,
2022

30. Temporal Logic guided Fuzzing
Fuzzing
Talk,
2022
30
Find
• Find acceptance states reachable
from initial states (DFS).
• Conduct fuzzing to reach an
accepting state s from initial
state
1
Find
• Find all such acceptance states
which are reachable from itself
(DFS).
• Conduct fuzzing to reach state s
back from state s
2
Counter-example
• Counter-example evidence (if any)
obtained by simply concatenating
the two DFS stacks.
• Construct violating input from
the two fuzzing runs
3

31. Büchi Automata Guided Fuzzing
31
Save Progress
Prefix Selection
Target Selection
Trace Evaluation
Progress Saving
31
Fuzzing Talk, 2022

32. Finding Zero-day Bugs
32
Fuzzing Talk, 2022

33. Stateful Fuzzing
Input 3
Fuzzing
Generate
test cases
Check output
Existing works:
Manual specification (IJON)
Return code as states
(AFLNet)
Challenge: To cover the state space without an explicit specification of the protocol.

34. IDLE
RECV_HEADERS
RECV_BODY
REQ_PENDING
SEND_HEADERS
SEND_BODY
BODY_IS_FINAL
END_STREAM
In fuzzing, we monitor the
changes of enumeration
variables’ values to construct
the state transition tree on the
fly.
State Transition Tree
END_STREAM
END_STREAM
END_STREAM
REQ_PENDING
SEND_HEADERS
SEND_BODY
BODY_IS_FINAL
END_STREAM
Monitor the variable:
stream->state

35. IDLE
RECV_HEADERS
RECV_BODY
REQ_PENDING
SEND_HEADERS
SEND_BODY
BODY_IS_FINAL
END_STREAM
State Machine Compare with the official document.
https://datatracker.ietf.org/doc/html/rfc7540#section-5.1

36. State Fuzzing Algorithm
IDLE
RECV_HEADERS
RECV_BODY
REQ_PENDING
SEND_HEADERS
SEND_BODY
BODY_IS_FINAL
END_STRE
AM
1. Give all state transitions equal opportunities to be fuzzed.
2. Assign more energy on the valuable state sequence.
3. Save the inputs that trigger new state transitions.
4. Correlate input bytes and state transitions, giving more
opportunities on mutating these bytes.
More effort to mutate
this part

37. New Bugs
CVE-2021-38381 (Live555)
SETUP
crash
PLAY SETUP PLAY

38. Handling Structured Data?
Make Greybox Fuzzing input-structure aware by
1. Changing the input representation (structured files)
 Use tree-like representation instead of bit string
2. Adding new mutation operators
 working at chunk level (e.g., chunk deletion, insertion and splicing)
3. Prioritizing more valid seed inputs
 More valid seeds are assigned higher fuzzing “energy”
4. Applying optimizations to retain fuzzing efficiency

39. Mutators
Test suite
Mutated files
Input Queue
Enqueue
Dequeue
Fuzzing Talk, 2022 39
AFLSmart
File Cracker
root
chunk 1
… …
chunk 2
Seed input
validity score (0->100)
100: the whole seed is
fully cracked/parsed
XML-based input model.
One input model for each file format.
(e.g., Peach pits)

40. Google FuzzBench
Fuzzing
Talk,
2022
40

41. Pointers
Fuzzing
Talk,
2022
41
Directed Greybox Fuzzing ( PDF )
24th ACM Conference on Computer and Communications Security (CCS) 2017.
Coverage-based Greybox Fuzzing as Markov Chain ( PDF )
23rd ACM Conference on Computer and Communications Security (CCS) 2016,
Also in IEEE Transactions in Software Engineering (TSE) 2019.
NUS Fuzzing page: https://abhikrc.com/projects/Fuzz/
“All” fuzzing papers: https://wcventure.github.io/FuzzingPaper/
ACKNOWLEDGEMENT: National Cyber Security Research program from NRF Singapore
Smart Greybox Fuzzing ( PDF )
IEEE Transactions on Software Engineering (TSE), September 2021.
Linear-time Temporal Logic guided Greybox Fuzzing
International Conference on Software Engineering (ICSE), 2022.
Fuzzing: Challenges and Reflections ( PDF )
IEEE Software, 38(3), pages 79-86, 2021, Outcome from a 2019 Shonan Meeting.
Stateful Greybox Fuzzing
Usenix Security Symposium, 2022.

42. Looking forward
Fuzzing
Talk,
2022
42
Protocol
Implementation
Lot of past works on fuzzing have focused on parsers or file format processors.
Stateful system fuzzing could be the next step – internet facing protocols.
Model Checking efficacy, without guarantees, and without state caching.
Protocol
Spec
Fuzzer
Advanced
Oracle
Bugs
Validate
True Bugs