Search-based Software Testing (SBST) '22
May 9, 2022, 9:15am (CEST) - UTC+2
https://sbst22.github.io/
@sbstworkshop
Giovani Guizzo and Sebastiano Panichella

Twitch stream → https://www.twitch.tv/koxp
(screen mirroring for students)
Twitter → @sbstworkshop
On Midspace all day!
Welcome to (virtual) Pittsburgh!
Giovani

Thanks to our sponsor!
Google Open-Source Security Group
Google Cloud credits for running the tool competition experiments
Giovani

General format
Participants:
Check your email for your Midspace and BeamNG.research invitations
Please remain muted until it is your turn to speak!
Non-registered participants:
Feel free to join our Twitch stream!
Everything on Twitch will be recorded and uploaded to YouTube!
Giovani

Schedule (CEST) / UTC+2
9:15 → Opening
9:30 → Live Keynote
Deep Learning Testing
by Paolo Tonella
10:30 → Break
10:45 → Tutorial
Learning and refining Input Grammars for Effective Fuzzing
by Rahul Gopinath
11:30 → Paper Session 1 (2 papers)
12:00 → Lunch Break
13:30 → Paper Session 2 (2 papers)
14:00 → SBST vs Fuzzing Discussion Panel
Andreas Zeller, Annibale Panichella, Lionel Briand, Marcel Böhme,
Mark Harman, Myra Cohen, Paolo Tonella
16:00 → Break
16:15 → Tool Competition
18:15 → Closing Giovani

Keynote (9:30am - 10:30am)
DEEP LEARNING TESTING
Deep neural networks have outperformed classical
techniques in domains such as natural language processing,
computer vision and speech recognition. They found several
real world applications, ranging from autonomous vehicles to
medical diagnosis. Correspondingly, the need for testing
approaches to ensure their dependability and quality has
increased. In recent years, we have seen an exponential
growth in the number of research papers that address
various aspects of deep learning testing. In this seminar, I will
describe a selected set of core problems in the field. In
particular, I will focus on the reasons why such problems
differ from the corresponding, traditional testing ones. I will
present some of the solutions that appeared recently in the
area and I will comment on the issues ("elephants in the
room") that still affect the existing approaches. Paolo Tonella
Bio:
Paolo Tonella is Full Professor at the Faculty of Informatics and at the
Software Institute of Università della Svizzera Italiana (USI) in Lugano,
Switzerland. He is Honorary Professor at University College London, UK
and he is Affiliated Fellow of Fondazione Bruno Kessler, Trento, Italy,
where he has been Head of Software Engineering until mid 2018. Paolo
Tonella holds an ERC Advanced grant as Principal Investigator of the
project PRECRIME. Paolo Tonella wrote over 150 peer reviewed
conference papers and over 50 journal papers. His H-index (according
to Google scholar) is 59. He is/was in the editorial board of the ACM
Transactions on Software Engineering and Methodology, of the IEEE
Transactions on Software Engineering, of Empirical Software
Engineering, Springer, and of the Journal of Software: Evolution and
Process, Wiley. His current research interests are in software testing, in
particular approaches to ensure the dependability of machine learning
based systems, automated testing of cyber physical systems, and test
oracle inference and improvement.
Sebastiano

Tutorial (10:45 - 11:30)
Fuzzing is one of the key techniques for evaluating robustness of
programs against malicious inputs. To fuzz the program logic effectively,
one needs the input specification of the program under fuzzing.
However, such input specifications are rarely available, and even when
present, can be obsolete, incomplete or incorrect leading to fuzzing blind
spots. In this tutorial, I will show how to mine the input specification from
a given program from the ground up, first generating sample inputs, then
using such inputs to mine the program input grammar, and finally using
the mined grammar to fuzz the program and find any bugs. What should
you do next once you find a bug? An input pattern rather than a particular
input is likely to result in the bug, and to have any confidence in a bug fix,
we should test the fix using the input pattern rather than a single input. In
this talk, I will show how to abstract such input patterns corresponding to
program behaviors such as bugs into a focused grammar, how to
combine multiple input patterns together, and use such patterns to fuzz.
The specialized grammars we generate can be used by any grammar
fuzzer for precise control of produced inputs and hence the expected
behavior.
Rahul Gopinath
Bio:
Rahul Gopinath is a lecturer at the University of Sydney, Australia. He
received his Ph.D. in 2017 from the Oregon State University, US. Rahul's
research focus is on static and dynamic analysis of programs, especially
mining input specifications, focused fuzzing, and debugging. He is one
of the co-authors of the "Fuzzing Book - Tools and Techniques for
Generating Software Tests". Rahul also has extensive industry
experience, having worked in diverse fields such as telco, publishing,
systems, and DevOps. He has also worked on empirical evaluation of
the effectiveness of different coverage techniques and Mutation
Analysis..
LEARNING AND REFINING INPUT GRAMMARS
FOR EFFECTIVE FUZZING
Giovani
Link Tutorial:
https://github.com/vrthra/SBST22-tutorial/tree/main

Research papers
Sessions:
- Paper Session 1 (2 papers) - 11:30am - 12:00pm
- Paper Session 2 (2 papers) - 1:30pm - 2:00pm
Long papers (15 minutes):
- 10 minutes for talk
- 5 minutes for questions
Short papers (10 minutes):
- 5 minutes for talk
- 5 minutes for questions
Vincenzo

Research paper authors/schedule
Learning to Rank for Test Case Prioritization
Safa Omri and Carsten Sinz
Wasserstein Generative Adversarial Networks for Online Test Generation for Cyber Physical
Systems
Jarkko Peltomäki, Frankie Spencer and Ivan Porres
Vincenzo

Lunch session (12:10 - 13:30)
You can (virtually) socialise.
Every room supports video- and text-chat.
Attendees can create public/private room for chatting!
Shuffle rooms - you click a button and you are matched with a number of people for
a random networking experience.
Giovani and Sebastiano

Research paper authors/schedule
A Comparative Evaluation on the Quality of Manual and Automatic Test Case Generation
Techniques for Scientific Software - A Case Study of a Python Project for Material Science
Workflows
Daniel Trübenbach, Sebastian Müller and Lars Grunske
Towards Run-Time Search for Real-World Multi-Agent Systems
Abigail Diller and Erik M. Fredericks
Gunel

Discussion Panel
Giovani and Sebastiano

Summary on panel Discussion
It is difficult to completely separate Fuzzing from SBST approaches/tools:
- They are used to solve the same problem and similar techniques have been proposed in the
literature
- Renaming the SBST workshop to something including both SBST and Fuzzing would be ideal:
- “Search-based Fuzzing” could be a potential proposal
- Merging Fuzzing and SBST requires having "agreed" review criteria (mainly because researchers
tend to do conduct case studies with different emphases)
- Having a Tool competition based on both Fuzzing and SBST tools would be ideal
“The SBST and Fuzzing communities should walk together“
- Organize common events together, organize seminars, and have hybrid PC members
- Security should be also another aspect/problem targeted by both SBST / Fuzzing
- We need to consider execution costs and attributes such as scalability of proposed tools (does an
approach scale in a particular context?)
- The usage of “surrogate models” is a topic that could be investigated by both SBST / Fuzzing
communities to test complex systems
- Human-into-the-loop testing could be an important challenging aspect to consider in the evaluation
of SBST/Fuzzing techniques
Combining SBST and Fuzzing could help in better addressing the oracle problem:
- We need to combine such techniques since both SBST and Fuzzing require oracles to expand the
types of bugs they can detect
- Flakiness is also another problem to be addressed, related to the oracle problem
Giovani and Sebastiano

Tools paper authors/schedule (1/2)
Overview of Tool Competition and Report Results
Alessio Gambi [1], Gunel Jahangirova [2], Fiorella Zampetti [3], Vincenzo Riccio [2]
University of Passau, Germany & IMC FH Krems, Austria [1], Università della Svizzera Italiana, Switzerland [2], University of Sannio, Italy [3]
EvoSuite at the SBST 2022 Tool Competition
Sebastian Vogl [1], Gordon Fraser [1], Andrea Arcuri [2]
University of Passau, Germany [1], Kristiania University College and Oslo Metropolitan University, Norway [2]
BBC at the SBST 2022 Tool Competition
Pouria Derakhshanfar [1], Xavier Devroey [2]
Delft University of Technology, Netherlands [1] and University of Namur, Belgium [2]
Kex & Kex-Reflection at the 2022 SBST Tool Competition
Azat Abdullin, Marat Akhin, and Mikhail Belyaev
JetBrains Research Verification or Program Analysis Lab, Russia
UtBot & UtBot-Mocks at the SBST2022 Tool Competition
Dmitry Ivanov, Nikolay Bukharev, Alexey Menshutin, Arsen Nagdalian, Gleb Stromov, and Artem Ustinov
Huawei, Russian Research Institute, St. Petersburg, Russia
Tool Chairs

Tools paper authors/schedule (2/2)
AdaFrenetic at the SBST 2022 Tool Competition
SongYang Yan, Ming Fan
Xi'an Jiaotong University, China
Ambiegen tool at the SBST 2022 Tool Competition
Dmytro Humeniuk, Giuliano Antoniol, and Foutse Khomh
Polytechnique Montreal, Canada
FreneticV tool at the SBST 2022 Tool Competition
Ezequiel Castellano, Stefan Klikovits, Ahmet Cetinkaya, Paolo Arcaini
National Institute of Informatics (NII), Japan
GenRL tool at the SBST 2022 Tool Competition
Luigi Libero Lucio Starace [1], Andrea Romdhana [2,3], Sergio Di Martino [1]
Università degli Studi di Napoli Federico II, Italy [1], Università degli Studi di Genova, Italy [2], Fondazione Bruno Kessler, Italy [3]
EvoMBT tool at the SBST 2022 Tool Competition
Raihana Ferdous, Chia-Kang Hung, Fitsum Kifetew, Davide Prandi, Angelo Susi
Fondazione Bruno Kessler, Italy
WOGAN tool at the SBST 2022 Tool Competition
Jarkko Peltomäki, Frankie Spencer, Ivan Porres
Åbo Akademi, Finland
Tool Chairs

Tools competition papers (4:15pm - 6:15pm)
Competition reports:
- 10 minutes per paper (inclusive)
Tool Chairs

Closing (18:15)
Thanks to:
- All of you for joining!
- Our PC for their support in
reviewing papers!
- Our steering committee for
their support in moving the
workshop online!
- Our sponsor (Google)
143 registered participants specifically for SBST
115 people joined throughout the day
80 (peak) attending participants at SBST
40 (peak) attending via Twitch!
110 (peak) concurrent viewers!
Giovani

Thanks to: Our Tool Competition Co-chairs
for organizing an exciting and relevant tool competition!
Sebastiano
Alessio Gambi Gunel Jahangirova Vincenzo Riccio Fiorella Zampetti

Thanks to: our Web-chair
for preparing our beautiful website!
Sebastiano
Rebecca Moussa

Thanks to: the Program Committee members
for their support in reviewing papers!
Nazareno Aguirre, Universidad Nacional de Río Cuarto - CONICET, Argentina
Aldeida Aleti, Monash University, Australia
Giuliano Antoniol, Ecole Polytechnique de Montréal, Canada
Kate Bowers, Oakland University, USA
Jose Campos, University of Washington, USA
Thelma E. Colanzi, State University of Maringá, Brazil
Byron DeVries, Grand Valley State University, USA
Gordon Fraser, University of Passau, Germany
Erik Fredericks, Oakland University, USA
Gregory Gay, Chalmers and the University of Gothenburg, Sweden
Alessandra Gorla, IMDEA Software Institute, Spain
Gregory Kapfhammer, Allegheny College, USA
Yiling Lou, Peking University, China
Mitchell Olsthoorn, Delft University of Technology, Netherlands
Justyna Petke, University College London, UK
Silvia R. Vergilio, Universidade Federal do Paraná, Brazil
Simone do Rocio Senger de Souza, University of São Paulo, Brazil
Thomas Vogel, Humboldt-Universität zu Berlin, Germany
Jie Zhang, University College London, UK
Sebastiano

Thanks to: our Viewers
For making this workshop interesting all around!
You are the best :)
Sebastiano
And many more…

What’s Next?
Special issue at Science of Computer Programming 2022: "SBST’22: Search-Based
Software Engineering – Tools"
Open Call!
Short papers with a great focus on software and replication packages
Submissions: 1st of June, 2022 to 1st of October, 2022
Recordings of our Workshop will be made available on Twitch and YouTube
Integration of Inputs from the Panel discussion for the SIGSOFT newsletter (informal
quarterly newsletter on Software Engineering), linking to the video of the discussion
Giovani

What’s Next?
Find more sponsors to support with cash prize(s) for the winners of the tool
competition? 🤑
Coordinate with similar academic and industrial workshops or venues (e.g.,
FuzzCon, Fuzzing Workshop @ NDSS) in other SE and non-SE venues to
continuously foster research in the field.
Establishment of a Fuzzing competition
Giovani

SBST 2023 - Cyber-physical Testing Themed
Keynote on challenges concerning the development and
testing Cyber-physical systems
Promote discussion panels around contemporary Cyber-physical challenges with
industrial and academic researchers (Tesla? Google? Manna Drones Delivery?)
Encourage the design, implementation, and public availability of usable and
high-quality tools to deal with SBST-related challenges.
Sebastiano

Thank you all for participating!
See you next year in Melbourne
at SBST 2023!