Opening remarks given at Cybersecurity R & D workshop at Singapore International Cyberweek 2020.

1. Trustworthy Systems to
Trusted AI
Prof. Abhik Roychoudhury
Provost’s Chair Professor
National University of Singapore
1
Cybersecurity R&D Workshop 2020

2. Outline
• Background: Singapore Cyber-security Consortium
• Vision of Trustworthy Systems
• Ongoing work on Trustworthy Systems and Trusted AI
2

3. Encourage
problem-inspired research
Singapore Cybersecurity Consortium (SGCSC)
Est. 1 September 2016
A nation-wide platform for engagement
between industry, academia, and government
towards greater awareness, adoption, and translation
of cybersecurity technologies
Upgrade capabilities
through technology adoption
Grow an
innovation ecosystem
Industry
Academia
Agencies
3
About

4. Singapore public agencies
Open participation
Industry members
Singapore-registered companies with interest or expertise in
cybersecurity are eligible to apply for membership
Agencies
Industry
Academia
Institutes of Higher Learning and Research Institutes
Open participation
4
Structure
Structure

5. S I LV E R
P L AT I N U M G O L D
5
Industry Members As of 15 Sep 2020

6. National
Satellites of
Excellence
Local and
International
Research Grants
National
Cybersecurity
R&D Laboratory
&
iTrust Labs
Singapore
Cybersecurity
Consortium
Cybersecurity
Postgraduate
Scholarship
National Cybersecurity R&D (NCR) Programme
https://www.nrf.gov.sg/programmes/national-cybersecurity-r-d-programme
SGCSC, a component of the NCR programme,
helps members gain awareness and exposure to
various resources and support for cybersecurity R&D
available under the programme.
6
Ecosystem

7. Annual
WILD & CRAZY IDEAS DAY
Research ideas
Problem statements
Annual
CYBERSECURITY CAMP
Workshop on trending topics
Industry applications
Hands-on learning
MEMBER RATE
Quarterly
TECHNOLOGY TALKS
Latest technologies and trends
Project showcases
EXPOSURE OPPORTUNITIES
SPECIAL INTEREST GROUPS
Knowledge and idea exchange
R&D partnership exploration
MEMBER ONLY
Annual
SEED GRANT CALL
Funding for joint R&D
(Industry-Academia pair)
Approx. $100 – 150K
1- to 1.5-year projects
MEMBER ONLY
CYBERSECURITY TRACK
Pre- / early start-up mentorship
Business + Technical discussions
Training and tech update
Discussions to alleviate
pain points in existing work
Dream up new projects –
Translation-oriented research
Maturity slope
7
Activities

8. Seed Grant 2020 Award
Deep Learning-based Side Channel Attacks on
SoC Architecture for Hardware Assurance
EarAuth: Designing Usable Security for the Next Billion Users
(NBUs):
A Novel Multi-Factor Authentication Solution using Smart
Earables
This project enables comprehensive and inexpensive
security evaluation for IoT devices.
This project aims to develop an authentication framework
using smart wearables around the ear, to enable
password-less logins for swift usability.
CONGRATULATIONS!!

9. Outline
• Background: Singapore Cyber-security Consortium
• Vision of Trustworthy Systems
• Ongoing work on Trustworthy Systems & Trusted AI
9

10. Trustworthy software
10
Creativity Precision+
- Solving differential equations for an examination
- Painting a landscape of the lush greenery or a landscape.
Compare these activities with crafting software systems

11. Engendering Trust
Formal Verification
• Formally verified Software Stack
• Verified Operating Systems: seL4 project
• Verified file systems: BesFS, work at NUS
Trust from COTS
11

12. Chronological Evolution of Capabilities
Point Projects
MINDEF, MoE…
[2009-12, 2011-14,
2013-15]
Targeted Capability
NCR 1
TSUNAMi
(2015 –20)
National Satellite of
Excellence
(2019- )
12

13. Our Capability Stack
13
Security Testing and
Analysis
(TSUNAMi, NRF NCR)
Formal Verification of Systems
(Securify, NRF NCR)
[Core] Certified Trustworthy Systems
– Call 1
Regression
analysis
(MoE)
Symbolic
analysis
(DIRP, DSO)
[App] Secure Smart Nation –
Call 2
Modeling
and
Verification
(FSTD)
Scalable
MC (NTU)
20092015201820192020
[App] Challenge from
Call 2
National Satellite of
Excellence

14. Vulnerability
Discovery
Binary
Hardening
Verification
Data
Protection
14
Agency
Collaboration
…
Industry
Collaboration
…
Education –
Universities,
…
Research Outputs – Publications, Tools, Academic
Collaboration, Exchanges, Seminars, Workshops
Enhancing local
capabilities
Overall Outlook

15. 15
Malware
&Rootkit
Analysis
Internet
File
System
Account &
Protection
Kernel &
Process
Function
Call
System
Call
Program &
Service
strace
Buffer
Overflow
Fuzzing
Binary
Analysis
gdb
SPIKE
BitBlaze/QEMU
ls, cd, mv,
ps, vi, …
Password
Cracking
john
Scanning
ping,
traceroute,
nmap
Sniffing
WireShark
Spoofing &
Session
Hijacking
netwox
nc
Denial of
Service
VM
simulation
Firewall &
NAT
iptables
Web attacks:
SQL injection,
CSRF, XSS
TamperData,
Paros Proxy
System
Security
Software
Security
Network Security
Web Security
Education: module at NUS

16. National Satellite of
Excellence
The NSoE-TSS aims to enhance Singapore's national capabilities in
trustworthy smart system infrastructures.
We seek to build on our combined strengths in software security, and smart
systems to build consolidated technologies, related to software assurance for
smart systems.
The certification can take on a range of flavours including functionality
certification, checking against crashes and vulnerabilities, measuring and
certifying resilience against malicious inputs and environments, as well as
checking and certifying for absence of information leakage via extra-
functional mechanisms such as side channels.
https://www.comp.nus.edu.sg/~nsoe-tss/index.htm

17. Mission
17
Technology
• Deep tech.
capabilities for
software sys.
certification
• Functional and
non-functional
properties
Innovation
• Show-case
innovative uses of
certified software
sys. for secure
smart nation
• Deployment
scenarios
Policy
• Enhance and aid
regulatory
processes for
critical software
systems
• Feedback to
public agencies

18. Outline
• Background: Singapore Cyber-security Consortium
• Vision of Trustworthy Systems
• Ongoing work on Trustworthy Systems & Trusted AI: Capabilities
• Spectre Attacks
• Fuzz Testing
• Fuzzing for DNNs
• Self-Healing Systems
18

19. Defense against Spectre attacks
19
Taint
Sources
list
Code
repair
<TB , RS, LS>
<TB, RS>
<TB> …
Binary
New
Binary
Source
code
Taint
analysis
BAP
Spectre
Detector
Report
Assembly
code (.s)
Assemble
& link
Repaired
assembly
code (.s)
Compile Code
Matcher
Disassembly
code (.asm)
Objdump
• Spectre attacks exploit the vulnerabilities of a program to steal the sensitive data through speculative execution.
• oo7 is a static analysis framework that can mitigate Spectre attacks by detecting potentially vulnerable code snippets in
program binaries and protecting them against the attack.
Spectre variant 1
The detection condition of Spectre variant 1
oo7

20. Fuzzing
20
� Model-Based
Blackbox
Fuzzing
Input model
Peach, Spike …
Seed Input
�
�
�
Pass al l check s
Sat i sf y so m e check s
Sat i sf y so m e check s
Mutated Inputs
Mutators
Test suite
Mutated files
Input Queue
EnqueueDequeue
ProgramInput

21. AFLFast
• Design power schedules to regulate the
“energy” to gravitate path exploration
towards low-frequency paths
• Integrated into AFL Fuzzer, used in DARPA
CGC.
• Intuition is simple – deprioritize the common
paths, works directly on binaries.
21
if (condition1)
return // frequented by inputs
else if (condition2)
exit // frequented by many inputs
else ….
• Directed Fuzzing as an optimization problem (No constraint so
• Program analysis moved to instrumentation time
to retain efficiency of greybox fuzzing.
• Distance to targets efficiently computed at runtime.
• Find global minimum using search meta-heuristic – Simulated An
• Results: outperforms KATCH and BugRedux. 17 CVEs assign
• Application: patch testing, crash reproduction, information flow
Mutators
Test suite
Mutated files
Input Queue
EnqueueDequeue

22. Deployment
22
Independent evaluation found crashes 19x faster on
DARPA Cyber Grand Challenge (CGC) binaries
Integrated into main-line of AFL fuzzer within a year of publication (CCS16), which is
used on a daily basis by corporations for finding vulnerabilities

23. Model Training and Model Robustness
� �
0 -20 -10 0 10 20 30
-30 -20 -10 0 10 20 30
-30 -20 -10 0 10 20 30
2.21
1.72
1.23
0.74
0.49
0.25
0.00
2.21
1.72
1.23
0.74
0.49
0.25
0.00
7.47
5.69
4.80
3.03
2.14
1.25
0.53
-3
-2
-1
0
1
2
3
-3
-2
-1
0
1
2
3
-3
-2
-1
0
1
2
3
rotaterotate
translate
translate
rotate
translate
� �
-30 -20 -10 0 10 20 30
-30 -20 -10 0 10 20 30
-30 -20 -10 0 10 20 30
2.21
1.72
1.23
0.74
0.49
0.25
0.00
2.21
1.72
1.23
0.74
0.49
0.25
0.00
7.47
5.69
4.80
3.03
2.14
1.25
0.53
-3
-2
-1
0
1
2
3
-3
-2
-1
0
1
2
3
-3
-2
-1
0
1
2
3
rotaterotate
translate
translate
rotate
translate
• Neural Network can be fooled with simple special transformation (rotation, translate)
rotate by
labels are different
Adversarial
learning
Program synthesis
Complete features
Complete
specifications
Test case
generation
Data
augmentation
• Model training can be regarded as AI-based program synthesis. Given a set of specs (training data), it
generates a program (model) satisfying all the specs.
23

24. Mutator
Mutated inputs
} Selector
model
Seed pool
Fuzz-based Data Augmentation to Improve Robustness
• Generate representative perturbations using genetic algorithm to augment training data
• The goal is to maximize the diversity of samples in the distribution
Dataset Standard Acc Random Augment Sensei
GTSRB 1.9% 73.3% 88.2%
CIFAR-10 1.8% 73.3% 81.5%
• Result in terms of robust accuracy[*]
[*] Exploring the Landscape of Spatial Robustness. L. Engstrom, B. Tran, D. Tsipras, L. Schmidt, and A. Madry ICML 19’ 24
Training data-
set (Seeds)
Interesting
inputs

25. Intelligent software!
25
In the absence of formal specifications, analyze the
buggy program and its artifacts to glean a specification
about what could have gone wrong!
Specification Inference
(application: self-healing)
Buggy
Program
Tests

26. (very Non-exhaustive) History of AI
Symbolic AI
• 1958 LISP
• 1965 Resolution theorem proving
• 1970 Prolog
• 1982-92 Fifth Generation Comp Sys
• 1995 - … Advances in SAT, SMT solving
• 2005 - … Symbolic Execution
Biologically inspired AI
• 1959 Perceptron
• 1970 - … Genetic Algorithm
• 1980 -… Neural Networks
• 1992 Genetic Programming
• 1997 Deep Blue
• 2012 AlexNet work on CNN
26

27. GENETIC programming
27
Lift semantic features from correct patches
and use learning to rank them.

28. Symbolic AI approach
28

29. Genetic approach
may not work here
29

30. Inference
30

31. The future for autonomous systems?
31
Can autonomous software test and repair itself autonomously to
cater for corner cases? Can autonomous software repair itself
subject to changes in environment?

32. https://sgcsc.sg/
cyber@comp.nus.edu.sg
https://www.facebook.com/sgcsc/
32