Daniel Stenberg's presentation done on June 30, 2020 about curl testing, CI, QA, fuzzing and more.

1. DanielStenberg-June30th
2020
for security!
Testing

2. Daniel Stenberg
@bagderhttps://daniel.haxx.se

3. Daniel Stenberg
@bagder

4. … ten billion installations
across the world
curl runs in...
@bagder@bagder

5. …… devices all of you heredevices all of you here
today own and use!today own and use!
curl runs in
@bagder@bagder

6. Writing safe codeWriting safe code
@bagder@bagder

7. Safe code is not a coincidence
Doesn’t happen randomly, we need to work on it!
We care about it!
It is about time we all address this topic!
Everything I’ll tell you today, you already know!
@bagder@bagder

8. Writing
Help users use the tool and APIs safelyHelp users use the tool and APIs safely
Defaults!Defaults!
Document. Everything, detailed and a lot.Document. Everything, detailed and a lot.
Get rid of warningsGet rid of warnings
UseUse commentscomments
assert()assert()
Releasing the code asReleasing the code as open sourceopen source
@bagder@bagder

9. Review
All codeAll code shallshall get reviewedget reviewed
CodeCode shallshall be easy to readbe easy to read and understandand understand
Use theUse the same code stylesame code style everywhereeverywhere
Commit message template andCommit message template and qualityquality
Make sure style and templates are followed!Make sure style and templates are followed!
@bagder@bagder

10. Test!
Unit tests
System tests
Integration tests
Test the documentation
Manual tests
Fixed a bug? Add a test. Or two.
@bagder@bagder

11. Torment
Valgrind, address/UB/integer sanitizersValgrind, address/UB/integer sanitizers
Static code analysersStatic code analysers
FuzzersFuzzers
@bagder@bagder

12. All that, all the time
For every commit
For every PR
All. The. Time
@bagder@bagder

13. curl CI
80+ builds and test “rounds” per commit
Tests code style, indenting etc
Thousands of tests per build
Builds and tests on tens of platforms
20-25 hours of CI per commit
@bagder@bagder

14. @bagder@bagder

15. @bagder@bagder

16. The curl project’s CI sponsors
Parts of the CI system run of free tiers
Teamviewer sponsors Appveyor powers
Travis gives us more power
Azure Pipelines gives us more power
@bagder@bagder

17. @bagder@bagder

18. @bagder@bagder

19. Writing test cases should be easy
A curl test case is a single file in a human readable well-documented
format
• conditions and features needed to run
• what command (line) to run
• what the test wants returned from a server
• how the protocol exchange should look like
• stdout and stderr contents
• expected error code
• … and more
@bagder@bagder

20. Custom test servers
We avoid using “real” servers for testing
Test servers are as dumb as possible
Controlled from the test case what it should send and expect
Allows “crazy” behaviors and send/receive “anything”
Makes the test servers smaller and simpler
Helps with test suite portability
TLS is done by stunnel-fronting
Servers run on random port numbers
@bagder@bagder

21. Tools of the trade
Valgrind
Clang address,
undefined, signed-
integer-overflow
sanitizers
Clang tidy
“torture tests”
Scan-build
Lgtm
Codacy
CodeQL
Monocle AI
Deepcode AI
Coverity
Travis CI
Appveyor
Cirrus CI
Github Actions
Azure Pipelines
Buildbots
OSS-Fuzz
CI-fuzz
@bagder@bagder

22. Torture tests – error injection
Build with a “debug” option
Add wrapper functions for
common “fallible” functions (that
allocate and free resources)
Each wrapper function logs what it
does and can optionally return
error.
The complete individual test case
is first run once
count how many fallible functions
rerun the test case that number of
times and for each iteration make
next fallible function return error
Verify nothing crashed and no
memory leaked
Repeat for all tests
@bagder@bagder

23. Source code policy
Fix all warnings (eye roll)Fix all warnings (eye roll)
No defects leftNo defects left
Use the strictest and most picky optionsUse the strictest and most picky options
As many tests as possibleAs many tests as possible
Fix security issues as soon as possibleqFix security issues as soon as possibleq
@bagder@bagder

24. A million build combos, 70+ OSes and 20 CPUs
Testing all combinations is simply not possible
Test the common setups
Test on as many platforms as possible
Test on several different CPU architectures
“white spots” in test coverage handled by review
Users keep finding untested areas and build combinations
@bagder@bagder

25. The security of curl is the sum of all
components combined.
- Lean on the right set!
@bagder@bagder

26. 32 third party dependencies
I/O layer
libcurl
URL parser libidn2winidn
HTTPHTTPS
OpenSSL
Mesalink
gskit
mbedTLS
wolfSSL
Schannel
SecureTransport
GnuTLS
NSS
boringssl
libressl
AmiSSL
SFTP SCP LDAP
WinLDAP
OpenLDAP
RTMP
librtmp
Name resolver c-ares
compression
libz brotli
cookies
libpsl
IMAP SMTP POP3
HTTP/2
nghttp2
authentication
winsspi Heimdal MIT kerberos
HTTP/3
quiche
ngtcp2
HTTP/1
SSH
wolfSSH
libssh2
libssh
@bagder@bagder
BearSSL
nghttp3

27. Fuzz Testing in wolfSSL
● Best Tested TLS due to our extensive internal
testing process
● Fuzz testing in our internal testing process is
one of the reasons why
● Fuzz testing bombards the program with
invalid, unexpected, and random data that
then allows for observing if there is
potential memory leaks or logic errors
● Allows us to catch bugs that could
turn into potential vulnerabilities
● We only use highly respected external
testers:
● Guido Vranken in Holland
● Robert Horr of T-Systemsin Germany
● Internally, we test using several different
software fuzzers such as:
● in-memory fuzzer
● a network fuzzer
● OSS-fuzz
● libfuzzer
● tlsfuzzer
● AFL
● To maximize security we run 7 fuzz testers
internally every night on our latest code on
over 20 platforms!
@bagder@bagder

28. The curl bug-bounty
https://hackerone.com/curl
Pays rewards using donated money
Rewards up to 12,000 USD for a critical flaw
Has paid 2,300 USD so far
Increasing bounty amounts
@bagder@bagder

29. @bagder@bagder

30. University researchers
looking for a class of bugs
Get involved!
Corporate security researchers
who’s companies consume curl
Pen testers
who want to find something big and widespread
Security enthusiasts
looking for a reward and “fame”
@bagder@bagder

31. There’s one more thing curl has
@bagder@bagder

32. Full-time curl maintainer
@bagder@bagder

33. Bad things will happen
We are not immune, no matter how hard we try!
Act immediately
Own the problem
Fix it and announce it
Learn from it
Make it harder to do the same mistake again
@bagder@bagder

34. But does it work?
10 billion installations is no proof
✔ Decreasing number of CVEs over time
✔ Decreasing number of OSS-Fuzz reports over time
✔ Increasing bounty rewards
@bagder@bagder

35. Reported security vulnerabilities per year
@bagder@bagder

36. CRTTA
@bagder@bagder

37. Clean code
Review
Test
Torment
Act on mistakes
@bagder@bagder

38. Daniel Stenberg
@bagder
https://daniel.haxx.se/
Thank you!Thank you!
Questions?Questions?
@bagder@bagder

39. License
This presentation and its contents are
licensed under the Creative Commons
Attribution 4.0 license:
http://creativecommons.org/licenses/by/4.0/
@bagder@bagder