Presented at MUC++, 6 August 2020 Links from this talk are at: https://github.com/claremacrae/talks/blob/main/Quickly_and_Effectively_Testing_Legacy_C++_Code_with_Approval_Tests.md#top Abstract: You've inherited some legacy code: it's valuable, but it doesn't have tests, and it wasn't designed to be testable, so you need to start refactoring. But you can't refactor safely until the code has tests, and you can't add tests without refactoring! How can you ever break out of this loop? Whether Legacy code for you means "old code", "code without tests", or "code you wish to redesign for new features or unit-tests", this talk will enable you to become productive and work safely, quickly. The simplicity, convenience, ease-of-use, power and flexibility of Llewellyn Falco's "Approval Tests" approach has long been proven in a dozen programming languages. And now all this is now available to C++ developers too! Clare will present a small but surprisingly effective C++11 library for applying "Approval Tests" to cross-platform C++ code - for both legacy and green-field systems, and with a range of testing frameworks. She will demonstrate its use in some real-world situations, including how to quickly lock down the behaviour of legacy code. She will show how to quickly achieve good test coverage, even for very large sets of inputs. Attendees will discover some quick, practical techniques to use for common challenges that can be applied very easily using Approval Tests.

1. 1
Quickly and Effectively
Testing Legacy C++ Code with Approval Tests
Clare Macrae (She/her)
clare@claremacrae.co.uk
6 August 2020
MUC++

2. 2
About Me
• C++ and Qt developer since 1999
• My mission: Sustainable and efficient testing and refactoring of legacy code
– Co-author of “Approval Tests for C++”
• Consulting & training
– https://claremacrae.co.uk
• All links from this talk via:
– github.com/claremacrae/talks

3. 3
Goal
Share the Power of
Approval Tests

4. 4
Contents
•Intro
• Getting Started
• Legacy Example
• Non-deterministic Output
• Customisability
• Visualising Differences
• Summary

5. 5

6. 6
Quickly and Effectively Testing
Legacy Code

7. 7
What does Legacy Code really mean?
• Dictionary
–“money or property that you receive from someone after they die ”
• Michael Feathers
–“code without unit tests”
• J.B. Rainsberger
–“profitable code that we feel afraid to change.”
• Kate Gregory
–“any code that you want to change, but are afraid to”

8. 8
Typical Scenario
• I've inherited some
legacy code
• It's valuable
• I need to add feature
• Or fix bug
• How can I ever break
out of this loop?
Need to
change
the code
No tests
Not
designed
for testing
Needs
refactoring
to add
tests
Can’t
refactor
without
tests

9. 9
Typical Scenario
•
•
•
•
•
Need to
change
the code
No tests
Not
designed
for testing
Needs
refactoring
to add
tests
Can’t
refactor
without
tests
Topics of
this talk

10. 10
Assumptions
• Value of automated testing
• Evaluate existing tests
– Code coverage tools
– Mutation testing
• No worries about types of tests
– (unit, integration, regression)

11. 11

12. 12

13. 13
“Approval Tests allow you to
verify a chunk of output (such as a file)
in one operation
as opposed to
writing test assertions for each element”

14. 14
Questions?

15. 15
Contents
• Intro
•Getting Started
• Legacy Example
• Non-deterministic Output
• Customisability
• Visualising Differences
• Summary

16. 16
Demo 1:
Hello Approvals

17. 17
Main Methods
• Approvals::verify(std::string, Options)
– 2 other overloads take template types
• Approvals::verifyAll()
– For use with containers
– Writes one element at a time
• All are in namespace ApprovalTests

18. 18
object verify()
writes
"received"
reads
"approved"
Same?
Pass
Fail Reporter
Stages of Approvals::verify()

19. 19
Easy to:
• … Add tests quickly
• … Visualise differences
• … Update output
• Convenience
– Separates test data from test code
– Ignore end-of-line differences

20. 20
Easy setup
• Single header
– https://github.com/approvals/ApprovalTests.cpp/releases
• Finds diff tool automatically
• Sensible filenames automatically
– source_directory/test_filename.test_name.approved.txt
• “It just works”

21. 21
Supported Test Frameworks
• Lots to choose from:
• Get to know chosen one well!

22. 22
Questions?

23. 23
Contents
• Intro
• Getting Started
•Legacy Example
• Non-deterministic Output
• Customisability
• Visualising Differences
• Summary

24. 24
Demo 2:
Legacy Code: Gilded Rose

25. 25
Approvals and Legacy Code
• Before you start refactoring…
• Achieve good coverage
• verifyAllCombinations()
– Not just for legacy code

26. 26
CombinationApprovals::verifyAllCombinations()
• One line from GildedRose .approved.txt file:
(Wibble, 0, 0) => name: Wibble, sellIn: -1, quality: 0
verifyAllCombinations() writes
one set of input values
and then the result of the lambda for
that input set: here, an Item object

27. 27
Lots of String Conversion Options
• String Conversions
– Pass in a std::string
– Write custom operator<<( std::ostream, … )
– Use Approvals::verify( object, lambda )
– Specialize ApprovalTests::StringMaker::toString( … )
– Use TApprovals<YourStringConvertingClass>
• How to Use the Fmt Library To Print Objects
– Use {fmt} library via FmtApprovals::verify()

28. 28
String Design Guidelines
• Objects print their relevant data
• The data is consistent between runs
– (no times, pointers, random)
• The data is easy to read, at a glance:
• Advice: Tips for Designing Strings
[0] = [x: 4 y: 50 width: 100 height: 61]
[1] = [x: 50 y: 5200 width: 400 height: 62]
[2] = [x: 60 y: 3 width: 7 height: 63]
(x,y,width,height) = (4,50,100,61)
(x,y,width,height) = (50,5200,400,62)
(x,y,width,height) = (60,3,7,63)

29. 29
Questions?

30. 30
Contents
• Intro
• Getting Started
• Legacy Example
•Non-deterministic Output
• Customisability
• Visualising Differences
• Summary

31. 31
Demo 3:
Log Files

32. 32
Key Points
• Approvals::verifyExistingFile()
– For when the file is already written
• Scrubbers for non-deterministic output
– More scrubber options coming soon…

33. 33
Questions?

34. 34
Contents
• Intro
• Getting Started
• Legacy Example
• Non-deterministic Output
•Customisability
• Visualising Differences
• Summary

35. 35
Customisation Points Overview

36. 36
Scrubber Namer Writer Comparator
Pass
Fail Reporter
Stages of Approvals::verify()

37. 37
Everything is customisable
• We take great pride in the docs
• All examples generated from compiled, tested code
github.com/approvals/ApprovalTests.cpp/blob/master/doc/ approvaltestscpp.readthedocs.io/en/latest/

38. 38
Questions?

39. 39
Contents
• Intro
• Getting Started
• Legacy Example
• Non-deterministic Output
• Customisability
•Visualising Differences
• Summary

40. 40
Demo 4:
SVG Files

41. 41
Key Points
• Custom Reporters can help understand failures
• Make it easy to visualise differences
• No need to version control .png files

42. 42
See Also ImageApprovals
• github.com/p-podsiadly/ImageApprovals
• Built on ApprovalTests.cpp
• Not yet released

43. 43
Questions?

44. 44
Contents
• Intro
• Getting Started
• Legacy Example
• Non-deterministic Output
• Customisability
• Visualising Differences
•Summary

45. 45
Summary

46. 46
“Approval Tests allow you to
verify a chunk of output (such as a file)
in one operation
as opposed to
writing test assertions for each element”

47. 47
Approval Tests for C++
• Easy to use
• Convenient
• Powerful
• Flexible
• It just works!

48. 48

49. 49

50. 50

51. 51
Quickly & Effectively
Test Legacy C++ Code
with Approval Tests

52. 52
References
• All links from this talk, and more, via:
– github.com/claremacrae/talks
• Sustainable and efficient testing and refactoring of legacy code
• Consulting & Training
– claremacrae.co.uk
– clare@claremacrae.co.uk
Any Questions?

53. 53
Contents
• Intro
• Getting Started
• Legacy Example
• Non-deterministic Output
• Customisability
• Visualising Differences
• Summary
•Appendix: Challenges

54. 54
Appendix: Common Challenges

55. 55
External Resources
• I/O, Networking, System API calls…
• Be creative!
• Can you intercept and log calls?
– And save them as “approved” results?
– Then intercept again during later runs, and compare results?
• You can even use this data as inputs to later tests!
• Excellent example from Angie Jones:
– https://angiejones.tech/verifying-entire-api-responses/

56. 56
Embedded Systems
• Separate as much logic as possible
• Test the logic on desktop machines and CI
• Then if you get a failure on device, you’ve got less to test
• It may become possible to run Approval Tests from cross-built code

57. 57
Output is OS-specific
• Include the OS in the file-names
• So verify() compares against output from same platform
• For example
– TestQtDialog.loginScreen.onMacOSX.approved.png
– TestQtDialog.loginScreen.onWindows.approved.png
– TestQtDialog.loginScreen.onLinux.approved.png
• See Multiple output files per test

58. 58
Approving is fiddly or takes too long
• Approving images?
• Approving hundreds of files?
• Try one of:
– AutoApproveIfMissingReporter
– AutoApproveReporter (and compare the diffs in version control!)