@ RubyKaigi 2017 (19th Sep.)

1. An introduction and future
of Ruby coverage library
RubyKaigi 2017 (19th Sep. )
Yusuke Endoh (@mametter)

2. Yusuke Endoh (@mametter)
• Ruby committer (2008—)
• Full-time Ruby committer (2017—)
• My goal: make Ruby programs robust
– Test coverage, and type system?

5. Method.
Put the cream cheese into the mixing bowl.
Put the sour cream into the mixing bowl.
Put the white sugar into the mixing bowl.
Put the yogrut into the mixing bowl.
Put the unsalted butter into the mixing bowl.
Combine the milk.
Put the cake flour into the mixing bowl.
Combine the corn starch.
Put the brown sugar into the mixing bowl.
Combine the egg whites.
Combine the egg yolks.
Put the lemon juice into the mixing bowl.
Stir for 7 minutes.
Liquify the contents of the mixing bowl.
Pour contents of the mixing bowl into the baking dish
Bake the cake mixture.
Watch the cake mixture until baked.
Serves 4.
Cheese cake in Chef.
Ingredients.
100 g cream cheese
97 g sour cream
107 g yogrut
112 g white sugar
11 g brown sugar
37 g unsalted butter
37 g cake flour
3 g corn starch
3 ml milk
3 egg yolks
3 egg whites
10 ml lemon juice
0 g cake mixture
Cooking time: 80 minutes.
‘d’
‘a’
‘k’
‘p’
11*3*3=99 ‘c’
37*3=111 ‘o’
¥n
‘o’
Push characters
‘¥n’ ‘d’ ‘a’ ‘p’ ‘k’ ‘o’ ‘o’ ‘c’
into the stack
Convert them to a string
and “serve” it.
data
code

6. Esoteric Recipe
• Polyglot of Chef and
real recipe
– Japanese version
https://cookpad.com/
recipe/4649810
– English version
https://cookpad.com/us/
recipes/3335222

7. My main contributions for Ruby
• Implementation of some features:
keyword arguments, deadlock detection, etc.
• Release management for Ruby 1.9.2 and 2.0.0
• Optcarrot: A NES emulator for Ruby3x3
benchmark
• Enhancement of
the test suite of Ruby
• coverage.so: the core library
for coverage measurement
’06B ’07A ’07B ’08A
60
70
80
90
100
coverage(%)
70%
85%
line coverage

8. Today’s theme
• An introduction of test coverage
• An improvement plan of Ruby’s
coverage measurement feature
towards 2.5

9. Survey [1/3]
• Q. Do you use Ruby/RoR in production?
– Raise your hand, please!

10. Survey [2/3]
• Q. In those, do you test your code?

11. Survey [3/3]
• Q. In those, do you use “coverage”?
– Do you check the result of SimpleCov?

12. Agenda
• What is coverage
• How to understand and use coverage
• The current status of Ruby coverage
feature
• The future plan of Ruby coverage
feature
• Conclusion

13. Agenda
☞ What is coverage
• How to understand and use coverage
• The current status of Ruby coverage
feature
• The future plan of Ruby coverage
feature
• Conclusion

14. What is coverage?
• A measure of “goodness” of a test suite
– Also called “test coverage” or “code coverage”
• Allows you:
– Find untested code
– Decide whether your test suite is good enough
or not yet
• (This is arguable, but I think we can use it as an
advice)
• Types of coverage
– Function coverage, line coverage, branch
coverage, …

15. Function coverage
• How many functions are executed by
the tests
# code
def foo; …; end # ✓
def bar; …; end # ✗
def baz; …; end # ✓
# test
foo
baz
2/3
(67%)
• Advantage
• Easy to understand
• Easy to visualize
• Disadvantage
• Too weak as a measure

16. Line coverage
• How many lines are executed
# code
def foo(x) # ✓
if x == 0 # ✓
p :foo # ✗
else
p :bar # ✓
end
end
# test
foo(1)
3/4
(75%)
Non-significant line is
ignored
• Advantage
• Easy to understand
• Easy to visualize
• Disadvantage
• Still weak as a measure
• foo() if x == 0

17. Branch coverage
• How many branches are taken true
and false
# code
def foo(x)
p :foo if x == 0 # ✓
p :bar if x < 2 # ✗
end
# test
foo(0)
foo(1)
1/2
(50%)
• Advantage
• Relatively exhaustive
• Disadvantage
• Difficult to visualize
true-case and false-case are
both executed

18. Coverage types
Coverage type Easy to
understand/
visualize
Exhaustive
Function
coverage
○ ✕
Line
coverage
○ △
Branch
coverage
△ ○
Currently, Ruby supports only line coverage

19. Other types of coverage
• Condition coverage
– How many conditions
(not branches) are taken
both true and false
• Path coverage
– How many paths are executed
– Combinatorial explosion
• Other advanced ones
– Data-flow coverage
– MC/DC coverage
if a && b
branch
conditioncondition

20. Trivia
• “C0/C1/C2 coverages” have difference
meanings to different people
– C0 coverage = line coverage
– C1 coverage = branch coverage or path
coverage?
– C2 coverage = condition coverage or
path coverage?

21. Coverage and Ruby
• In Ruby, Coverage is crucial!
– A test is the only way to ensure quality
– Coverage is important to measure test goodness
• Considering it, coverage is not used so
much…
– Coverage is not well-known?
– It is not well-known how to use coverage?
– Ruby’s coverage measurement feature is not
enough?
• I’d like to improve the situation with this talk

22. Agenda
• What is coverage
☞ How to understand and use
coverage
• The current status of Ruby coverage
feature
• The future plan of Ruby coverage
feature
• Conclusion

23. What is a good test suite?
• Covers the code
– Coverage measures this
• Also covers the design of program
– Coverage does not measure this directly

24. How to understand coverage
• Coverage is just a measure
• Coverage is not a goal

25. If coverage XX% is required as a
goal…
• Developers will
1. Pick untested code that looks easiest to
cover
2. Write a trivial test just for the code
3. Iterate this procedure until XX% is achieved
• It will result in trivial, not-so-good test
suite
– It may be exhaustive for the code itself, but
– It won't be exhaustive for the design

26. A good way to improve
coverage
• Developers should
1. Look at untested code
2. Consider what “test design” is insufficient
3. Write them
– In consequence of them, the untested code
is executed
• It will result in good test suite
– It will be exhaustive not only for the code
but also for the design

27. How many % is
needed/enough?
• It depends upon the module being tested
– Aim 100% for a significant module (e.g., injure
someone)
– Don't worry too much for a non-significant
module
• It also depends upon cost performance
– For non-significant module, write a test only if it
is not so hard
• Again: Coverage is not a goal

28. Agenda
• What is coverage
• How to understand and use coverage
☞ The current status of Ruby
coverage feature
• The future plan of Ruby coverage
feature
• Conclusion

29. Ruby's coverage ecosystem
• SimpleCov
• coverage.so
• Concov

30. SimpleCov
• A wrapper library for coverage.so
• Visualization with HTML
• Useful features: merging, filtering, for Rails app
• Author: Christoph Olszowka (@colszowka)

31. Usage of SimpleCov
• Write this at the top of
test/test_helper.rb
• Run the test suite
• coverage/index.html will be produced
– Note: SimpleCov cannot measure already-
loaded files before SimpleCov.start
require "simplecov"
SimpleCov.start

32. coverage.so
• The only implementation of coverage
measurement for Ruby 1.9+
• SimpleCov is a wrapper for
coverage.so
• Author: me

33. Basic usage
# test.rb
require "coverage"
Coverage.start
load "target.rb"
p Coverage.result
#=> {"target.rb"=>
# [nil,nil,1,1,1,nil,
# 0,nil,nil,nil,1]}
Start measuring coverage
Load the target file
Stop measuring
and get the result
Coverage data

34. Coverage data
# target.rb
def foo(x)
if x == 0
p 0
else
p 1
end
end
foo(1)
[nil,
nil
1
1
0
nil
1
nil
nil
nil
1]
nil:
Non-significant line
Number:
Count executed
Untested line!

35. Method definition is code in
Ruby
# target.rb
def foo(x)
if x == 0
p 0
else
p 1
end
end
[nil,
nil
1
0
0
nil
0
nil
nil]
Method definition is
counted as an
execution
(It is not a count of
method invocation!)

36. I regret the design of
coverage.so
• Support only line coverage
• Excuse: I introduced it just for test
enhancement of Ruby itself
– I didn't deliberate the API for external
project…
• I have wanted to make the next version
better
ext/coverage/coverage.c:
69 /* Coverage provides coverage measurement feature for Ruby.
70 * This feature is experimental, so these APIs may be changed in future.
71 *

37. Concov
• CONtinuous COVerage
– Detects temporal change (degradation) of
coverage
• Developed for monitoring Ruby's coverage
• Author: me
• Presented at RubyKaigi 2009, and then…
– It has not been used by everyone (including me)
– It was based on Ramaze (old web framework)!

38. Concov reveals reality of Ruby dev.
（Enumerable#join, 2009/07/07, M***）
New feature
introduced
with no tests!

39. Concov reveals reality of Ruby dev.
（File#size, 2009/02/25, M***）

40. Concov reveals reality of Ruby dev.
（Etc::Passwd.each, 2009/02/19, N*****）

41. Concov reveals reality of Ruby dev.
（Dir.home, 2009/02/03, N*****）

42. Concov reveals reality of Ruby dev.
（Array#sort_by!, 2009/02/03, M***）

43. Concov reveals reality of Ruby dev.
（rb_to_float, 2008/12/30, M***）

44. Coverage ecosystem for other
languages
• C/C++: GCOV/LCOV
– Integrated with gcc:
gcc -coverage target.c
• Java: A lot of tools
– Cobertura, Emma,
Clover, JaCoCo
– Integrated with CI tools
and/or IDEs
• JavaScript: Istanbul
Jenkins Cobertura plugin
LCOV result

45. Agenda
• What is coverage
• How to understand and use coverage
• The current status of Ruby coverage
feature
☞ The future plan of Ruby coverage
feature
• Conclusion

46. A plan towards Ruby 2.5
• Support function and branch coverage
– There have been multiple requests and
some PoC patches…
• To make the API better, any
comments are welcome
– https://bugs.ruby-lang.org/issues/13901

47. API: to start measuring
# compatible layer
Coverage.start
Coverage.result
#=> {"file.rb"=>[nil, 1, 0, …], … }
# new API
Coverage.start(lines: true)
Coverage.result
#=> {"file.rb" => { :lines => [nil, 1, 0, …] } }

48. API: to start other types of
coverage
# enable branch and function coverage
Coverage.start(lines:true,
branches:true,
methods:true)
Coverage.result
#=> {"file.rb" => { :lines => [nil, 1, 0, …],
# :branches => {…},
# :methods => {…} } }
# shorthand
Coverage.start(:all)

49. Coverage.result for branch
coverage
{"target1.rb"=>
{:lines=>[…],
:branches=>{
[:if, 0, 2]=>{
[:then, 1, 3]=>2,
[:else, 2, 5]=>1
}
},
:methods=>{
[:test_if, 1]=>3
}}}
# target1.rb
1: def test_if(x)
2: if x == 0
3: p "x == 0"
4: else
5: p "x != 0"
6: end
7: end
8:
9: test_if(0)
10: test_if(0)
11: test_if(1)
From if at Line 2
Jumped to
then clause
at Line 3
twice
Jumped to
else clause
at Line 5
once

50. Coverage.result for branch
coverage
{"target2.rb"=>
{:lines=>[1, 1, 1, nil, nil, 1],
:branches=>
{[:if, 0, 2]=>{
[:then, 1, 2]=>1,
[:else, 2, 2]=>0},
[:if, 3, 3]=>{
[:then, 4, 3]=>0,
[:else, 5, 3]=>1}},
:methods=>{
[:test_if, 1]=>3
}}}
# target2.rb
1: def test_if_oneline(x)
2: p "x == 0" if x == 0
3: p "x != 0" if x != 0
4: end
5:
6: test_if_oneline(0)
Line coverage 100%
Branch coverage tells you
there are untested cases

51. Discussion of API design
• 100% compatible
• [<label>, <numbering>, <line-no>]
– e.g., [:if, 0, 1], [:while, 1, 1], [:case, 2, 1]
– <numbering> is a unique ID to avoid conflicts for the
case where there are multiple branches in one line
• LCOV-style
– Other candidates:
• [<label>, <line-no>, <column-no>]
– How to handle TAB character
• [<label>, <offset from file head>]
– Looks good, but hard to implement (I'll try later)

52. Overhead of coverage
measurement
(just preliminary experiment)
# Example 2
1: foo()
2: foo()
…
99: foo()
100: foo()
Benchmark w/o cov. w/ cov. Overhead
Example 1 0.322 μs 6.21 μs x19.3
Example 2 1.55 μs 7.16 μs x4.61
make test-all 485 s 550 s x1.13
# Example 1
1: x = 1
2: x = 1
…
99: x = 1
100: x = 1

53. Demo
• Applied the new coverage.so to Ruby
• Integrated with C code coverage by
GCOV and Visualized by LCOV
Ruby code
in stdlib
make exam with
gcc -coverage
make exam
COVERAGE=1
test-
coverage
.dat
*.gcda gcov
my script
run test
C code
of MRI
cov. datasource aggregate
lcov HTML

54. Jenkins Cobertura Plugin

55. Agenda
• What is coverage
• How to understand and use coverage
• The current status of Ruby coverage
feature
• The future plan of Ruby coverage
feature
☞ Conclusion

56. Acknowledgement
• @_ko1
• @t_wada
• @kazu_cocoa
• @moro
• @makimoto
• @dev_shia
• @tanaka_akr
• @nalsh
• @spikeolaf
• @k_tsj

57. Conclusion
• What is coverage, how important in Ruby,
and how to understand coverage
• The current status of Ruby's coverage
measurement and ecosystem
• A plan towards Ruby 2.5 and preliminary
demo
– Any comments are welcome!
– https://bugs.ruby-lang.org/issues/13901

58. Future work
• Determine the API
• define_method as method coverage
• &. as branch coverage
• Callsite coverage
• Block coverage
obj.foo.bar
ary.map { …… }