More Related Content Similar to Property-based Testing and Generators (Lua) (20) More from Sumant Tambe (11) Property-based Testing and Generators (Lua)2. Agenda
• Property-based Testing
• The need for generators
• Design/Implementation of the generator library
• Using mathematics for API design
• Value constraints in type definitions
• Reactive generators
• Generators at compile-time (type generators)
8/25/2015 © 2015 RTI 2
3. Why should you care?
• If you write software for living and if you are
serious about any of the following
– Bug-free code
– Productivity
• Modularity
• Reusability
• Extensibility
• Maintainability
– Performance/latency
• Multi-core scalability
– and having fun while doing that!
8/25/2015 © 2015 RTI 3
5. What’s a Property
• Reversing a linked-list twice has no effect
– reverse(reverse(list)) == list
• Compression can make things smaller
– compress(input).size <= input.size
• Round-trip serialization/deserialization has no
effect
– deserialize(serialize(input)) == input
8/25/2015 © 2015 RTI 5
6. Property-based Testing
• Complementary to traditional example-based testing
• Specify post-conditions that must hold no matter
what
• Encourages the programmer to think harder about
the code
• More declarative
• Need data generators to produce inputs
• Free reproducers!
– Good property-based testing frameworks shrink inputs to
minimal automatically
• Famous: Haskell’s QuickCheck, Scala’s ScalaTest
8/25/2015 © 2015 RTI 6
7. A property-test in RefleX
8/25/2015 © 2015 RTI 7
template <class T>
void test_roundtrip_property(const T & in)
{
T out; // empty
DDS::DynamicData dd(...); // empty
reflex::update_dynamicdata(dd, in);
reflex::read_dynamicdata(out, dd)
assert(in == out);
}
• What’s the input data?
• What are the input types?
9. Data Generators
8/25/2015 © 2015 RTI 9
• Simplest data generator = math.random
math.randomseed(os.time())
local r = math.random(6)
print(r) -- one of 1, 2, 3, 4, 5, 6
r = math.random(20, 25)
print(r) -- one of 20, 21, 22, 23, 24, 25
10. Data Generators
8/25/2015 © 2015 RTI 10
• A boolean generator
function boolGen()
return math.random(2) > 1
end
• An uppercase character generator
function upperCaseGen() -- A=65, Z=90
return string.char(math.random(65, 90))
end
• An int16 generator
function int16Gen()
return math.random(MIN_INT16, MAX_INT16)
end
11. Data Generators
8/25/2015 © 2015 RTI 11
• A range generator
function rangeGen(lo, hi)
return math.random(lo, hi)
end
• Inconvenient because args must be passed every time
Use state!
12. Closure Recap
8/25/2015 © 2015 RTI 12
• A simple closure
local f = function ()
return boolGen()
end
print(f()) -- true or false
• Another closure
local a = 20
local f = function ()
print(a)
end
f() -- 20
a = 30
f() -- 30
13. Closures capture state
8/25/2015 © 2015 RTI 13
• Back to the range generator
function rangeGen(lo, hi)
return function()
return math.random(lo, hi)
end
end
local f = rangeGen(20, 25)
print(f()) -- one of 20, 21, 22, 23, 24, 25
14. Closures capture state
8/25/2015 © 2015 RTI 14
• The oneOf generator
function oneOfGen(array)
local len = #array –- get length of the array
return function()
return array[math.random(1, len)]
end
end
local days = { “Sun”, “Mon”, “Tue”, “Wed”, “Thu”, “Fri”, “Sat” }
local f = oneOfGen(days)
print(f()) -- one of Sun, Mon, Tue, Wed, Thu, Fri, Sat
15. 8/25/2015 © 2015 RTI 15
“Closures are poor man's
objects; Objects are poor
man's closure”
-- The venerable master Qc Na
Source Google images
16. Closure vs Object
• Is function a sufficient abstraction for any
generator?
function () {
return blah;
}
• Answer depends upon your language
– No. In Lua, C++, …
– Yes. In Haskell
8/25/2015 © 2015 RTI 16
17. The Generator Library
8/25/2015 17
local Gen = require(“generator”)
local rangeGen = Gen.rangeGen(20,25)
for i=1, 5 do
print(rangeGen:generate()) –- 5 values in range(20, 25)
end
Github: https://github.com/rticommunity/rticonnextdds-ddsl
local stepperGen = Gen.stepperGen(1, 10)
for i=1, 5 do
print(stepperGen:generate()) –- 1, 2, 3, 4, 5
end
local infiniteGen = Gen.stepperGen(1, 10, 1, true)
while true do
print(infiniteGen:generate()) –- 1, 2, 3, 4, ...
end
18. The Generator Library
8/25/2015 18
local Gen = require(“generator”)
local charGen = Gen.uppercaseGen()
local maxLen = 64
local strGen = Gen.stringGen(maxLen, charGen)
for i=1, 5 do
print(strGen:generate()) -- 5 uppercase strings upto size 64
end
Github: https://github.com/rticommunity/rticonnextdds-ddsl
sequenceGen is quite similar
19. The Generator library (batteries included)
8/25/2015 © 2015 RTI 19
boolGen posFloatGen emptyGen
charGen posDoubleGen singleGen
wcharGen floatGen constantGen
octetGen doubleGen oneOfGen
shortGen lowercaseGen inOrderGen
int16Gen uppercaseGen stepperGen
int32Gen alphaGen rangeGen
int64Gen alphaNumGen seqGen
uint16Gen printableGen arrayGen
uint32Gen stringGen aggregateGen
uint64Gen nonEmptyStringGen enumGen
20. The Generator “Class”
• Lua does not have “classes” but that’s not important
8/25/2015 © 2015 RTI 20
local Generator = {} –- an object that will serve as a class
function Generator:new(generateImpl)
local o = {}
o.genImpl = generateImpl -- A method assignment
-- some additional metatable manipulation here
return o
end
function Generator:generate()
return self.genImpl()
end
function rangeGen(lo, hi)
return Generator:new(function ()
return math.random(lo, hi)
end)
end
22. Composing Generators
• Producing more complex Generators from one
or more simpler Generators
8/25/2015 © 2015 RTI 22
Generator
map
reduce
append
where
zipMany
concatMap
take
scan
23. Mapping over a generator
8/25/2015 © 2015 RTI 23
local days = { “Sun”, “Mon”, “Tue”, “Wed”, “Thu”, “Fri”, “Sat” }
local fiboGen = Gen.fibonacciGen()
-- 0 1 1 2 3 5 8 ...
local plus1Gen = fiboGen:map(function (i) return i+1 end)
-- 1 2 2 3 4 6 9 ...
local dayGen = plus1Gen:map(function (j) return days[j] end)
-- Sun, Mon, Mon, Tue, Wed, Fri, nil, ...
for i=1, 6 do
print(dayGen:generate()) -- Sun, Mon, Mon, Tue, Wed, Fri
end
24. Zipping generators
8/25/2015 © 2015 RTI 24
local xGen = Gen.stepperGen(0,100)
local yGen = Gen.stepperGen(0,200,2)
local pointGen =
Gen.zipMany(xGen, yGen, function (x, y)
return { x, y }
end)
for i=1, 5 do
print(pointGen:generate()) –- {0,0}, {1,2}, {3,4}, ...
end
25. Zipping generators
8/25/2015 © 2015 RTI 25
local xGen = Gen.stepperGen(0,100)
local yGen = Gen.stepperGen(0,200,2)
local colorGen = Gen.oneOfGen({ “RED”, “GREEN”, “BLUE” })
local sizeGen = Gen.constantGen(30)
local shapeGen =
Gen.zipMany(xGen, yGen, colorGen, sizeGen,
function (x, y, color, size)
return { x = x,
y = y,
color = color,
shapesize = size }
end)
• The mandatory Shapes example
26. Using Data Domain Specific Language (DDSL)
8/25/2015 © 2015 RTI 26
local ShapeType = xtypes.struct{
ShapeType = {
{ x = { xtypes.long } },
{ y = { xtypes.long } },
{ shapesize = { xtypes.long } },
{ color = { xtypes.string(128), xtypes.Key } },
}
}
local shapeGen = Gen.aggregateGen(ShapeType)
for i=1, 5 do
print(shapeGen:generate()) –- output next slide
end
• Use Lua DDSL to define types
28. Use a Collection of Generators
8/25/2015 © 2015 RTI 28
local ShapeType = xtypes.struct{
ShapeType = {
{ x = { xtypes.long } },
{ y = { xtypes.long } },
{ shapesize = { xtypes.long } },
{ color = { xtypes.string(128), xtypes.Key } },
}
}
local shapeGenLib = {}
shapeGenLib.x = Gen.stepperGen(0, 100)
shapeGenLib.y = Gen.stepperGen(0, 200, 2)
shapeGenLib.color = Gen.oneOf({ "RED", "GREEN", "BLUE" })
shapeGenLib.shapesize = Gen.rangeGen(20, 30)
local shapeGen = Gen.aggregateGen(ShapeType, shapeGenLib)
for i=1, 5 do
print(shapeGen:generate()) –- output next slide
end
30. In not so distant future…
8/25/2015 © 2015 RTI 30
local ShapeType = xtypes.struct{
ShapeType = {
{ x = { xtypes.long,
xtypes.constraint{ “[ x | x <- stepperGen(0,100) ]” }}},
{ y = { xtypes.long,
xtypes.constraint{ “[ x*x | x <- stepperGen(0,200,2) ]” }}},
{ size = { xtypes.long,
xtypes.constraint{ “[ x | x <- rangeGen(20,30) ]” }}},
{ color= { xtypes.string(128),
xtypes.constraint{ “[ x | x <- constantGen('BLUE') ]” }}}
}
}
• Direct applicability in
• MEP Message Emulation Processes (i.e., generate data)
• MCT Generator expressions may be specified directly in the UI
32. Generators
• General design principal
– Create a language to solve a problem
– API is the language
– Reuse parsing/compilation of the host language (duh!)
• The “language” of Generators
– Primitive values are basic generators
– Complex values are composite generators
– All APIs result in some generator
• Especially, map, zip, reduce, etc.
– I.e., generators form an algebra
8/25/2015 © 2015 RTI 32
34. 8/25/2015 © 2015 RTI 34
What Math Concepts?
• Algebraic Structures from Category Theory
– Monoid
– Monad
– Functor
– Applicative
– Group
– Endofunctor
• Where to begin?
– Haskell
– But innocent should start with Scala perhaps
35. 8/25/2015 © 2015 RTI 35
Does it have anything to do with DDS?
Yes!
Composable API for DDS exists. It’s Rx4DDS
Also, RPC-over-DDS (futures). Both are humble
beginnings
And remember, sky is the limit
36. Next Items
• Value constraints in type definitions
• Reactive generators
• Generators at compile-time (type generators)
• (Back to) Testing RefleX API properties
8/25/2015 © 2015 RTI 36
37. Constraints in Data Models
• Database Schemas
• XSD Restrictions
8/25/2015 © 2015 RTI 37
CREATE TABLE products (
product_no integer UNIQUE NOT NULL,
name text,
price numeric CHECK (price > 0)
);
<xs:element name="age">
<xs:simpleType>
<xs:restriction base="xs:integer">
<xs:minInclusive value="0"/>
<xs:maxInclusive value="120"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element name="prodid">
<xs:simpleType>
<xs:restriction base="xs:integer">
<xs:pattern value="[0-9][0-9][0-9]"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
38. Constraints in Data Models
• STANAG 4607
– Ground Moving Target Indicator
• Object Constraint Language
– Defined by OMG: part of the UML standard
– Huge!
8/25/2015 © 2015 RTI 38
39. In not so distant future…
8/25/2015 © 2015 RTI 39
local ShapeType = xtypes.struct{
ShapeType = {
{ x = { xtypes.long,
xtypes.constraint{ “[ x | x <- stepperGen(0,100) ]” }}},
{ y = { xtypes.long,
xtypes.constraint{ “[ x*x | x <- stepperGen(0,200,2) ]” }}},
{ size = { xtypes.long,
xtypes.constraint{ “[ x | x <- rangeGen(20,30) ]” }}},
{ color= { xtypes.string(128),
xtypes.constraint{ “[ x | x <- constantGen('BLUE') ]” }}}
}
}
• Direct applicability in
• MEP Message Emulation Processes (i.e., generate data)
• MCT Generator expressions may be specified directly in the UI
40. General Syntax for Data Generation
• Mathematical Set Builder Notation
• Suggested Generator Syntax
– [ x | x <- stepperGen() ]
– [ 2*x | x <- fibonacciGen() ]
– [ “ID” .. i | i <- stepperGen(1, 256) ]
• Translated mechanically to the Generator API
– map, flatMap, etc.
– List Comprehension in disguise
• Python, Haskell has it. Lua does not.
8/25/2015 © 2015 RTI 40
41. Value Dependencies
• List Comprehension alone is not sufficient
• What if data members have value dependencies
– o.y = o.x * o.x
– o.len = length(o.somearray)
– o.min = minimum(o.somearray)
– o.max = maximum(o.somearray)
8/25/2015 © 2015 RTI 41
42. Syntax for Value Dependencies
• Generator Syntax seems to work for value
dependencies too
• Just refer to the generator of the parent attribute
– [ a*a | a <- $(x) ]
– [ len(a) | a <- $(somearray) ]
– [ min(point.x) | point <- t,
t <- $(trajectory) ]
(Assuming trajectory is an array of points )
8/25/2015 © 2015 RTI 42
43. Pull Generators aren’t enough
• Pull Generators don’t handle dependencies
well
– At least not without state
– Requires multiple passes through state for every
generated sample
• Seems expensive
– Dependencies must be “interpreted” for every
sample
• I.e., needs to refer type description to do its job
• Dependencies can’t be “compiled” for performance
– A relatively simple and powerful alternative exists
8/25/2015 © 2015 RTI 43
44. How dependencies break
• The ShapeType generator
asks leaf generators to
produce a new value.
• Dependencies are
immediately lost
8/25/2015 © 2015 RTI 44
X
Y
ShapeType
Y = X2
Circle represents a generator not state
45. Reactive Generators
• Work “just like DDS!”
• Propagate values “downstream”
– Push-based instead of pull-based
• Subject-Observer (pub/sub) pattern
– A single value is propagated to one or more dependent
attributes
• Dependent generators take the “right” action
– Such as doubling the value, counting min, etc.
• map, flatmap, reduce etc. work exactly like
pull generators
• Composable
– Generator expressions can be mechanically translated to
reactive generator API
8/25/2015 © 2015 RTI 45
46. Basics of Reactive Generators
8/25/2015 © 2015 RTI 46
class Generator<T>
{
void listen(callback);
};
Where callback is a closure
callback = function (t) {
// provided by the user
}
You can register multiple callbacks
47. Create Listen Push Dispose
Lifecycle of a reactive (push) generator
n n
Create a
Pull
Generator
Subject
transform,
…
Optional
48. Composing Reactive Generators
• Producing more complex Generators from one
or more simpler Generators
8/25/2015 © 2015 RTI 48
ReactGen
map
reduce
append
where
zipMany
concatMap
take
scan
51. Round-trip property-test for ShapeType
8/25/2015 © 2015 RTI 51
void test_roundtrip_shape(const ShapeType & in)
{
ShapeType out; // empty
DDS::DynamicData dd(...); // empty
reflex::write_dynamicdata(dd, in);
reflex::read_dynamicdata(out, dd)
assert(in == out);
}
• The assertion must hold for all instances.
void test_many_shapes()
{
auto generator = gen::make_aggregate_gen<ShapeType>();
for (int i = 0;i < 100; ++i) {
test_roundtrip_shapetype(gen.generate());
}
}
52. A generic property-test in RefleX
8/25/2015 © 2015 RTI 52
template <class T>
void test_roundtrip_property(const T & in)
{
T out; // empty
DDS::DynamicData dd(...); // empty
reflex::update_dynamicdata(dd, in);
reflex::read_dynamicdata(out, dd)
assert(in == out);
}
• What are the input types?
• Given a type, how to produce data?
54. Random Numbers at Compile-time
• Demo
– In C
– Then in C++
8/25/2015 © 2015 RTI 54
Live code
55. Linear Feedback Shift Register (LFSR)
• Simple random number generator
• 4 bit Fibonacci LFSR
• Feedback polynomial = X4 + X3 + 1
• Taps 4 and 3
8/25/2015 © 2015 RTI 55
56. Linear Feedback Shift Register (LFSR)
• Live code feedback polynomial (16 bit)
– X16 + X14 + X13 + X11 + 1
• Taps: 16, 14, 13, and 11
8/25/2015 © 2015 RTI 56
57. Type Generators Demystified
• Random seed as command line #define
• Compile-time random numbers using LFSR
• C++ meta-programming for type synthesis
• std::string generator for type names and
member names
• RefleX to map types to typecode and dynamic data
8/25/2015 © 2015 RTI 57
58. RefleX Property-test Type Generator
• “Good” Numbers
– 31415 (3 nested structs)
– 32415 (51 nested structs)
– 2626 (12 nested structs)
– 10295 (15 nested structs)
– 21525 (41 nested structs)
– 7937 ( 5 nested structs)
– ...
• “Bad” Numbers
– 2152 (test seg-faults) sizeof(tuple) > 2750 KB!
8/25/2015 © 2015 RTI 58
Github: https://github.com/rticommunity/rticonnextdds-reflex/blob/develop/test/generator