2. Lua in Testing
Paul Du Bois
Double Fine Productions
Thursday, March 18, 2010
3. About us
» Paul: Senior Programmer
» Bert: Software Test Engineer
» RoBert:
Robot brainchild
Automated tester
Thursday, March 18, 2010
4. 120-second pitch
» Unit testing is well understood
» “But how do we test game logic…”
» We implemented a prototype
Thursday, March 18, 2010
5. 120-second pitch
The result
» Framework for writing very high-level
code to exercise game
» Runs on any idle devkit
» Used directly by
❖ Test
❖ Gameplay, System programmers
❖ Designers
Thursday, March 18, 2010
6. 120-second pitch
The result
» Everyone at Double Fine loves RoBert
(even though it gives them bugs)
» Game would be significantly smaller
without it
» Permanent part of our process
Thursday, March 18, 2010
15. Reflection
Entity* e = ...;
Co* co = e->GetComponent(“CoDamageable”);
Class* cl = co->GetClass();
Attribute* a = cl->GetAttribute(“Health”);
Any any = a->GetValueAny(co);
float health = any.GetAs<float>();
Thursday, March 18, 2010
16. Lua implementation
» Class system with inheritance
» Wrapper for Entity instances
» Smidgen of coroutines
» No debugger ☹
Thursday, March 18, 2010
17. Lua and Tests
» Class system with simple inheritance
» See Programming in Lua § 13.4.1
class = {
Run = <function>,
Teardown = <function>,
bFastSpeed = true
}
instance = {
<mt> = { __index = class }
}
Thursday, March 18, 2010
18. Lua and Entities
player.CoDamageable.Health
Entity* e = ...;
Co* co = e->GetComponent(“CoDamageable”);
Class* cl = co->GetClass();
Attribute* a = cl->GetAttribute(“Health”);
Any any = a->GetValueAny(co);
float health = any.GetAs<float>();
Thursday, March 18, 2010
19. Lua and Entities
player.CoDamageable.Health
Lua table Entity A02...
CoScript
_cpp_ref: 0x1ea7f000 Table ref:
<opaque>
Thursday, March 18, 2010
20. Lua and Entities
player.CoDamageable.Health
player -> { __cpp_ref = 0x1ea7f00,
<mt> = ent_mt }
Thursday, March 18, 2010
24. Lua and Entities
» Entity may be deleted at will
» A destructor clears out out _cpp_ref
Lua table Entity A02...
CoScript
_cpp_ref: false Table ref:
<opaque>
Thursday, March 18, 2010
25. Lua and Tests
function Class:waitForActiveLine(self, ent)
while true do
self:sleep(0)
if not ent.CoVoice.HasActiveVoiceLine then
return
end
end
end
Thursday, March 18, 2010
26. Lua and Tests
» Test can function as input source
» Mutate a state block
» Use blocking calls to make API
convenient
» Manipulate joystick in “world coordinates”
Thursday, March 18, 2010
27. Example: providing input
-- push some button for time t1
self.input.buttons[btn] = true
self:sleep(t1)
self.input.buttons[btn] = false
-- move towards world-space pos x,y,z
self.input.joy1 = test.GetInputDir(x,y,z)
Thursday, March 18, 2010
28. Example: simple mission
function Class:Run()
function fightSpiders(entity)
self:attackSmallSpiders()
self:killHealerSpiders()
self:basicFightFunc(entity)
self:waypointAttack(
"P1_050_1", "Monster", 40, fightSpiders)
self:attackEntitiesOfTypeInRadius(
"Monster", 50, fightSpiders)
self:attackBarrier("A_WebBarrierA", 100)
self:waypointTo{"P1_050_ChromeWidowLair"}
Thursday, March 18, 2010
29. Example: reproduce a bug
function Class:Run()
function waitForActiveLine()
while true do
self:sleep(0)
if not player.CoVoice.HasActiveVoiceLine then
return
streams = sound.GetNumStreams()
while true do
game.SayLine( 'MIIN001ROAD' )
game.SayLine( 'MIIN001ROAD' )
waitForActiveLine()
if sound.GetNumStreams() > streams then
self:sleep(1)
self:ASSERT(sound.GetNumStreams() <= streams)
Thursday, March 18, 2010
30. Role of the human
» Bot farm means more time writing bugs
» Half time writing new tests, updating old
tests, writing/regressing bugs
» Half time on infrastructure work
Thursday, March 18, 2010
32. Uses and Examples
» Does the game start/quit/leak memory?
» Do these entities spawn properly?
» Can this unit pathfind properly?
Thursday, March 18, 2010
33. Uses and Examples
» Can player interact with this unit?
» Can bot fly across the world without the
game crashing?
» Can I go to each mission contact and talk
to them?
Thursday, March 18, 2010
34. Uses and Examples
» Can I complete each contact's mission?
» Can I successfully fail the mission?
» Multiplayer!
Thursday, March 18, 2010
35. Diagnostic “tests”
» What is our memory usage as a function
of time?
» How does it change from build to build?
» Where are the danger spots?
Thursday, March 18, 2010
37. Diagnostic “tests”
» What does our performance look like as a
function of time?
» How does it change from build to build?
» What is it like in certain troublesome
scenes?
Thursday, March 18, 2010
39. Non-test tests
» Reproduce tricky bugs
» Typically involve feedback between test
and programming
» Guess at the fail case, try to exercise it
Thursday, March 18, 2010
40. Use by programmers
» Pre-checkin verification
» Soak testing for risky changes
» Can use Debug builds!
Thursday, March 18, 2010
42. Use by designers
» Write a series of balance “tests”
» Throw permutations of unit groups at each
other
» Print out results in a structured fashion
» Examined by a human for unexpected
results
Thursday, March 18, 2010
43. Use by artists
» They don’t run it themselves…
» …but they do see it running
» See parts of the game they normally
wouldn’t
» Notice things that don’t look right
Thursday, March 18, 2010
44. Takeaway
Why was this project successful?
» Rapid development: doing a prototype
was cheap and low-risk
» Lua doesn’t crash the game
» Coroutines and higher-order functions let
you write powerful, concise code
» Performance not an issue
Thursday, March 18, 2010
46. ‘’
Questions?
dubois@doublefine.com
Thursday, March 18, 2010
47. Number of bugs found
Date through bot total
2006-05-01
2006-09-01
2007-01-01
2007-05-01
2007-09-01
2008-01-01
2008-05-01
2008-09-01
2009-01-01
(to date) 2009-05-01
(projected) 2009-05-01
0 750 1,500 2,250 3,000
Thursday, March 18, 2010
48. Number of bugs found
» Raw bug count undersells RoBert
» Query didn’t catch all RoBert bugs
» Not all problems found get entered
Thursday, March 18, 2010
49. Types of bugs found
» Almost all crashes and asserts
» Middleware bugs
» Logic bugs manifest as “Bot stuck in
mission” failures
» Complementary to bugs found by human
testers
Thursday, March 18, 2010
50. What we test
» Most tests merely exercise behavior
» Unsuccessful at verifying behavior
» Correctness of test is an issue
Thursday, March 18, 2010
51. What we don’t test
» No testing of visuals
» Limited testing of performance
» Specific behaviors, game logic
Thursday, March 18, 2010
52. Problems and future work
» Big tests can take hours to complete
» Still a lot of human-required work
» Bot cheats a lot
Thursday, March 18, 2010
53. Our takeaway
» Doesn’t replace a test team
» Does take tedious work off their plate
» Hillclimbing development strategy worked
well
» Very curious what others are doing!
Thursday, March 18, 2010
