1. Test-Driven
Language Development
Lennart Kats Delft University of Technology
Rob Vermaas LogicBlox
Eelco Visser Delft University of Technology

2. Language Workbenches

3. Testing
“DOES THE TYPE CHECKER CATCH THIS?”

4. Testing
“DOES THIS HYPERLINK POINT TO THE RIGHT PLACE?”

5. Syntax
Semantics IDE

6. parsing
abstract syntax
syntax error marking
Syntax
Semantics IDE
type checking errors/warnings
compilation reference resolving
interpretation content completion
execution refactoring
views

7. How to test
language services?

8. IMPLEMENT
FEATURE, TRY WITH DOESN’T WORK
EXAMPLE
DISCARD
CLOSE ENOUGH. EXAMPLE

9. How can we
systematically test
language definitions?

10. General-Purpose
Testing Tools?

11. A Test Input
module Example
function foo() {
bar();
}
function bar() {
}

12. Another Test Input
module Example
function foo() {
foo();
}
function bar() {
}

13. A Negative Test Case
module Example
function foo() {
notfoo();
}
function bar() {
}

14. Automated Testing
Infrastructure
Parser
Test Test Compiler
input script
IDE

15. Automated Testing
Infrastructure
Parser
Test Test Compiler
input script
IDE
+ simple
– language-specific script
– limited expressiveness
– boilerplate in tests
– ...

16. Can we design a
general, reusable
solution for language testing?

17. Yes We Can
Generic test specification language
+
Parametrization

18. Language-Parametric
Testing Language (LPTL)
module my-tests
language mobl
test Cannot assign an integer to a string [[
module Example
module Example
<test input>
function test() {
var s s : String = 1;
var : String = 1;
}}
]] 1 error

19. LPTL Opportunities
Expressiveness
Tool support

20. LPTL Opportunities (1)
Module system
+
GUI Test Runner

21. LPTL Opportunities (2)
IDE support
for test
specifications

22. LPTL Opportunities (3)
Immediate
test
evaluation

23. LPTL Opportunities (4)
IDE support
for test
inputs

24. LPTL Opportunities (5)
setup [[
module Example
imports stuff
Reduced function test() {
[[...]]
boilerplate }
]]
test Cannot assign ... [[
var s : String = 1;
]] 1 error

25. LPTL Opportunities (6)
1 error
2 warnings
/expected here/
parse fails
Wide set of
test conditions complete ... to ...
resolve ... to ...
refactor ... to ...
build ...
run ...

26. Testing Syntax (1)
test Proper declaration [[
var s : String = "a";
]] parse
test Java-like declaration [[
String s = "a";
]] parse fails

27. Testing Syntax (2)
test Proper declaration [[
var s : String = "a";
]] parse to VarDecl("s", _)
test Precedence [[
3+1*2
]] parse to [[
3 + (1 * 2)
]]

28. Testing Error Markers
test Variable declaration [[
var s : String = "a";
]] 0 errors
test Bad variable declaration [[
var s : String = 25;
]] 1 error /wrong type/

29. Testing References
test [[
module Example
function foo() {
bar();
}
function bar() {
}
]] resolve ... to ...

30. Testing References
test [[
module Example
function foo() {
bar();
[[bar]]();
}
function [[bar]]() {
bar() {
}
]] resolve ... to ...
#1 to #2

31. Testing Code
Generation..?

32. Testing Code
Generation..?
test [[
function foo() {
return 3;
}
]] build generate-javascript to [[
[[bar]]();
function bar() {
}
]]

33. Testing Code
Generation..?
test [[
function foo() {
return 3;
}
to [[
]] build generate-javascript to [[
var foo = function() {
[[bar]]();
return 3;
};
function bar() {
]]
}
]]

34. Testing Execution
setup [[
application execution
function main() : Num {
// init
[[...]]
}
]]
test Arithmetic [[
return 1 + 1;
]] run run-test to 2

35. Implementation
Spoofax Testing Language
(spoofax.org)

36. Implementation
Techniques
Language embedding
Dynamic instantiation of language services

37. Language Embedding:
Syntax
module tests
language mobl
test Java-like declaration [[
String s = "a";
]] parse fails

38. Language Embedding:
Syntax
1 module tests
language mobl
test Java-like declaration [[
String s = "a";
]] parse fails

39. Language Embedding:
Syntax
1 module tests
2 language mobl
test Java-like declaration [[
String s = "a";
]] parse fails

40. Language Embedding:
Syntax
1 module tests
2 language mobl
test Java-like declaration [[
3 ]]String sfails
parse
= "a";

41. Language Embedding:
Semantics & IDE (1)
test A function call [[
function foo() {
}
fo|
]]
Content complete

42. Language Embedding:
Semantics & IDE (1)
test A function call [[
function foo() {
Mobl
}
fo|
]]
Content complete

43. Language Embedding:
Semantics & IDE (2)
test A function call [[
function foo() {
Mobl
}
notfoo();
]]
No condition;
error not expected

44. Dynamic Language Service
Instantiation
Mobl

45. Dynamic Language Service
Instantiation
Language consists of separate services
Services have functional interfaces

46. Reflection
+ simple
+ no scripting required
+ IDE helps avoid errors
+ little boilerplate code
+ expressiveness

47. WRITE 1 TEST
IMPLEMENT FEATURE
TESTS FAIL
TESTS SUCCEED!

48. ADDITIONAL
SLIDES

49. Self-Application
language Spoofax-Testing
test Testing testing [[[
language mobl
test Testing [[
module y
]]
]]]

50. Dynamic Language Service
Instantiation
Test
“mobl” input
Language Mobl
Parser
registry language
Completion
service

51. Related:
Automatic Test Generation
• Generate tests from grammar
• Requires oracle
• Complementary to our approach

52. The Spoofax Language
Workbench [OOPSLA 2010]
• Integrated environment for language
definition
• Define syntax, semantics, IDE
• Based on Eclipse
www.spoofax.org