http://www.etaps.org/2013/fase13 I talked about this slide at FASE 2013.

1. FASE 2013 in Rome
22/03/2013
Waseda University Kazunori Sakamoto
Google Japan Inc. Tomohiro Kaizu
Google Japan Inc. Daigo Hamura
Waseda University Hironori Washizaki
Waseda University Yoshiaki Fukazawa

2. Introduction
• Test oracles for web apps in integration testing
– How should we judge test cases are passed or not ?
e.g.) Title of web page? Visibility of specific elements?
Contributions
1. Defining fragments should be checked
2. New coverage for html templates
3. Tool for generating skeleton test code
4. Evaluation results of coverage and tool
3/22/2013 FASE 2013 in Rome 2

3. Template engine
• Combine fixed and variable fragments in HTML
• Many web frameworks employ template engines
– Closure Templates, Razor, Smarty, Django, eRuby
{template .addition} Calculation page
<p> + = Calc
{$l}+{$r}={$ans}</p>
Result page Result page
{/template}
Closure Templates (JavaScript) 1+2=3 2+3=5
Contests depend
• Should check variable fragments on input values
– Expressions embedded in templates
– Faults in fixed fragments can be detected easily (*
*) Ali Mesbah and Arie van Deursen, “Invariant-based automatic testing of
AJAX user interfaces,” ICSE 2009.
3/22/2013 FASE 2013 in Rome 3

4. WB unit / BB integration testing
WB (White-box) unit testing BB (Black-box) integration testing
Client side Check working Client side
on Android OS
Web browser or iOS Web browser
Sele
Platform (OS) nium Platform (OS)
HTML Selenium HTML
JUnit test
HTML template HTML template
Junit test
Server program Server program
Check working
DB Web Auth on real env DB Web Auth
server server server server server server
Server side Server side
3/22/2013 FASE 2013 in Rome 4

5. WB / BB Testing example
{template .addition}
• Testing target <p>{$l}+{$r}={$ans}</p>
– Result page {/template}
• White-box unit testing
We call @Test void test() { WB Test code with JUnit in Java
checking
code as
int ans = add(1, 2); // Test scenario
test oracle assertEquals(ans, is(3)); };
• Black-box integration testing
@Test void test() { BB Test code with Selenium in Java
// Open calculation result page (abbr.)
WebElement p = d.css_selector(“p”);
assertEquals(“1+2=3”, p.getText()); };
3/22/2013 FASE 2013 in Rome 5

6. Problems on BB testing
P1: It is unclear what oracles we should test
S1: Extract template variable fragments
P2: No indicator of oracle quality for web apps
S2: Define template variable coverage
P3: High costs to write test code
S3: Tool for generating skeleton test code
P4: High costs to maintain test code
S4: Accessors based on expression strings
3/22/2013 FASE 2013 in Rome 6

7. P1: Test oracles are unclear
P2: No indicator of oracle quality for web
• It’s unclear what fragments we should check
• Traditional coverage cannot assess oracles
Fixed fragment
It shows different contents
Variable
depending on contexts so
fragment
that it’s hard to detect faults
How can we extract variable fragments from HTML ?
How can we check variable fragments thoroughly ?
3/22/2013 FASE 2013 in Rome 7

8. P3,4: High costs to write/maintain tests
Maintenance cost
• Test code depending on HTML structure
– It frequently break when changing web design
drv.findElement(By.cssSelector(“p > div”));
{template .addition} {template .addition}
<p>{$l}+{$r}= <p>{$l}+{$r}=
<div>{$ans}</div> <span>{$ans}</span>
</p>{/template} </p>{/template}
• Mark elements containing variables Writing cost
– It is hard to add marks such as ID by hand
drv.findElement(By.id(“ans”));
<div id=“ans”>{$ans} <span id=“ans”>{$ans}
3/22/2013 FASE 2013 in Rome 8

9. Template variable coverage
• Based on checked variable fragments
– Ctmp = Vtest / Vall: #checked / #variable fragments
– E.g. {$left}, {$(x * 2)}, {$(”ab” + “c”)}
– Consider only variable fragments to present
{template .result} Closure Template
<p>{$left}+{$right}=<div>{$answer}</div></p>
<ul>{call .items} {param list : $histories /}{/call}</ul>
{/template}
@Test void test() { Test code with Selenium and JUnit
WebElement answer = ... // get div elems Coverage
assertEquals(“3”, answer.getText()); }; value: 33%
3/22/2013 FASE 2013 in Rome 9

10. Coverage definition in detail
• Templates can contain same fragments multiple times
• Distinguish same fragments because of different
concatenations between fixed and variable fragments
• Not distinguish fragments in loop statements
because they have same concatenations
{template .result} Closure Template
<p>{$value}</p> Distinguish same
<ul>{foreach $item in $items} variable fragments
<li>{$item}</li>
{/foreach}</ul>
<div>{$value}</div> Not distinguish variable
{/template} fragments in loop statements
3/22/2013 FASE 2013 in Rome 10

11. Gray-box integration testing
Client side • Input template files
Web browser – More than BB info
Sele
nium Platform (OS) – less than WB info
Selenium HTML – Can test on real env
test
HTML template • Analyze templates
Server program – Generate test code
• We propose POGen
DB Web Auth
server server server
– Measure coverage
Server side • Monitor with AspectJ
3/22/2013 FASE 2013 in Rome 11

12. Testing process using POGen
Contain methods for Add user-specific attributes to
~~----
getting DOM elements ~--~~ access DOM elements which
and text values ~~~~
--~~~ have template variable fragments
depending on variable
names and expressions Modified
template
Web
pass
~~~~
~~~~
server fail
pass
~~~~ Web app
POGen Test result
HTML Selenium
template
~~~~
~~~~
~~~~
~~~~
JUnit
Test
Users ~~~~
Test case2
log
0110 case1
0011 Skeleton
1000 test code Execution log 75%
0001 Test
Monitor
code Coverage
Coverage
information AspectJ
3/22/2013 FASE 2013 in Rome 12

13. Architecture of POGen
Can add
analyzers
Element location Insert user-
HTML template analyzer
~~~~ ~~----
~~~~
Templates Expression string ~--~~
specific attrs
analyzer
~~~~ ~~~~
~~~~
Closure --~~~
Accessors
transformer
Closure Modified HTML
dependign on
template
Templates 00110
template
exp strings
HTML
01000
00010
generator
info ~~~~
Skeleton
test code
~~~~
00110 ~~~~
analyzer
~~~~
~~~~ 01000
ejs
~~~~ 00010
01000
00010
Skeleton
ejs info test code
Coverage info
Expression strings in templates (e.g. {$v + 1}) are more
robust than HTML structures (e.g. XPath or CSS selector)
3/22/2013 FASE 2013 in Rome 13

14. Test code generated by POGen
<p> Original HTML template @Test void test() { Test case
by hand
{$left}+{$right}= WebDriver d = ...
<div>{$answer}</div> // Conducting 1+2 Calculation
</p> AdditionPage p = ... Call below
Template assertEquals(“3”, accessors
in Closure POGen p.getTextOfAnswer()); }
Templates
<p id=“_po0”> class AdditionPage exnteds AbstractPage {
{$left}+{$right}= /* ---- GENERATED CODE START ---- */
@FindBy(id = “_po1”)
<div id=“_po1”> WebElement _answer;
{$answer}</div> WebElement getElementOfAnswer(){..}
<!-- _po {$left} --> String getTextOfAnswer() {..} //abbrev.
... /* ---- GENERATED CODE END ---- */
</p> // user code Generated test code designed
Modified HTML temaplte } by PageObject pattern
3/22/2013 FASE 2013 in Rome 14

15. Research questions
• RQ1: Is template variable coverage
correlated with a test's capability to
detect faults?
• RQ2: How can our technique improve
a test's capability to find faults?
• RQ3: How can our technique facilitate
writing test code?
• RQ4: How can our technique facilitate
maintaining test code?
3/22/2013 FASE 2013 in Rome 15

16. Evaluation: Coverage effectiveness
Apply mutation testing to Seam Framework sample(booking)
H: Coverage is positively correlated with #killed mutants
Ratio of killed mutants to mutants
25 15
generated by Simple Jester [%]
Pearson’s test Pearson’s test
cor: 0.930 2 tests cor: 0.657
20 p: 0.0007 p: 0.076
4 tests
6 tests
10
15
8 tests
10 10 tests
5
12 tests
5 Median of 3 coverage 8 coverage values
values selecting some 14 tests selecting 2 test
0 test cases randomly 16 tests 0 cases randomly
0 10 20 30 40 0 10 20 30
Template variable coverage [%]
3/22/2013 FASE 2013 in Rome 16

17. Evaluation: Test code generation
Apply POGen to online code execution web apps
LOC without comments and spaces
500 Generate
450
68% test code
400 Test code
38 143
350
No need to maintain by hand
code when changing
300
250
200
design (HTML structure)
150
13 304 105 304
12 7
100
6 0 Used test
50
82 47 88 44 43 code
0
0
Index Create Edit Solve Result Pages Test Sum generated
sum case by POGen
3/22/2013 FASE 2013 in Rome 17

18. Evaluation: Reduction of costs
• 8 people wrote 3 test cases 2 times by filling blanks
• Time to write: left, Checked variable fragments: right
#Checked variable fragments
10
Time to write 3 test cases [min]
70
p: 0.030 p: 0.084 A
9
60 B 8
7.8
50
C 7 6.3
40
35.3 34.8 D 6
5
E
30 4
F 4 3.5
3
20
G 2
10 H 1 p: 0.0011 p: 0.0150
13.3 11.3
0 Avg. 0
Test cases 1 Test cases 2 Test cases 1 Test cases 2
w/ POGen:left, w/ POGen:left, w/ POGen:left, w/ POGen:left,
w/o POGen:right w/o POGen:right w/o POGen:right w/o POGen:right
3/22/2013 FASE 2013 in Rome 18

19. Limitations
• Dependency on template engines:
Our technique cannot elucidate the
dynamic components which are created
using web frameworks or the DOM API
without template engines.
• Robustness of accessor methods’
names: When a template variable occurs
more than once in an HTML template,
POGen names the corresponding
accessor methods with sequential
numbers (1,2...). So some changes cause
changes of accessor methods’ names.
3/22/2013 FASE 2013 in Rome 19

20. Related work
• Staats et al., “Programs, tests, and oracles: the
foundations of testing revisited,” ICSE 2011.
– Argued test researches should consider test oracles
• Schuler et al., Assessing Oracle Quality with
Checked Coverage, ICST 2011.
– New statement coverage in consideration of oracles
• Sreedevi et al., “Coverage Criteria for Testing Web
Applications,” TechnicalPaper 2005. No oracle
assessment
– Control / data flow coverage by a page
• Kodaka et al., “Automated testing for Web applications
using Aspect-oriented technology,” SIGSE 2007.
– Comparing variable fragments with expected strings in JSP
Only comparisons
3/22/2013 FASE 2013 in Rome 20

21. Conclusion
• Template variable coverage
– Ratio of checked template variable fragments
• POGen: Tool for generating test code
– Accessors for getting variable fragments
• Evaluation
– Positive correlation, Reduction of costs
• Future work
– Evaluate coverage for web apps in real world
– Dynamic analysis for extracting variable fragments
3/22/2013 FASE 2013 in Rome 21