Learn how to build a robust, modular, automated test suite for your website's UI that models real users traversing the application. Built with the free ColdFusion tools CFSelenium and MXUnit, tests can pull in parameterized input data from external sources to make them more dynamic and flexible. Using the Page Object Design Pattern, test suites can be made more maintainable and less brittle while allowing for faster test development by abstracting how a user interacts with a website, thus minimizing the effort to fix tests when the website UI changes. Pull it all together with ANT and Jenkins for hands free, reliable test suite execution and test result notification on a daily schedule.
1. Automated System
Testing
For Web Applications
Using CFSelenium, MXUnit, & Jenkins CI
Steven Erat
FirstComp Insurance
talkingtree@gmail.com
CF.Objective() 2012
5. What to Test?
Functional Testing
Unit, Integration, System, Regression,
Acceptance
does a speciïŹc feature work?
can the user do {this}?
Non Functional Testing
Security, Performance, Usability
cuts across features
6. Unit Testing
A type of Functional Testing
Written & run by software developers
Finds problems early in Development
Tests components in isolation
Dependencies usually mocked
Canât test legacy (non-CFC) code
7. System Testing
A type of Functional Testing
Tests the completed software product
Often performed manually
Can be automated with testing tools:
QTP - The $6000 Gorilla
Selenium - FOSS; Has many language APIs
8. Continuous Integration
Continuous process of quality control
Development - Creates âThe Buildâ
QA - Test Suite Execution
Performed Frequently
after each commit, or
scheduled regularly (daily, hourly, etc)
CI Software: Jenkins, Bamboo,
CruiseControl
9. How to Test?
System Testing
mimics user interaction with GUI
QTP - Expensive, complex, scriptable (VB)
TestNG Java Test Framework with
Selenium
Free. Write tests in Java.
MXUnit ColdFusion Framework with
CFSelenium
10. The Landscape
Test Suite is independent of AUT
AUT 1
Test Suite AUT 2
Framework
AUT 3
ColdFusion
MXUnit
CFSelenium Application(s) Under Test
11. What is Selenium?
Selenium IDE
record/playback user actions in a browser
recorded in Selenese, a special test scripting
language
Selenium Client API
alternative to Selenese, tests can also be
written in various programming languages
Selenium (RC) Server
Java-based server that accepts commands for
browser
16. Selenese Commands
Actions
open a URL, click a link, wait for page to load
Accessors
store results of actions in a variable: storeTitle
Assertions
assert (failure stops further test method)
verify (test method continues after failure)
17. Selenese HTML Format
<tr>
<td>open</td>
<td>/blog/admin/index.cfm?</td>
<td></td>
</tr>
<tr>
Selenium
<td>assertTitle</td>
<td>BlogCFC Administrator: Logon</td>
performs the <td></td>
</tr>
<tr>
Assertion
<td>click</td>
<td>css=input[type=âsubmitâ;]</td>
during playback <td></td>
</tr>
<tr>
<td>waitForPageToLoad</td>
<td>30000</td>
<td></td>
</tr>
19. Object Locators
Strategies for locating objects/elements
on pageselenium.isElementPresent('id=username')
selenium.isElementPresent('name=username')
id
selenium.select(âdocument.forms['myForm'].cb1
name â)
dom selenium.click(â//a[contains(@href,'#id1')]â)
xpath selenium.click(âlink=Add Commentâ)
link selenium.click('css=input[type="submit"]')
css selenium.getText(âregexpi:^[A-Z0-9+_.-]
+@[A-Z0-9.-]+$â)
24. What are the beneïŹts so
far?
Quick testcase creation in familiar
language
Framework for test execution & reporting
System Regression Testing
25. What are the problems?
Object locators hard coded in each
testcase
Object locators repeated thru testcase/
suite
Assertions interwoven with Selenium
commands
Static tests embedded with literal input
26. Solutions: POM
The Page Object Model design pattern
Models web page or page part
independently
navigation bar, sidebar, pods, footer
Object identiïŹers exist in only one place
Object oriented approach to testcase
creation
29. Modeling a Part / Screen
Add Comment pop-up
One function for each
form ïŹeld, button, or
widget
Functions isolate the
object locator
Method chaining
33. The Landscape
Logical divisions in AUT can be tested
Test Suite
Framework
Applications Under Test
34. TestCase« » Page
Test doesnât know
about object
locators
Page Object
doesnât know
about Asserts
Page can return
instance of next
page: loginAs
returns Home
36. Parameterized Test Data
External data read for use as MXUnit
dataproviders
Easy to modify data by non-technical users
JExcel letâs you easily use multiple tables
37. WorkïŹow
Record user interaction in a browser
Performed by QA or Business User
Use Selenium IDE with CFML Format
Produces linear recordings with static data
Enhance static scripts
Performed by QA Software Engineer
Convert to Page Object Model
Convert for Parameterized Data
(Dataproviders)
Commit to Source Control
38. Recording User
Interaction
Selenium IDE records object interactions
Can change priority of which locators it
uses
id, name, link, XPath, DOM, etc
During recording you may add Assertions
manually
43. Ant Properties Files
Can use a global properties ïŹle
shared aspects across entire test framework
Can also use local properties ïŹles
store conïŹg values unique to each AUT test
suite
Flexible conïŹguration - Itâs up to you
44. Global Properties
DeïŹne a property to switch testing
environments
Dev or Staging?
Tell Selenium which browser to launch
45. Local Properties
Properties unique to each UAT being tested
DeïŹne properties to initialize Selenium -
browserURL
$ENV swapped to match global env property (Dev or
Staging)
46. MXUnit Entry Points
MXUnit kicks off Test Suite via a Remote Method
call
Plugin test runner calls RemoteProxy.cfc
Ant test runner calls HttpAntRunner.cfc
47. How are Tests Run So
Far?
Manually
Locally per Individual
Execute Plugin or Ant Test Runner
51. Jenkins
CI
Create âjobsâ to do work
Call Ant Tasks
Run Source Control
Schedule jobs using cron syntax
Chain jobs to run in sequence
Status / Trends / History
Exceptions Stack Traces
Email NotiïŹcations
60. Lessons Learned
Stub out Page Objects up front
Organize test suite Page and Test directory
structure by logical areas in the AUT
Blog Home, Blog Entry, Comment, Contact,
Search, RSS ...
Blog Admin Login, Home, Edit Entry, Refresh,
Stats ...
Test writers can then work in parallel
without duplication
61. Lessons Learned
Convince AUT Developers Testing Works
Develop AUTs with better object id/
naming
unique IDs better than CSS, DOM, XPath
locators
Disambiguate signiïŹcant data in AUT
body
62. Lessons Learned
Design Page Objects for method chaining
LoginPage.enterUserID(âmeâ).enterPass(âpwâ).submit();
Easier to read
Takes advantage of code insight
CF Builderâs code insight not quite there yet
Other IDEs may provide better code insight
(IntelliJ)
63. Lessons Learned
Always perform LogOut() in beforeTests()
method
Run Login() in the setUp() method
Run LogOut() in the tearDown() method
Tests no longer need to perform login
actions
(you should already have tests just for Login)
65. Troubleshooting
Response of the Selenium RC is invalid:
ERROR: Element css=p >
input[name="search"] not found
Response of the Selenium RC is invalid:
Current window or frame is closed!
Get the AUTâs HTML Source via Selenium
Have Selenium grab a screenshot of AUT
69. Troubleshooting
Slow the test execution down
Rerun the test via MXUnit Plugin
Watch the browser as test runs
Add sleep(NNNN) statements at key points
70. Troubleshooting
// Runs once before all test methods in file
public void function beforeTests() {
// dataproviders must be in the variables scope
variables.TwoCommenters = application.dataproviders['TwoCommenters'];
variables.selenium = application.selenium;
variables.HomePage = new HomePage(selenium);
variables.defaultSpeed = selenium.getSpeed();
selenium.setSpeed(1000);
}
// Runs once after all test methods in file
public void function afterTests() {
HomePage.clickNavButton_Home();
assertEquals(HomePage.getExpectedTitle(), selenium.getTitle());
selenium.setSpeed(variables.defaultSpeed);
}
72. Troubleshooting
A TestCase method failed
Did the TestCase change?
Did a Page Object change?
Check TestSuite Source Control for changes
since last success
Did the AUT change?
73. Troubleshooting
A Job in Jenkins Failed to Build
Check the Test Suiteâs ColdFusion
exception.log