My presentation on Continuous Integration and single user performance with Compuware's APM tool DynaTrace as presented at the 2013 Selenium Conference.

1. Cars.com – Revolutionizing
Continuous Integration with
Selenium
Rick Darcy
Lead AutomationTest Engineer
rdarcy@Cars.com
www.linkedin.com/in/rickdarcy/

2. History
2

3. IT then and now…
3

4. Cars.com Fast Facts
4

5. Dealer Tools
5
100
Hrs
2 Hrs
98%

6. Sell and Trade
6
46 Hrs 30 Min
99%

7. Explore
7
40 Hrs 30 Min
99%

8. Classified Search
8
40 hrs 30 Min
99%

9. CRM
9
4 hrs 30 Min
92%

10. Mobile
10
8 Hrs 30 Min
96%

11. Overall
11
235
Hrs
4.5 Hrs
98.2%

12. Current Cars.com Automation
2011 technical
debt
Smoke tests and
Core tests are
run daily
Regression tests
are run when we
push code to IT
Java-based
Selenium was
chosen since
we’re a Java
shop
Selenium 2.0 /
WebDriver
managed by
Maven
We have a very
robust automation
framework
12

13. How do we integrate with Agile
Participate in planning
• Understand story points
• Determine Automation
necessities
Create Automation tasks
Develop and execute the
Automation
Reporting analysis and
potential refactor
13

14. Our Automation Framework
6 months to build
 We are in a state of continuous script tuning for optimization and speed
 Our framework achieves maintainability, organization, faster script
development
Pages
 We identify all page objects and include these in our scripts
Scripts
 We group our cases together, and separate them by product, functionality, and
suite type
Suites
 Where we set browser types/versions and select which test scripts to run
Flows
 Repeated steps we call into our scripts; makes test case development faster
14

15. Toolset
15

16. Integration with BuildForge for CI
Find performance issues without having to re-run
multiple times
The product team will be notified of any issues
after the tests are run by BuildForge
Automation will be run against every new build
pushed to any environment
16

17. Adding timers to code
17

18. Continuous Integration
• Integrated with Compuware’s APM tool
 Can be run in any environment
 Gathers data from automation suite runs
• Page load times, purepaths etc…
• Monitors our JVM’s
 210 – Prod
 208 – Staging
 61 – Integration
 30/32 – Dev. Environments
18

19. My little Regressions…
19
Compuware’s
APM tool

20. 20
Identifying regressions. Early on.
Continuous Performance
Tests!

21. Identifying regressions. Early on.
21
Version Control History Lookup

22. A Regression due to resources
22
Regression
Naming Convention
Resource
Exception

23. Exceptions
23
Increased
Exceptions

24. Root Cause
24
Missing
Query
??

25. Export to Development
25
Export & Send to
Development

26. Scalable architecture. Simple.
26
Performanc
e
Warehouse
dynaTrace
Server
Exporte
d
Session
File
Offline
Session Analysis
dynaTrace
Client
PurePat
h
Collect
or
Easy install:
Browser plug-in
Browser Web app

27. Integrate into workflows. Easy.
Developer
Code Repository Build Server
Test Automation Engine,
e.g. Selenium
RESTInterface
Com
m
it
Build Test
Browser with
dynaTrace Agent
PurePath Data
dynaTrace Client dynaTrace Server
Performance
Warehouse
Execute
Automated
Tests

28. dynaTrace
Server
Selenium
Tests
HTML Report
1 2 3
0
Application Build
Integrate into Buildsystem
(Hudson, BuildForge, QuickBuild, ...)
Continuous Integration
(Maven/Hudson/Ant)

29. Best Practices
29

30. Defining our Strategy
30
Start Early
Test Continuously
Break in Pieces

31. Creating Focus
31

32. Define KPI’s
32
Throughput
Response Time
Memory Consumption
Other KPIs …

33. Lessons Learned
33

34. Questions
34