This slide deck illustrates Constant Contact's approach to running sets of regression test suites on a regular schedule using SeleniumRC and Hudson. Hudson is a continuous integration server designed for building applications and running unit test unattended, and storing run report and metric artifacts. Hudson can also process a variety of different types of jobs, including running SeleniumRC Java test cases using JUnit as the test runner. SeleniumRC can be run unattended on multiple slave computers in parallel, each running SeleniumRC, to quickly run through many test cases.

1. Functional Continuous Integration SeleniumRC and Hudson David Jellison: Director, Quality Engineering Collin Riley: Principal Quality Engineer January 25, 2011

2. Agenda About US Continuous Integration (CI) Concepts Continuous Integration (CI) Concept Functional Continuous Integration (FCI) Extends the concept SeleniumRC is part of the capability for an FCI strategy Continuous Integration Servers make CI easy Constant Contact FCI Approach Hudson is a rich dashboard to orchestrate traditional CI & FCI How Constant Contact uses Hudson and SeleniumRC for FCI What other pieces are in the FCI framework at Constant Contact Q&A

3. Speakers David Jellison: Director, Quality Engineering David has 18 years experience in test and development management, and over the last 4.5 years he led agile development and test organizations with .Net, Rails, and Java stacks in SaaS organizations. David is passionate about test leadership and efficient software development practices. He is consistently has his hands in technological and agile practice continuous improvement. Collin Riley: Principal Quality Engineer Collin is the architect of Constant Contact's use of Selenium RC in our Java automation framework utilizing Hudson. This framework integrates with Quality Center, uses page objects, and other scalable features. Collin has 8 years Quality Engineering and automation experience and has been using Selenium RC for over two years.

4. Who is Constant Contact? (CTCT) Constant Contact Corporate Helps small businesses, associations, and nonprofits connect with their customers, clients, and members. (>400,000 subscribers) Launched in 1998, Constant Contact champions the needs of small organizations and provides them with an easy and affordable way to build successful, lasting customer relationships. Constant Contact Engineering ~200 people, ~16 Agile teams (½ product & ½ services) ~30 in Quality Engineering, 3:1 Dev:QE SaaS (Software as a Service), Java/JBoss/CentOS/DB2 stack, live up-time deployments to production Practice ScrumBan (blend of Scrum release management and KanBan work item management) Use Java/Groovy test scripts with SeleniumRC and Java libraries for functional tests

5. CI Concepts

6. Assumptions Familiar with the following: Selenium Remote Control (SeleniumRC) Agile Practices (what Scrum and eXtreme Programming are) Source Code Repository (e.g. ClearCase, CVS, SVN, Git, etc.) Integrated Development Environment (IDE - e.g. Eclipse) Infrastructure Monitoring (e.g. Nagios) xUnit test runner family (e.g. JUnit, NUnit, Test:Unit etc.)

7. Continuous Integration “Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.” – Martin Fowler (martinfowler.com) Approach (Developer Focused) Developer creates unit tests along with new and re-factored code Developer builds and runs unit tests locally often Developer often commits code to a source code repository that is auto-built, tested, and packaged on a CI server Benefits Significantly reduce integration problems Allows teams to develop cohesive software more rapidly Note: The term ‘Continuous Integration’ was introduced as one of the 12 practices of eXtreme Programming.

8. Continuous Delivery Deployment Automation Theme of reducing cycle time between idea and usable software Culture of frequent reliable deliveries Configuration management for OS, Server, and Application settings Manifest description of each environment Machines and their allocation of applications Database and resource permissions Load balancer configuration Test Environments Represent configuration of production (manifest for each) Rely on manifest and deployment automation for changes Dedicated CI test environment to validate deployment automation with each change

9. Functional Continuous Integration (FCI) Prerequisites Continuous Integration (CI) practices Continuous Delivery (Deployment Automation) practices Dedicated CI test environment (minimum configuration) CI Functional Tests Deployment tests that prove a system is properly configured Application smoke tests that prove an application is testable Extend the culture of break-the-build urgency Passing tests can be delivered to test environments or production Failing tests require immediate attention to resolving problems Benefits Robust deployments to stable test environments Frequent deliveries to production Increased release confidence, lowering business risk

10. Challenges of scaling CI and FCI Multiple Branches Rebasing often (merge back from trunk) Merge back to trunk as often as feasible Multiple repositories Package integration installers often (automated packaging) Package library includes and dependencies carefully Many services and applications Distributed application architecture Clearly defined interfaces (with versioning) Test fixtures (test components in isolation) Health check requests for infrastructure monitoring (e.g. Nagios) Deploy and functional test often Require functional regression tests with new features

11. Agile Testing Collaborating with Developers Agile Testers collaborate with Developers Developers participate in test planning Testers participate in feature design Testers test as early as code changes are available Domain expertise Traditional and exploratory testing Develop automated test cases Developers work with testers to keep page objects consistent with application pages as code changes are delivered Developers run regression tests prior to check-in Regression tests consistently pass with automated test runs

12. With passing tests, commit changes Agile Testing QE & Dev work together to create, re-factor, and update functional regression tests Dev deploys to sandbox system QE runs tests against the sandbox Dev updates the application code for problems found – no defects written When all tests are complete and working, Dev commits changes FCI

13. Constant Contact

14. Functional Test Needs Test Case Manager (Quality Center) Representation of what is automated & automation runs Test Runner (SeleniumRC / Junit / Page Objects) Browser & OS Testing Scalability Run Manager (Hudson) Push-button runs Parallel test runs Run reporting and artifacts Drill-in Run Reports (internal reporting application) Reporting Application leveraging Quality Center Aggregate run reports Dynamic charts

15. Choosing Hudson for FCI Dashboard Familiar Java-based server Hudson was already CI server choice by Release Engineering Job flexibility Can configure job parameters, run externally, run a collection of other jobs, produce artifacts, permission levels Job Queuing Can create slave pools to queue many jobs to cycle through the target slave pool Job Agents Control over agent machines Iframe views Can create dashboard applications on a report server and integrate into Hudson views. Support many plug-ins to CI tools

16. Test Automation Stack

17. Test Automation Stack

18. FCI Flow – Authoring Tests Quality Center Add Test Case(s) / Test Plan Test Templates Pull Templates from QC into the Java Framework Write the Test Code Page Object classes Helper methods Web Class Reporting Class Test locally Once Pass, mark in QC for Hudson

19. FCI Flow – Before Running in Hudson Build the Java Project in Hudson Push to Hudson slaves Java Project Jars Custom Firefox profile Host file entries Restart Selenium Server Instances on Slaves Publish Hudson Jobs Talk to QC and Push Jobs to Hudson Test Plans automated Selenium based Webservice based Run Types

20. FCI Flow – Running with Hudson Launch a Run Hudson talks to QC for what to run Hudson adds Testplans to the build queue n Number of slaves pull from the queue in parallel Test Plans report back To Hudson Red or Green Archive html report with screenshots Junit output To QC Full results for TestCases Status with framework Run Id Detail info for each step of the TestCase

21. Running Tests Repository Pool of Win Machines Environment 1 Build Test Jar Pool of Apple Machines Smoke Daily QC Suite 01 Suite 02 Suite 03 Suite 04

22. Reporting View Reporting Via a simple Java web application integrated into Hudson View Aggregate reporting for all TestPlans in a Run Graphing Trends, Pies Drill in grid based reports Compare run over past runs Act on Reporting After Failure is investigated Mark Failure as Known Defect Script Issue Configuration Issue

23. Run Results Wiki regression status charts Email drill-in grid report

24. Drill into run comparison grid & detail report

25. Q&A

26. http://ConstantContact.com/Careers Quality Engineering Senior Quality Engineer - Event Marketing - Waltham, MA Senior Quality Engineer - Contact Management - Waltham, MA Senior Quality Engineer - Scalable Infrastructure - Waltham, MA Senior Quality Engineer - Web Services - Waltham, MA Software Development Principal Software Engineer - Online Survey - Waltham, MA Principal Software Engineer - Website - Waltham, MA Principal Software Engineer, Social Media - San Francisco, CA Senior Software Engineer - Event Marketing - Waltham, MA Senior Software Engineer - San Francisco, CA Senior Software Engineer - Waltham, MA Senior Software Engineer - Waltham, MA Senior Software Engineer - Waltham, MA Software Development Manager - Waltham, MA Software Engineer - Waltham, MA Software Engineer - Social Media - San Francisco, CA