2. The Stories
• John The Tester
is a manual tester,
• He is ordered to
do an automation,
• So he does,
• He develop some
tests, he runs it
on his box
3. The Stories
• The more tests he develops, the more time tests lasts,
• At first, it takes couple minutes,
• Later, it takes dozen,
• It is a lot of time!
He cannot afford to
stare at running tests
9. ...but at the beginning there was
Eternity...
• server hosting Hudson,
• test execution testbed for
Selenium IDE tests,
• but one executor means
one test at a time.
10. VM and Hyper-V
• QA team uses Hyper-V
virtualization software,
• It let us to run ~14 VM,
• CPU is not the problem,
• Memory is the problem,
• 80% free memory rule – swapping leads to serious
performance lost,
• We have GEB and BES servers available for us:
• BES – manual tests environments for testers and developers
(different OS, different browsers, mobiles),
11. VM and Hyper-V
• GEB – automated tests environments.
• Slaves for us and our clients,
• Jenkins,
• Applications – QA Lab uses this machine to provide tools and
services for our test framework. So far we have W3C Markup
Validation service – our Validation Tool uses it.
12. VM and Hyper-V
All machines are under Nagios supervision,
It let us to control if the environment is up and ready to go,
Notifies by e-mail if something is wrong.
13.
14. VM Future
• More tools available for our testing framework (W3C tools),
• Different browsers/browsers version,
• Maybe different virtualization system – bare metal hyper-
visors seems to look promising,
• Cloud – as a main resource, as a backup resource.
15. Centralized Automated test
driving with Jenkins CI
• QA Team use Jenkins CI tool to drive tests,
• We have Jenkins deployed in a Tomcat container,
• We use SLAVE AGENTS as a communication channel,
• It’s not greatest solution because CI tools are rather build
tools not testing tools.
16. How we use Jenkins...
• We use Views to manage projects
17. How we use Jenkins...
• We use “distributed builds”
feature to manage parallel
testing (speed up) with several
slaves,
18. How we use Jenkins...
• We use labels to nickname a slaves
(ff36, ff4, winxp).
• We use “Restrict where this project can be
run” to configure a test job run.
19. How we use Jenkins...
• For performance testing we use JMeter and JMeter plugin for
Jenkins.
20. How we use Jenkins...
• We use “Configuration Matrix” for some jobs.
21. How we use Jenkins...
• We use “Discard old builds” to save disk space.
22. How we use Jenkins...
• We use Jenkins Remote Access API,
• We use “Trigger build remotely” to run build from outside:
• ...and tools like curl/wget to trigger it:
curl --user user:password JENKINS/view/project/job/build?token=token
23. How we use Jenkins...
• We try follow some time line guidelines:
00:00 to 08:00 - Automated/scheduled tests, nightly test,
08:00 to 21:00 - work day, manual/scheduled test,
21:00 to 00:00 - daily/weekly maintenance time,
24. Maintenance problem
• We suffer with maintenance problem while managing so
many VM – we use STAF for performing actions on all
machines.
26. Maintenance problem
• Daily tasks (21:00): Weekly tasks (21:00 Sunday):
• Restart, Cleaner,
• Set screen resolution,
• Resources.
Defragmentation,
Anti-Virus Scan,
Windows updates.
27.
28. Jenkins future
• Browsers auto updating,
• Dynamic slave management,
• Custom framework – leave Jenkins bottlenecks behind and
build your own,
29. The answer
• Distributed testing environment on virtual machines,
• Centralized automated test driving with Jenkins CI.
30. Benefits of presented solution
• Easy way to extend test environment (just clone VM and
voila),
• ...which let us to speed up tests by splitting and paralleling,
• Easy way to restore corrupted system (revert snapshot, copy
VM disk file),
• HQ for managing tests with different tools (Selenium,
Webdriver, Jmeter, Wapiti, Test Complete, AutoIt...),
• Built-in features and plugins (e.g.: SVN client),
