4. Setup
Mobile
Global Portland Global
Product USA Product
San Jose Chennai
USA India
EU
Quality
Engineering
Global Local
Product
(Zürich, London,
Berlin, Paris)
Product
Shanghai Zürich,
China London
Adjacencies
(i.e. Brands
for Friends)
4
Produced by: Michael Palotas
5. Not only the Web
Windows
Desktop
Web
Mobile
5
Produced by: Michael Palotas
7. Testing Today – Ownership of Quality
Manual Test
Consulting
+
Coaching
Automation
7
Produced by: Michael Palotas
8. Manual Test
Still very important, mainly exploratory type
All new features are tested manually first
Edge cases
Not automatable tests
Manual testing still yields the highest number of bugs
8
Produced by: Michael Palotas
9. Automation
Automation is not about eliminating jobs !!!
Save time and reinvest into manual testing
Regression Testing (fully) automated
Programmatic approach (JAVA)
Open source only (WebDriver/Selenium 2 GRID)
3500 end to end testcases running in less than 1 hour
9
Produced by: Michael Palotas
10. GRID
eBay Europe:
- 3500 e2e regression testcases
- 2-3 minutes execution time (each)
executions
Execution time 7000 minutes (~116 hours = ~5 days)
Sequential
Parallel
Execution Test
1
Test
2
Test
3
Test
…
Test
…
Test
3500
Execution Time
Parallel Test Test Test Test Test
Parallel executions
150 Virtual Machines / Browsers in
execution in Test Test
the WebDriver GRID
Test Test Test
Execution time: 60 minutes
Grid
Test Test Test Test Test
Execution Time
10
Produced by: Michael Palotas
11. Coaching / Consulting
EU team conducts project audits
Improvement initiatives
Consulting for dev and test teams
11
Produced by: Michael Palotas
20. Obstacles and opportunities
Mobile Web is easy…
Native App is not…
Planning to work with Google on open source project “nativeDriver”:
http://code.google.com/p/nativedriver/
Produced by: Michael Palotas
22. Overview
What is Selenium
The different tools under the selenium umbrella
Integrating selenium for web testing
Leveraging the protocol for other technologies
Conclusion
Produced by: Michael Palotas
23. What is Selenium today ?
Web testing tool
Highly successful open source project, supported by big companies
Stable, and fast evolving
Protocol describing the user interactions
Comes with most browsers implementations
Works with most common languages
Produced by: Michael Palotas
24. What it's not
A test framework
A network testing/monitoring tool
A performance testing tool
Produced by: Michael Palotas
25. History
Started as a pure JS implementation
• 1 implementation for all browsers
• browser specific features had to be hacks
• XSS issues
Webdriver merged to Selenium
• real events
• new, documented, API
• backward compatibility of sel1 protocol
• faster,easier to maintain
• 1 implementation per browser
Produced by: Michael Palotas
26. Webdriver JSON protocol
The one page to read :
http://code.google.com/p/selenium/wiki/JsonWireProtocol
The core of the product
Built to be a standard
Based on the HTTP protocol
Specifies :
• all the “normal” browsing interactions:
GET /session/:sessionid/url
POST /session/:sessionid/url
• and the new generation interactions :
POST /session/:sessionid/touch/longclick
Produced by: Michael Palotas
27. The vision behind selenium
The cure :)
Vendors are responsible for the implementation
Protocol becomes the standard for web testing
Produced by: Michael Palotas
28. When to use selenium
Reproduce use cases
Unit test of the web components for x-browser validation
End to end validation / regression ( eBay EU )
Produced by: Michael Palotas
29. When not to use it
Everything else :
• Performance testing
• Driver for anything with a UI...
Produced by: Michael Palotas
30. Component 0/3 : Selenium IDE
Firefox plugin
Super easy to start
Usually first interaction with selenium
Produced by: Michael Palotas
31. Component 1/3 : Selenium client driver
What testers see as “selenium”
Doesn't do much : generates the API calls
The part that will be called :
• from the test framework
• from the CI
Official selenium support
• Java, c#, python, ruby
Not Officially supported
• Perl, php
Produced by: Michael Palotas
32. Component 2/3 : Selenium server
The more interesting part
Where the contributors spend their times
Drives the AUT
Officially supported by the selenium team:
• IE, Firefox,iphone,android
Supported by vendors:
• Chrome,Opera
Produced by: Michael Palotas
33. Component 3/3 : Selenium grid
Proxy the HTTP requests to do 3 things :
• Load balancer
• Environment manager
• Disaster recovery
Produced by: Michael Palotas
34. Writing Selenium tests
IDE
• Record / replay
• (too) easy
• Selenese / html
Client / server
• Code
• No record and replay
• Testing requires basic dev skills
Produced by: Michael Palotas
35. Selenium IDE - SIDE
Record and replay design
Entry barrier very low
Maintenance is a nightmare
Building abstraction layers for SIDE = reinventing the wheel
Produced by: Michael Palotas
36. Selenium client driver
Clear separation on the test code and the driver code
Stable API : update the driver without impacting the clients
Allows to run remotely
The tests are just another piece of software
Produced by: Michael Palotas
37. Tests are code !
Spend time on the design
Use page object
Don't test everything
Produced by: Michael Palotas
38. Next step : scaling using grid
Scalability :
• manage multiple browsers / OS installations
• run tests in parallel
• hooks to deploy everything in a production environment
Produced by: Michael Palotas
40. Grid – create plugins !
Extension points
Change the default behavior :
• Registration
• Test
• Command
Or tune it :
• Custom priority
• Custom matching
Plugins will :
• execute on the hub
• access the internal hub state ( mutable )
• allow to overwrite methods
Produced by: Michael Palotas
41. Plugin example : eBay
We moved away from Grid1
• Self healing features covered mostly connectivity issues
• Nodes were taken out after a crash, but no automatic recovery
• We needed to manage more than just the browser : the OS too.
• Managing a growing grid is error prone
Produced by: Michael Palotas
42. eBay grid requirements
Maintenance :
• (re)start everything automatically
• Adding 1 node = 1 line in a config file
• Nodes are restarted before having issues
• Test isolation
Debugging :
• Showcase for grid , adding features without changing the tests : VNC
Produced by: Michael Palotas
43. Default grid startup overview
Launcher
Hub
registration request webdriver
server
node
Produced by: Michael Palotas
44. Grid : logic of the eBay plugin
The plugin ( proxy ) we currently use implement 2 interfaces :
• Registration listener
Restores the VM to a clean state
Starts the display and VNC
Starts webdriver server
Validation
• Test session listener :
Counts the tests
Directs the browsers to the right display
Produced by: Michael Palotas
45. Integration with virtualization
Hub • Custom node proxy
• Registration doesn’t come from the node
• New proxy code in the classpath. No need to re-compile grid.
• Extra params for the proxy
Produced by: Michael Palotas
48. More integration : TWIN
http://code.google.com/p/twin
• Webdriver implementation for automating Windows graphical applications.
• Open source project from eBay
Produced by: Michael Palotas
49. eBay example : TWIN for Turbo Lister 2
Produced by: Michael Palotas
50. Challenges of Turbo lister2 Automation
On top of driving the application, the test infrastructure needs to handle :
• the software installation
• the config files
This can be handled by a plain grid hub by adding a plugin :
• When a node registers :
it has a vm ID property
the hub check for existing snapshots
the hub triggers the installation for each site
TurboLister2 is started for the first time
the hub create a snapshot
• When a new test is started :
a VM is assigned
the hub revert to the appropriate snapshot
Produced by: Michael Palotas
51. Advantages of a single grid
All the tests follow the same high level life cycle :
• automatically manage the context around the test
• run the tests until the OS / AUT crashes
• try to recover
• or restart
Products release at different times :
• re-assign the VMs
Produced by: Michael Palotas
52. Next step : mobile
Native driver + link
Early stage
Android
• working for basic features
• Linux makes it easy to integrate
Iphone
• working for basic features
• Mac OS makes it harder to integrate
Produced by: Michael Palotas
53. Conclusion
Not the silver bullet
It allows your QE team to :
• use the language they're comfortable to write their test
• reuse the helpers
• clean API allows you to extends to new technology “easily”
Produced by: Michael Palotas
54. Resources
https://github.com/freynaud/grid-spine-common
• Depends on selenium 2.4.0
• VMWare integration + restart every N tests logic
https://github.com/freynaud/grid-spine-selenium
• Depends on common
• All the selenium specific logic, VNC integration, Linux vs Windows code.
Built using
• noVNC : http://kanaka.github.com/noVNC/
• tightVNC
• VIX ( VMWare API ) http://labs.vmware.com/flings/vix-java-toolkit
• VMWare WS http://vijava.sourceforge.net/
Produced by: Michael Palotas
