QaOps - NYC Selenium Users Meetup - 03/15/12

Christopher Hoey

■ Senior IT Manager - Wireless Generation

■ Started in QA

■ Switched over to Operations

■ Selenium abuser since 2006

■ Got to see the Selenium vs Webdriver Cage Fight live
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012

Thanks!
Ashley Wilson
Lindsey Baron
Sauce Labs
Conductor
NYC Metro Selenium Users

Selenium Developers

Jmeter Developers

Founded 12/14/06 Closed 07/16/09

March 15, 2012

Show of hands

Numbers collected and updated before sharing slides

About 35 attended and participated in informal poll

March 15, 2012

Show of hands - primary job title

Quality Assurance ?
12
March 15, 2012


Developer ?
10
March 15, 2012


Operations ?
5
March 15, 2012

Show of hands - experience with Selenium

Have not used it?
5
March 15, 2012


Tried Selenium
IDE?
8
March 15, 2012


Mastered Selenium
IDE?
1
March 15, 2012


Tried Selenium
RC/Webdriver?
13
March 15, 2012


Mastered Selenium
RC/Webdriver?
6
March 15, 2012


Selenium tied to CI
build system?
10
March 15, 2012


Running a
Selenium Grid?
7
March 15, 2012

Show of hands - use of Selenium

Functional Testing
New Features?
9
March 15, 2012


Regression
Testing?
10
March 15, 2012


Load/Stress
Testing? really?

1
March 15, 2012


Benchmarking
Performance?
1
March 15, 2012


Automating Data
Entry?
1
March 15, 2012


Monitoring for
Operations?
1
March 15, 2012


Something Strange?

2
Filling out forms to renew trial licenses when expired
Running build jobs for a build tool that does not have a cli
March 15, 2012


Disclaimers
Not a talk about best practices

Forcing Selenium to do things not originally intended

A lot of talk about Selenium IDE

QAOps term kind of meant as a joke but might have meaning?

Times have changed, tools improved and storms have passed

March 15, 2012

About Wireless Generation

● Create tools that make teacher's jobs easier

● Primarily focused around assessment and reporting apps

● Mix of mature code/platforms and new development projects

● Experiencing rapid growth year over year

March 15, 2012

Going MacGyver in early days of Ops
Faith in monitors pretty low

Manual configuration management

Serious bugs in third party tools

Large increases in user traffic

Lots of new projects to juggle while
keeping existing work online

Sysadmins manually checking site =(

Reuse anything that helps!

Selenium to the rescue!
March 15, 2012

Suites of Smoke Tests for Operations
● Use Selenium IDE so anyone in Operations with Firefox can use

● Suites are happy path smoke tests that are not brittle

● Focus on touching all moving parts of infrastructure

● All suites can run with any username

● All suites can run on any environment

● Sysadmins running Selenium tests -- still done manually =/

March 15, 2012

Suites for Operations

DEMO
TIME
Removing brittleness from Selenium
IDE tests against a site with difficult
locators

March 15, 2012

Suites for Operations - DEMO TIME
All we really want to know is if we threw any errors while loading pages that touch on all parts of our
platform.

Two simple regex searches for every page gets us the desired minimal coverage

Yes that looks way too simple but it did the job very well.

These tests started finding issues before our customers did.

Confidence increased dramatically!

March 15, 2012

Suites for Operations - DEMO TIME
https://zzz.yyy.net/wgen/tpri/LoadTpriClasseSummary.do?
ppkDefaultVersionId=7&ppkPeriodId=20&ppkClassId=550987013&ppkCompetencyDisplayId=1&ppkInstI
d=33645685&ppkSchoolGroupId...
We know the starting URL pattern.

The rest of the URL is really dynamic and the page locators that happen to be available are not Selenium
friendly. We could use XPath but that has proven to be really brittle for us.

Solution - Store the html source and use storeEval with a regex to find the correct link every time

March 15, 2012

Automation - judges.py
■ Headless Selenium servers

■ Suites of tests for each production webapp using
smoke tests

■ Run through suites serially in a never ending loop

■ Any failure in suites marks the status as failed for
that server

■ Look at each servers total result time running the
suite based on thresholds

■ Running helps see what server is currently being
tested
■ Results projected on wall in Ops area

March 15, 2012

We have a fundamental problem

Sysadmins Customers

March 15, 2012

Solution - Any browser any platform

Bromine

Grid
March 15, 2012

Loadtesting Selenium vs JMeter

● Lots of start up overhead ● Loads quickly

● ~30 concurrent sessions before ● ~300 concurrent sessions before
strange things happen strange things happen

● Lots of requirements / coordination ● Java and a jar, headless is easy
headless is a hack

March 15, 2012

Loadtesting with JMeter
Queries to find
interesting users

Selenium Scripts IN
Scripts to replace variables
for Selenium suites

Playback Selenium scripts
through JMeter proxy

JMeter Scripts OUT

March 15, 2012

Loadtesting with JMeter

DEMO
TIME
Setting up jmeter as a proxy
recorder for Selenium
playback

March 15, 2012

Loadtesting with JMeter - DEMO TIME

Start JMeter and create a new basic WorkBench
March 15, 2012


Add an HTTP Proxy Server: Edit -> Add -> Non-Test Elements -> HTTP Proxy Server
March 15, 2012


Set your port, any record filters of what to include or exclude and click start to fire up the proxy
March 15, 2012


Set Firefox network settings to use localhost and the port you configured JMeter proxy to use
March 15, 2012


Playback your tests using Selenium and watch as JMeter records the hits
March 15, 2012


Add a thread group by right clicking Test Plan -> Add -> Threads -> Thread Group
March 15, 2012


Name your thread group, pick how many threads you want, ramp up time and how many loops
March 15, 2012


Run your tests and set servers ablaze. Run -> Start

March 15, 2012

Finding hard bugs

Uh oh! Just throwing more memory at the problem will not solve this!
March 15, 2012

Finding hard bugs

Did we see this in the environments below Production? Not very noticeable unless looking hard for it.
March 15, 2012

Finding hard bugs

Ramping up JMeter threads of a specific user in an endless loop reproduces issue in every environment
March 15, 2012

Dashboard Setup?

Use Selenium to login and navigate to your various dashboards for TV's and sharing with teams.
March 15, 2012

QAOps?

A lot of quick wins can
be made here.

Getting most of the focus these days.
Certainly best of all worlds coming
together. Takes a while to be a good Selenium IDE makes it trivial
practitioner based on tech debt | servers | for Ops to start running and
timeboxes | complexity | culture developing tests

March 15, 2012

Questions?

We are hiring!
http://www.wirelessgeneration.com
March 15, 2012

Picture Credits
http://uploads.neatorama.com/wp-content/uploads/2010/06/macgyver.jpg

http://www.packagingsuppliesbymail.com/images/C/Duct%20Tape.jpg

http://www.russpainter.com/shop/images/clip.jpg

http://g-ecx.images-amazon.com/images/G/01/stores/sport-goods/1209-huntsman.jpg

http://www.agilegeoscience.com/storage/post-images/python-logo-small.png?
__SQUARESPACE_CACHEVERSION=1316752792052

March 15, 2012

Picture Credits
http://www.pptbackground.net/plog-content/images/powerpoint/education-backgrounds/raising-hands-
together-powerpoint-backgrounds.jpg

http://seleniumhq.org/images/selenium-ide-logo.png

http://upload.wikimedia.org/wikipedia/en/0/07/Jakarta_jmeter_logo.jpg

http://cdn.babble.com/momcrunch/files/2012/02/SauceLabs_Logo_Vert_150px.gif

http://gigaom2.files.wordpress.com/2011/01/firefox-ie.png

March 15, 2012

Picture Credits
http://3.bp.blogspot.com/_t5Xqp0AYeg4/S74FdH-SO-I/AAAAAAAAAOU/OlMrI9m8VIE/s320/smileyface1.jpg

http://blogs.babycenter.com/wp-content/uploads/2008/02/crying-face.jpg

http://www.portlandmercury.com/imager/dr-strangelove/b/original/3791589/80b9/mwbw-drstrangelove.jpg

http://stylreesta.files.wordpress.com/2011/07/caution_big.png

http://en.wikipedia.org/wiki/File:Devops.png

March 15, 2012

Picture Credits
http://upload.wikimedia.org/wikipedia/commons/thumb/a/af/Tux.png/220px-Tux.png

http://www.wisedatasecurity.com/imagenes/ie-8.jpg

http://seleniumhq.org/images/selenium-grid-logo.png

http://www.studiosysadmins.com/site_media/attachments.
gallery/2009/05/12/ServerRoomFireWeb.jpg

http://blogs.splunk.com/wp-content/uploads/2010/02/Picture-65.png

March 15, 2012

QaOps - NYC Selenium Users Meetup - 03/15/12

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Empfohlen

Empfohlen (20)

QaOps - NYC Selenium Users Meetup - 03/15/12