Mark meninger-feb-2010

An Examination of Test Automation Use Cases
(and the factors behind the examination)

January 19, 2010

Mark Meninger

1

January 20, 2010

An Examination of Test Automation Use Cases

Contents
Objectives and Perspective
• The Speaker
• Presentation Objectives

Test Automation Philosophy
• Objectives of test automation
• GUI vs non-GUI
• A look at 2 products

Test Automation Use Cases

How does this all fit in?
2 Confidential McAfee Internal Use Only

January 20, 2010



January 20, 2010

The Speaker (Mark_Meninger@McAfee.com)

• Current role @ McAfee: Driving test automation on Consumer side
• Functional GUI automation (support for L10n) across 3 sites
• Functional non-GUI automation (development focused)
• Automated Performance testing
• Previously @ RIM
• As test automation manager for end-user BlackBerry testing
• RIM Test Automation Conference Chair

• A student of test automation and testing
• Worked (and work) with some very capable individuals
• Made some good mistakes
• Learned from my mistakes
• Little experience with Agile
• Presentations @ StarEast, QAI, KWSQA
• My4Blog (as of this Friday): http://automatedtestingblog.blogspot.com
Confidential McAfee Internal Use Only

January 20, 2010

Presentation Objectives

1. Talk about my learnings and philosophy of test automation
2. Hear your perspective on what I’m talking about
3. Evaluate test automation use cases/examples (group-
exercise)
a) Interface complexity
b) Speed of test automation
c) Speed of execution
d) Derived value
e) How this fits into the testing cycle
4. In the context of Agile, I would…


January 20, 2010

Mark’s Test Automation Philosophy


January 20, 2010

Manual Testing - A path in the trees

• Strategy/Plan (resourcing, infrastructure, approach)
• Need to learn the SUT:
• User-stories
• Use Cases
• Tools
• Technologies
• Testing types & cycles (regression, defect, smoke)
• Sprints
• Apply common techniques
• Scripted test cases (with different methodologies),
• Exploratory testing
• Then testing can begin

• I’ve seen most testing organizations do this well to varying degrees

January 20, 2010

Automated testing – Find a path in the forest

• Test automation is less of a cookie-cutter approach than manual testing
• Common tasks in test automation
• Interface, framework, integration
• Very specific considerations:
• The technology of the system under test (SUT)
• The technologies used to test the SUT
• Impact on the development/testing schedule

• These details impact the success of the
implementation, the derived value, the
time to delivery, availability etc
• This makes each test automation
implementation unique and challenging

• Not all orgs do test automation well

January 20, 2010

Evolution of the Approach to Test Automation

In the beginning:

Here is a test automation tool… now automate!
(seen it)


January 20, 2010


And Then:

Evaluate tools, choose one and automate!
(done it)


January 20, 2010


After much learning (now do it!) :
Examine my test automation requirements
a) Look at
• Objectives
• SUT technology, roadmap
• Available budget, resources
• Available tool technologies, capabilities and constraints
• SDLC methodology (Agile, Waterfall etc.)
• Development culture
• Testing culture
• Relationships
b) Put together my business case
c) Start with a careful plan

January 20, 2010

Objectives of test automation

1. Executed successfully Test functionality sooner
relatively early within the rather than later in the cycle
development/testing cycle Interface used is available
and stable
2. Reduction in manual testing Automated execution as soon
time as a build is released:
Evenings, Weekends
Automated execution run in
parallel
Execution across platforms and
test suites
Limited by availability of
infrastructure

January 20, 2010

Test Automation Objectives

3. Provides reliable results on Tests can be executed repeatedly
1st run with minimal false negatives
Regular testing of interface and
framework essential
Changing interface not breaking
tests

We can increase coverage of
4. The solution is scalable & manual test cases without excessive
maintainable framework growth
Grow test case numbers with
confidence:
10’s -> 100’s
100’s -> 1000’s

January 20, 2010

Test Automation Objectives

5. Agile Environment Out-of-the-box, light-weight test
automation supports planned sprints
Test automation team structured,
organized to support agile testing
requirements
Solution, approach is flexible and
expandable


January 20, 2010

GUI vs non-GUI Technology Considerations

Non-GUI
GUI (some
API)


January 20, 2010

Test Automation – The Big Question

Do we
UI
start
here? UI-Logic Interface
Or here? First Business Logic Layer Software
Re Abstractions
Or here? Another Abstraction

And Another
Where?
The Basement


January 20, 2010

Test Automation - GUI Fat Client (Generalities)

GUI - Fat Client
• Interface • Skill-set
• Typically complex - more points of failure • Requires solid technical skill-set with
• Changes more frequent good design and programming abilities
• Increased maintenance • Deeper test automation skill-set and
experience required
• Automation - dependent upon being
stable • More expensive
• Technology • Localization
• Vendor tools provide better support • Running identical functional tests across
localized builds
• Scripting technology usually has limited
support (i.e. VBScript) • Language independence support is
essential to support L10n automation
• Performance
• Management for unresponsive GUI
• GUI adds performance hit each time
accessed
• Framework can add considerable
overhead

January 20, 2010

Test Automation – GUI Web (Generalities)

GUI - Web
• Interface
• Typically simpler – fewer complexities to manage
• Changes frequently over time
• Automation – dependent upon being stable
• Technology
• Good open source (Selenium, Watir)
• Performance
• Fewer if no performance issues
• Good available open source frameworks
• Skill-set
• Better use of ‘scripters’ rather than developers
• Less investment required in more experienced test automation skill-sets
• Less expensive
• Localization
• Language independent testing can be supported

January 20, 2010

Test Automation – non-GUI (Generalities)

Non-GUI (API) Automation
• Interface • Performance
• Stable and reliable earlier in the • No GUI impact - much faster
dev/testing cycle • Choose integrated framework with little
• Less changes over time overhead
• Fewer points of failure • Skill-set
• High-level testing can be supported • Requires knowledge of underlying
• Usually needs to be modified for increased business logic, application architecture and
testability over time (fits Agile well IMO) design
• Technology • More expensive
• Choose scripting technology with robust • Localization
library support • Issues around language independence
• Integrate into development cycle (localized strings integrated into GUI
abstractions)


January 20, 2010

GUI Automation

GUI
• Automation Implication
• Coverage
• Good top-to-bottom testing solution
• Broader
• # of automated test cases dependent upon GUI comlexity
• Development (GUI complexity)
• Higher maintenance costs
• More effort to write tests
• Longer development times
• Execution
• Dependent upon stable GUI
• Slower execution times
• More expensive (resources, equipment and license cost)
• Value-add later in dev/testing cycle for products with major GUI changes


January 20, 2010

Non-GUI Automation

Non-GUI Automation
• Automation Implication
• Coverage
• Not top-to-bottom
• Deeper testing
• Larger # of automated test cases
• Development
• Easier to write tests
• Shorter development times
• Lower maintenance costs
• Execution
• Ready when the API are ready
• Test much sooner in the dev/test
• Test results faster; test more often
• Gate before release to testing
• Cheaper (resources, no licenses)
• Add quality paradigm to development organization (code for testability)
•21Value-add sooner in dev/testing cycle Confidential McAfee Internal Use Only

API Automation
Manual Testing
GUI Automation

Automated Automated
API Testing API Testing

No No Tests
Tests
Pass? Pass?
Yes
Yes Automated GUI
Manual Testing & Manual
Quality Testing Testing
Assurance
Starts
??
!!!
Start End

22 January 20, 2010

RTQA Build Release Cycle

January 20, 2010

Complex GUI - Cost for Coverage

• Complex GUI applications will incur an increasing cost to
automate larger #’s of test cases
• Cost: Effort, Infrastructure, Dollars, Time for Execution

Automation
Cost Ceiling

GUI
Costs become
prohibitive

Automated Testing Coverage

January 20, 2010

Automation Philosophy Review – Cost for Coverage

• Non-GUI Automation has lower cost for coverage
• Cost: Effort, Infrastructure, Dollars, Time for Execution

Automation
Cost Ceiling

GUI
Less
expensive
Non-
Non-
GUI24
Automated Testing Coverage

Consumer Test Automation Framework (CTAF)/QTP/MAGI

Lots of VBScript Test Script
libraries
CTAF External Libraries CTAF Internal
to build GUI/
Assert Language
Dependent
Libraries
Unit Test
Table
(and MPF MVS
Helper

Global
support)
r Our own
MAGI Framework CTAF
Complex Services
Core
Extensions extensions
Functions
Framework
(yet very
HP Quick Test Pro
helpful)
GUI HTML ID HTML ID HTML ID
Control Control Control


Example API Test Automation – Using
Python

Python Test Script
Framework
exists
Python Python py.test/nose(?) Test Automation which
has Framework
nicely
Python CTAF API Python
numerous Libraries Libraries integrates
rich and with
diverse Python
libraries COM
High-
Level
High-
Level
High-
Level
Interface Class Class Class

Implement IDispatch


January 20, 2010


A look at some interfaces
• Google Search
• McAfee Security Center (2010 release)

Disclaimer
• These thoughts are my own and are not necessary correct
• Part of the process would be for me to understand the
technology and determine the best approach
(Finding a path through the forest I find myself in)


January 20, 2010

Points to Evaluate
• What approach would you take?
• What are the risks of doing it this way?
• What are the costs/investment requirements of doing this? (be
specific)
• What are the gains of doing this?
• Why would you do this?
• How would you integrate this into a testing cycle?
• Why would you integrate this into the testing cycle this way?
• How could this approach be fit into an Agile/Lean
methodology?
• What type of buy-in would you need to achieve this?
• What relationships would you need to establish to be
successful?

January 20, 2010

What I will focus on

• Interface complexity
• Speed of test automation
• Speed of execution
• Derived value
• How this fits into the testing cycle


January 20, 2010


January 20, 2010

Test Automation Use Cases – Google Search

• Google Search
– From: http://en.wikipedia.org/wiki/Google_search
– PageRank Logic
– synonyms, weather forecasts, time zones, stock quotes,
maps, earthquake data, movie showtimes, airports, home
listings, and sports scores
– prices, temperatures, money/unit conversions ("10.5 cm in
inches"), calculations ( 3*4+sqrt(6)-pi/2 ), package tracking,
patents, area codes,[6] and rudimentary language
translation of displayed pages
– ‘I’m feeling lucky’
– Privatization logic (Switzerland)
– Ajax code
– ….

January 20, 2010


Our job to automate Search logic
• Core functionality
• Page rank is an algorithm that evaluates an index using
supposedly over 200 factors


January 20, 2010


•What I’d do
– Back-end (non-gui) automation
– Focus just on the engine and it’s data
– Evaluate the probability and weighting of each factor
– Generate a list of results and probably would fit into
some level of confidence of accuracy
– Rendering of data could be easily verified manually


January 20, 2010


• How
• Work with developers to build a testing engine that could handle
‘plug-ins’ of new testing factors
• A common testing pattern would be for each factor
• Determine how each factor would return results on its own or in
interaction with another factor
• Integrate this all into the testing engine
• Establish a common mechanism for executing tests, gathering
expected results and comparing actual results
• Utilize the common testing pattern
• Utilize the same pattern for evaluation in the testing engine
• Drive a common testing interface across those adding new ranking
functionality to the google search engine


January 20, 2010


Interface complexity
• Interface would be simple – an API
• The combination of algorithms (factors) would make this
complex

Speed of test automation delivery
• Fast
– Could start writing tests for each factor
– Could start building in some relationships for each factor
• Slower
– Integration of everything together
– This would also include the test engine


January 20, 2010


Speed of execution
• Very fast
• Could run very often

Derived value
• Substantial
• Could fully automate the algorithm
• If the integration of factors together could be done
successfully, this would be substantial

How this fits into the testing cycle
• Immediately
• Write code, test code

• No throw-over to SV&V/QA

January 20, 2010


January 20, 2010

Cialis - erectile dysfunction drug
Radical Prostatectomy - is an operation to
remove the prostate gland and some of
the tissue around it Very fast


January 20, 2010


January 20, 2010

Test Automation Use Cases – McAfee Security
Center

• McAfee Consumer Product (2010)
– Security Center
– Firewall, AntiVirus, AntiSpam, Parental Controls…
– Focus on AntiVirus
– Verify GUI and Functional behaviour


January 20, 2010


– GUI – thick GUI


January 20, 2010


• How
• Vendor tool


January 20, 2010




January 20, 2010

Center

•What I’d do & How
– Get a good GUI automation tool
– Find a manageable way to integrate with the GUI DOM
• Re-usability
• Maintainability
– First go: limit my automation to easiest test cases
– Find a good framework to integrate with my GUI
automation tool
• Automate the automation
– Aim would be reduce manual testing with an eye to
reduce maintenance


January 20, 2010

Center
• Interface is very complex
– Lots of objects to manage (lots of points of failure)
– How we work with the interface is complex under the
covers (constrained by tool)

• Slow!!
– Most off-the-shelf have limited library support
– Have common set of libraries across the products
– Need to build libraries for each product at the GUI & functional
level
– Integration into the framework could take even longer

January 20, 2010

Center
Speed of execution
• Slow!
• Fat client GUIs are usually slower
• Hooks from application driving the GUI adds overhead
• Integrating a framework that drives all this adds additional
time

Derived value
• Some!
• New GUI; some changes – later in the cycle
• Same GUI; little changes – earlier in the cycle
• Need more infrastructure to support

January 20, 2010

Center
• If no GUI changes - > Right Away
• GUI changes - > Later
• Testing cycles will take longer
• Slower to execute


January 20, 2010

Test Automation Use Cases – McAfee Security Center

Test Automation – The Big Question

Do we
UI
start
here? UI-Logic Interface

First Business Logic Layer Software
Re Abstractions
Or here? Another Abstraction

And Another

The Basement


January 20, 2010

Center

•What I’d do & How
– Examine the layers below the GUI
– Can I use any of them to automate testing?
– Work to drive this idea within the organization
– Most likely some developer has thought the same thing
– That is my foothold and I build on that


January 20, 2010

Center
• Interface is less complex
– New technology learning curve
– Easier to call some API than manage GUI objects
– Less change; more stability

– Fast!!
– Available frameworks (don’t need to build your own) (i.e. py.test)
– Access to rich scripting environments & libraries (i.e. Python) – don’t
need to build your own
– Less points of failure to manage


January 20, 2010

Center
Speed of execution
• Fast!! No GUI overhead
• Integrated framework will also add little overhead (i.e.
TestNG, Py.test)

Derived value
• Higher value

• Almost always test earlier in the cycle
• Test more frequently
• Integrate into the development cycle rather than QA
– Quality moving upstream

January 20, 2010


January 20, 2010


• Have your test automation specialist begin to develop a methodology that
will fit your agile development cycle
• Tell her/him what your requirements are and ask for a solution that will
meet this
• Optimize your test automation execution for Agile!
• This will most likely be an out-of-the-box approach to test automation

• All pay-offs should be well understood:
• Light-weight, quick to execute, easy to develop, not-too-deep solution
• Heavier, complex, longer-to execute, harder to develop, deeper test
automation solution


January 20, 2010


• They should have a good understanding of the available technologies to
use and what the trade-offs are for each
• What solution will best meet the above requirements?
• Let that test automation specialist know what’s needed and why. This will
hopefully inspire them to build the solution that meets these needs

• Integrate automated testing into your testing cycles
• Fit the automated testing for a given sprint/release
• Each automated testing approach will have a given set of benefits and
costs
• Choose the automated testing items that make most sense


January 20, 2010
Consumer Applications QA
Test Automation Strategy – Detailed Review

Thank-you
Mark_Meninger@McAfee.com

As of this Friday
http://automatedtestingblog.blogspot.com

Mark meninger-feb-2010

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (19)

Ähnlich wie Mark meninger-feb-2010

Ähnlich wie Mark meninger-feb-2010 (20)

Mark meninger-feb-2010