SlideShare ist ein Scribd-Unternehmen logo
1 von 27
Downloaden Sie, um offline zu lesen
Context-Driven Test Automation
How to Build the System you Really Need
Pete Schneider
P.Schneider@F5.com
F5 Networks
License
The content of these slides is made available
under the Create Commons Attribution
Share-Alike 3.0 United States license.
Where this talk came from…
• F5 Networks has lots of product teams who
have their own automation tools.
• We were taking inventory of the tools and
trying to see where we could share and
eliminate duplication.
• We looked at 11 different test automation
tools.
We saw 6 common tasks…
• All the tools had ways of addressing common
tasks:
– Test distribution and run control
– Test case set up
– Test case execution
– Test case evaluation
– Test case tear down
– Results reporting
We discussed the tasks…
• Test distribution and run-time control
– Some tools had sophisticated controls, some were rudimentary.
– Some were automatic, some required manual effort.
• Set up / tear down
– Some tools took care of a lot of the work in setting up and
tearing down a test, others left it all to the test case.
• Test execution and verification
– Except for data-driven tests, all tools left this entirely to the test
case.
– We saw huge variations in complexity of verification.
• Reporting
– All tools sent results via email.
– Some had web GUIs, some didn’t.
We argued…
• Everyone agreed that the six tasks existed and
were important.
• We did not agree on the relative importance
of each task.
• We did not agree on what was needed to
meet requirements for each task.
The light bulb came on…
• We realized that we were approaching test
automation from different directions, with
different intentions.
• In short, we had different contexts.
We looked at the tools again…
• We tried to figure out how to group the tools
• The context of the tool was the key
• Who writes the tests?
• Who looks at the results?
• What decisions do the results influence?
We came up with 4 contexts in our setting…
Context Tests written by Results looked at
by
Decisions
influenced by
Individual
Developer
Developers Developers Code check-in
Development
Team
Developers
and/or testers
Testers,
developers, PM’s
Branch merges,
releases
Project Testers Testers, PM’s Project
milestones,
releases
Product Line Testers Testers, PM’s,
senior
management
Updates and
maintenance
releases
Will the refactoring I just
finished break anything?
Individual Developer context…
• A common example is Unit Tests.
• These tests need to be very quick, duration
measured in seconds.
• They test very small pieces of functionality—
e.g. a single procedure in an API.
• Writing them requires deep knowledge of
product code.
• They should be considered part of the product
code deliverable, i.e. the code isn’t finished if
there are no unit tests
• See xUnit Test Patterns, by Gerard Meszaro
If we merge our feature
branch to main, will we
break anything?
Development Team context…
• These tests focus on a specific area of
functionality or a subsystem of the product.
• They still need to be fast, but speed is not as
critical.
• The tests may use an interface that is not directly
available to product users.
• Writing these kinds of tests requires significant
expertise in the specific protocol/feature.
• Once fully implemented, the tests can be
migrated to project/product-line testing.
Are the builds becoming
more stable or less stable?
Project context…
• Focus on user functionality of the system
• Speed is desirable, but not essential
• Requires a more complex infrastructure
– Hardware dependencies
– Variations in expected results from release to
release
– Other external dependencies
• Reporting is critical
• Can be migrated to Product Line easily
Will this patch work for
customers running Basic, Pro,
and Premiere editions with
Service Packs 1, 2 or 3?
Product Line context…
• This automation is intended to run on releases
that are out in the field.
• The automation may take a very long time to run.
• Goals:
– Ensure that patches fix the problem they claim to fix.
– Ensure that they don’t break something else.
• Reliability is critical.
• These tests are challenging to maintain.
• Run-time control is a big deal.
Case Study: ITE…
• Summary:
– The ITE is STAF/STAX based.
– It was developed by testers for use by other testers,
with developers as a secondary target.
• ITE Design Criteria:
– Allow hands-off execution of tests.
– Allow the test harness to automatically determine
which tests to run.
– Reduce the set up/tear down burden on test writers.
ITE…
• Distribution / Runtime control
– Tests and framework are distributed as a linux chroot
that includes all dependencies.
– Both tests and framework stored in source control.
– Tests tagged with meta-data used to control runs.
• Test Setup
– The ITE offers services to configure DUT and various
test services.
• Execution
– This is largely left to test writer. The ITE is beginning
to support data-driven tests.
ITE…
• Verification
– This is largely left to the test writer.
– The ITE performs “health checks” on DUT.
• Teardown
– It performs more extensive cleanup after “subjobs”
complete.
• Results Reporting
– The ITE sends email after completion of runs, it also
stores results in database.
– Web pages are available for viewing results.
Case Study: xBVT…
• Summary:
– The xBVT is a Perl based system.
– It was developed by a developer for use by other
developers, with testers as secondary target.
• xBVT Design Criteria:
– Tests should be able to run inside or outside the
tool.
– Impose little/no overhead on test writers and
runners.
xBVT…
• Distribution / Runtime control
– Tests and framework are stored in source control.
– Tests are stored with the product code
– Runtime execution is determined by “test manifests.”
Manifests can be nested to arbitrary depths.
• Setup
– The xBVT provides the test with login credentials and
ip address. Test is responsible for configuring system.
• Test execution
– Execution is left to test writer.
xBVT…
• Results verification
– Verification is left to test writer.
• Teardown
– Teardown is left to test writer. The expectation is that
each test will clean up completely, leaving system as it
was prior to test.
• Reporting
– A text file is generated containing pass/fail results for
each test.
– The text files are emailed out when run is completed.
They are also stored on a web page.
What I Learned…
• If you have trouble agreeing, take a step back.
• There are many different approaches that will
work, the one that will work best for you
depends on your test writers, framework
writers, and automation customers.
• Rather than build “one framework to test
them all”, consider building sharable
components.
How you can use this…
• Ask yourself:
– Who is going to write and maintain the
framework?
– Who will build and maintain the tests?
– How are the tests going to be used?
– How long will the tests live?
In conclusion…
Define your context:
– Who is going to write the tests?
– Who is going to look at the results?
– What decisions will the test results influence?
Determine how your automation will implement the 6 actions:
– Test distribution and run control
– Test set up / tear down
– Test execution / Results evaluation
– Reporting
Acknowledgements …
Thanks to the members of F5’s cross-functional tools team
• Brian Sullivan, Chris Rouillard, Ephraim Dan, Patrick Walters, Sebastian Kamyshenko,
Bob Conard, Terry Swartz
Thanks to the members of F5’s automated test team
• Henry Su, Rex Stith, Randy Holte, Richard Jones, James Saryerwinnie
Special Thanks to
• John Hall, Brian DeGeeter, Ryan Allen, and Brian Branagan

Weitere ähnliche Inhalte

Was ist angesagt?

Fundamentals of Software Engineering
Fundamentals of Software Engineering Fundamentals of Software Engineering
Fundamentals of Software Engineering Madhar Khan Pathan
 
Test Automation
Test AutomationTest Automation
Test Automationrockoder
 
Softwaretestingstrategies
SoftwaretestingstrategiesSoftwaretestingstrategies
Softwaretestingstrategiessaieswar19
 
Agile Testing Strategy
Agile Testing StrategyAgile Testing Strategy
Agile Testing Strategytharindakasun
 
Automation testing introduction for FujiNet
Automation testing introduction for FujiNetAutomation testing introduction for FujiNet
Automation testing introduction for FujiNetHai Tran Son
 
Automated visual-regression-testing (1)
Automated visual-regression-testing (1)Automated visual-regression-testing (1)
Automated visual-regression-testing (1)Sriram Angajala
 
Fundamentals of software part 1
Fundamentals of software part 1Fundamentals of software part 1
Fundamentals of software part 1Siddharth Sharma
 
Unit Testing, TDD and the Walking Skeleton
Unit Testing, TDD and the Walking SkeletonUnit Testing, TDD and the Walking Skeleton
Unit Testing, TDD and the Walking SkeletonSeb Rose
 
Software Engineering Lec 10 -software testing--
Software Engineering Lec 10 -software testing--Software Engineering Lec 10 -software testing--
Software Engineering Lec 10 -software testing--Taymoor Nazmy
 
Ncerc rlmca202 adm m3 ssm
Ncerc rlmca202  adm m3 ssmNcerc rlmca202  adm m3 ssm
Ncerc rlmca202 adm m3 ssmssmarar
 
Frank Cohen - Are We Ready For Cloud Testing - EuroSTAR 2010
Frank Cohen - Are We Ready For Cloud Testing - EuroSTAR 2010Frank Cohen - Are We Ready For Cloud Testing - EuroSTAR 2010
Frank Cohen - Are We Ready For Cloud Testing - EuroSTAR 2010TEST Huddle
 
N-version programming
N-version programmingN-version programming
N-version programmingshabnam0102
 
Performance testing with jmeter
Performance testing with jmeter Performance testing with jmeter
Performance testing with jmeter Knoldus Inc.
 

Was ist angesagt? (19)

Manual testing - Introduction to Manual Software testing
Manual testing - Introduction to Manual Software testingManual testing - Introduction to Manual Software testing
Manual testing - Introduction to Manual Software testing
 
Automation Testing
Automation TestingAutomation Testing
Automation Testing
 
Chapter 6
Chapter 6Chapter 6
Chapter 6
 
Patch Review
Patch ReviewPatch Review
Patch Review
 
Fundamentals of Software Engineering
Fundamentals of Software Engineering Fundamentals of Software Engineering
Fundamentals of Software Engineering
 
Test Automation
Test AutomationTest Automation
Test Automation
 
Softwaretestingstrategies
SoftwaretestingstrategiesSoftwaretestingstrategies
Softwaretestingstrategies
 
Software Architecture
Software ArchitectureSoftware Architecture
Software Architecture
 
Agile Testing Strategy
Agile Testing StrategyAgile Testing Strategy
Agile Testing Strategy
 
Automation testing introduction for FujiNet
Automation testing introduction for FujiNetAutomation testing introduction for FujiNet
Automation testing introduction for FujiNet
 
Automated visual-regression-testing (1)
Automated visual-regression-testing (1)Automated visual-regression-testing (1)
Automated visual-regression-testing (1)
 
Fundamentals of software part 1
Fundamentals of software part 1Fundamentals of software part 1
Fundamentals of software part 1
 
Unit Testing, TDD and the Walking Skeleton
Unit Testing, TDD and the Walking SkeletonUnit Testing, TDD and the Walking Skeleton
Unit Testing, TDD and the Walking Skeleton
 
Software Engineering Lec 10 -software testing--
Software Engineering Lec 10 -software testing--Software Engineering Lec 10 -software testing--
Software Engineering Lec 10 -software testing--
 
Ncerc rlmca202 adm m3 ssm
Ncerc rlmca202  adm m3 ssmNcerc rlmca202  adm m3 ssm
Ncerc rlmca202 adm m3 ssm
 
Frank Cohen - Are We Ready For Cloud Testing - EuroSTAR 2010
Frank Cohen - Are We Ready For Cloud Testing - EuroSTAR 2010Frank Cohen - Are We Ready For Cloud Testing - EuroSTAR 2010
Frank Cohen - Are We Ready For Cloud Testing - EuroSTAR 2010
 
Continuous Integration
Continuous IntegrationContinuous Integration
Continuous Integration
 
N-version programming
N-version programmingN-version programming
N-version programming
 
Performance testing with jmeter
Performance testing with jmeter Performance testing with jmeter
Performance testing with jmeter
 

Andere mochten auch

The New Era of Cognitive Computing
The New Era of Cognitive ComputingThe New Era of Cognitive Computing
The New Era of Cognitive ComputingIBM Research
 
Breaking Azure for Fun and Profit
Breaking Azure for Fun and ProfitBreaking Azure for Fun and Profit
Breaking Azure for Fun and ProfitPavelMichailov
 
Cognitive Computing : Trends to Watch in 2016
Cognitive Computing:  Trends to Watch in 2016Cognitive Computing:  Trends to Watch in 2016
Cognitive Computing : Trends to Watch in 2016Bill Chamberlin
 
Event driven-automation and workflows
Event driven-automation and workflowsEvent driven-automation and workflows
Event driven-automation and workflowsDmitri Zimine
 
The Hive Think Tank: The Future Of Customer Support - AI Driven Automation
The Hive Think Tank: The Future Of Customer Support - AI Driven AutomationThe Hive Think Tank: The Future Of Customer Support - AI Driven Automation
The Hive Think Tank: The Future Of Customer Support - AI Driven AutomationThe Hive
 
Naghi Prasad at AI Frontiers: Building AI systems to automate enterprise proc...
Naghi Prasad at AI Frontiers: Building AI systems to automate enterprise proc...Naghi Prasad at AI Frontiers: Building AI systems to automate enterprise proc...
Naghi Prasad at AI Frontiers: Building AI systems to automate enterprise proc...AI Frontiers
 
The Hive Think Tank: Machine Learning Applications in Genomics by Prof. Jian ...
The Hive Think Tank: Machine Learning Applications in Genomics by Prof. Jian ...The Hive Think Tank: Machine Learning Applications in Genomics by Prof. Jian ...
The Hive Think Tank: Machine Learning Applications in Genomics by Prof. Jian ...The Hive
 

Andere mochten auch (7)

The New Era of Cognitive Computing
The New Era of Cognitive ComputingThe New Era of Cognitive Computing
The New Era of Cognitive Computing
 
Breaking Azure for Fun and Profit
Breaking Azure for Fun and ProfitBreaking Azure for Fun and Profit
Breaking Azure for Fun and Profit
 
Cognitive Computing : Trends to Watch in 2016
Cognitive Computing:  Trends to Watch in 2016Cognitive Computing:  Trends to Watch in 2016
Cognitive Computing : Trends to Watch in 2016
 
Event driven-automation and workflows
Event driven-automation and workflowsEvent driven-automation and workflows
Event driven-automation and workflows
 
The Hive Think Tank: The Future Of Customer Support - AI Driven Automation
The Hive Think Tank: The Future Of Customer Support - AI Driven AutomationThe Hive Think Tank: The Future Of Customer Support - AI Driven Automation
The Hive Think Tank: The Future Of Customer Support - AI Driven Automation
 
Naghi Prasad at AI Frontiers: Building AI systems to automate enterprise proc...
Naghi Prasad at AI Frontiers: Building AI systems to automate enterprise proc...Naghi Prasad at AI Frontiers: Building AI systems to automate enterprise proc...
Naghi Prasad at AI Frontiers: Building AI systems to automate enterprise proc...
 
The Hive Think Tank: Machine Learning Applications in Genomics by Prof. Jian ...
The Hive Think Tank: Machine Learning Applications in Genomics by Prof. Jian ...The Hive Think Tank: Machine Learning Applications in Genomics by Prof. Jian ...
The Hive Think Tank: Machine Learning Applications in Genomics by Prof. Jian ...
 

Ähnlich wie Context Driven Automation Gtac 2008

The QA/Testing Process
The QA/Testing ProcessThe QA/Testing Process
The QA/Testing ProcessSynerzip
 
Lecture3.se.pptx
Lecture3.se.pptxLecture3.se.pptx
Lecture3.se.pptxAmna Ch
 
unit-2_20-july-2018 (1).pptx
unit-2_20-july-2018 (1).pptxunit-2_20-july-2018 (1).pptx
unit-2_20-july-2018 (1).pptxPriyaFulpagare1
 
testing strategies and tactics
 testing strategies and tactics testing strategies and tactics
testing strategies and tacticsPreeti Mishra
 
Software Engineering (Testing Overview)
Software Engineering (Testing Overview)Software Engineering (Testing Overview)
Software Engineering (Testing Overview)ShudipPal
 
Automated testing overview
Automated testing overviewAutomated testing overview
Automated testing overviewAlex Pop
 
39.Testing-Part3.ppt
39.Testing-Part3.ppt39.Testing-Part3.ppt
39.Testing-Part3.pptSapnaRastogi3
 
Se 381 - lec 28 -- 34 - 12 jun12 - testing 1 of 2
Se 381 -  lec 28 -- 34 - 12 jun12 - testing 1 of 2Se 381 -  lec 28 -- 34 - 12 jun12 - testing 1 of 2
Se 381 - lec 28 -- 34 - 12 jun12 - testing 1 of 2babak danyal
 
Software Engineering (Software Quality Assurance & Testing: Supplementary Mat...
Software Engineering (Software Quality Assurance & Testing: Supplementary Mat...Software Engineering (Software Quality Assurance & Testing: Supplementary Mat...
Software Engineering (Software Quality Assurance & Testing: Supplementary Mat...ShudipPal
 
Cucumber jvm best practices v3
Cucumber jvm best practices v3Cucumber jvm best practices v3
Cucumber jvm best practices v3Ahmed Misbah
 
Software testing part
Software testing partSoftware testing part
Software testing partPreeti Mishra
 
Object Oriented Testing(OOT) presentation slides
Object Oriented Testing(OOT) presentation slidesObject Oriented Testing(OOT) presentation slides
Object Oriented Testing(OOT) presentation slidesPunjab University
 
Agile Software Testing the Agilogy Way
Agile Software Testing the Agilogy WayAgile Software Testing the Agilogy Way
Agile Software Testing the Agilogy WayJordi Pradel
 
Software coding & testing, software engineering
Software coding & testing, software engineeringSoftware coding & testing, software engineering
Software coding & testing, software engineeringRupesh Vaishnav
 
Test Driven Development – What Works And What Doesn’t
Test Driven Development – What Works And What Doesn’t Test Driven Development – What Works And What Doesn’t
Test Driven Development – What Works And What Doesn’t Synerzip
 
Software Quality Assurance
Software Quality AssuranceSoftware Quality Assurance
Software Quality AssuranceSaqib Raza
 
Agile Testing Introduction
Agile Testing IntroductionAgile Testing Introduction
Agile Testing IntroductionHai Tran Son
 

Ähnlich wie Context Driven Automation Gtac 2008 (20)

Software automation
Software automationSoftware automation
Software automation
 
The QA/Testing Process
The QA/Testing ProcessThe QA/Testing Process
The QA/Testing Process
 
Lecture3.se.pptx
Lecture3.se.pptxLecture3.se.pptx
Lecture3.se.pptx
 
unit-2_20-july-2018 (1).pptx
unit-2_20-july-2018 (1).pptxunit-2_20-july-2018 (1).pptx
unit-2_20-july-2018 (1).pptx
 
Software testing
Software testingSoftware testing
Software testing
 
testing strategies and tactics
 testing strategies and tactics testing strategies and tactics
testing strategies and tactics
 
Software Engineering (Testing Overview)
Software Engineering (Testing Overview)Software Engineering (Testing Overview)
Software Engineering (Testing Overview)
 
Automated testing overview
Automated testing overviewAutomated testing overview
Automated testing overview
 
39.Testing-Part3.ppt
39.Testing-Part3.ppt39.Testing-Part3.ppt
39.Testing-Part3.ppt
 
Se 381 - lec 28 -- 34 - 12 jun12 - testing 1 of 2
Se 381 -  lec 28 -- 34 - 12 jun12 - testing 1 of 2Se 381 -  lec 28 -- 34 - 12 jun12 - testing 1 of 2
Se 381 - lec 28 -- 34 - 12 jun12 - testing 1 of 2
 
Software Engineering (Software Quality Assurance & Testing: Supplementary Mat...
Software Engineering (Software Quality Assurance & Testing: Supplementary Mat...Software Engineering (Software Quality Assurance & Testing: Supplementary Mat...
Software Engineering (Software Quality Assurance & Testing: Supplementary Mat...
 
Cucumber jvm best practices v3
Cucumber jvm best practices v3Cucumber jvm best practices v3
Cucumber jvm best practices v3
 
Chapter6
Chapter6Chapter6
Chapter6
 
Software testing part
Software testing partSoftware testing part
Software testing part
 
Object Oriented Testing(OOT) presentation slides
Object Oriented Testing(OOT) presentation slidesObject Oriented Testing(OOT) presentation slides
Object Oriented Testing(OOT) presentation slides
 
Agile Software Testing the Agilogy Way
Agile Software Testing the Agilogy WayAgile Software Testing the Agilogy Way
Agile Software Testing the Agilogy Way
 
Software coding & testing, software engineering
Software coding & testing, software engineeringSoftware coding & testing, software engineering
Software coding & testing, software engineering
 
Test Driven Development – What Works And What Doesn’t
Test Driven Development – What Works And What Doesn’t Test Driven Development – What Works And What Doesn’t
Test Driven Development – What Works And What Doesn’t
 
Software Quality Assurance
Software Quality AssuranceSoftware Quality Assurance
Software Quality Assurance
 
Agile Testing Introduction
Agile Testing IntroductionAgile Testing Introduction
Agile Testing Introduction
 

Kürzlich hochgeladen

Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 

Kürzlich hochgeladen (20)

Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 

Context Driven Automation Gtac 2008

  • 1. Context-Driven Test Automation How to Build the System you Really Need Pete Schneider P.Schneider@F5.com F5 Networks
  • 2. License The content of these slides is made available under the Create Commons Attribution Share-Alike 3.0 United States license.
  • 3. Where this talk came from… • F5 Networks has lots of product teams who have their own automation tools. • We were taking inventory of the tools and trying to see where we could share and eliminate duplication. • We looked at 11 different test automation tools.
  • 4. We saw 6 common tasks… • All the tools had ways of addressing common tasks: – Test distribution and run control – Test case set up – Test case execution – Test case evaluation – Test case tear down – Results reporting
  • 5. We discussed the tasks… • Test distribution and run-time control – Some tools had sophisticated controls, some were rudimentary. – Some were automatic, some required manual effort. • Set up / tear down – Some tools took care of a lot of the work in setting up and tearing down a test, others left it all to the test case. • Test execution and verification – Except for data-driven tests, all tools left this entirely to the test case. – We saw huge variations in complexity of verification. • Reporting – All tools sent results via email. – Some had web GUIs, some didn’t.
  • 6. We argued… • Everyone agreed that the six tasks existed and were important. • We did not agree on the relative importance of each task. • We did not agree on what was needed to meet requirements for each task.
  • 7. The light bulb came on… • We realized that we were approaching test automation from different directions, with different intentions. • In short, we had different contexts.
  • 8. We looked at the tools again… • We tried to figure out how to group the tools • The context of the tool was the key • Who writes the tests? • Who looks at the results? • What decisions do the results influence?
  • 9. We came up with 4 contexts in our setting… Context Tests written by Results looked at by Decisions influenced by Individual Developer Developers Developers Code check-in Development Team Developers and/or testers Testers, developers, PM’s Branch merges, releases Project Testers Testers, PM’s Project milestones, releases Product Line Testers Testers, PM’s, senior management Updates and maintenance releases
  • 10. Will the refactoring I just finished break anything?
  • 11. Individual Developer context… • A common example is Unit Tests. • These tests need to be very quick, duration measured in seconds. • They test very small pieces of functionality— e.g. a single procedure in an API. • Writing them requires deep knowledge of product code. • They should be considered part of the product code deliverable, i.e. the code isn’t finished if there are no unit tests • See xUnit Test Patterns, by Gerard Meszaro
  • 12. If we merge our feature branch to main, will we break anything?
  • 13. Development Team context… • These tests focus on a specific area of functionality or a subsystem of the product. • They still need to be fast, but speed is not as critical. • The tests may use an interface that is not directly available to product users. • Writing these kinds of tests requires significant expertise in the specific protocol/feature. • Once fully implemented, the tests can be migrated to project/product-line testing.
  • 14. Are the builds becoming more stable or less stable?
  • 15. Project context… • Focus on user functionality of the system • Speed is desirable, but not essential • Requires a more complex infrastructure – Hardware dependencies – Variations in expected results from release to release – Other external dependencies • Reporting is critical • Can be migrated to Product Line easily
  • 16. Will this patch work for customers running Basic, Pro, and Premiere editions with Service Packs 1, 2 or 3?
  • 17. Product Line context… • This automation is intended to run on releases that are out in the field. • The automation may take a very long time to run. • Goals: – Ensure that patches fix the problem they claim to fix. – Ensure that they don’t break something else. • Reliability is critical. • These tests are challenging to maintain. • Run-time control is a big deal.
  • 18. Case Study: ITE… • Summary: – The ITE is STAF/STAX based. – It was developed by testers for use by other testers, with developers as a secondary target. • ITE Design Criteria: – Allow hands-off execution of tests. – Allow the test harness to automatically determine which tests to run. – Reduce the set up/tear down burden on test writers.
  • 19. ITE… • Distribution / Runtime control – Tests and framework are distributed as a linux chroot that includes all dependencies. – Both tests and framework stored in source control. – Tests tagged with meta-data used to control runs. • Test Setup – The ITE offers services to configure DUT and various test services. • Execution – This is largely left to test writer. The ITE is beginning to support data-driven tests.
  • 20. ITE… • Verification – This is largely left to the test writer. – The ITE performs “health checks” on DUT. • Teardown – It performs more extensive cleanup after “subjobs” complete. • Results Reporting – The ITE sends email after completion of runs, it also stores results in database. – Web pages are available for viewing results.
  • 21. Case Study: xBVT… • Summary: – The xBVT is a Perl based system. – It was developed by a developer for use by other developers, with testers as secondary target. • xBVT Design Criteria: – Tests should be able to run inside or outside the tool. – Impose little/no overhead on test writers and runners.
  • 22. xBVT… • Distribution / Runtime control – Tests and framework are stored in source control. – Tests are stored with the product code – Runtime execution is determined by “test manifests.” Manifests can be nested to arbitrary depths. • Setup – The xBVT provides the test with login credentials and ip address. Test is responsible for configuring system. • Test execution – Execution is left to test writer.
  • 23. xBVT… • Results verification – Verification is left to test writer. • Teardown – Teardown is left to test writer. The expectation is that each test will clean up completely, leaving system as it was prior to test. • Reporting – A text file is generated containing pass/fail results for each test. – The text files are emailed out when run is completed. They are also stored on a web page.
  • 24. What I Learned… • If you have trouble agreeing, take a step back. • There are many different approaches that will work, the one that will work best for you depends on your test writers, framework writers, and automation customers. • Rather than build “one framework to test them all”, consider building sharable components.
  • 25. How you can use this… • Ask yourself: – Who is going to write and maintain the framework? – Who will build and maintain the tests? – How are the tests going to be used? – How long will the tests live?
  • 26. In conclusion… Define your context: – Who is going to write the tests? – Who is going to look at the results? – What decisions will the test results influence? Determine how your automation will implement the 6 actions: – Test distribution and run control – Test set up / tear down – Test execution / Results evaluation – Reporting
  • 27. Acknowledgements … Thanks to the members of F5’s cross-functional tools team • Brian Sullivan, Chris Rouillard, Ephraim Dan, Patrick Walters, Sebastian Kamyshenko, Bob Conard, Terry Swartz Thanks to the members of F5’s automated test team • Henry Su, Rex Stith, Randy Holte, Richard Jones, James Saryerwinnie Special Thanks to • John Hall, Brian DeGeeter, Ryan Allen, and Brian Branagan