SlideShare a Scribd company logo
1 of 41
Download to read offline
Quality Assurance
            in
            Distributed Continuous Delivery
            By
            Jin Ming|@金明i
            Continuous Delivery Lead, ThoughtWorks



Wednesday, August 8, 12
Text
              Background: Top One Property Listing Site in OZ




Wednesday, August 8, 12
Business Overview


                             Seller            Consumer




                                      Agency




Wednesday, August 8, 12
Architecture Overview




Wednesday, August 8, 12
How to Do QA?




Wednesday, August 8, 12
Quality is Value to Some Person.
                                     ---Gerald Weinberg




Wednesday, August 8, 12
Challenges We’re Facing




Wednesday, August 8, 12
Legacy Systems




Wednesday, August 8, 12
Cross Systems




Wednesday, August 8, 12
Distributed Development




Wednesday, August 8, 12
Also, We Want to Be




Wednesday, August 8, 12
Wednesday, August 8, 12
We Need “Total Quality”




Wednesday, August 8, 12
Test Quadrants
                    Automat                     Business Facing
                    ed
                                                             Manual
                    & Manual

                                Functional Tests
             Exploratory Testing
                                   Examples
                      Scenarios
                                  Story Tests
                 Usability Testing
                                  Prototypes
            UAT(User Acceptance Testing)
                                  Simulations
                   Alpha / Beta
                                       ...
                            ...
                                                    Q2
 Q3
                              Critique
 Supporting
 Team                                               Q1
 Q4
                              Product

                                   Unit Tests
            Performance & Load Testing
                                Component Tests
                Security Testing
                                       ...
                            ...

                    Automat
                                                                                    Tools
                    ed
                                              Technology Facing
Wednesday, August 8, 12
Functional Test Boundaries




Wednesday, August 8, 12
We Need to Ensure End-to-End




Wednesday, August 8, 12
We Need to Ensure Single System




Wednesday, August 8, 12
Units Inside Single System




Wednesday, August 8, 12
Automatic Those Tests




Wednesday, August 8, 12
Test Pyramid

              Efficiency                              Scope
                Cost             End-to-End Test   End-User
                                                   Simulation
                                  System Test
                    Scope
             End-User               Unit Test      Efficiency
             Simulation                              Cost


Wednesday, August 8, 12
Capybara
                          Functional Test Samples
                                             Selenium

                                            WebDriver
Wednesday, August 8, 12
Performance Test Samples




Wednesday, August 8, 12
Deployment Validation




Wednesday, August 8, 12
Test Plan




Wednesday, August 8, 12
Vision & High-
                          Level Stories



                                                     $


                            Iteration                         $

                                           Release                      $



                                                                                   $




                                                         Test Plan
                                                                  Release Test
                                                                  Plan
                                                                  Iteration Test
                                                                  Plan
                                                                  Daily Test
                                                                  Plan




Wednesday, August 8, 12
Iterations
                                                                                                   Stabilize (opt)

             I0               I1                  I2             I3                 I4 … In        Pre-production    Production

                          Acceptance          Exploratory,                                               UAT            PVT
        Test Analysis
                            Testing           Automation
              I1
                              I1                   I1
                                                                                                        iIities
                                                                                                      operations
                                                                                                     &Compliance
                                              Acceptance     Exploratory,
                          Test Analysis
                                                Testing      Automation                                 Data
                                I2
                                                  I2              I2                                  Migration


                                                                                                     Deployment
                                                             Acceptance        Exploratory,
                                             Test Analysis
                                                               Testing         Automation
                                                   I3
                                                                 I3                 I3                   Full
                                                                                                      Regression
                                                                                                         Test

                                                                 Automated Regression Test
                                               Integration   End-To-End        Integration     End-To-End
                                                   Test         Test               Test           Test
                                                                  Performance Profiling Test

                                                             Load/Stress                          Load/Stress




Wednesday, August 8, 12
Good Enough?




Wednesday, August 8, 12
It’s Just Validation.
                          Where is Verification?




Wednesday, August 8, 12
Wednesday, August 8, 12
Customer Domain Expert                    QA    Developer




                                                      BDD

                          Stories                 Behaviour            Application

                                                            Exercise
                                    Matches to
                                                       expectations against



Wednesday, August 8, 12
Co-Education




Wednesday, August 8, 12
Quality Assurance Activities
                                                           Business
                                                           Analytics

                                                                                 Test
                                        Showcase
                                                                               Strategy




                                                                                             Risk
                          Exploratory
                                                                                          Assessmen
                             Test
                                                                                              t
                                                      Agile Quality
                                                       Assurance




                              Common
                                                                                       Test
                              Vision on
                                                                                     Analytics
                               Quality



                                                                         Process
                                                  Test
                                                                       Improveme
                                               Framework
                                                                            nt




Wednesday, August 8, 12
Wednesday, August 8, 12
Build Quality In




Wednesday, August 8, 12
Shared Responsibility




Wednesday, August 8, 12
Communication & Collaboration




Wednesday, August 8, 12
QA is NOT JUST Testing




Wednesday, August 8, 12
Eradicating Non-Determinism in Tests




Wednesday, August 8, 12
Visualizing Quality




Wednesday, August 8, 12
Wednesday, August 8, 12
?
            ?                 ?
                          ?   ?
                   ?
Wednesday, August 8, 12

More Related Content

Similar to Quality assurance in distributed continuous delivery

Creating Maintainable Automated Acceptance Tests
Creating Maintainable Automated Acceptance TestsCreating Maintainable Automated Acceptance Tests
Creating Maintainable Automated Acceptance TestsJez Humble
 
Pivotal Labs Open View Presentation Quality Assurance And Developer Testing
Pivotal Labs Open View Presentation Quality Assurance And Developer TestingPivotal Labs Open View Presentation Quality Assurance And Developer Testing
Pivotal Labs Open View Presentation Quality Assurance And Developer Testingguestc8adce
 
Inverting The Testing Pyramid
Inverting The Testing PyramidInverting The Testing Pyramid
Inverting The Testing PyramidNaresh Jain
 
Ashwini - Effective use of CI by QA
Ashwini - Effective use of CI by QAAshwini - Effective use of CI by QA
Ashwini - Effective use of CI by QAvodQA
 
Programmer testing
Programmer testingProgrammer testing
Programmer testingJoao Pereira
 
Gabriele Lana: Testing Web Applications
Gabriele Lana: Testing Web ApplicationsGabriele Lana: Testing Web Applications
Gabriele Lana: Testing Web ApplicationsFrancesco Fullone
 
Testing Android
Testing AndroidTesting Android
Testing AndroidMarc Chung
 
ebay @ Hasso Plattner Institut Potsdam
ebay @ Hasso Plattner Institut Potsdamebay @ Hasso Plattner Institut Potsdam
ebay @ Hasso Plattner Institut PotsdamMichael Palotas
 
ICTSS 2010 - Iterative Software Testing Process for Scrum and Waterfall Projects
ICTSS 2010 - Iterative Software Testing Process for Scrum and Waterfall ProjectsICTSS 2010 - Iterative Software Testing Process for Scrum and Waterfall Projects
ICTSS 2010 - Iterative Software Testing Process for Scrum and Waterfall ProjectsEliane Collins
 
Test Driven Development via Agile Testing
Test Driven Development via Agile TestingTest Driven Development via Agile Testing
Test Driven Development via Agile TestingAnand Bagmar
 
Qtp-training A presentation for beginers
Qtp-training  A presentation for beginersQtp-training  A presentation for beginers
Qtp-training A presentation for beginersDhavamani Prakash
 
Open Source Test Workshop for CIOs, CTOs, Managers
Open Source Test Workshop for CIOs, CTOs, ManagersOpen Source Test Workshop for CIOs, CTOs, Managers
Open Source Test Workshop for CIOs, CTOs, ManagersClever Moe
 
Unit Testing in Java
Unit Testing in JavaUnit Testing in Java
Unit Testing in Javaguy_davis
 
How we Test at eBay Europe
How we Test at eBay EuropeHow we Test at eBay Europe
How we Test at eBay EuropeMichael Palotas
 
San Jose Selenium Meet-up PushToTest TestMaker Presentation
San Jose Selenium Meet-up PushToTest TestMaker PresentationSan Jose Selenium Meet-up PushToTest TestMaker Presentation
San Jose Selenium Meet-up PushToTest TestMaker PresentationClever Moe
 
Miratech software testing and QA services
Miratech software testing and QA servicesMiratech software testing and QA services
Miratech software testing and QA servicesMiratech
 
Six Sigma is Possible with R
Six Sigma is Possible with RSix Sigma is Possible with R
Six Sigma is Possible with REmilio L. Cano
 

Similar to Quality assurance in distributed continuous delivery (20)

Creating Maintainable Automated Acceptance Tests
Creating Maintainable Automated Acceptance TestsCreating Maintainable Automated Acceptance Tests
Creating Maintainable Automated Acceptance Tests
 
Emerasoft Day 2012 - Test Automation con Polarion
Emerasoft Day 2012 - Test Automation con PolarionEmerasoft Day 2012 - Test Automation con Polarion
Emerasoft Day 2012 - Test Automation con Polarion
 
Pivotal Labs Open View Presentation Quality Assurance And Developer Testing
Pivotal Labs Open View Presentation Quality Assurance And Developer TestingPivotal Labs Open View Presentation Quality Assurance And Developer Testing
Pivotal Labs Open View Presentation Quality Assurance And Developer Testing
 
Inverting The Testing Pyramid
Inverting The Testing PyramidInverting The Testing Pyramid
Inverting The Testing Pyramid
 
Test Automation and Keyword-driven testing af Brian Nielsen, CISS/AAU
Test Automation and Keyword-driven testing af Brian Nielsen, CISS/AAUTest Automation and Keyword-driven testing af Brian Nielsen, CISS/AAU
Test Automation and Keyword-driven testing af Brian Nielsen, CISS/AAU
 
Agile QA Process
Agile QA ProcessAgile QA Process
Agile QA Process
 
Ashwini - Effective use of CI by QA
Ashwini - Effective use of CI by QAAshwini - Effective use of CI by QA
Ashwini - Effective use of CI by QA
 
Programmer testing
Programmer testingProgrammer testing
Programmer testing
 
Gabriele Lana: Testing Web Applications
Gabriele Lana: Testing Web ApplicationsGabriele Lana: Testing Web Applications
Gabriele Lana: Testing Web Applications
 
Testing Android
Testing AndroidTesting Android
Testing Android
 
ebay @ Hasso Plattner Institut Potsdam
ebay @ Hasso Plattner Institut Potsdamebay @ Hasso Plattner Institut Potsdam
ebay @ Hasso Plattner Institut Potsdam
 
ICTSS 2010 - Iterative Software Testing Process for Scrum and Waterfall Projects
ICTSS 2010 - Iterative Software Testing Process for Scrum and Waterfall ProjectsICTSS 2010 - Iterative Software Testing Process for Scrum and Waterfall Projects
ICTSS 2010 - Iterative Software Testing Process for Scrum and Waterfall Projects
 
Test Driven Development via Agile Testing
Test Driven Development via Agile TestingTest Driven Development via Agile Testing
Test Driven Development via Agile Testing
 
Qtp-training A presentation for beginers
Qtp-training  A presentation for beginersQtp-training  A presentation for beginers
Qtp-training A presentation for beginers
 
Open Source Test Workshop for CIOs, CTOs, Managers
Open Source Test Workshop for CIOs, CTOs, ManagersOpen Source Test Workshop for CIOs, CTOs, Managers
Open Source Test Workshop for CIOs, CTOs, Managers
 
Unit Testing in Java
Unit Testing in JavaUnit Testing in Java
Unit Testing in Java
 
How we Test at eBay Europe
How we Test at eBay EuropeHow we Test at eBay Europe
How we Test at eBay Europe
 
San Jose Selenium Meet-up PushToTest TestMaker Presentation
San Jose Selenium Meet-up PushToTest TestMaker PresentationSan Jose Selenium Meet-up PushToTest TestMaker Presentation
San Jose Selenium Meet-up PushToTest TestMaker Presentation
 
Miratech software testing and QA services
Miratech software testing and QA servicesMiratech software testing and QA services
Miratech software testing and QA services
 
Six Sigma is Possible with R
Six Sigma is Possible with RSix Sigma is Possible with R
Six Sigma is Possible with R
 

Recently uploaded

Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed
 
My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024The Digital Insurer
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Patryk Bandurski
 
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage CostLeverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage CostZilliz
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsMiki Katsuragi
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks..."LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...Fwdays
 
Vector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector DatabasesVector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector DatabasesZilliz
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machinePadma Pradeep
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piececharlottematthew16
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubKalema Edgar
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxNavinnSomaal
 

Recently uploaded (20)

Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
 
My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
 
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage CostLeverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering Tips
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks..."LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
 
Vector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector DatabasesVector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector Databases
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piece
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding Club
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptx
 

Quality assurance in distributed continuous delivery