BUILD THE “RIGHT”
REGRESSION SUITE
USING BEHAVIOR
DRIVEN TESTING (BDT)
vodQA
ABOUT US
about.me/anand.bagmar priyankh@thoughtworks.com
@BagmarAnand
vodqancr@thoughtworks.com
WHAT DO YOU EXPECT FROM THIS SESSION?
CASE STUDY
•  Multiple, long running projects
•  Legacy applications
•  Integrated
•  Limited / flaky / long-running Automation
•  Long regression cycle
•  Huge cost of fixing defects
FUNCTIONAL TESTING CHAL...
•  Brittle
•  Little / less value
•  No visibility into what is tested
•  Maintenance nightmare
•  Outdated
•  Expensive
A...
•  Trust deficit on the team
•  In existing Automation
•  In manual testing
•  Finger pointing / blame game
BIGGEST PROBLEM
My Case Study
Concept #1
Given -> When -> Then
Expected Functionality:
The customer should be able to
withdraw money from his account via
an ATM machine
BDD Example:
Given the account has balance of 5000
When the customer requests 1000
Then the account is debited by 1000
13
Workshop Begins – Iteration 1
Case Study
PHASE 1 REQUIREMENTS
•  Guest User is able to search for a flight for a
single traveller
•  Select option from Search Resul...
A Guest User can …
WORKFLOW
Search
Search
Results
Contact
Information
SEARCH MODULE
SEARCH RESULTS
CONTACT DETAILS
CONTACT DETAILS
PHASE 1 REQUIREMENTS
•  Guest User is able to search for a flight for a
single traveller
•  Select option from Search Resul...
A Guest User can …
WORKFLOW
Search
Search
Results
Contact
Information
LETS GET STARTED … WRITE 2 TEST SCENARIOS
Iteration #1
Showcase
Concept #2
Test Specification Styles
•  Imperative
•  Declarative
TEST SPECIFICATION STYLES
A Guest User can …
AN EXAMPLE
Search
Search
Results
Contact
Information
IMPERATIVE STYLE
Given I am a guest user on the home page
And I choose “round” trip option
And I select “Chicago” from the...
IMPERATIVE STYLE …
…
When I select the “first” option
Then I am on the Contact Information page
When I enter first name as “...
DECLARATIVE STYLE
Given I am a guest user
When I search for flight options for a “one-way” trip for “1”
“Adult” from “Chica...
•  Ben Mabey -
http://benmabey.com/2008/05/19/
imperative-vs-declarative-scenarios-in-
user-stories.html
•  Alister Scott ...
Workshop – Iteration 2
Case Study Continues
•  User registration from Contact information
page
•  Registered user can login from home page
(below search criteria)
•  ...
Login Search
Search
Results
Contact
Information
•  Pre-filled
WORKFLOW … ENHANCED
REGISTERED USER
•  User registration from Contact information
page
•  Registered user can login from home page
(below search criteria)
•  ...
Login Search
Search
Results
Contact
Information
•  Pre-filled
WORKFLOW … ENHANCED
LETS GET STARTED … WRITE 2 TEST SCENARIOS
Iteration #2
Showcase
WHY IS THIS IMPORTANT?
Concept #3
The Test Pyramid
AUTOMATION – A SAFETY NET
•  “Succeeding with Agile” – Mike Cohn
•  Martin Fowler – Test Pyramid
martinfowler.com/bliki/TestPyramid.html
THE TEST PY...
Time
Value
Cost
IDEAL TEST AUTOMATION PYRAMID
Manual / Exploratory
UI
Web Service
View
JavaScript
Integration
Unit
Technol...
REALITY
UI
Web Service
View
JavaScript
Integration
Unit
Manual / Exploratory
ICE CREAM CONE ANTI PATTERN
Developers Test Pyramid QA Team Test Pyramid
Web Service
View
JavaScript
Integration
Unit
Manual / Exploratory
UI
Web Serv...
Concept #4
Behavior Driven Testing (BDT)
BEHAVIOR DRIVEN TESTING (BDT)
•  How is this functionality going to be used
by the end-user?
•  What is their thought process going to be
when using thi...
Manual / Exploratory
UI
Web Service
View
JavaScript
Integration
Unit
Technology-
facing Tests
Business-facing
Tests
Identi...
My Case Study
HOW TO SOLVE THE PROBLEM?
•  Remove the ambiguity
•  Same visibility for all
SOLUTION
•  Whiteboards
•  Mind maps
•  Flow charts
•  Collaboration!!
TOOLS TO ENABLE BDT
HOW DID WE ACCOMPLISH THIS?
IDENTIFY BUSINESS FLOWS
•  Identify business flows based on personas,
and how the personas use the business
functionality, or,
•  Identify business...
PRIORITIZE BUSINESS FLOWS
1st candidate
•  Automate!
•  High Value, Low Cost
2nd candidate
•  Automate!
•  High Value, High Cost
Can Automate?
•  Ma...
AUTOMATE THE TOP LAYER USING
•  BDD Tools, or,
•  Directly in any appropriate programming
language / tools
•  Identification of Tests
•  Automate it!
WHAT’S NEXT?
AUTOMATE TESTS THAT ARE VALUABLE,
NOT BECAUSE YOU CAN OR LIKE TO!
CAUTION!
Who is doing the action?
What is the business
functionality?
Depicts a user flow / user
journey
Actions & Verifications
happ...
Delete
profile
UPDATE BUSINESS FLOWS
1st candidate
•  Automate!
•  High Value, Low Cost
2nd candidate
•  Automate!
•  High Value, High Cost
Can Automate?
•  Ma...
Project Management tool Mingle
Test case repository (manual
+ automated)
Cucumber .feature files with
appropriate tagging
F...
•  Builds and deployments were automated
•  Tests ran on demand, automatically!
•  Smoke / Regression / Javascript / Unit
...
How did this Technique (BDT) become
successful?
CONTEXT IS KING!
Domain
Distributed
Teams
Skills
Capabilities Product-under test
Tech Stack
Process
Tools
NOT work in isolation
BDT WILL …
BDT REQUIRES …
Lower layers of Test Pyramid to be well
automated!
BDT REQUIRES …
Dedicated and Focused testing (manual /
exploratory)
BDT helps identify the “right” type
of regression tests!
Executable
Specs
BDT
Scenarios
Business Idea
Features
Epics
Stories
Iterations
Releases
BDT VALUE PROPOSITION
BDT VALUE PROPOSITION …
Incrementally
build
Business Flows
Regress
Business Value
Test in-sync with
Product Functionality
...
BDT VALUE PROPOSITION …
Remove
ambiguity
Understand
new functionality
Living
Documentation
Onboarding
ABOUT US
about.me/anand.bagmar priyankh@thoughtworks.com
@BagmarAnand
vodqancr@thoughtworks.com
​Build the ‘Right’ Regression Suite using Behavior Driven Testing (BDT)
​Build the ‘Right’ Regression Suite using Behavior Driven Testing (BDT)
​Build the ‘Right’ Regression Suite using Behavior Driven Testing (BDT)
​Build the ‘Right’ Regression Suite using Behavior Driven Testing (BDT)
​Build the ‘Right’ Regression Suite using Behavior Driven Testing (BDT)
​Build the ‘Right’ Regression Suite using Behavior Driven Testing (BDT)
Nächste SlideShare
Wird geladen in …5
×

​Build the ‘Right’ Regression Suite using Behavior Driven Testing (BDT)

7.803 Aufrufe

Veröffentlicht am

​Build the ‘Right’ Regression Suite using Behavior Driven Testing (BDT)

Veröffentlicht in: Technologie

​Build the ‘Right’ Regression Suite using Behavior Driven Testing (BDT)

  1. 1. BUILD THE “RIGHT” REGRESSION SUITE USING BEHAVIOR DRIVEN TESTING (BDT) vodQA
  2. 2. ABOUT US about.me/anand.bagmar priyankh@thoughtworks.com @BagmarAnand vodqancr@thoughtworks.com
  3. 3. WHAT DO YOU EXPECT FROM THIS SESSION?
  4. 4. CASE STUDY •  Multiple, long running projects •  Legacy applications •  Integrated
  5. 5. •  Limited / flaky / long-running Automation •  Long regression cycle •  Huge cost of fixing defects FUNCTIONAL TESTING CHALLENGES
  6. 6. •  Brittle •  Little / less value •  No visibility into what is tested •  Maintenance nightmare •  Outdated •  Expensive AS A RESULT
  7. 7. •  Trust deficit on the team •  In existing Automation •  In manual testing •  Finger pointing / blame game BIGGEST PROBLEM
  8. 8. My Case Study
  9. 9. Concept #1 Given -> When -> Then
  10. 10. Expected Functionality: The customer should be able to withdraw money from his account via an ATM machine
  11. 11. BDD Example: Given the account has balance of 5000 When the customer requests 1000 Then the account is debited by 1000
  12. 12. 13
  13. 13. Workshop Begins – Iteration 1 Case Study
  14. 14. PHASE 1 REQUIREMENTS •  Guest User is able to search for a flight for a single traveller •  Select option from Search Results page •  Specify contact information for person booking the flight
  15. 15. A Guest User can … WORKFLOW Search Search Results Contact Information
  16. 16. SEARCH MODULE
  17. 17. SEARCH RESULTS
  18. 18. CONTACT DETAILS
  19. 19. CONTACT DETAILS
  20. 20. PHASE 1 REQUIREMENTS •  Guest User is able to search for a flight for a single traveller •  Select option from Search Results page •  Specify contact information for person booking the flight
  21. 21. A Guest User can … WORKFLOW Search Search Results Contact Information
  22. 22. LETS GET STARTED … WRITE 2 TEST SCENARIOS
  23. 23. Iteration #1 Showcase
  24. 24. Concept #2 Test Specification Styles
  25. 25. •  Imperative •  Declarative TEST SPECIFICATION STYLES
  26. 26. A Guest User can … AN EXAMPLE Search Search Results Contact Information
  27. 27. IMPERATIVE STYLE Given I am a guest user on the home page And I choose “round” trip option And I select “Chicago” from the origin dropdown And I select “San Francisco” from the destination dropdown And I select departure date as “5 December 2013” And I select returning date as “25 December 2013” When I click on Search Then I should see the search results page And I should see at least 1 option for my criteria ...
  28. 28. IMPERATIVE STYLE … … When I select the “first” option Then I am on the Contact Information page When I enter first name as “Anand” And I enter last name as “Bagmar” … … And I click the “Select and Continue” button Then I should be on the next page
  29. 29. DECLARATIVE STYLE Given I am a guest user When I search for flight options for a “one-way” trip for “1” “Adult” from “Chicago” to “San Francisco” And I select the “first” flight And I enter “valid” contact details for “traveller1” Then I am able to Save and Continue
  30. 30. •  Ben Mabey - http://benmabey.com/2008/05/19/ imperative-vs-declarative-scenarios-in- user-stories.html •  Alister Scott - http://watirmelon.com/2010/12/10/ cucumber-imperative-or-declarative-that- is-the-question/ REFERENCES
  31. 31. Workshop – Iteration 2 Case Study Continues
  32. 32. •  User registration from Contact information page •  Registered user can login from home page (below search criteria) •  For logged-in user, when booking ticket, contact information is pre-filled with logged-in user information PHASE 2 REQUIREMENTS
  33. 33. Login Search Search Results Contact Information •  Pre-filled WORKFLOW … ENHANCED
  34. 34. REGISTERED USER
  35. 35. •  User registration from Contact information page •  Registered user can login from home page (below search criteria) •  For logged-in user, when booking ticket, contact information is pre-filled with logged-in user information PHASE 2 REQUIREMENTS
  36. 36. Login Search Search Results Contact Information •  Pre-filled WORKFLOW … ENHANCED
  37. 37. LETS GET STARTED … WRITE 2 TEST SCENARIOS
  38. 38. Iteration #2 Showcase
  39. 39. WHY IS THIS IMPORTANT?
  40. 40. Concept #3 The Test Pyramid
  41. 41. AUTOMATION – A SAFETY NET
  42. 42. •  “Succeeding with Agile” – Mike Cohn •  Martin Fowler – Test Pyramid martinfowler.com/bliki/TestPyramid.html THE TEST PYRAMID
  43. 43. Time Value Cost IDEAL TEST AUTOMATION PYRAMID Manual / Exploratory UI Web Service View JavaScript Integration Unit Technology- facing Tests Business-facing Tests
  44. 44. REALITY
  45. 45. UI Web Service View JavaScript Integration Unit Manual / Exploratory ICE CREAM CONE ANTI PATTERN
  46. 46. Developers Test Pyramid QA Team Test Pyramid Web Service View JavaScript Integration Unit Manual / Exploratory UI Web Service DUAL TEST PYRAMID ANTI-PATTERN
  47. 47. Concept #4 Behavior Driven Testing (BDT)
  48. 48. BEHAVIOR DRIVEN TESTING (BDT)
  49. 49. •  How is this functionality going to be used by the end-user? •  What is their thought process going to be when using this product? •  What “core-business-value” should be delivered to the end-user? THE BIG PICTURE
  50. 50. Manual / Exploratory UI Web Service View JavaScript Integration Unit Technology- facing Tests Business-facing Tests Identify the “right” type of tests that sit on top of the Test Pyramid
  51. 51. My Case Study
  52. 52. HOW TO SOLVE THE PROBLEM?
  53. 53. •  Remove the ambiguity •  Same visibility for all SOLUTION
  54. 54. •  Whiteboards •  Mind maps •  Flow charts •  Collaboration!! TOOLS TO ENABLE BDT
  55. 55. HOW DID WE ACCOMPLISH THIS?
  56. 56. IDENTIFY BUSINESS FLOWS
  57. 57. •  Identify business flows based on personas, and how the personas use the business functionality, or, •  Identify business flows, and how different user personas can use the business flows NOTE: Business flows are different from UI flows HOW TO IDENTIFY BUSINESS FLOWS ?
  58. 58. PRIORITIZE BUSINESS FLOWS
  59. 59. 1st candidate •  Automate! •  High Value, Low Cost 2nd candidate •  Automate! •  High Value, High Cost Can Automate? •  Manual Regression? •  Low Value, Low Cost Do Not Automate •  Manual Regression! •  Low Value, High Cost High Value High Cost Low Value Low Cost
  60. 60. AUTOMATE THE TOP LAYER USING •  BDD Tools, or, •  Directly in any appropriate programming language / tools
  61. 61. •  Identification of Tests •  Automate it! WHAT’S NEXT?
  62. 62. AUTOMATE TESTS THAT ARE VALUABLE, NOT BECAUSE YOU CAN OR LIKE TO! CAUTION!
  63. 63. Who is doing the action? What is the business functionality? Depicts a user flow / user journey Actions & Verifications happen implicitly Given auctioneer creates an auction And 5 vehicles are added to the auction When auctioneer starts the auction And auctioneer sets a starting bid of 5000 And buyer_1 bids And buyer_2 bids Then auctioneer Sells the vehicle And buyer_2 wins the vehicle EXAMPLE
  64. 64. Delete profile UPDATE BUSINESS FLOWS
  65. 65. 1st candidate •  Automate! •  High Value, Low Cost 2nd candidate •  Automate! •  High Value, High Cost Can Automate? •  Manual Regression? •  Low Value, Low Cost Do Not Automate •  Manual Regression! •  Low Value, High Cost High Value High Cost Low Value Low Cost
  66. 66. Project Management tool Mingle Test case repository (manual + automated) Cucumber .feature files with appropriate tagging Feature files / Scenarios 33 / 65 # of Manual Tests 0 Tests running on CI (Jenkins) Since the 1st test was automated Smoke tests execution time 15 minutes Regression tests execution time 45 minutes FINAL STATE … IN MY CASE STUDY
  67. 67. •  Builds and deployments were automated •  Tests ran on demand, automatically! •  Smoke / Regression / Javascript / Unit •  Functional Performance •  Concurrent Testing ADDITIONALLY …
  68. 68. How did this Technique (BDT) become successful?
  69. 69. CONTEXT IS KING! Domain Distributed Teams Skills Capabilities Product-under test Tech Stack Process Tools
  70. 70. NOT work in isolation BDT WILL …
  71. 71. BDT REQUIRES … Lower layers of Test Pyramid to be well automated!
  72. 72. BDT REQUIRES … Dedicated and Focused testing (manual / exploratory)
  73. 73. BDT helps identify the “right” type of regression tests!
  74. 74. Executable Specs BDT Scenarios Business Idea Features Epics Stories Iterations Releases BDT VALUE PROPOSITION
  75. 75. BDT VALUE PROPOSITION … Incrementally build Business Flows Regress Business Value Test in-sync with Product Functionality Effective Tests
  76. 76. BDT VALUE PROPOSITION … Remove ambiguity Understand new functionality Living Documentation Onboarding
  77. 77. ABOUT US about.me/anand.bagmar priyankh@thoughtworks.com @BagmarAnand vodqancr@thoughtworks.com

×