BDD Scenarios in a Testing Strategy

BDD Scenarios in a
Testing Strategy
A case study
16/11/2023 – AgileTD
Gáspár Nagy
coach • trainer • bdd addict • creator of specflow
https://bddbooks.com • “Discovery” • “Formulation”
@gasparnagy • gaspar@specsolutions.eu
ChatGPT
free

Copyright © Gaspar Nagy
Today…
• Super brief BDD & SpecSync intro
• The SpecSync quality strategy
• Living Documentation and Traceability
• Wrap-up, Q&A

Let’s get prepared…

This is a BDD scenario
Scenario: Discount is applied when item added to the cart
Given a user without submitted order
And they selected a book that costs €50
When the user adds the book to the cart
Then the cart sub-total should be €45
BDD IS NOT
TESTING!

BDD Practices
Image source: https://cucumber.io/docs/bdd/
Discovery
Shared understanding is established through
collaboration and structured conversations
Formulation
The examples of system behaviour are
documented as scenarios
Automation
Scenarios are automated to be able to verify
the behaviour of the system

SpecSync for Azure DevOps
synchronize • share • trace
<TRX>
<JUnit XML>
<Cucumber JSON>
<…>
synchronize
(push)
(pull)
run tests
BUG
1234
share
(publish test results)
trace

Started in 2015
Cross-platform (Win/Linux/macOS/Docker)
Supports all major BDD Frameworks (SpecFlow, Cucumber, Behave, …)
Supports 5 Azure DevOps versions, Jira Data Center & Jira Cloud
60k+ line of code (C#)
Survived 3 major updates and an ADO API change
Build with BDD/SpecFlow (~300 scenarios)
Uses SpecSync for traceability – eat your own dogfood

Rule: Can create new Test Case work items from scenarios
Scenario: A new scenario is linked to a new Test Case
Given there is an Azure DevOps project
And there is a scenario in the local repository
"""
Scenario: Sample scenario
Given there is something
When I do something
Then something will happen
"""
When the local repository is synchronized with push
Then a new Test Case work item "Scenario: Sample scenario" is created in the project
And the Test Case contains the following test steps
| step |
| Given there is something |
| When I do something |
| Then something will happen |
And a tag "@tc:[id-of-new-test-case]" is added to the scenario in the local repository

The SpecSync Quality
Strategy

The
Pyramid
is
not
enough
Test Automation Pyramid: Mike Cohn – Succeeding with Agile
Pic: http://gasparnagy.com/2017/02/balancing-scenarios-and-unit-tests-case-study-with-specflow/

There
is
a
carnival
of
pyramids!

More Agile Testing: One-page test plan
Source: More Agile Testing, page 95, Bernice Niel Ruhland

The four-page quality strategy
• A quality strategy is like an overall test plan – gives a guidance for the whole
team about how we address quality
• Custom & specific to your project, but keep it short
• SpecSync’s strategy* has the following sections:
• Challenges (Risks)
• Quality Aspects
• Testing Methods
• Each with: purpose, automation strategy, structure, out-of-scope
• Feedback channels
THE STRATEGY
EVOLVES!
*Download current snapshot from: http://speclink.me/webinar190221-strategy

Challenges (Risks)
• SpecSync is an integration product
• The quality cannot be fully ensured
• Quality changes externally (e.g., new version of ADO comes out)
• The majority of the business value cannot be tested without integration
• Supports 8 platforms, 5 target ADO versions, 15+ BDD Frameworks and 8
SpecFlow versions
• ADO versions: TFS 2017, TFS 2018, ADO 2019, ADO 2020, ADO Services (cloud)
• Critical for clients:
• chance of blocking traceability (e.g., traceability cannot be enforced if SpecSync does not
work)
• chance of obscuring traceability (e.g., generating unnecessary changes)
• chance of data loss (e.g., SpecSync deletes important data due to a bug)

Why the Pyramid® is not enough as a strategy
WHAT WE WANT
IS QUALITY!

Breaking down the undefinable quality:
quality aspects (sample)
Functional
• Works as
expected
• Expectations
are good
• Expectations
are
documented
Operational
• Secure
• Fast
• Convenient
• Pretty
• Consistent
• Predictable
Strategic
• Maintainable
• Architecture
• Code quality
• Easy to
integrate
• Flexible
Source: Kevlin Henney

SpecSync quality aspects
Functional
• Features documented and
work
• Supports different BDD
frameworks
• Supports different ADO
versions
• Handles variations in
configuration
• Handles variations in BDD
framework test results
• Handles ADO integration
edge cases (version, size
specialties)
Operational
• Works on different platforms
• Secure
• Understandable output, easy
to diagnose
user/configuration errors
• Provides guidance for
configuration
• Works with acceptable speed
(processes at least 100
scenario per minute)
Strategic
• Has a fast & offline
development environment
• Easy to make a fix, hotfix,
proposed fix or spike
• Easy to maintain long-term
• Flexible to support custom
development processes of the
clients
• Easily extensible for new BDD
frameworks and new ADO
versions

Choosing
test
methods
Automated BDD Scenarios
(SpecFlow, Integration)
SpecSync
ADO
ADO
2019
TFS
2017
TFS
2018
STUB
ADO
SpecSync Algorithmic Logic
Tests (MsTest, Unit)
ADO Integration Tests
(MsTest, Integration)
sync
engine
service classes
console API
Smoke Tests
(SpecFlow, E2E)
Exploratory
Features documented and work
Supports different ADO versions
Has fast & offline dev env
Works on different platforms
Handles variations in BDD frw test results Handles ADO integration edge cases
Understandable output

Code Coverage Results
Type Test Count Local execution time Coverage
Unit 462 2 sec 30%
ADO Integration 117 7 min 19%
Specs (ADO) 263 9.6 min 76%
Specs (Stub) 263 15.5 sec 73%
Smoke (ADO) 51 5.9 min N/A
Aggregated 1156 N/A 89%

Testing
methods
&
Feedback
channels

Living Documentation
and Traceability

Living Documentation
Scenarios describe the requirements in a business-readable form. As
they are automated, by running them as tests, they can tell us when the
implementation diverges from our expectations – so it becomes living
documentation that the whole team can rely on.

The living documentation is a documentation!
http://speclink.me/specsyncdoc
STRUCTURE IT
AS A DOC!

Feature – Rule – Scenario
Feature: Linking Work Items and other artifacts using tags
Rule: Establish work item links based on scenario tags
Scenario: A work item link tag added to the scenario
Scenario: A work item link tag added to the feature
Rule: Link type can be specified
Scenario: "Tests" link is created by default
Scenario: The work item is linked as "Parent“
Scenario: The link type has changed
Rule: Existing links are preserved and links should remain when tag is removed
Scenario: The work item tag is removed from the scenario
Rule: Multiple prefixes can be linked
Scenario: A product backlog link and a bug link tag is added to the scenario
Rule: Pull Requests can be linked […]
Rule: Adding a work item link should trigger an update […]

Details: hide incidental, show real data
@tc:155
Given there is an Azure DevOps project with a Bug work item
And the synchronizer is configured as
| setting | value |
| synchronization/links[]/tagPrefix | bug |
And a @bug tag has been added to an existing synchronized scenario as:
"""
@tc:[id-of-test-case] @bug:[id-of-pbi]
When I do something
"""
Then the test case is linked to the Bug work item

Details: hide incidental, show real data
@tc:155
Given there is an Azure DevOps project with a Bug work item
| setting | value |
And a @bug tag has been added to an existing synchronized scenario as:
"""
@tc:[id-of-test-case] @bug:[id-of-pbi]
When I do something
"""
Then the test case is linked to the Bug work item
--- table step argument ---
| setting | value |
Applying config settings:
{
"synchronization": {
"links": [
{
"tagPrefix": "bug"
}
]
}
}
-> done: ConfigurationSteps.GivenTheSynchronizerIsConfiguredAs(<table>) (0.3s)

Tags – break out from strict hierarchy
Three examples from SpecSync tags:
• @customization – cross-cutting feature
• Describes “Customization” features: used to support special development process
requirements of the SpecSync clients.
• @notsupported-tfs2017 – exclusion criteria
• The particular scenario is not supported on the target platform (here TFS2017). These
scenarios are excluded from the CI builds testing that target
• @formulated – SpecSync development process
• Allows specifying (formulating) scenarios and commit them before automation. These
scenarios are excluded from CI builds (they would fail anyway).

Wrap-up

• How to build and document a quality strategy
• BDD scenarios are phrased using the business language, and therefore the same
scenario can be executed against different targets
• Well-structured BDD scenarios can form a living documentation and enable
traceability
What have we seen?
MAKE YOUR
OWN STRATEGY!

Gáspár Nagy
coach • trainer • bdd addict • creator of specflow
http://bddbooks.com • “Discovery” • “Formulation”
@gasparnagy • gaspar@specsolutions.eu

BDD Scenarios in a Testing Strategy

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie BDD Scenarios in a Testing Strategy

Ähnlich wie BDD Scenarios in a Testing Strategy (20)

Mehr von Gáspár Nagy

Mehr von Gáspár Nagy (13)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

BDD Scenarios in a Testing Strategy