For enterprise projects, maintaining automation tests suite is always a challenge.
* Detailed walkthru of automation pyramid containing different types of specification driven tests layer.
* Framework built using RSpec, Spinach, Cucumber, webkit and Webdriver with examples
* Process of evolving and building specification tests
* Tips to maintain test suite and minimize test run durations
4. Keywords
Specification – More than test – Feature/Requirement
Cucumber – Software behavior in plain text
Capybara – Tests application by simulating a real user
Spinach - Encapsulation and modularity of your step definitions
Headless webkit - A capybara driver - WebKit via QtWebKit.
R-Spec - Test-Driven Development
5. Let’s talk about
Why specifications
Approach to specifications
Specifications pyramid
Learning's
Q&A
6. Trying to solve these challenges?
Defects leakage - missing
Requirements
Tests and requirements
Frequent changes -
documents becomes stale
breaks existing
functionalities
Long Regression Spends more time on defect
Cycles management and triage
Less confidence -
Existing test coverage
7. Trying to solve these challenges?
Defects leakage - missing
Requirements
Less confidence -
Existing test coverage
8. Let’s talk about
Why specifications
Approach to specifications
Specifications pyramid
Learning's
Q&A
10. Evolving Implementing Executing Maintaining
Story
• Intial Cukes creation Development • Manual testing for
• Inputs from Business, each Story
Technical and Testing • For TDD, writing • Evolved as part of
• Tests from Unit to RSpec exploratory testing
End-to-End • Checklist for story
completion
• Referred during dev-
Story box
Testing
Creation
11. Evolving Implementing Executing Maintaining
Story
• Intial Cukes creation Development • Manual testing for
• Inputs from Business, each Story
Technical and Testing • For TDD, writing • Evolved as part of
• Tests from Unit to RSpec exploratory testing
End-to-End • Checklist for story
completion
• Referred during dev-
Story box
Testing
Creation
12. Evolving Implementing Executing Maintaining
Story
• Intial Cukes creation Development • Manual testing for
• Inputs from each Story
Business, Technical • For TDD, writing • Evolved as part of
and Testing RSpec exploratory testing
• Tests from Unit to • Checklist for story
End-to-End completion
• Referred during dev-
Story box
Testing
Creation
13. Let’s talk about
Why specifications
Approach to specifications
Specifications pyramid
Learning’s
Q&A
15. Evolving Implementing Executing Maintaining
Algorithms / business logic
validation
API methods validation
Non-UI
RSpec + API Created as part of story
development
16. Evolving Implementing Executing Maintaining
Algorithms / business logic
validation
API methods validation
Non-UI
RSpec + API Created as part of story
development
17. Evolving Implementing Executing Maintaining
Presentation layer validations
Story level validations
Spinach + Jobs related functionalities
Headless Webkit
Created after story development
RSpec + API complete (QA’s / Dev’s)
18. Evolving Implementing Executing Maintaining
Presentation layer validations
Story level validations
Spinach + Jobs related functionalities
Headless Webkit
Created after story development
RSpec + API complete (QA’s / Dev’s)
19. Evolving Implementing Executing Maintaining
End user entry and exit scenarios
Cucumber +
Capybara
Webdriver Story level UI validations
Spinach + Headless End-To-End business scenarios
Webkit
Created after story development
RSpec + API complete (QA’s / Dev’s)
20. Evolving Implementing Executing Maintaining
End user entry and exit scenarios
Cucumber +
Capybara
Webdriver Story level UI validations
Spinach + Headless End-To-End business scenarios
Webkit
Created after story development
RSpec + API complete (QA’s / Dev’s)
21. Evolving Implementing Executing Maintaining
Cucumber + Developer/Tester
Laptop
CI pipeline
Capybara
• Smoke Tests • Smoke and E2E tests
Webdriver • Less mocks
Spinach + Headless
Webkit
Developer/Tester CI pipeline
Laptop
• Respective modules • After check-in
• Before each check-in • Before smoke and
RSpec + API • Mocks and Cassettes end-to-end
23. Let’s talk about
Why specifications
Approach to specifications
Specifications pyramid
Learning's
Q&A
24. Key learning's
Specifications - Before development
Fail fast – 95% of functionalities gets validated before check-in
Specifications verified during dev-box (including multiple
browsers)
Categorize specifications - Appropriate layer
Parallelize test runs
Keep test code equivalent to developer’s code
Executable specifications
Defects backlog not beyond a threshold (for example 20 defects)
Defects root-cause - Feedback to the team
25. Let’s Recap
Defects leakage - missing
Requirements
Tests and requirements
Frequent changes -
documents becomes stale
breaks existing functionalities
Long Regression Spends more time on defect
Cycles management and triage
Less confidence -
Existing test coverage
26. Tools Available
• Jbehave, Cucumber, Cuke4Duke with
Java different drivers (Selenium,Sahi)
• Twist (Thoughtworks Studios)
• Specflow, Cucumber, Cuke4Nuke,
.NET Coypu with drivers like Selenium, White,
WatiN