JCN Conference 2015

2. Zephyr Test Tool Deployment | Procountor
Knowit Oy, Jussi Rautio 6.10.2015
PAGE 2 [CONFIDENTIAL] THIS IS KNOWIT6.10.2015
www.knowit.fi

3. 6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.03
Senior Quality Consultant, Service Productization Responsible, Service
Manager, Test Tool Technical support, Test Tool Deployments and
Migrations…
• Speaks, train, coach and mentor about
• How to take new testing tool in use and best practices.
• User trainings for testers, test managers, product owners and quality managers
• Metrics and KPI`s benchmarking and design
• Data Quality
• Process enhancements
• Etc.
JUSSI RAUTIO, KNOWIT
www.knowit.fi

4. KNOWIT
Scandinavian company that is listed on Nordic
Exchange Stockholm
Knowit offering in Finland:
• Test and quality management
• Solutions for digital business and information
management
• Development and maintenance of client-
specific applications
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.04
www.knowit.fi

5. 02.09.15 [CONFIDENTIAL] THIS IS KNOWITPAGE 5

6. • Leading provider of cloud accounting software
• Used by over 10 000 companies and over 400
accounting offices
• Available in Finland, Norway, Sweden and
Denmark
02.09.15 [CONFIDENTIAL] THIS IS KNOWITPAGE 6

7. PROCOUNTOR ATLASSIAN ENVIRONMENT
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.07
Crowd
JIRA FisheyeConfluence
Refinedwiki
JIRAagile
ZephyrforJIRA
Timesheetreports
Crucible
Confluence
• Social Intranet
• Knowledge Management
JIRA
• Requirement management
• Product development (scrum)
• Test management
Fisheye/Crucible
• Source code browsing
• Code review
Crowd
• User management
*500
*50*100*250
Hosted by: www.ambientia.net
www.knowit.fi

8. 10/6/2015 © Copyright Knowit Oy 2014 | Confidential | Version 1.08
Problem
Description
www.knowit.fi

9. PROCOUNTOR NEED
• Procountor is a rapidly growing firm
• New people are hired constantly
• Testing tool and processes need to be updated to meet the new demands
• Knowit was asked to enhance Procountor testing
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.09 www.knowit.fi

10. TEST PROCESS ANALYSIS
Knowit test process analysis for Procountor
Means of analysis were:
• Workshops
• Interviews
Two weeks after the release development teams are very busy on fixing bugs. Why?
• Too tight schedule
• Fixed content
• Not enough time for regression testing
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.010 www.knowit.fi

11. TEST PROCESS ANALYSIS CONCLUSION
Insufficient regression testing time ->
Whole team approach should be introduced ->
Testing tasks are in the different testing tool ->
Better tool to enforce task sharing ->
Jira ensures already a common way of working ->
Test tool Zephyr was chosen to be used in regression testing
because of seamless integration to Jira among other things
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.011 www.knowit.fi

12. © Copyright Knowit Oy 2014 | Confidential | Version 1.012
Introducing
the Test Tool
www.knowit.fi

13. TEST TOOL DEPLOYMENT PRINCIPLES
Fail fast
• If tool is not suited for you, don’t waste your time, look other way
Start light
• Put only necessary test data in to system in pilot phase
• Do only minimum tweaking to the tool at the beginning
Embrace the change
• Give tool a chance
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.013 www.knowit.fi

14. www.knowit.fi
Enhanced
Quality
and
Effectivity
Create a usage model for test tool
Test tool in pilot use for regression testing
Usage model tweaking
Test tool in use for wider testing
Test tool in use internationally
Continuous improvement activities
Integrations to other systems
Test tool used through organization if Finland
Virtualization & automation

15. GENERAL ABOUT ZEPHYR
• Brings test management and execution functionalities to Jira
New issue type Test
• Possibility to have as an add on for Jira or as an own product
• First Version released in the beginning of 2012
• Latest version 2.6.0 Released on 09.2015
Characteristics
• Easy to use for Jira users
• Customizable
• No test case version control
• If lots of add-on’s are in use the update process of Jira and Zephyr becomes challenging
• No folder structure
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.015 www.knowit.fi

16. THINGS TO DO BEFORE DEPLOYMENT 1/3
• Fields for tests must be agreed
• default fields can be used at start
• Screens must be agreed
• Tests workflow should be done when testing process with the new tool is mature enough
• User groups and rights should be put in order
• For example who can modify test executions should be restricted
• Procountor has one tester per group so there is no need to restrict user rights
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.016 www.knowit.fi

17. THINGS TO DO BEFORE DEPLOYMENT 2/3
Labels must be decided
• Label is users self chosen tag for tests
• Labels should be agreed so that their amount would not explode
• Labels could be for example: Functional testing, Regression testing, Usability, Load…
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.017 www.knowit.fi

18. THINGS TO DO BEFORE DEPLOYMENT 3/3
Components must be decided
• Components are user defined parts of software
• For example: Accounting, User Management, Billing..
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.018 www.knowit.fi

19. TEST STATUSES
One part of test process definition is specifying test statuses
• Statuses can be modified based on test process
Default test statuses of Zephyr are
• PASS
• FAIL
• WIP
• BLOCKED
• UNEXECUTED
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.019 www.knowit.fi

20. LINKING TESTS TO USER STORIES
Tests can be linked to User Stories
• “Is blocked by” -link statuses are NOT affecting if the test can be run or not
Linking should be done always so changes in User Story can be followed easily
• Changes in user story are NOT sending signal for re-testing
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.020 www.knowit.fi

21. TEST CASE DIVISION
1. By Project
• Project is a way to division test cases in a very high level. Procountor has only one Project
2. By Version
• Fix Versions –field can be used to division tests. Test Summary Screen shows tests by version
automatically
3. By Component
• Test can be attached to a component (multivalue field.) Test Summary shows also these.
4. By Label
• Anyone can create new labels so this should be agreed in organization to keep it controlled
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.021 www.knowit.fi

22. CYCLES – TEST ROUNDS
Tests are executed in Cycles and one test can be in many Cycle
Tests can be taken to test Cycles either by:
• One at a time
• By Filter
• By Component
• By Label
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.022 www.knowit.fi

23. NAMING AND DIVISION OF CYCLES
Cycle name should present their division and support reporting.
Cycle names can be for example:
• Regression test for version xx (Procountor)
• Sprint
• Week
• Tester
• Group
• Combination of above
…
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.023 www.knowit.fi

24. © Copyright Knowit Oy 2014 | Confidential | Version 1.024
Monitoring
Test Tool
Deployment
Success
www.knowit.fi

25. TEST TOOL CONCEPT OWNER
Someone from the organization should be named as a
test tool concept owner after the test tool deployment
Test tool concept owner tasks:
• Follow test tool efficiency continuously
• Follow test tool development roadmap
• Introduce new test tool features to organization
• Gather user requests for test tool customization and integrations
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.025 www.knowit.fi

26. MEASURING OF TEST TOOL EFFICIENCY
Test tool efficiency should be followed in during the long period of time
Has the quality of the product enhanced?
• Number of bugs found in production
• Customer satisfaction
Has the efficiency increased?
• Is the tool really used?
• What is the gut feeling? Has the tool helped testing? Is regression testing easier now?
• How wide has the tool usage spread?
• Has the hurry after the release decreased?
• What is the usability of the tool?
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.026 www.knowit.fi

27. FUTURE SCENARIO FOR PROCOUNTOR
Testing during development could be done with exploratory testing
• Link exploratory testing session to User Story
• Leaves a trail what has been already tested
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.027
• Actor: intended user of the system
• Purpose: the theme of the charter including what particular objective the actor wants to
achieve, i.e., the test conditions
• Setup: what needs to be in place in order to start the test execution
• Priority: relative importance of this charter, based on the priority of the associated user story or
the risk level
• Reference: specifications (e.g., user story), risks, or other information sources
• Data: whatever data is needed to carry out the charter
• Activities: a list of ideas of what the actor may want to do with the system and what would be
interesting to test (both positive and negative tests)
• Oracle notes: how to evaluate the product to determine correct results
• Variations: alternative actions and evaluations to complement the ideas described under
activities
www.knowit.fi

28. EXPLORATORY TESTING – TEST SESSION
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.028
Session based testing aims to combine accountability and exploratory testing
Test sessions:
• Based on charter
• Uninterrupted session from 60 to 120 minutes
• Evaluation of the functionality or characteristics
• Each session is reported in detail
Test Design
Test Execution
Test Results
Learn Test Charter
Test Session
Test Design
Test Execution
Test Results
Learn Test Charter
Test Session
Test Design
Test Execution
Test Results
Learn Test Charter
Test Session
Session-Based Test Management
www.knowit.fi

29. TOOLS FOR EXPLORATORY TESTING
Record Steps to Reproduce a Problem
• Windows Out Of the Box tool to take screenshots
• Creates screenshots on each click and typing and creates one file from the recording
• Free in all Windows versions
JIRA Capture
• Web –application to create issues easily
• Records Test Session length, issues and tester notes
• Needs to be purchased
6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.029 www.knowit.fi

30. THANK YOU!
QUESTIONS?
www.knowit.fi

31. Knowit Oy
Tehtaankatu 27-29 D
00150 Helsinki
tel. +358 40 3400600
www.knowit.fi

32. What Test Management challenges and
possibilities are there in the agile/DevOps
model?
www.ambientia.net
Let’s talk!
© Copyright Knowit Oy 2015 | Confidential | Version 1.0www.knowit.fi