Presentation in Ques No.13

1. Struggles and challenges in STLC
May 17, 2019 (令和元年)
Leisure Product Dept.
Rakuten, Inc.

2. 2
1.About Our Team
2.About Our Product
3.STLC
4.RTM(Struggle& Challenge#1)
5.Flow(Struggle& Challenge#2)
6.Conclusion
Agenda

3. 3
1.About Our Team
2.About Our Product
3.STLC
4.RTM(Struggle& Challenge#1)
5.Flow(Struggle& Challenge#2)
6.Conclusion
Agenda

4. 4
[Career Summary]
2004-2010 : Experienced 2 internet based companies
Aug 2010 : Joined Rakuten (3rd company for me)
2010-2013 : Lead Developer for Ad-tech platform
2013-2016 : Manager for Social tool platform dev team
2017~ : Manager for SQA Group for leisure services
Self-Introduction
@freddiefujiwara

5. 5
Product Quality
Assurance
By Testing
Reaching
ZERO Customer Dissatisfaction
Our Mission
Best Experience
Assurance
By Data Science
Our Team
Reliability
Assurance
By Stress Test

6. 6
Product Quality
Assurance
By Testing
Reaching
ZERO Customer Dissatisfaction
Our Mission
Best Experience
Assurance
By Data Science
Our Team
Reliability
Assurance
By Stress Test
Today’s scope

7. 7
Our Team

8. 8
1.About Our Team
2.About Our Product
3.STLC
4.RTM(Struggle& Challenge#1)
5.Flow(Struggle& Challenge#2)
6.Conclusion
Agenda

9. 9
Our Product
*As of Feb 28 2018
*As of Feb 28 2018
*As of Feb 28 2018
*As of Feb 28 2018

10. 10
1.About Our Team
2.About Our Product
3.STLC
4.RTM(Struggle& Challenge#1)
5.Flow(Struggle& Challenge#2)
6.Conclusion
Agenda

11. T1(Unit Test)Implementation
DevSpec T2(Integration Test)
1.Test
Plan
2.Test
Design
3.Test
Implement
ation
5.Test
Execution
6.Release
Judgement
PRD
Affected Area/
Test Environment Information
System Test(QA)
• Test Strategy
• Test Scope
• Estimation(Draft
)
• RTM
• HLS
• Estimation(Fix)
• Case
• Data
• Env Setup
SQA Task
Output
For stakeholders
Legend
• Incident
Report
• Final Report
Signoff for Reliability , Maintainability….
4.Smoke
Test
Sharing
• Smoke
Test
Cases
Smoke Test
Result
Output
w/o sharing
Input from
stakeholders
STLC
PRD(Requirement)

12. 12
1.About Our Team
2.About Our Product
3.STLC
4.RTM(Struggle& Challenge#1)
5.Flow(Struggle& Challenge#2)
6.Conclusion
Agenda

13. 13
1.About Our Team
2.About Our Product
3.STLC
4.RTM(Struggle& Challenge#1)
5.Flow(Struggle& Challenge#2)
6.Conclusion
Agenda

14. 1.Test
Plan
2.Test
Design
3.Test
Implement
ation
5.Test
Execution
6.Release
Judgement
• Test Strategy
• Test Scope
• Estimation(Draft
)
• RTM
• HLS
• Estimation(Fix)
• Case
• Data
• Env Setup
• Incident
Report
• Final Report
4.Smoke
Test
Sharing
• Smoke
Test
Cases
Struggle#1

15. • Test Strategy
• Test Scope
• Estimation(Draft
)
• RTM
• HLS
• Estimation(Fix)
• Case
• Data
• Env Setup
• Incident
Report
• Final Report
• Smoke
Test
Cases
Struggle#1
1.Difficult to manage both of Requirement and Test cases
2.Difficult to utilize our activity.
----- ---- -----
----- -----
-----
---- ---- ----
----- ----
-----
---------
------
1.Test
Plan
2.Test
Design
3.Test
Implement
ation
5.Test
Execution
6.Release
Judgement
4.Smoke
Test
Sharing
RTM

16. • Test Strategy
• Test Scope
• Estimation(Draft
)
• RTM
• HLS
• Estimation(Fix)
• Case
• Data
• Env Setup
• Incident
Report
• Final Report
• Smoke
Test
Cases
Struggle#2
1.Test
Plan
2.Test
Design
3.Test
Implement
ation
5.Test
Execution
6.Release
Judgement
4.Smoke
Test
Sharing

17. • Test Strategy
• Test Scope
• Estimation(Draft
)
• RTM
• HLS
• Estimation(Fix)
• Case
• Data
• Env Setup
• Incident
Report
• Final Report
• Smoke
Test
Cases
Struggle#2
1.Test
Plan
2.Test
Design
3.Test
Implement
ation
5.Test
Execution
6.Release
Judgement
4.Smoke
Test
Sharing

18. • Test Strategy
• Test Scope
• Estimation(Draft
)
• RTM
• HLS
• Estimation(Fix)
• Case
• Data
• Env Setup
• Incident
Report
• Final Report
• Smoke
Test
Cases
Struggle#2
1.Difficult to make it on time about automation task
1.Test
Plan
2.Test
Design
3.Test
Implement
ation
5.Test
Execution
6.Release
Judgement
4.Smoke
Test
Sharing

19. 19
1.About Our Team
2.About Our Product
3.STLC
4.RTM(Struggle& Challenge#1)
5.Flow(Struggle& Challenge#2)
6.Conclusion
Agenda

20. 20
Self-Introduction
My Name is Sonal Singh ソナル シング
I’m from India
I have more than 10 years of experience as QA
I have worked into Telecom, Investment Banking,
Insurance, Re-insurance domains
I was into mobile game testing for 3 years
I’m ISTQB Foundation level certified
I have worked into Service based as well as
product based companies

21. 21
Have you ever experienced
Production Issues? I never had 

22. 22
Now?

23. 23
And Production issue is always scary!

24. 24
Retrospective result showed some facts:
 Many defects were missed because some specifications were missed in test design.
Requirement Test Design
Requirement A

Requirement B 
Requirement C 
Requirement D 
Requirement E 
Requirement F 
Requirement G

Requirement H 

25. 25
Retrospective result showed some facts:
 There were many specification sheets kept at different locations.
 Multiple requirements were changed after Test design was finished.
Req v1.0 Req v1.1 Req v1.2

26. 26
Major Problem
What if requirement changes
multiple times after test design
is finished and test implementation
is in process?

27. 27
Is there any solution?
RTM
YES


28. 28
What is RTM 

29. 29
RTM – Requirement traceability Matrix
 It a matrix which is used to ensure test coverage.
S.No. Features/Module Requirements Priority HLS – High level scenario Test case # Coverage
1 Login Req#1 - Login &
logout
1 Successful login. TC001-
TC010 
1 Incorrect credentials not able to
login.
TC011-
TC015 
2 New Member Req#2 - New Member
creation
1 New member registration
successfully.
TC016-
TC025 
3 RP form <Req#3 -Example 3> 2 Non member can submit form. TC026-
TC045 
<Req#4 - Example 4> 2 Member can withdraw. TC046-
TC085 
4 Create a Hall <Req#5 - <Example
5>
1 Hall can be created successfully. TC086-
TC095 
5 Book a Hall <Req#6 - <Example
6>
2 Successful book a hall. TC096-
TC110 
6 Cancel a Hall <Req#7 - <Example 3 Successful cancel a hall. TC111-


30. 30
RTM – Requirement traceability Matrix
 You can create it in Microsoft excel or any test management tool which support Requirements.

31. 31
RTM – Requirement traceability Matrix
 Each requirement is linked with a high level scenarios(HLS).
S.No. Features/Module Requirements Priority HLS – High level scenario Test case # Coverage
1 Login Req#1 - Login & logout Successful login.
Incorrect credentials not able to login.
2 New Member Req#2 - New Member
creation
New member registration successfully.
3 RP form <Req#3 -Example 3> Non member can submit RP form.
<Req#4 - Example 4> Member can withdraw.
4 Create a Hall <Req#5 - <Example 5> Hall can be created successfully.
5 Book a Hall <Req#6 - <Example 6> Successful book a hall.
6 Cancel a Hall <Req#7 - <Example 7> Successful cancel a hall.

32. 32
RTM – Requirement traceability Matrix
 Each HLS is mapped to the corresponding test cases.
S.No. Features/Module Requirements Priority HLS – High level scenario Test case # Coverage
1 Login Req#1 - Login & logout 1 Successful login. TC001-TC010

1 Incorrect credentials not able to login. TC011-TC015

2 New Member Req#2 - New Member
creation
1 New member registration successfully. TC016-TC025

3 Form <Req#3 -Example 3> 2 Non member can submit form. TC026-TC045

<Req#4 - Example 4> 2 Member can withdraw. TC046-TC085

4 Create a Hall <Req#5 - <Example 5> 1 Hall can be created
successfully.
TC086-TC095

5 Book Hall <Req#6 - <Example 6> 2 Successful book a hall. TC096-TC110

6 Cancel Hall <Req#7 - <Example 7> 3 Successful cancel a hall. TC111-TC156


33. 33
 For test case if there is any defect then link that as well.
S.No. Features/Module Requirements HLS – High level scenario Test case # Defect Id # Status
1 Login Req#1 - Login & logout Successful login. TC001-TC010 Defect#001-010 Fail
Incorrect credentials not able
to login.
TC011-TC015 Defect#0011 Pass
2 New Member Req#2 - New Member
creation
New member registration
successfully.
TC016-TC025 Defect#0012 Pass
3 Form <Req#3 -Example 3> Non member can submit form. TC026-TC045
<Req#4 - Example 4> Member can withdraw. TC046-TC085
4 Create a Hall <Req#5 - <Example 5>Hall can be created
successfully.
TC086-TC095
5 Book a Hall <Req#6 - <Example 6>Successful book a hall. TC096-TC110
6 Cancel a Hall <Req#7 - <Example 7>Successful cancel a hall. TC111-TC156
RTM – Requirement traceability Matrix

34. S.No. Features/Module Requirements Priority HLS – High level scenario Test case # Defect Id
#
Status URL
1 Login Req#1 - Login &
logout
1 Successful login. TC001-
TC010
Defect#00
1
Fail
1 Incorrect credentials not able to
login.
TC011-
TC015
Pass
2 New Member Req#2 - New
Member creation
1 New member registration
successfully.
TC016-
TC025
Pass
3 Form <Req#3 -Example
3>
2 Non member can submit form. TC026-
TC045
Pass
<Req#4 - Example
4>
2 Member can withdraw. TC046-
TC085
Pass
4 Create a Hall <Req#5 - <Example
5>
1 Hall can be created successfully. TC086-
TC095
Pass
5 Book a Hall <Req#6 - <Example
6>
2 Successful book a hall. TC096-
TC110
Pass
6 Cancel a Hall <Req#7 - <Example
7>
3 Successful cancel a hall. TC111-
TC156
Pass
To be provided by PDM Not Required
RTM – Requirement traceability Matrix

35. S.No. Features/Module Requirements Priority HLS – High level scenario Test case # Defect Id
#
Status URL
1 Login Req#1 - Login &
logout
1 Successful login. TC001-
TC010
Defect#00
1
Fail
1 Incorrect credentials not able to
login.
TC011-
TC015
Pass
2 New Member Req#2 - New
Member creation
1 New member registration
successfully.
TC016-
TC025
Pass
3 Form <Req#3 -Example
3>
2 Non member can submit form. TC026-
TC045
Pass
<Req#4 - Example
4>
2 Member can withdraw. TC046-
TC085
Pass
4 Create a Hall <Req#5 - <Example
5>
1 Hall can be created successfully. TC086-
TC095
Pass
5 Book a Hall <Req#6 - <Example
6>
2 Successful book a hall. TC096-
TC110
Pass
6 Cancel a Hall <Req#7 - <Example
7>
3 Successful cancel a hall. TC111-
TC156
Pass
To be updated by QA TeamRTM – Requirement traceability Matrix

36. S.No. Features/Module Requirements Priority HLS – High level scenario Test case # Defect Id
#
Status URL
1 Login Req#1 - Login &
logout
1 Successful login. TC001-
TC010
Defect#00
1
Fail
1 Incorrect credentials not able to
login.
TC011-
TC015
Pass
2 New Member Req#2 - New
Member creation
1 New member registration
successfully.
TC016-
TC025
Pass
3 Form <Req#3 -Example
3>
2 Non member can submit form. TC026-
TC045
Pass
<Req#4 - Example
4>
2 Member can withdraw. TC046-
TC085
Pass
4 Create a Hall <Req#5 - <Example
5>
1 Hall can be created successfully. TC086-
TC095
Pass
5 Book a Hall <Req#6 - <Example
6>
2 Successful book a hall. TC096-
TC110
Pass
6 Cancel a Hall <Req#7 - <Example
7>
3 Successful cancel a hall. TC111-
TC156
Pass
Updated by offsiteUpdated by onsite
Handling of RTM by Onsite & Offsite

37. S.No. Features/Module Requirements Priority HLS – High level scenario Test case # Defect Id
#
Status URL
1 Login Req#1 - Login &
logout
New Requirement
1 Successful login. TC001-
TC010
Defect#00
1
Fail
1 Incorrect credentials not able to
login.
TC011-
TC015
Pass
2 New Member Req#2 - New
Member creation
1 New member registration
successfully.
TC016-
TC025
Pass
3 Form <Req#3 -Example
3>
2 Non member can submit form. TC026-
TC045
Pass
<Req#4 - Example
4>
2 Member can withdraw. TC046-
TC085
Pass
4 Create a Hall <Req#5 - <Example
5>
1 Hall can be created successfully. TC086-
TC095
Pass
5 Book a Hall <Req#6 - <Example
6>
2 Successful book a hall. TC096-
TC110
Pass
6 Cancel a Hall <Req#7 - <Example
7>
3 Successful cancel a hall. TC111-
TC156
Pass
More Advantage of RTM
If any requirement changes then its easier to find out the test cases id which needs modification.

38. S.No. Features/Module Requirements Priority HLS – High level scenario Test case # Defect Id
#
Status URL
1 Login Req#1 - Login &
logout
1 Successful login. TC001-
TC010
Defect#00
1
Fail
1 Incorrect credentials not able to
login.
TC011-
TC015
Pass
2 New Member Req#2 - New
Member creation
1 New member registration
successfully.
TC016-
TC025
Pass
3 Form <Req#3 -Example
3>
2 Non member can submit form. TC026-
TC045
Pass
<Req#4 - Example
4>
2 Member can withdraw. TC046-
TC085
Pass
4 Create a Hall <Req#5 - <Example
5>
1 Hall can be created successfully.
5 Book a Hall <Req#6 - <Example
6>
2 Successful book a hall. TC096-
TC110
Pass
6 Cancel a Hall <Req#7 - <Example
7>
3 Successful cancel a hall. TC111-
TC156
Pass
More Advantage of RTM
It ensures coverage, so easy to find out if any requirement is without test cases.

39. S.No. Features/Module Requirements Priority HLS – High level scenario Test case # Defect Id
#
Status URL
1 Login Req#1 - Login &
logout
1 Successful login. TC001-
TC010
Defect#00
1
Fail
1 Incorrect credentials not able to
login.
TC011-
TC015
Pass
2 New Member Req#2 - New
Member creation
1 New member registration
successfully.
TC016-
TC025
Pass
3 Form <Req#3 -Example
3>
2 Non member can submit form. TC026-
TC045
Pass
<Req#4 - Example
4>
2 Member can withdraw. TC046-
TC085
Pass
4 Create a Hall <Req#5 - <Example
5>
1 Hall can be created successfully.
5 Book a Hall <Req#6 - <Example
6>
2 Successful book a hall. TC096-
TC110
Pass
6 Cancel a Hall <Req#7 - <Example
7>
3 Successful cancel a hall. TC111-
TC156
Pass
More Advantage of RTM
It shows which requirement had major no of defects.

40. 40
Suggestion

41. 41
1.About Our Team
2.About Our Product
3.STLC
4.RTM(Struggle& Challenge#1)
5.Flow(Struggle& Challenge#2)
6.Conclusion
Agenda

42. 42
Who am I
Name : Sadaaki Emura (nickname M)
Join in Rakuten : 2007
Career : Embedded engineer (2000-2002)
Web engineer (2002-2015)
Product manager (2015-2016)
QA (2016~)
Role: Test Automation Engineer Lead
Birthplace : Kanazawa-city
Hobby: jog , climbing , horse racing

43. 43
How do we use automation?

44. 44
How do we use automation testing for project
Regression test
• Current function
• Affect
• Not affect
• New function

45. 1.Test
Plan
2.Test
Design
3.Test
Implementatio
n
5.Test
Execution
6.Release
Judgement
• Incident
Report
• Final Report
4.Smoke Test
Sharing
should be all green
used for release judgement
How do we use automation testing for project
STLC

46. 1.Test
Plan
2.Test
Design
3.Test
Implementatio
n
5.Test
Execution
6.Release
Judgement
4.Smoke Test
Sharing
Difficult to make it on time about automation task
Struggle#2 (再掲)

47. 47
Reason for struggle
1. Web design is changeable include important functions
2. Coding is available after application is ready in test environment
3. Release cycle is short and automation engineer resource is less

48. 48
How do we handle?

49. 49
Manual & automation scope
Manual test Automation test
Current function
New function
△ ◎
◎ △

50. 50
Manual & automation QA process
Manual Team Automation Team
Test Design
Catch up automation
scope
execute manual test
Test Implementation
Modify regression script
Test design for
automation
Script run by CI
New functionCurrent function
Create script
Marge to regression script
QA done Fixed by project
done
Use next project
1-2
weeks

51. 51
Project & Regression test timeline
time
project1
project2
project3
project4
Project1 current function
Automation
cover
project
Project1 new function
Project2 current function
Project2 new function
Project3 current function
Project3 new
function
Project4
current
function

52. 52
Project scale ratio
62%
38%
Small Project
Big Project
※Small Project = Release within 2 weeks

53. 53
Automation test coverage
20%
16%
64%
new function
current function need modification
current function
※ target : SMALL project
※ scope : function to be automated
Focus on this

54. 54
Result
1. Changeable design can be support
2. Regression test is done on schedule
3. Green Jenkins when release

55. 55
1.About Our Team
2.About Our Product
3.STLC
4.RTM(Struggle& Challenge#1)
5.Flow(Struggle& Challenge#2)
6.Conclusion
Agenda

56. 56
1.RTM(Struggle& Challenge#1)
2.Flow(Struggle& Challenge#2)
Conclusion

57. 57
1.RTM(Struggle& Challenge#1)
2.Flow(Struggle& Challenge#2)
Conclusion
----- ---- -----
----- -----
-----
---- ---- ----
----- ----
-----
---------
------
RTM

58. 58
Conclusion
Manual Team
Automation TeamTest Design
Catch up automation scope
execute manual test
Test Implementation
Modify regression script Test design for automation
Script run by CI
New functionCurrent function
Create script
Marge to regression script
QA done Fixed by project
done Use next
project
1-2
weeks
1.RTM(Struggle& Challenge#1)
2.Flow(Struggle& Challenge#2)