Agile Mumbai 2022 AI-driven TestOps : A Need or Necessity for DevOps Saikat Prabhakar Director Engineering, Guavus - A Thales Company

1. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
1
OPEN
www.thalesgroup.com
AI-Driven TESTOPS
A NECESSITY IN THE DEEP-TECH VUCA WORLD - AN
ENGINEERING PERSPECTIVE
Saikat Prabhakar
Director Engineering
Guavus, A Thales Company
https://www.agilemumbai.com/

2. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
2
OPEN
www.thalesgroup.com
Welcome to the VUCA world
It is the reality, unfortunately..
• Changes in SDLC can be disruptive and unpredictable both in terms
of volume and velocity.
• The ecosystem doesn’t always offer complete and reliable
information to make the most intelligent and future-proof decisions.
• The environment is increasingly dynamic and complex with multiple
moving pieces.
• The information available can be interpreted in more than one ways
leading to ambiguity.
• Leadership must evolve yet needs to be enabled !
https://www.agilemumbai.com/

3. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
3
OPEN
www.thalesgroup.com
It isn't getting any better really
The Deep Tech space demands more
• Pre-established Agile transformation “best practices” and
frameworks might not transcend naturally into the futuristic world of
Deep Tech.
• Deep Tech projects involve continuous exploration and
experimentation and that too parallelly at times.
• The ‘X’Ops movement just got bigger – now we have DataOps,
MLOps , AIOps etc..
• VUCA is amplified !
https://www.agilemumbai.com/

4. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
4
OPEN
www.thalesgroup.com
Quality is the key !
And the best part is that everyone has an opinion about
it
• We know we have “built the right thing” when we achieve
Customer’s User Experience.
• However, we need to “build things the right way” in order to counter
VUCA.
• Cruft leads to Tech Debt
• We need to continuously measure the level of compromise.
• “Internal Quality” v/s “External Quality”
• TestOps is the way forward !
https://www.agilemumbai.com/

5. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
5
OPEN
www.thalesgroup.com
What is TestOps ?
The answer to the ever-increasing demand for efficiency
and agility in quality
• Well DevOps operationalizes your SDLC for faster deliveries.
• But TestOps operationalizes its test needs to ensure those deliveries
are reliable and scalable too.
• Testing never stops, it is continuous and is the glue for Agile +
DevOps
https://www.agilemumbai.com/

6. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
6
OPEN
www.thalesgroup.com
A practical perspective
Test operations need to scale..
• An effective DevOps strategy rides on the success of its individual
atomic stages.
• Software Testing, in particular, is complex - Non-functional aspects
such as Performance, Security, Resiliency, ML-oriented testing etc are
important too.
• We need the right arsenal
• Complexity increases when we talk about Test automation across all
these levels.
https://www.agilemumbai.com/

7. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
7
OPEN
www.thalesgroup.com
An analytical perspective
Allowing fully automated transparent view of continuous
testing to stakeholders
• We must plan ‘what’ should be operationalised and ‘how’ ?
• ’What’ helps answer what should be prioritised considering at what
stage of the project we are.
• ‘How’ can be achieved by a mix of indigenous or commercial
platforms.
• The level of quality should be visible and comprehensible to every
stake holder.
https://www.agilemumbai.com/

8. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
8
OPEN
www.thalesgroup.com
Is TestOps shift-left or shift-right ?
We need to stretch both ways
• Shift-left is much needed for involvement of Developers in Quality
much early in the SDLC.
• Shift-right is needed to be closer to the User and Production needs.
• TestOps must pave the path to stretch both ways.
https://www.agilemumbai.com/

9. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
9
OPEN
www.thalesgroup.com
Artificial intelligence quotient
Ability and efficiency of intelligent systems in terms of
knowledge, learning, use, and creation.
• An intelligent system should be able to act based on what it learns.
• Automation should be driven by the principles of self-learning and
self-optimisation.
• Each step should supplement the cause of AIOps
https://www.agilemumbai.com/

10. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
10
OPEN
www.thalesgroup.com
AI-driven Testops
Eating your own dog food..
• Targeting the intelligence derived from the key metrics from the test
execution data to empower efficiency in future runs.
• Deep Tech space calls for specific test tools & ML-driven test
approaches to learn problem areas because the Tech is entirely
different.
• TestOps is a major driving-factor for AIOps as it feeds the necessary
data to predict risks.
https://www.agilemumbai.com/

11. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
11
OPEN
www.thalesgroup.com
Where can I dig ?
Data Data Data..
• Data from Project Mgmt, Test Mgmt & Defect Mgmt Tools
• Data from the functional and non-functional testing tools.
• Data from the Test Automation Frameworks , Logs and Reports.
• Data from Test and Production Environments
https://www.agilemumbai.com/

12. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
12
OPEN
www.thalesgroup.com
Intel gathering
Metrics should be comprehensible and reusable
• Insufficient Requirement Coverage
• Insufficient Test Coverage
• Automation v/s Manual coverage
• Gaps in Test environment v/s Staging or Production
• Test Automation Execution Efficiency
• Slowness in Test Execution causing delay in releases
• Excessive Test Maintenance
• Test Flakiness
• Anomalies and Outliers in Application Performance Monitoring
• Application downtime
https://www.agilemumbai.com/

13. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
13
OPEN
www.thalesgroup.com
Insights
Slowness causing delay in releases
https://www.agilemumbai.com/

14. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
14
OPEN
www.thalesgroup.com
Insights contd..
Test Automation Execution Efficiency
https://www.agilemumbai.com/

15. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
15
OPEN
www.thalesgroup.com
Insights contd..
Insufficient Test Coverage
https://www.agilemumbai.com/

16. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
16
OPEN
www.thalesgroup.com
Insights contd..
Reducing excessive Test & Defect Management
https://www.agilemumbai.com/

17. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
17
OPEN
www.thalesgroup.com
Tech Debt : Blast from the past
Cruft slows us down..
• According to Robert C. Martin, recognized for developing Software
Design Principles, there are 4 important characteristics of a bad
design. Those are:
Rigidity
Immobility
Fragility
Viscosity
• Rings a bell ? Sounds like VUCA..
https://www.agilemumbai.com/

18. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
18
OPEN
www.thalesgroup.com
Cruft in the Deep Tech Arena
Creates ‘Deep’ Tech Debt..
• Technical Stories take a whole new dimension
• Choice of Tech Stack could be decisive
• Data Science squad members needs to be aligned with the sprints.
https://www.agilemumbai.com/

19. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
19
OPEN
www.thalesgroup.com
TestOps can help fuel strategic decisioning
Quality should be everyone’s responsibility
• Time lost is money lost !
• Outliers and anomalies must be understood by all key stake holders
for alignment.
• Issues raised should fuel prioritisation
https://www.agilemumbai.com/

20. This
d
ocument
may
not
be
reproduced,
modified,
adapted,
published,
translated,
in
any
way,
in
whole
or
in
part
or
disclosed
to
a
third
party
without
the
prior
written
consent
of
THALES
-
©
2022
THALES.
All
rights
reserved
.
20
OPEN
www.thalesgroup.com
THANK YOU
Q & A
https://www.agilemumbai.com/