By leveraging highly-analyzed operational data - the voice of customers, machines and tests - quality assurance (QA) and IT groups can derive major gains in quality of apps and in user experience.

1. From Data to Insights: How IT
Operations Data Can Boost Quality
By listening to the voice of customers, machines and test data,
organizations can advance their digital business objectives and
position quality as a key competitive advantage.
Executive Summary
The digital economy has transformed the way
organizations do business. To survive in a
complex and competitive marketspace, it has
become imperative for enterprises to rely on a
variety of software systems that drive customer
experience and express brand value. The stakes
today are higher than ever before, which makes
software failure cost an unforgiveable mistake
for business.
This has triggered a consequent realignment in
business strategies, bringing a renewed focus on
technology as a key enabler of digital transfor-
mation. Big data and analytics are the engines
powering such transformations as they together
yield intelligent solutions enabling better
decisions, improving customer experience and
driving innovation. In fact, by 2018, 67% of the
CEOs of global 2000 enterprises will position
digital at the center of their corporate strategies.1
This white paper offers an in-depth analysis
of the benefits accrued to QA when analytical
tools and strategies are applied to data from
downstream operations systems. The insights
generated by identifying root causes of testing
holes and leaked defects, in our view, can be used
to improve upstream quality and optimize testing.
Error-prone code modules can also be identified,
which should eventually lead to a streamlined test
strategy before code is released into production.
Moreover, by analyzing historical data and
developing trends using live data, previously
unanticipated defects can be predicted, enabling
preemptive quality intervention.
Gaining Actionable Insights
from Operational Data
The data residing in systems across the lifecycle
can be utilized to map code to tests to defects
and to incidents in production systems, making
it possible to instantly isolate the root cause of
a failure.
Terabytes of data generated every day by
production systems can yield results that can be
used across the spectrum of testing to enable
advanced prediction of failures due to defects
or environmental or configuration issues. This
predictive ability empowers quality assurance in
the digital economy.
cognizant 20-20 insights | november 2016
• Cognizant 20-20 Insights

2. cognizant 20-20 insights 2
How We Got Here
Even with 100% test coverage, applications still
end up failing in production due to an assortment
of reasons that current test methodologies
cannot address. As has been observed, a sig-
nificant number of production incidents tie back
to an oversight in application testing, making
it the topmost priority when developing a cost-
avoidance strategy.
Systems utilized by IT operations professionals
daily generate massive amounts of data. A wealth
of information resides in such systems, such as
log data, app or infrastructure monitoring data,
transaction data, service ticket data, mobile
device usage and app crash data, user preferenc-
es, app settings, network conditions, etc. While
operational personnel, through a variety of
advanced analytical tools, regularly leverage this
information to enhance operational efficiency,
QA teams cannot consume the output with the
tools that they use. A significant amount of effort
is required to transform this data before it can
be leveraged to generate QA insights. In order to
enhance quality efficiency, it is imperative that
QA teams be able to gather insights from this
data, without getting lost when rifling through
voluminous operations logs.
Extracting Intelligence from Ops Data
Log files can be classified into three broad
categories, based on the data source and SDLC
process from which they are generated:
• Voice of customers: Incidents logged in tra-
ditional incident management systems from
production systems.
• Voice of machines: System logs, server logs,
database logs and application logs generated
automatically by machines in production when
an application is up and running.
• Voice of tests: Application logs generated in
the test environment combined with defects,
test cases and test execution logs from test
management tools and code to test correla-
tions generated by other specialized tools.
DEVELOPMENT
CODE
Code to Tests Tests to Defects Defects to Logs
CORRELATE CORRELATE CORRELATE
TESTS DEFECTS OPERATIONS LOGS
TESTING RELEASE
Upstream processes Downstream processes
Build correlations across the lifecycle
Deriving Insights from ‘Ops’ Data
Figure 1
Derive root cause analysis for
incidents (voice of customers,
tests & voice of machines).
Correlate incident logs to gaps
in testing (voice of customers).
Correlate event logs to gaps in
testing (voice of machines).
Correlate test event logs to
gaps in testing (voice of tests).
Reduce defect slippage.
Insights from
Operations
Dashboard View
(Real-time Alerts,
Visual & Predictive
Analytics,
Intelligent search,
Healthinsight)
Impacted
Test Cases
Data Rules
Business Rules
Analytics
Engine
Log Cleanup/Analysis
Text Analytics
Application logs
Database logs
App server logs
Event logs
Perfmon
Test
Management
Incident logs
Workflow for Capturing Insights from Operations
Figure 2

3. cognizant 20-20 insights 3
A simple four-step process can be used to
transform and analyze log data:
• Data massaging: Input data typically comes
in a variety of formats, ranging from unstruc-
tured free text to structured XML. This data
has to be cleaned; missing data then needs to
be represented through data imputation for
easy manipulation in the next step.
• Data analysis: This involves applying text
analytics techniques to identify key predictors
and descriptors, which can then be used in a
clustering algorithm to identify leading trends.
• Data transformation: Analyzed data can be
transformed in accordance to business rules to
discern correlations and generate a map tying
code to tests to defects and to production
incidents, thereby identifying probable root
causes and possible resolutions.
• Visualization and representation: Correlated
data can then be depicted graphically or
tabulated in an easy-to-understand format
that can be used by QA teams for quick
analysis and action.
This process can be further extended through
supporting utilities to automate failure resolution
retroactively or proactively, thereby preempting
failure through predictive analytics based on
defect and test execution histories.
The Benefits
By automatically correlating data across the
software lifecycle and applying analytics, IT orga-
nizations can generate quality insights that tradi-
tional testing processes fail to provide. They can
effectively collaborate and deliver by maximizing
velocity and minimizing risk and also reduce IT
spend on incident management. Sample use
cases from the point of view of different stake-
holders in the software value chain include:
• QA analyst: Can I implement a self-healing
solution to create automated test scripts and
execute these tests based on defects found in
production logs?
>> Identify testing holes through analysis of
applications lifecycle management (ALM),
operations and incident logs.
>> Sample testing holes reports:
»» Coverage analyzer based on logs.
»» Defect map matrix based on nonstruc-
tured data.
»» Tailored reports to solve specific client
problems.
Once these testing holes or gaps are
identified by correlating ops and test data, it
can be further integrated with external test
case generation utilities to auto-generate
automated test scripts. These test scripts can
then be executed using continuous integration
solutions, and the results recorded to verify
defect resolution, making a fully automated,
self-healing process from a QA perspective.
• Operations analyst: Can I build system intel-
ligence to automate defect triage?
>> Identify the root cause of a production inci-
dent from the voice of customers and associ-
ate it with testing holes or gaps.
>> Use advanced machine learning algorithms
to learn and predict “triage root cause” val-
ues, thereby eliminating the manual com-
ponent of root cause identification and en-
abling quicker defect triage.
• Developer: Can I automatically identify
functional gaps to feed into unit tests based
on application logs?
>> Identify and highlight functional gaps in
code using baseline application binary scans.
>> Trace back testing holes or gaps to code ele-
ments to identify functional gaps, which can
then be added to unit tests. Improved test cov-
erage enhances the quality of the deliverable.

4. Insights from Operations: A Working Model
Quick Take
A U.S. discount footwear retailer wanted to make
its QA process more intelligent. The retailer was
looking at ways to test more intelligently and
effectively to ensure minimal leakage of defects
to a mission-critical production system.
By analyzing the operations logs, actionable
insights were generated that helped improve
future testing iterations through the processes
outlined in this white paper. The client was able to
test coverage and reduce cycle time by focusing
on high-value tests. Moreover, minimized defect
leakage helped the client deliver an improved
customer experience. Figure 4 offers a snapshot
of the log files that were taken as input and the
inferences that were generated.
Helping a U.S. Footwear Retailer Minimize Defect Leakage
4cognizant 20-20 insights
Analytics Engine
(Text Analytics & NLP Algorithms)
Operations Engineer
Can I build systems intelligence
to automatically update my
virtual services based on
Incident logs?
Developer
Can I automatically identify
functional gaps to feed into
unit tests based on application
logs?
Quality Engineer
Can I automatically create test
scripts and execute tests based
on defects found in production
logs?
Identify root cause of the
incident from voice of
customers
Identify and highlight
functional gaps in code
using baseline application
binary scans
Intelligently identify
gaps in testing by building
correlations with
baseline test assets
and footprints
INCIDENT LOGS
TESTING
HOLES/GAPS
TESTING
HOLES/GAPS
TESTING
HOLES/GAPS
UPDATED
VIRTUAL SERVICE
COMPONENT
SELF-HEALING
COGNIZANT
ADPART
FUNCTIONAL GAPS IN CODE ELEMENTS
USING TRACEABILITY FROM
PRODUCTION ERRORS
APPLICATION LOGS
PRODUCTION LOGS AUTOMATICALLY
GENERATE
TEST SCRIPTS
& EXECUTE
Figure 3
INPUTS
PROCESSES
INFERENCES
CODE RELATED
TEST RELATED
DEFECT RELATED
Application server logs from production
systems
Defect summary logs from the test
management tool, HP ALM
Tests & code were correlated using the
mocked data from the CodeInsight solution
Application server errors were correlated
with the defect summary
Test cases were further mapped to code
elements to derive E2E mapping from
defects to code
Identified potential defect recurrence from the testing phase.
Identified new errors which were missed in the testing phase,
read 13,250 error messages.
Found functional gaps in code which was subsequently used by
the development team for analysis and remediation.
Log analysis classified the error messages as 90% testing
holes and 10% defect leakage.
Mapped production errors to respective test cases.
Defect description — 63% traceable to test case description.
Test case description — 70% traceable to defect description.
Figure 4
From Data to Insights

5. About Cognizant
Cognizant (NASDAQ: CTSH) is a leading provider of information technology, consulting, and business process services,
dedicated to helping the world’s leading companies build stronger businesses. Headquartered in Teaneck, New Jersey
(U.S.), Cognizant combines a passion for client satisfaction, technology innovation, deep industry and business
process expertise, and a global, collaborative workforce that embodies the future of work. With over 100 develop-
ment and delivery centers worldwide and approximately 255,800 employees as of September 30, 2016, Cognizant is a
member of the NASDAQ-100, the S&P 500, the Forbes Global 2000, and the Fortune 500 and is ranked among the top
performing and fastest growing companies in the world. Visit us online at www.cognizant.com or follow us on Twitter:
Cognizant.
World Headquarters
500 Frank W. Burr Blvd.
Teaneck, NJ 07666 USA
Phone: +1 201 801 0233
Fax: +1 201 801 0243
Toll Free: +1 888 937 3277
Email: inquiry@cognizant.com
European Headquarters
1 Kingdom Street
Paddington Central
London W2 6BD
Phone: +44 (0) 20 7297 7600
Fax: +44 (0) 20 7121 0102
Email: infouk@cognizant.com
India Operations Headquarters
#5/535, Old Mahabalipuram Road
Okkiyam Pettai, Thoraipakkam
Chennai, 600 096 India
Phone: +91 (0) 44 4209 6000
Fax: +91 (0) 44 4209 6060
Email: inquiryindia@cognizant.com
­­© Copyright 2016, Cognizant. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in any form or by any
means, electronic, mechanical, photocopying, recording, or otherwise, without the express written permission from Cognizant. The information contained herein is
subject to change without notice. All other trademarks mentioned herein are the property of their respective owners.
Footnote
1
Culled from IDC’s IT Industry, Digital Transformation and CIO Agenda webcasts.
About the Author
Vikul Gupta is Director for Digital Assurance and DevOps with Cognizant’s Quality Engineering &
Assurance business unit. He has more than 16 years of experience in strategy, management, delivery
and development on multiple technologies, and deep domain experience in DevOps, cloud and infra-
structure automation, with roles ranging from product developer to key strategist. Currently, Vikul is
responsible for creating differentiated DevOps service offerings and intellectual property that defines
the go-to-market strategy for these offerings. He is a graduate of the National Institute of Technology,
Surat. Vikul can be reached at Vikul.Gupta@cognizant.com.
Codex 2311
Moving Forward: Shifting Gears
As enterprises balance between legacy and
digital IT, addressing new paradigms of QA will
require an ecosystem thinking for assurance
solutions that are open, community-based and
consumed in an as-a-model manner. Building
on next-gen automation and intelligence capa-
bilities, these solutions should be able to drive
business and technology assurance outcomes
for the enterprise.
Quality at speed requires enterprises to look
beyond traditional realms of testing to improve
test cycle times, while delivering predictable
quality. The utilization of operations data has in
our view resulted in improved quality of applica-
tions and has created positive user experiences.
The evolving maturity of analytics engines
built on artificial intelligence, natural language
processing and machine learning techniques
empowers enterprises to use their operations
data to create a quality-led competitive
advantage. This not only translates into lower
software development costs but also has
widespread implications on customer experience
and resulting brand value that are exceptionally
valued in the digital world.