The web-conference hosted by CRISIL Global Research & Analytics on “Big Data’s Big Impact on Businesses” on January 29, 2013, saw participation from senior officials of global multinationals from 9 countries. The presentation described how data analytics is helping businesses make “evidence-based” decisions, thereby creating a positive impact. It also spoke about the opportunities opening up in the Big Data space in India and across the globe.
Hosted by:
Sanjeev Sinha, President, CRISIL Global Research & Analytics
Gaurav Dua, Director & Practice Leader (Technology, Media & Telecom), CRISIL Global Research & Analytics
Russian Call Girls In Gurgaon ❤️8448577510 ⊹Best Escorts Service In 24/7 Delh...
Big Data’s Big Impact on Businesses
1. Big Data’s Big Impact on Businesses
To join the call, please dial the below toll-free phone line for your country:
– USA 1 866 746 2133
– UK 0 808 101 1573
– Singapore 800 101 2045
– Hong Kong 800 964 448
– India 1 800 200 1221
– Australia 1 800 053 698
– Poland 00 800 112 4248
– Netherlands 0 800 022 9808
– UAE 800 017 5282
– Argentina 0 800 444 1557
– China 10 800 140 1383
– South Korea 003 081 32 503
– Sweden 020790997
If you are not based in any of the above locations, you can dial the following numbers to participate in the discussion.
Primary number: 0013233868721
Secondary number: 00442031067123
1
2. Big Data’s Big Impact
on Businesses
Webconference : Jan 29, 2013
3. Key Takeaways
Slide 3
Introduction to Big Data
Slide 5
Global Landscape and Trends
Slide 12
The Big Data Opportunity
Slide 20
Big Data’s Big
Impact on
Businesses
4. Key Takeaways
Big Data market opportunity is expected to witness strong growth in the next 5 years
– Expected to touch US$25 billion globally; the ‘BIG’ opportunity for India lies in the IT & IT-enabled
Services space, which is likely to be ~US$ 10-11 billion market globally in 2015
– India is likely to garner a ~10% share of the ~US$ 10-11 billion global Big Data IT Services Market by
2015
– Data-related regulations like Dodd-Frank and Basel III to impact Big Data implementations
Initially, North America & Europe are likely to drive the Big Data opportunity since
over 85% of the world’s data is today residing in these 2 regions
New database architectures and innovative analytics tools & techniques to facilitate
Big Data implementations
By end of 2012, around 90% of Fortune 500 companies had some initiatives underway
related to Big Data
Key verticals driving demand for Big Data analytics: Financial services, Retail,
Telecom, Healthcare and Manufacturing
Key risk – potential shortfall of 1.5 million Data-Savvy Managers and 140,000-190,000
Data Scientists in the US by 2018
Source: CRISIL GR&A analysis
4
5. Key Takeaways
Slide 3
An Introduction to Big Data
Slide 5
Global Landscape and Trends
Slide 16
The Big Data Opportunity
Slide 23
Definition of Big Data
Big Data ecosystem
Benefits of Big Data to enterprises
Key applications for end
consumers
6. Big Data is Defined by Volume, Variety and Velocity
What is Big Data ?
Big Data relates to rapidly growing, Structured and Unstructured datasets with sizes beyond the ability of
conventional database tools to store, manage, and analyze them. In addition to its size and complexity, it refers to
its ability to help in “Evidence-Based” Decision-making, having a high impact on business operations
Speed, Accuracy and Complexity of Intelligence
3Vs 1 Large quantity of data
Small Data Sets Big Data which may be enterprise-
Volume specific or general and
public or private
Advanced Big Data
analytics analytics
2
Diverse set of data
being created, such
Variety as social networking
Small Data Sets Big Data feeds, video and
audio files, email,
sensor data and
Traditional Traditional other raw data
analytics analytics Velocity
3
Speed of data inflow as
Gigabytes Terabytes Petabytes Zetabytes well as rate at which this
fast-moving data needs to
Size of Data be stored
Source: CRISIL GR&A analysis
Source: CRISIL GR&A analysis
6
7. The Global Data likely to Grow at a CAGR of 41%
Growth of global data, 2009-2020
Zetabytes CAGR
35.0
(2009-2020)
41.0%
7.9
1.9
0.8
2009 2011 2015 2020
Implication for an organization
Need for large storage capacity and quick retrieval of data
Enable informed decision-making effectively, leveraging large data sets
– Turn 12 TB of Tweets created each day into improved product sentiment analysis
– Convert 350 billion annual meter readings to better predict power consumption
Note: ZB stands for Zetabytes;
Source: IDC; CRISIL GR&A analysis
7
8. Volume
Today 80% of Data Existing in any Variety
Enterprise is Unstructured Data Velocity
Introduction Resides in formal data stores – RDBMS and Data
Structured Data Warehouse; grouped in the form of rows or columns
Variety of sources from where data is being
Accounts for ~10% of the total data existing currently
generated has also undergone a shift
RDBMS (e.g., Data Microsoft Project
The types of data being created has changed from ERP and CRM Warehousing Plan File
structured to semi-structured to unstructured data
A form of structured data that does not conform with the
Implication for organization Semi-
formal structure of data models
Structured Data
Need to manage broad range of data types Accounts for ~10% of the total data existing currently
Process analytic queries across numerous data
types
Comprises data formats which cannot be stored in row/
Unstructured column format like audio files, video, clickstream data,
Data
Solutions required Accounts for ~80% of the total data existing currently
Weather Location
Need to extract meaningful analysis from this data Video Audio Text message Blogs patterns co-ordinates
has led to several technologies to gain traction
Examples include NoSQL databases to store
unstructured data as well as innovative processing
Web logs & Sensor data/ Geospatial
methods like Hadoop and massive parallel clickstreams M2M Email Social media data
processing (MPP)
Source: Industry reporting; CRISIL GR&A analysis
8
9. Volume
Big Data will Enable Real Time Analytics Variety
Velocity
Big Data velocity enabling real
600+ time use of data
videos on
YouTube 200 Big Data is also characterized by
1,500+
blog posts million+ velocity or speed i.e. frequency of
emails sent data generation or the frequency of
data delivery
2 New age communication channels
7,000+ million+
photos on Google such as mobile phones, emails, social
flickr search networking has increased the rate of
Data queries information flows
velocity
per Examples:
minute
700,000+ 400,000+ Telcos adopting location based
minutes of
Facebook
Skype marketing based on user location
updates sensed by mobile towers
calling
Satellite images can help monitor and
US$ 3500+ analyze troop movements, a flood
300,000+ ticks per
are spent minute in plane, cloud patterns, or forest fires
on online 400,000+ securities
shopping trading Video analysis systems could monitor
tweets on
Twitter a sensitive or valuable facility,
watching for possible intruders and
alert authorities in real time
Source: Industry reporting; CRISIL GR&A analysis
9
10. Big Data Analytics is Application of Advanced Techniques on Big
Datasets; Answers Questions Previously Considered Beyond Reach
Evolution of analytics Big Data analytics
Behavioral analytics Advanced
Big Data analytics is
analytics
where advanced Stochastic Analytic
analytic techniques Complex optimization database Why did it
Prescriptive are applied on Big event happen?
functions
analytics Data sets processing Constraint When will it
Optimization
The term came into Extreme SQL
based BI happen
play late 2011 – early Visualization again?
Level of Complexity
2012 Predictive
Social network analytics What
modeling
caused it to
Predictive Semantic analytics
Forecast happen?
analytics
- ing What can be
Time series analysis
Statistical done to
analysis Natural Language Processing avoid it?
Multivariate statistical analysis
Alerts Online analytical processing (OLAP)
Descriptive Query Data mining
analytics drill
down
Adhoc Basic analytics
reports What happened?
Standard When did it happen?
reports What was the its impact ?
Late 1990s 2000 onwards
Time
Analytics as a separate value chain function In-database analytics
Source: CRISIL GR&A analysis
10
11. Big Data Management, Analytics, IT Services & Applications
are the Key Constituents of Big Data Ecosystem
Four key elements: What does the Big Data Ecosystem Constitute ?
1. Big Data Components of Big Data Ecosystem
Management &
storage: End users
Data storage Big Data Analytics
application and use
Data analytics & its
infrastructure Applications
and technologies Developer Environments Analytics (mobile, search, web)
(Languages (Java), products
(SI,customization, consulting, system design)
2. Big Data Analytics Environments (Eclipse &
NetBeans), programming (Avro, Apache BI
Includes the Thrift)
interfaces (MapReduce)) &visualization
technologies and
tools
tools to analyze the
data and generate Input data
insight from it
Business analysts
3. Big Data’s Data
IT services
Application & Use Sources Big Data
Operational Data
Involves enabling
Data management &
Unstructured Data Architecture NoSQL
the Big Data data Hadoop/ Big Data NoSQL MPP
insights to work in (Text, web tech’y framework Hadoop RDBMS
storage
BI and end-user pages, social (MapReduce etc.) based DW
applications media content,
4. IT services including video etc.)
System Integration
Structured Data administration tools
Consulting
data
ETL & Data Workflow/
Project System
(stored in integration scheduler
management and tools
MPP, RDBMS products products
customization
and DW*)
*MPP – Massively parallel processing; RDBMS - Relational Data Base Management Systems; DW – Data warehouse
Source: CRISIL GR&A analysis
11
12. Key takeaways
Slide 3
An Introduction to Big Data
Slide 5
Global Landscape and Trends
Slide 16
The Big Data Opportunity
Slide 23
Big Data – Geographic Analysis
Market Trends & Developments
13. North America & Europe Drives the Big Data
Opportunity with over 85% of the World’s Data
As North America and Europe account for the lion’s share of the world’s data the initial opportunity of both Big
Data implementations and analytics lies in the these geographies i.e. developed economies
Key verticals: Healthcare, Key verticals: Technology, Financial services,
Manufacturing, Retail, Digital Oil & Gas, Utilities, Manufacturing
Marketing Demand trend: European MNC’s are still in
Demand trend: High demand the early stages of the adoption cycle
North of Big Data analytics
America Europe
>2,000 Japan
>400
>3,500 China
Middle East
>250
>200 India
>50 Key verticals: Manufacturing,
Telecom, Health & Life Sciences
Demand trend: Demand for BI
Demand trend: Current demand to derive operational efficiency
appears to be limited, however,
South America lack of skills may drive Key verticals: Telecom, Bioinformatics,
>40 outsourcing of Big Data analytics Retail
Demand trend: Industry is in nascent stage
with demand catching up, particularly in retail
Low awareness levels
Key verticals: Telecom, Retail, Banking
Demand trend: Still embryonic; most
organizations have wait and watch approach
Data generated: High to low
Amount of new Big Data stored (Petabytes), 2010
Source: McKinsey Global Institute; CRISIL GR&A analysis
13
14. Emergence of Niche Startups and Large IT Players Enhancing
their Big Data Capabilities are key enablers for the Industry
Market Trends and Developments
1 Emergence of niche Big Data startups driving technological innovation
2 Large IT players leveraging M&As to add Big Data capabilities to their service portfolios
3 Financial Services, Retail and Telecom are likely to be the early adopters in the Big Data space
4 Talent shortage is one of the biggest challenges of the Big Data space
Source: CRISIL GR&A analysis
14
15. Emergence of niche Big Data start-ups to boost 1
technological innovation
A new class of companies, specializing in Big Data technologies have emerged, to capitalize on the
opportunities in the Big Data domain
Technology Area Players*
Big Data start-ups – Key characteristics
Specialized in niche Big Data technologies like Hadoop distributions
Hadoop, NoSQL systems, in-memory analytics,
1
multiple parallel processing, and analytical
platforms
Non Hadoop Big Data
Majority of start-ups generate revenue less than Platforms
2 USD 50 million and exhibit double digit revenue
growth annually
Most start-ups raising funding by private ventures Analytic Platforms
3 and Applications
or being acquired by large IT players
Cloud-based Big
Data Applications
*Indicative list of players
Source: Industry reporting; CRISIL GR&A analysis
15
16. Large IT Players Leveraging M&As to add Big Data 2
Capabilities to their Service Portfolios
Target
Area Acquirer Date Deal value Rationale
Company
Data Oct. '11 USD 1.1 billion Develop a comprehensive platform to analyze Big Data
Management
USD 263 Strengthen position in data warehousing market through
Mar. '11
million expertise in SQL and MapReduce-based analysis
Extend Smarter Commerce suite with qualitative analytics
Jun. '12 N.A.
software
Leverage data navigation technologies for Big Data by
May. '12 N.A. automating discovery of through innovative index and search
capabilities
May. '12 N.A. Addition of sales performance analytics
Advanced
analytics
May. '12 N.A. Enhance Big Data marketing analytics
Apr. '12 N.A. Acquisition of spend and procurement analytics
Mar. '12 N.A. Accelerate development of Big Data analytic applications
Mar. '11 N.A. Enhance real time business analytics for Big Data
Key highlights
M&As in the Big Data space had tripled in M&As with bigger deal value are happening in data
the first half of 2012 management
Acquisition targets are mainly innovative Big Data
start-ups
N.A. is not available. Source: Industry reporting; CRISIL GR&A analysis
16
17. 1. Retail: Sears is leveraging Big Data analytics internally and 3
is also keen on offering analytics services externally
Sears Holding is a leading integrated retailer with ~4,000 full-line and specialty retail stores in the US
and Canada. It operates through its subsidiaries including Sears, Roebuck and Co. and Kmart Corp.
Challenge/Business
Solution Benefits
Need
IT need • Leverages its global In-house center in Across IT environment
• Manage Increasing volumes of data Pune, India for Big Data Analytics
• Utilization of 100% of collected data
like customer personal information, • Implemented a Big Data architecture against 10% utilization earlier
PoS data, online purchases, etc., using Hadoop
• Ability to run price elasticity
posing a challenge • Used MapReduce algorithms to analyze algorithms in one week, as opposed
• Capacity run-out on its mainframe, and data and feed results back into the to eight weeks previously
adding more capacity proving to be mainframe, on individual customer
expensive activity, across all 4,000 locations • Cost-savings of USD 600,000 per year
Business need Across business
• The need to set prices quickly and in • More relevant and personalized
real time customer communications and offers to
• The need to drive customer loyalty an active customer base (~80 million)
• Increased shopping and higher spend
per transaction by active members
Looking at the current and potential benefits of Big Data analytics, Sears aims to expand into newer areas and sell its data
management and analytics services technology to other companies, through its subsidiary MetaScale
*Massively Parallel Processing
Source: Industry reporting; CRISIL GR&A analysis
18. 2. Financial Services: Witnessing increased adoption of Big Data 3
analytics, to reduce risk and uncover new market opportunities
• The need to meet growing regulatory compliances, detect fraud and create new market opportunities is driving the growth for Big Data
analytics in the financial services sector
• Customer & transaction data from multiple channels like branch, kiosks, mobile and web; social media; emails; credit cards data;
insurance claims data; stock market data; statistical data, PDF & excel files, videos, government filings, etc. are key Big Data sources
Big Data application across Financial Services sub-sector
Capital
Banking Markets/ Insurance
Trading
Predict client longevity,
Credit line optimization Trading surveillance along with analyzing
perspective clients
Credit reward program medical status
Intraday analysis
analysis Using weather and
calamity information for
Trading pattern analysis managing exposures
and losses
Pre-trade decision support analytics
Risk management/assessment
Fraud detection
Portfolio analytics
Compliance & regulatory reporting
CRM,, Entering new markets
Source: Industry reporting; CRISIL GR&A analysis
19. Potential Shortfall of 1.5 million Data-Savvy Managers and 4
~150,000 Data Scientists in the US in 2018
Demand-supply gap for data scientists* Requisite educational
Role in Ecosystem Other expertise
in US, 2018 qualifications
440K-490K
Big Data analytics Advanced degree like Expertise in data
Data Business intelligence
M.S. or Ph.D., in analytics skills to extract
mathematics, statistics, data, use of modeling &
300K Scientists
Visualization economics, computer simulations
science or any decision
140K – 190K Multi-disciplinary
sciences
knowledge of business to
50%-60% find insights
gap relative
to supply Knowledge of statistics
Project management Advanced business
and/or machine learning
Data-savvy across the Big Data degree such as MBA,
M.S. or managerial to frame key questions
Managers ecosystem and analyze answers
diplomas
– Consulting
2018E Supply 2018E Demand services Conceptual knowledge of
– Implementation business to interpret and
– Infrastructure challenge the insights
Demand-supply gap for data-savvy
management
managers* in US, 2018 – Analytics Ability to make decisions
using Big Data insights
4.0 million
Technical support in Having a degree in Possessing data
Technical hardware & software computer management knowledge
2.5 million across the Big Data science, information
Engineers
ecosystem for: technology, systems IT skills to
1.5 million – Data architecture engineering. or related develop, implement, and
60% gap disciplines maintain hardware and
– Data
relative to administration software
supply – Developer
environment
– Applications
2018E Supply 2018E Demand
*Analysts with deep analytical training; **Managers to analyze Big Data and make decisions based on their findings; Source: McKinsey Global Institute; CRISIL GR&A analysis
19
20. Key Takeaways
Slide 3
An Introduction to Big Data
Slide 5
Global Landscape and Trends
Slide 16
The Big Data Opportunity
Slide 23
Forecasted market size
Future outlook
21. Global Big Data market to reach ~USD 25 billion by
2015,with a 45% share of IT & IT-enabled services
The global Big Data market is expected to grow by about a CAGR of 46% over 2012-2015
IT & ITES, including analytics, is expected to grow the fastest, at a rate of more than 60%
– Its share in the total Big Data market is expected to increase to ~45% in 2015 from ~31% in 2011
The USD 25 billion opportunity represents the initial wave of the opportunity. This opportunity is set to expand
even more rapidly after 2015 given the pace at which data is being generated.
Global Big Data Market Size, 2011 – 2015E Global Big Data Market Size, 2015F
US$ billion
~US$25 billion
25.0-26.0 Opportunity for India
lies in capturing the
Big Data analytics & slice of IT services that
US$ 10-11
IT & IT-enabled includes Big Data
billion
services analytics and IT & IT-
enabled services
US$ 7-7.5 Lion’s share of the Big
8.0-8.5 Software billion Data hardware and
software market is
5.3-5.6 expected to be
occupied by IT giants
US$ 6-6.5 like
Hardware billion IBM, HP, Microsoft, SA
P, SAS, Oracle, etc.
2011E 2012E 2015F
2015
Source: Industry reporting; CRISIL GR&A analysis
21
22. Conclusion
Big Data market opportunity is expected to witness strong growth in the next 5 years
– Expected to touch US$25 billion globally; the ‘BIG’ opportunity for India lies in the IT & IT-enabled
Services space, which is likely to be ~US$ 10-11 billion market globally in 2015
– Data-related regulations like Dodd-Frank and Basel III to impact Big Data implementations
New database architectures and innovative analytics tools & techniques to facilitate
Big Data implementations
Key verticals driving demand for Big Data analytics: Financial services, Retail,
Telecom, Healthcare and Manufacturing
Key risk – potential shortfall of 1.5 million Data-Savvy Managers and 140,000-190,000
Data Scientists in the US by 2018
Source: CRISIL GR&A analysis
22
25. India’s ‘BIG’ opportunity is in IT and
IT-enabled services
India Big Data outsourcing opportunity, 2011 – 2015E India Big Data outsourcing opportunity, by
US$ billions category, 2015F, Percent
100%= ~US$1.1 billion
1.1-1.2
24%-27% Pure-play Analytics
firms
Integrated IT/ BPO
players
~0.2
~0.1
73%-76%
2011E 2012E 2015F
Source: CRISIL GR&A analysis Source: CRISIL GR&A analysis
India’s Big Data market is expected to grow at a 83% CAGR over 2011-2015 to reach ~US$ 1.1-1.2 billion
India’s share in the ~USD 10-11 billion global Big data IT and IT-enabled services market is expected to
be ~10% in 2015 , where:
– In 2015, integrated IT and BPO players will dominate the US$1.1 billion opportunity with close to 73-76%
Source: Industry reporting; CRISIL GR&A analysis
25
26. Key Players Across the Traditional and Big Data
Technology Stack
Key players in BI/Traditional Analytics vs. Big Data Analytics technology stack
Big Data Analytics
BI/Traditional Analytics
Big Data Use
E-commerce, Search , Social gaming
Traditional BI suites and OLAP
End-user
applications
IT Services – Data Management
Big Data Analytics Basic visualization apps. Advanced visualization apps. MapReduce Programs
Visualization
tools
Traditional Analytics Advanced Analytics
Analytical tools
Parallel Relational Hadoop
Big Data RDBMS NoSQL Databases Database
SAP HANA
Data
management
systems
Conventional
file systems HDFS
Infrastructure &
storage systems Monolithic Hardware Distributed Hardware
Note: This is a representative list of players
Source: Industry reporting; CRISIL GR&A analysis
26
27. Financial Services and Telecom to be the early
adopters of the Big Data
Indian service providers like Infosys, Fractal are enabling Big Data analytics in the area of fraud detection, CRM
Financial and customer loyalty program, trading pattern analysis, risk calculation on large portfolio of loans
Services Key Adopters: JPMorgan Chase, Merrill Lynch, HSBC, American Express, Goldman
Sachs, Barclays, Bank of America, Citigroup, and Wells Fargo
Telecom players are increasingly focusing on Big Data to limit churn rates, build loyalty and provide multi-
Telecom channel and multi-service applications by proactively analyzing the subscriber and network data
Key Adopters: Airtel , Vodafone
Both brick and mortar as well as online retailers are increasing their adoption of Big Data analytics for real time
Retail analysis of purchase behavior and buying patterns, enhanced customer segmentation and customer loyalty
Key Adopters: Walmart & Sears
Indian service providers are enabling manufacturing companies through Big Data analytics in the areas of
Manufacturing accurate demand forecasting, optimization of operations, inventory management, open innovation and better
analysis of post sales feedback in real time
Key benefits of big data in public sector include: Intelligence to counter national threats, Forecast economic
Public Sector events, Traffic management, Environment monitoring, energy/ waste management, etc.
Healthcare players use Big Data Next-generation sequencing and mapping for genomics, analysis of correlation
Healthcare between treatments & outcomes and real time data from medical devices for better patient care
Source: Industry reporting; CRISIL GR&A analysis
27
Hinweis der Redaktion
Companies worldwide are turning their attention to Big Data as they scramble to derive insights from the deluge of information generated from various sources. In the past few years, the global marketplace has seen exponential growth in data volumes, created and consumed by a diverse cross-section of stakeholders. The term “Big Data” signifies large data sets in multiple formats, growing at an enormous rate and posing problems for traditional storage and analytical platforms. Big Data is distinct from large existing data stored in various relational databases, as it warrants a more advanced mechanism for both storage and analysis. Technologies such as NoSQL databases and MapReduce/Hadoop frameworks are at the core of the solutions heralding a paradigm shift. So Big Data is characterized by three attributes of data: volume, variety and the velocity at which it is generated.Traditional analytics on transactional or structured data have helped data-driven organizations gain insights from various enterprise data. As data from weblogs, social media posts, sensors, images, e-mails, audio and video files emerge as sources of insights, it presents a huge competitive opportunity for businesses. The need to derive predictive and actionable insights from this data for improved business operations and better decision-making is what drives Big Data analytics.
Data volume is the primary characteristic of Big Data. With data becoming an indispensable part of every economy, industry, organization, business function and individual, it is being actively captured by companies to better understand their customers, suppliers, partners and operations. Large data sets yield more information and hence improved analysis compared to limited records of data, leading to better competitive advantage and business operations. This data is being generated at a rapid pace: around 2.5 billion GB of data is generated every day, and more than 90% of the data available today has been created in the past 3-4 years. According to IDC, data generated globally is expected to witness a 41.0% CAGR between 2009 and 2020 to reach 35.0 zetabytes. Moreover, the technological landscape has changed with innovation in both managing and storing large data. As companies move away from the traditional data storage systems such as file systems and databases to newer technologies such as cloud-based storage and open source software, data storage and management costs are seeing a downward trend. According to IDC, storage costs have plummeted from US$18.9/gigabyte in 2005 to US$1.6/gigabyte in 2011, and are expected to further decline to 0.7/gigabyte by 2015. Apart from storage costs, the evolution of several open-source analytical tools and platforms has made data analytics flexible, reliable and relatively affordable for Big Data.
Organizations worldwide are increasingly realizing that unstructured data, if analyzed, can provide a competitive edge. While structured data is transactional and can be stored in rows and columns with an identifiable structure, unstructured data such as audio, video, and social media messages is raw or semi-structured. This data is generated in several forms such as web clicks, e-mails, phone conversations, weather data, audio and video files, location co-ordinates and pictures. Moreover, unstructured data is highly dynamic and does not have a particular format, i.e., it may be in different languages, have several terminologies, and may exist in the form of x-ray sheets, voice mails, digital photographs, or phone conversations.Companies are overwhelmed by the volume of unstructured data and are looking at ways to manage and analyze them in a systematic manner. As a result, one of the key focus areas for companies wanting to leverage Big Data is to handle unstructured data and adopt new technologies to deal with them.It is imperative to develop technologies that can enable storage of such huge data as well as maintain transactional consistency between structured and unstructured data. Newer technologies such as NoSQL databases to store unstructured data and processing methods such as Hadoop and massively parallel processing are gaining prominence in the area of Big Data and Big Data analytics.
The proliferation of the Internet and the mobile era has increased the rate at which data is created and stored; hence, there is a need for tools and technologies to analyze data at an equal speed. The shelf life of data has dropped from months to hours and seconds. The ubiquitous nature of the Internet, coupled with massive computing power and accessibility, has transformed data processing from an auxiliary function into an essential mechanism that enables organizations to transform their businesses. Big Data service providers are increasingly leveraging technologies such as streaming processing and in-memory computing that mitigates the shortcomings of batch processing and enable faster storage and data processing.Earlier, these technologies were popular in verticals considered more critical, such as the financial and government sectors. However, as the criticality of analyzing data in real time emerges, several other industries are also adopting solutions based on these technologies.
Big Data analytics is an evolving and multifaceted area for analytics players. The key differentiating factors between traditional analytics, advanced analytics and Big Data analytics are:Big Data analytics differs from advanced analytics in terms of different data formats and structures, and new application requirements for Big Data.While traditional analytics performs rear-view analysis on structured data, advanced analytics and Big Data analytics provide a progressive view, enabling organizations to anticipate and deal with future opportunities i.e. Big Data analytics has a definitive predictive end-result in its use.Big Data analytics has enabled cross-channel analytics and real-time insights at greater speed, access and collaboration. For example, detection of consumer emotions on a call on mentioning a competitor or conversion of a service call into an opportunity by leveraging Big Data analytics are more relevant in real time rather than after the interaction ends.
The Big Data ecosystem includes multiple elements from the data that is analyzed using the IT infrastructure that supports it and the applications that enable its analysis and usage. Elements of Big Data include:Data Management refers to systems where the data resides. It comprises the legacy systems as well as Hadoop-based systems and NoSQL databases. Legacy systems include databases that store and manage structured data, i.e., RDBMS to store and analyze structured data, and MPP systems to scale up for large structured data sets. Hadoop is an open-source software framework to support applications that enable analysis of petabyte- and xetabyte-sized data. Given Hadoop’s popularity and wide adoption, several other open-source projects have become associated with it, adding new functionality and enterprise-ready features to make it a compelling enterprise solution. These sub-projects include Hadoop Distributed File System (HDFS), Hbase, Hive, Mahout, Pig, ZooKeeper, Avro, Cassandra, and Chukwa. Once Big Data is collected and processed, it becomes operational data, i.e., it represents Big Data outcomes or serves as an input data for analytics. Big Data Analytics includes the technologies and tools to analyze the operational data and generate insight from it. After the data is analyzed, it becomes available for business users through various visualization techniques.Data Consumption involves enabling the Big Data insights to work in Business Intelligence (BI) and end-user applications IT Services enable integration of Big Data framework with the traditional business intelligence infrastructure.
North America and Europe, the two major data hubs of the world, account for a substantial portion of the global demand potential for Big Data analytics. Big Data service providers and leading IT players have significantly ramped up their capabilities in these developed regions that embraced the concept of Big Data, particularly in data-intensive industries such as digital media, manufacturing, healthcare, retail and financial services. While North America and Europe are poised to drive the growth of Big Data for the next two-three years, developing economies such as India and China are expected to catch up soon riding high on the rapid expansion of multimedia content, increasing popularity of social media, and proliferation of mobile devices. Further, while developed economies are likely to continue to be the major Big Data contributors in terms of revenue opportunity, emerging economies, particularly India, are all set to emerge as the preferred Big Data analytics and associated IT service providers.
Tools and technologies required to manage and analyze Big Data present a growth opportunity for startups to innovate and come up with new products. New companies across the Big Data technology stack have been thriving on the back of some robust investments anticipated in the Big Data space. The centerpiece of Big Data technology innovation, the Hadoop distribution, has been put to commercial use by many startups such as Cloudera, HortonWorks, Zettaset, and MaPR, with some customization of the open source software. Furthermore, the business environment is witnessing a slew of startups in the non-Hadoop systems such as NoSQL, Next Generation (MPP) Data Warehousing like CouchBase, Splunk, and VoltDB. The industry also has many startups emerging in the analytics platforms and cloud-based applications as well as in the advanced data visualization space. While the past two-three years have mainly seen new companies coming up in the data management space, analytics applications is the impetus for growth in the next few years. Some of the startups in this field include Karmasphere, Kognitio, 1010Data, Revolution Analytics, and QlikView.The Big Data technology space is witnessing a lot of venture capital activity, with funding in Big Data startups reaching ~USD 2.5 billion in 2011, compared with ~USD 1.5 billion in 2010. These startups are innovation hubs that are gaining importance across industry verticals. Most of these companies are witnessing high-double-digit revenue growth driven by the huge demand for their solutions. Moreover, many startups are being acquired by larger IT players given the growth opportunities and the need to build Big Data capabilities. For instance, IBM has acquired Tealeaf Technologies, Vivisimo and Varicent; Teradata acquired eCircle, and EMC2 acquired Greenplum.
The Big Data space is witnessing a string of M&A driven by the need to quickly ramp up capabilities and also to have a complete set of capabilities to service clients who are keen to have Big Data implementation. Leading technology players such as Oracle, IBM, SAP, and EMC are aggressively acquiring smaller independent software vendors (ISVs) and data analytics firms to strengthen their Big Data portfolio. IBM is in the forefront of this phenomenon through multiple acquisitions over 2010-12 in the Big Data space. It acquired Vivisimo and TeaLeaf Technology in 2012, i2 Limited in 2011 and Coremetrics and Netezza Corporation in 2010, for bolstering its Big Data capabilities. Further, HP acquired Autonomy for more than USD 10 billion, making it the largest deal in the Big Data industry. HP aims to cater to the Big Data market by leveraging Autonomy’s pattern-matching technology that recognizes and processes Big Data.
Financial services is considered to be a very data-intensive sector, with more data per million of revenue/operating expenditure or per employee, than almost all other sectors. Within the sector, structured and unstructured data is available from a variety of sources such as customer and transaction data from various channels such as branch, kiosks, mobile and web; social media; emails; credit cards data; insurance claims data; stock market data; statistical data, PDF & excel files, news, videos, and government filings. With the industry facing a multitude of challenges such as higher customer expectations, uncertain operating environment, strict regulations, stiff competition, and slowing economic growth, Big Data analytics can help banks, capital markets and insurance companies by providing tools to reduce costs and improve productivity. Increasing regulatory compliances and the need for collecting every piece of data and standardizing them is driving the growth of Big Data analytics. Several areas within the financial services sector are expected to gain from Big Data technologies. They include:Banking:Credit reward program analysis: Banks are increasingly using unstructured data to understand customer profile and introduce successfulcredit cards with innovative rewards programFor E.g. A national bank used a Big Data solution to analyze data from sources such as call centers, customer service emails, and social media conversations to create a credit card offering with a rewards program to attract a young, professional demographic. This helped in providing information to the marketing department to create a targeted promotion campaign, including strategically placed social messaging and monitoring.Capital Markets:Trading surveillance: The financial sector leverages Big Data to monitor trading activities and identify abnormal trading patterns. In surveillance, Big Data analytics allow on-line access to trade-by-trade history for investigation, trending, and discovery to be combined with real-time data to provide a real-time and historical context to behavior.For E.g. Companies combine data about the parties that participate in a trade with the complex data that describes relationships among those parties and how they interact with one another. The combination allows the bank to recognize unusual trading activity and to flag it for review.Insurance:Insurance companies are increasingly using unstructured data to predict client longevity, along with examining the prospective client’s medical status by analyzing their general comments, visits to particular websites, and enquiry about some specific products.Using weather and calamity information for managing claims exposures and losses based on unstructured data from weather measurements, and soil observations.E.g. An insurance company sells Total Weather Insurance, which pays local farmers when they are impacted by weather events that affect their profits. The company uses a cloud-driven Big Data analytics service to predict the possibility of extreme weather, along with the potential impact. It prices its insurance policies accordingly, based on 2.5 million daily weather measurements, 150 billion soil observations, and 10 trillion scenario data points to build and price their products.Big Data is being extensively used across all domains of the financial services for risk management, fraud detection, compliance and customer relationship management:Risk management: Predictive modeling of customer behavior and scoring techniques enable financial sector companies to access and minimize default risks at an individual level and make customized offerings, in line with the customer’s risk profile. E.g. A large bank wanted to use 12 years of monthly account-level credit card data, credit bureau information and bank account information to better assess the risk before granting loans or raising credit limits. Ideally, it wanted this information in real time. To speed the computing, it used an in-database Big Data approach, which helped the bank to calculate risk 70 times faster.Fraud detection: Big Data technologies give financial service companies the ability to run exploratory modeling and discovery on data, thereby increasing the accuracy of fraud detection models. The faster processing capability enables companies to quickly build or refresh fraud detection models, and also helps in detecting fraud in real time by analyzing and streaming transaction data.Compliance and regulatory reporting: Increased oversight and scrutiny of the companies’ operations, funding and investment portfolio has led financial services companies to adopt sophisticated Big Data technologies to store and process vast amount of data to simplify and streamline their regulatory and compliance reporting.For E.g. Reserve Bank of India (RBI) has directed all Indian banks to standardize their regulatory reporting by following an automated data flow (ADF) approach – to ensure 100% accuracy and zero human intervention in every stage of reporting: right from data extraction from source systems to the actual submission of returns. Firms that could not utilize complete information and firms that believed reporting did not really require management attention are increasingly focusing on Big Data analytics.Customer relation management: Big Data analytics also help financial service companies in acquiring new customers and cross-selling their offerings to existing customers by using Big Data to identify the most profitable customers and run effective marketing campaigns. The large volume of unstructured data from social media is combined with the CRM systems to study customer behavior and optimize customer experience. Apart from customer acquisition, companies can improve customer retention by using predictive analytics to detect early signs of disengagement.Financial service companies are gaining business advantage by mining and analyzing Big Data to stay ahead of the competition, improve customer service, detect fraud, accurately calculate risks and maximize operational efficiencies, along with adhering to stringent regulations and compliances.Indian service providers are enabling Big Data analytics in the area of fraud detection, client behavior analysis, trading pattern analysis, risk calculation on large portfolio of loans, and improved and targeted marketing campaigns. Further, Indian financial sector companies are increasingly favoring Big Data analytics to tackle terabytes of unstructured data:YES Bank is finding out solutions to handle the increasing pile of unstructured data from mobile devices and social media networks, customer transaction starting from withdrawal of money from bank, and ATM. The bank feels the regulatory requirement of storing internally generated data is driving banks to adopt Big Data.
The Big Data phenomenon has led to an increasing demand for ‘data scientists’ – professionals conversant with both the business context and data analytics – who play a crucial role in extracting insights from large data sets, analyzing these and then presenting the value-added information to business users or non-data experts. Big Data needs a new breed of professionals with a deep expertise in statistics and machine learning, as well as managers and analysts who can leverage insights for Big Data. The shortage of such talent is a significant challenge that companies need to address for successful Big Data implementation. According to McKinsey, the US alone faces a shortage of 140,000-190,000 analysts and 1.5 million managers who can analyze Big Data. To address the shortage, companies have embarked on initiatives to train their existing employees and develop new talent. Companies such as EMC2, Oracle, and IBM are partnering with universities to offer courses on various elements of Big Data. Internally, enterprises are creating organizational cultures that are favorable for data-driven decisions by hiring employees from academic fields such as statistics, and mathematics, as well as through on-the-job training on emerging technologies in the Big Data space.
As enterprises undertake pilots for Big Data implementation and large IT companies and startups compete for market share, the global Big Data market is expected to grow by about 46% to more than USD 25 billion by 2015. The IT & IT-enabled services, including analytics, are expected to grow the fastest, at a rate of more than 60%), with their share in the total Big Data market expected to increase to ~45% in 2015 from ~31% in 2011. Big Data analytics is likely to be driven by the near-ubiquitous nature of the data and proliferation of technologies and applications such as mobile sensors, smart phones and social networking, along with the growing realization of the benefits of Big Data by enterprises. While Big Data could add momentous value in the coming years, it might have to overcome certain roadblocks. Though early movers are formulating Big Data strategies, mass adoption may be hindered by the lack of best practices and the significant cultural change organizations require for sharing data. However, as companies leverage large datasets from within and outside, Big Data is likely to continue to grow as an area which can deliver substantial benefits. Finally, the aggressive efforts of service providers – both large IT companies and niche startups – to demonstrate their domain expertise and ability to derive valuable insights from Big Data would be an enabler to this opportunity.
India’s Big Data outsourcing opportunity is likely to grow by about 83% annually to ~US$1.0 billion during 2011-15. India is expected to be the preferred destination for analytics and IT services for Big Data due to its pre-eminence in IT/BPO services, knowledge services outsourcing and analytics as well as for its intellectual pool of talent. The share of analytics in the overall Big Data opportunity is expected to rise from ~16% in 2011 to 25% in 2015. The key drivers for India include the efforts of service providers to develop talent and increase their domain expertise and breadth of services. Moreover, a number of Indian service providers are leveraging partnerships with Big Data technology players to facilitate delivery of Big Data solutions. Finally, while the current demand for Big Data analytics is generated from global clients, domestic demand in India is also gaining traction. For example, Asian Paints and Star India have leveraged Big Data analytics to track and analyse large datasets.
India’s Big Data outsourcing opportunity is likely to grow by about 83% annually to ~US$1.0 billion during 2011-15. India is expected to be the preferred destination for analytics and IT services for Big Data due to its pre-eminence in IT/BPO services, knowledge services outsourcing and analytics as well as for its intellectual pool of talent. The share of analytics in the overall Big Data opportunity is expected to rise from ~16% in 2011 to 25% in 2015. The key drivers for India include the efforts of service providers to develop talent and increase their domain expertise and breadth of services. Moreover, a number of Indian service providers are leveraging partnerships with Big Data technology players to facilitate delivery of Big Data solutions. Finally, while the current demand for Big Data analytics is generated from global clients, domestic demand in India is also gaining traction. For example, Asian Paints and Star India have leveraged Big Data analytics to track and analyse large datasets.
As Big Data technologies become mainstream, the vendor landscape is evolving rapidly. Data management includes vendors of Hadoop-based solutions, other MapReduce technology suppliers as well as cloud and data center providers. The increased demand for Big Data analytics has changed the competitive landscape for the Big Data analytics service providers. In addition to the incumbent IT/BPO/knowledge service players, there are now more pure-play analytics players, some of whom provide sector-specific analytics solutions. Some of the larger organizations have set up captives, which provide data analytics solutions to the other divisions and subsidiaries of those organizations. Even the breadth of the services provided by analytics companies has substantially increased from data storage and management to delivering real-time insights and end-to-end data analytics services.Big Datamanagement and storage: Many new companies have emerged as providers of Apache open source Hadoop distributions, with various levels of proprietary customization for data management. Cloudera and Hortonworks are the major players for Hadoop distributions. While Cloudera contributes significantly to Apache HBase, the Hadoop-based non-relational database that enables low-latency, Hortonworks mainly offers next-generation MapReduce architecture. Other pure players include MapR, Hadapt, and Zettaset. Moreover, mega IT vendors have also entered the Big Data market through acquisitions. The Big Data warehouse market is mainly led by four players – IBM Netezza, EMC2Greenplum, HP Vertica and Teradata Aster Data. Non-Hadoop vendors are also significantly contributing to the Big Data market opportunity – Splunk, HPCC Systems, and Datastax are some of the key players.Big Data analytics: With the deluge of data, it has become pertinent to have applications and platforms that leverage the underlying Hadoop infrastructure for data analytics. Some of the key players in this segment are: Karmasphere, which offers an analytical development platform to perform ad-hoc queries on Hadoop-based data via an SQL interface; Datameer, which provides a Hadoop-based business intelligence platform that leverages a spreadsheet-like interface to analyze data; and service providers such as QlikView, Revolution Analytics, Informatica, 1010data, and ClickFox which offer cloud-based Big Data applications and services. Big Data use: Big Data analytics engage with large data sets which may be difficult to understand for business users. A number of companies such as Amazon Web Services, Google, and Intellicus are launching new user applications which facilitate the usage of Big Data analytics.Additionally, the landscape for Big Data IT services is growing exponentially, with established service providers such as Oracle, IBM, and CSC building their Big Data service portfolio. Moreover, Indian IT/BPO players such as TCS, Infosys, and Wipro are also bolstering their capabilities in Big-Data-specific software development and implementation.