SlideShare ist ein Scribd-Unternehmen logo
1 von 24
© 2014 Impetus Technologies1
July 25, 2014
Accelerating the Big Data Solution
Lifecycle and Improving ROI
© 2014 Impetus Technologies2
Agenda
Big Data
Analytics:
Implementation
patterns
Challenges
faced
Jumbune –
an open source
lifecycle
accelerator
Enterprise
solution
lifecycle
Ways to
address the
challenges
Recorded version available at http://bit.ly/1nMw8nQ
© 2014 Impetus Technologies3
Big Data Analytics
Primary drive
for performing
analytics
Rise of the
enterprise
data lake
Utilization of
analytical
resources
Recorded version available at http://bit.ly/1nMw8nQ
© 2014 Impetus Technologies4
Primary Purposes of an Analytical Solution
Optimize the
business
Reduce time
taken by analytics
Result in effective
analytics
Compete and
win
Recorded version available at http://bit.ly/1nMw8nQ
© 2014 Impetus Technologies5
Rise of the Enterprise Data Lake
BIG DATA
Sources of Data: ETL from every
source - RDBMS, flat files, queues,
legacy off loading, logs
Arrival of Data: Intermittent, bulk,
incremental
Theme : “Leave no Data unused”
Recorded version available at http://bit.ly/1nMw8nQ
© 2014 Impetus Technologies6
Utilization of Analytics Resources
• Capitalize on all analytics resources (engines) available
• Access data with a variety of processing engines – Storm, Spark, Yarn
etc.
• Model in data science analytical systems – R, Octave, SAS, etc.
• Write complex logic in custom MapReduce
• Reuse code as User Defined Functions (UDFs)
• Create ad hoc queries using Hive and PIG
• Customization of Mahout algorithms, machine learning libraries
Recorded version available at http://bit.ly/1nMw8nQ
© 2014 Impetus Technologies7
Enterprise Big Data Solution Trends
• No more single purpose Hadoop clusters
• Enterprise Data Lake: Data flowing from many sources
• Integrated platforms using variety of analytical engines
• Serving multiple business applications
• Resource sharing is a must across applications and engines
Recorded version available at http://bit.ly/1nMw8nQ
© 2014 Impetus Technologies8
Enterprise Solution Lifecycle (High level view)
Business
Requirement
Designing /
Modelling
Development
and Testing
Production and
Monitoring
Recorded version available at http://bit.ly/1nMw8nQ
© 2014 Impetus Technologies9
Enterprise Solution Lifecycle (Ground level
view)
xxx
xxx
Business User Data Analyst Development
Quality Test
DevOpsData Lake
Production and
Monitoring
Recorded version available at http://bit.ly/1nMw8nQ
© 2014 Impetus Technologies10
Challenges in Enterprise Analytical Solutions
No common
platform to detect
root causes
Incremental
imports may ingest
bad data
Cluster resources are
shared and optimal
utilization is the key
Implementing
models in custom
MR without errors
is like hitting the
bull’s eye
Bad logic or bad
data
Recorded version available at http://bit.ly/1nMw8nQ
© 2014 Impetus Technologies11
Scenario: Digitization of Newspaper for
Analyzing News
xxx
xxx
Team: 5 Dev, 3 QA, 2 DevOps
Simple Problem: ‘q’ was misread by OCR as 9
TIME
• A single code fault on TB of data can consume 24 work hours total
for 2 Developers + 1 QA
COST
• Additional hours by engineers + The cost of unproductive cloud
instances, storage and resources
Recorded version available at http://bit.ly/1nMw8nQ
© 2014 Impetus Technologies12
Scenario: Hive Queries Interpreted as
MapReduce Executions on a Hadoop Cluster
xxx
xxx
Team: 2 Dev, 1 QA, 1 DevOps
Simple Problem: Data imbalance across cluster, low performance by Hive queries.
TIME
• Development team were refactoring Hive queries for improving the
performance
COST
• Additional hours by engineers + The cost of unproductive cloud
instances, storage and resources
Recorded version available at http://bit.ly/1nMw8nQ
© 2014 Impetus Technologies13
Impact on ROI
Delayed Analytics Increase in CostsProductivity Loss
Defeats one of the
prime purpose of
analytics
Defeats the purpose
of business cost
optimization
Iterations reduce the
productivity of
dependent teams in
the cycle
Recorded version available at http://bit.ly/1nMw8nQ
© 2014 Impetus Technologies14
Current Iterative Development Approach
Local
Debug/ Unit
Tests
HDFS Data Check
Performance
• Localized subset
of data
• Non parallel
execution
• Practically
unfeasible
• Error prone
• Difficult to find
bad code
• Difficult to
collaborate
across
environments
Recorded version available at http://bit.ly/1nMw8nQ
© 2014 Impetus Technologies15
A Complete Enterprise Platform
Data Lake
Enterprise Engines
Solutions
Governance
Security
Validate,Profile,DebugandMonitor
Recorded version available at http://bit.ly/1nMw8nQ
© 2014 Impetus Technologies16
Introducing Jumbune: An Open Source
Solution
“A catalyst to accelerate realization of Big Data Analytics
solutions”
Flow AnalyzerData Validation Cluster Monitor Job Profiler
Recorded version available at http://bit.ly/1nMw8nQ
© 2014 Impetus Technologies17
© 2014 Impetus Technologies18
© 2014 Impetus Technologies19
© 2014 Impetus Technologies20
© 2014 Impetus Technologies21
Full Lifecycle Support - Jumbune
xxx
xxx
Development Quality
DevOpsData Ingestion
Recorded version available at http://bit.ly/1nMw8nQ
© 2014 Impetus Technologies22
Jumbune - Key Features
• In depth code level analysis of cluster wide flow
• Record and field level data violation reports
• No deployment on worker nodes - Ultra light agent installation on the gateway node
• Ability to turn on/off cluster monitoring at will – reduces resource load
• Customizable rack aware monitoring
• Correlated profiling analysis of phases, throughput and resource consumption
• Ability to work with all Hadoop distributions
• Coming up support for Yarn, Spark, Mesos
• Available as Open Source
Recorded version available at http://bit.ly/1nMw8nQ
© 2014 Impetus Technologies23
2
3
For general inquiries about other Impetus solutions and services
reach us at bigdata@impetus.com
Recorded version available at http://bit.ly/1nMw8nQ
© 2014 Impetus Technologies24
Thank You!
Website
• http://jumbune.org
Contribute
• http://github.com/impetus-opensource/jumbune
• http://jumbune.org/jira/JUM
Social
• Follow @jumbune Use #jumbune
• Jumbune Group: http://linkd.in/1mUmcYm
Forums
• Users: users-subscribe@collaborate.jumbune.org
• Dev: dev-subscribe@collaborate.jumbune.org
• Issues: issues-subscribe@collaborate.jumbune.org
Downloads
• http://jumbune.org
• https://bintray.com/jumbune/downloads/jumbune
Recorded version available at http://bit.ly/1nMw8nQ

Weitere ähnliche Inhalte

Ähnlich wie Accelerating Hadoop Solution Lifecycle and Improving ROI- Impetus On-demand Webcast

Real-world Applications of Streaming Analytics- StreamAnalytix Webinar
Real-world Applications of Streaming Analytics- StreamAnalytix WebinarReal-world Applications of Streaming Analytics- StreamAnalytix Webinar
Real-world Applications of Streaming Analytics- StreamAnalytix WebinarImpetus Technologies
 
Performance Testing of Large-scale Systems- Impetus Webinar
Performance Testing of Large-scale Systems- Impetus WebinarPerformance Testing of Large-scale Systems- Impetus Webinar
Performance Testing of Large-scale Systems- Impetus WebinarImpetus Technologies
 
Enterprise CI as-a-Service using Jenkins
Enterprise CI as-a-Service using JenkinsEnterprise CI as-a-Service using Jenkins
Enterprise CI as-a-Service using JenkinsCollabNet
 
Il paradigma DevOps e Continuous Delivery Automation
Il paradigma DevOps e Continuous Delivery Automation Il paradigma DevOps e Continuous Delivery Automation
Il paradigma DevOps e Continuous Delivery Automation HP Enterprise Italia
 
Comment déployer une stratégie microsoft en mode appliance
Comment déployer une stratégie microsoft en mode applianceComment déployer une stratégie microsoft en mode appliance
Comment déployer une stratégie microsoft en mode applianceMicrosoft Ideas
 
Smarter z/OS Software Delivery using Rational Enterprise Cloud Solutions
Smarter z/OS Software Delivery using Rational Enterprise Cloud SolutionsSmarter z/OS Software Delivery using Rational Enterprise Cloud Solutions
Smarter z/OS Software Delivery using Rational Enterprise Cloud SolutionsJean-Yves Rigolet
 
Accelerating SDLC for Large Public Sector Enterprise Applications
Accelerating SDLC for Large Public Sector Enterprise ApplicationsAccelerating SDLC for Large Public Sector Enterprise Applications
Accelerating SDLC for Large Public Sector Enterprise ApplicationsSplunk
 
IoT Scale Event-Stream Processing for Connected Fleet at Penske
IoT Scale Event-Stream Processing for Connected Fleet at PenskeIoT Scale Event-Stream Processing for Connected Fleet at Penske
IoT Scale Event-Stream Processing for Connected Fleet at PenskeVMware Tanzu
 
A Reference Architecture to Enable Visibility and Traceability across the Ent...
A Reference Architecture to Enable Visibility and Traceability across the Ent...A Reference Architecture to Enable Visibility and Traceability across the Ent...
A Reference Architecture to Enable Visibility and Traceability across the Ent...CollabNet
 
The Path to a Pain-Free Control System Upgrade
The Path to a Pain-Free Control System UpgradeThe Path to a Pain-Free Control System Upgrade
The Path to a Pain-Free Control System UpgradeInductive Automation
 
Does Big Data Spell Big Costs- Impetus Webinar
Does Big Data Spell Big Costs- Impetus WebinarDoes Big Data Spell Big Costs- Impetus Webinar
Does Big Data Spell Big Costs- Impetus WebinarImpetus Technologies
 
Pure App + Patterns + Prolifics = Feeding Change
Pure App + Patterns + Prolifics = Feeding Change Pure App + Patterns + Prolifics = Feeding Change
Pure App + Patterns + Prolifics = Feeding Change Prolifics
 
Top 5 .NET Challenges, Performance Monitoring Tips & Tricks
Top 5 .NET Challenges, Performance Monitoring Tips & TricksTop 5 .NET Challenges, Performance Monitoring Tips & Tricks
Top 5 .NET Challenges, Performance Monitoring Tips & TricksAppDynamics
 
Deployment Automation for Hybrid Cloud and Multi-Platform Environments
Deployment Automation for Hybrid Cloud and Multi-Platform EnvironmentsDeployment Automation for Hybrid Cloud and Multi-Platform Environments
Deployment Automation for Hybrid Cloud and Multi-Platform EnvironmentsIBM UrbanCode Products
 
Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines
Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data PipelinesPutting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines
Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data PipelinesDATAVERSITY
 
Planning Your Hadoop NoSQL Projects For 2011
Planning Your Hadoop NoSQL Projects For 2011Planning Your Hadoop NoSQL Projects For 2011
Planning Your Hadoop NoSQL Projects For 2011Impetus Technologies
 
Enterprise DevOps Transformation
Enterprise DevOps TransformationEnterprise DevOps Transformation
Enterprise DevOps TransformationBart Driscoll
 
Spring Mainframe VUG 2015: How to google your way through your mainframe appl...
Spring Mainframe VUG 2015: How to google your way through your mainframe appl...Spring Mainframe VUG 2015: How to google your way through your mainframe appl...
Spring Mainframe VUG 2015: How to google your way through your mainframe appl...Serena Software
 

Ähnlich wie Accelerating Hadoop Solution Lifecycle and Improving ROI- Impetus On-demand Webcast (20)

Real-world Applications of Streaming Analytics- StreamAnalytix Webinar
Real-world Applications of Streaming Analytics- StreamAnalytix WebinarReal-world Applications of Streaming Analytics- StreamAnalytix Webinar
Real-world Applications of Streaming Analytics- StreamAnalytix Webinar
 
Performance Testing of Large-scale Systems- Impetus Webinar
Performance Testing of Large-scale Systems- Impetus WebinarPerformance Testing of Large-scale Systems- Impetus Webinar
Performance Testing of Large-scale Systems- Impetus Webinar
 
Adopting DevOps for 2-Speed IT
Adopting DevOps for 2-Speed ITAdopting DevOps for 2-Speed IT
Adopting DevOps for 2-Speed IT
 
Enterprise CI as-a-Service using Jenkins
Enterprise CI as-a-Service using JenkinsEnterprise CI as-a-Service using Jenkins
Enterprise CI as-a-Service using Jenkins
 
Il paradigma DevOps e Continuous Delivery Automation
Il paradigma DevOps e Continuous Delivery Automation Il paradigma DevOps e Continuous Delivery Automation
Il paradigma DevOps e Continuous Delivery Automation
 
Comment déployer une stratégie microsoft en mode appliance
Comment déployer une stratégie microsoft en mode applianceComment déployer une stratégie microsoft en mode appliance
Comment déployer une stratégie microsoft en mode appliance
 
Smarter z/OS Software Delivery using Rational Enterprise Cloud Solutions
Smarter z/OS Software Delivery using Rational Enterprise Cloud SolutionsSmarter z/OS Software Delivery using Rational Enterprise Cloud Solutions
Smarter z/OS Software Delivery using Rational Enterprise Cloud Solutions
 
Accelerating SDLC for Large Public Sector Enterprise Applications
Accelerating SDLC for Large Public Sector Enterprise ApplicationsAccelerating SDLC for Large Public Sector Enterprise Applications
Accelerating SDLC for Large Public Sector Enterprise Applications
 
IoT Scale Event-Stream Processing for Connected Fleet at Penske
IoT Scale Event-Stream Processing for Connected Fleet at PenskeIoT Scale Event-Stream Processing for Connected Fleet at Penske
IoT Scale Event-Stream Processing for Connected Fleet at Penske
 
A Reference Architecture to Enable Visibility and Traceability across the Ent...
A Reference Architecture to Enable Visibility and Traceability across the Ent...A Reference Architecture to Enable Visibility and Traceability across the Ent...
A Reference Architecture to Enable Visibility and Traceability across the Ent...
 
The Path to a Pain-Free Control System Upgrade
The Path to a Pain-Free Control System UpgradeThe Path to a Pain-Free Control System Upgrade
The Path to a Pain-Free Control System Upgrade
 
Does Big Data Spell Big Costs- Impetus Webinar
Does Big Data Spell Big Costs- Impetus WebinarDoes Big Data Spell Big Costs- Impetus Webinar
Does Big Data Spell Big Costs- Impetus Webinar
 
Pure App + Patterns + Prolifics = Feeding Change
Pure App + Patterns + Prolifics = Feeding Change Pure App + Patterns + Prolifics = Feeding Change
Pure App + Patterns + Prolifics = Feeding Change
 
Test Automation in Agile
Test Automation in AgileTest Automation in Agile
Test Automation in Agile
 
Top 5 .NET Challenges, Performance Monitoring Tips & Tricks
Top 5 .NET Challenges, Performance Monitoring Tips & TricksTop 5 .NET Challenges, Performance Monitoring Tips & Tricks
Top 5 .NET Challenges, Performance Monitoring Tips & Tricks
 
Deployment Automation for Hybrid Cloud and Multi-Platform Environments
Deployment Automation for Hybrid Cloud and Multi-Platform EnvironmentsDeployment Automation for Hybrid Cloud and Multi-Platform Environments
Deployment Automation for Hybrid Cloud and Multi-Platform Environments
 
Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines
Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data PipelinesPutting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines
Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines
 
Planning Your Hadoop NoSQL Projects For 2011
Planning Your Hadoop NoSQL Projects For 2011Planning Your Hadoop NoSQL Projects For 2011
Planning Your Hadoop NoSQL Projects For 2011
 
Enterprise DevOps Transformation
Enterprise DevOps TransformationEnterprise DevOps Transformation
Enterprise DevOps Transformation
 
Spring Mainframe VUG 2015: How to google your way through your mainframe appl...
Spring Mainframe VUG 2015: How to google your way through your mainframe appl...Spring Mainframe VUG 2015: How to google your way through your mainframe appl...
Spring Mainframe VUG 2015: How to google your way through your mainframe appl...
 

Mehr von Impetus Technologies

Data Warehouse Modernization Webinar Series- Critical Trends, Implementation ...
Data Warehouse Modernization Webinar Series- Critical Trends, Implementation ...Data Warehouse Modernization Webinar Series- Critical Trends, Implementation ...
Data Warehouse Modernization Webinar Series- Critical Trends, Implementation ...Impetus Technologies
 
Future-Proof Your Streaming Analytics Architecture- StreamAnalytix Webinar
Future-Proof Your Streaming Analytics Architecture- StreamAnalytix WebinarFuture-Proof Your Streaming Analytics Architecture- StreamAnalytix Webinar
Future-Proof Your Streaming Analytics Architecture- StreamAnalytix WebinarImpetus Technologies
 
Building Real-time Streaming Apps in Minutes- Impetus Webinar
Building Real-time Streaming Apps in Minutes- Impetus WebinarBuilding Real-time Streaming Apps in Minutes- Impetus Webinar
Building Real-time Streaming Apps in Minutes- Impetus WebinarImpetus Technologies
 
Smart Enterprise Big Data Bus for the Modern Responsive Enterprise- StreamAna...
Smart Enterprise Big Data Bus for the Modern Responsive Enterprise- StreamAna...Smart Enterprise Big Data Bus for the Modern Responsive Enterprise- StreamAna...
Smart Enterprise Big Data Bus for the Modern Responsive Enterprise- StreamAna...Impetus Technologies
 
Impetus White Paper- Handling Data Corruption in Elasticsearch
Impetus White Paper- Handling  Data Corruption  in ElasticsearchImpetus White Paper- Handling  Data Corruption  in Elasticsearch
Impetus White Paper- Handling Data Corruption in ElasticsearchImpetus Technologies
 
Real-world Applications of Streaming Analytics- StreamAnalytix Webinar
Real-world Applications of Streaming Analytics- StreamAnalytix WebinarReal-world Applications of Streaming Analytics- StreamAnalytix Webinar
Real-world Applications of Streaming Analytics- StreamAnalytix WebinarImpetus Technologies
 
Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus...
Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus...Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus...
Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus...Impetus Technologies
 
Deep Learning: Evolution of ML from Statistical to Brain-like Computing- Data...
Deep Learning: Evolution of ML from Statistical to Brain-like Computing- Data...Deep Learning: Evolution of ML from Statistical to Brain-like Computing- Data...
Deep Learning: Evolution of ML from Statistical to Brain-like Computing- Data...Impetus Technologies
 
SPARK USE CASE- Distributed Reinforcement Learning for Electricity Market Bi...
SPARK USE CASE-  Distributed Reinforcement Learning for Electricity Market Bi...SPARK USE CASE-  Distributed Reinforcement Learning for Electricity Market Bi...
SPARK USE CASE- Distributed Reinforcement Learning for Electricity Market Bi...Impetus Technologies
 
Enterprise Ready Android and Manageability- Impetus Webcast
Enterprise Ready Android and Manageability- Impetus WebcastEnterprise Ready Android and Manageability- Impetus Webcast
Enterprise Ready Android and Manageability- Impetus WebcastImpetus Technologies
 
Real-time Streaming Analytics: Business Value, Use Cases and Architectural Co...
Real-time Streaming Analytics: Business Value, Use Cases and Architectural Co...Real-time Streaming Analytics: Business Value, Use Cases and Architectural Co...
Real-time Streaming Analytics: Business Value, Use Cases and Architectural Co...Impetus Technologies
 
Leveraging NoSQL Database Technology to Implement Real-time Data Architecture...
Leveraging NoSQL Database Technology to Implement Real-time Data Architecture...Leveraging NoSQL Database Technology to Implement Real-time Data Architecture...
Leveraging NoSQL Database Technology to Implement Real-time Data Architecture...Impetus Technologies
 
Maturity of Mobile Test Automation: Approaches and Future Trends- Impetus Web...
Maturity of Mobile Test Automation: Approaches and Future Trends- Impetus Web...Maturity of Mobile Test Automation: Approaches and Future Trends- Impetus Web...
Maturity of Mobile Test Automation: Approaches and Future Trends- Impetus Web...Impetus Technologies
 
Big Data Analytics with Storm, Spark and GraphLab
Big Data Analytics with Storm, Spark and GraphLabBig Data Analytics with Storm, Spark and GraphLab
Big Data Analytics with Storm, Spark and GraphLabImpetus Technologies
 
Webinar maturity of mobile test automation- approaches and future trends
Webinar  maturity of mobile test automation- approaches and future trendsWebinar  maturity of mobile test automation- approaches and future trends
Webinar maturity of mobile test automation- approaches and future trendsImpetus Technologies
 
Next generation analytics with yarn, spark and graph lab
Next generation analytics with yarn, spark and graph labNext generation analytics with yarn, spark and graph lab
Next generation analytics with yarn, spark and graph labImpetus Technologies
 
The Shared Elephant - Hadoop as a Shared Service for Multiple Departments – I...
The Shared Elephant - Hadoop as a Shared Service for Multiple Departments – I...The Shared Elephant - Hadoop as a Shared Service for Multiple Departments – I...
The Shared Elephant - Hadoop as a Shared Service for Multiple Departments – I...Impetus Technologies
 
Performance Testing of Big Data Applications - Impetus Webcast
Performance Testing of Big Data Applications - Impetus WebcastPerformance Testing of Big Data Applications - Impetus Webcast
Performance Testing of Big Data Applications - Impetus WebcastImpetus Technologies
 
Real-time Predictive Analytics in Manufacturing - Impetus Webinar
Real-time Predictive Analytics in Manufacturing - Impetus WebinarReal-time Predictive Analytics in Manufacturing - Impetus Webinar
Real-time Predictive Analytics in Manufacturing - Impetus WebinarImpetus Technologies
 
Webinar real-time predictive analytics in manufacturing
Webinar  real-time predictive analytics in manufacturingWebinar  real-time predictive analytics in manufacturing
Webinar real-time predictive analytics in manufacturingImpetus Technologies
 

Mehr von Impetus Technologies (20)

Data Warehouse Modernization Webinar Series- Critical Trends, Implementation ...
Data Warehouse Modernization Webinar Series- Critical Trends, Implementation ...Data Warehouse Modernization Webinar Series- Critical Trends, Implementation ...
Data Warehouse Modernization Webinar Series- Critical Trends, Implementation ...
 
Future-Proof Your Streaming Analytics Architecture- StreamAnalytix Webinar
Future-Proof Your Streaming Analytics Architecture- StreamAnalytix WebinarFuture-Proof Your Streaming Analytics Architecture- StreamAnalytix Webinar
Future-Proof Your Streaming Analytics Architecture- StreamAnalytix Webinar
 
Building Real-time Streaming Apps in Minutes- Impetus Webinar
Building Real-time Streaming Apps in Minutes- Impetus WebinarBuilding Real-time Streaming Apps in Minutes- Impetus Webinar
Building Real-time Streaming Apps in Minutes- Impetus Webinar
 
Smart Enterprise Big Data Bus for the Modern Responsive Enterprise- StreamAna...
Smart Enterprise Big Data Bus for the Modern Responsive Enterprise- StreamAna...Smart Enterprise Big Data Bus for the Modern Responsive Enterprise- StreamAna...
Smart Enterprise Big Data Bus for the Modern Responsive Enterprise- StreamAna...
 
Impetus White Paper- Handling Data Corruption in Elasticsearch
Impetus White Paper- Handling  Data Corruption  in ElasticsearchImpetus White Paper- Handling  Data Corruption  in Elasticsearch
Impetus White Paper- Handling Data Corruption in Elasticsearch
 
Real-world Applications of Streaming Analytics- StreamAnalytix Webinar
Real-world Applications of Streaming Analytics- StreamAnalytix WebinarReal-world Applications of Streaming Analytics- StreamAnalytix Webinar
Real-world Applications of Streaming Analytics- StreamAnalytix Webinar
 
Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus...
Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus...Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus...
Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus...
 
Deep Learning: Evolution of ML from Statistical to Brain-like Computing- Data...
Deep Learning: Evolution of ML from Statistical to Brain-like Computing- Data...Deep Learning: Evolution of ML from Statistical to Brain-like Computing- Data...
Deep Learning: Evolution of ML from Statistical to Brain-like Computing- Data...
 
SPARK USE CASE- Distributed Reinforcement Learning for Electricity Market Bi...
SPARK USE CASE-  Distributed Reinforcement Learning for Electricity Market Bi...SPARK USE CASE-  Distributed Reinforcement Learning for Electricity Market Bi...
SPARK USE CASE- Distributed Reinforcement Learning for Electricity Market Bi...
 
Enterprise Ready Android and Manageability- Impetus Webcast
Enterprise Ready Android and Manageability- Impetus WebcastEnterprise Ready Android and Manageability- Impetus Webcast
Enterprise Ready Android and Manageability- Impetus Webcast
 
Real-time Streaming Analytics: Business Value, Use Cases and Architectural Co...
Real-time Streaming Analytics: Business Value, Use Cases and Architectural Co...Real-time Streaming Analytics: Business Value, Use Cases and Architectural Co...
Real-time Streaming Analytics: Business Value, Use Cases and Architectural Co...
 
Leveraging NoSQL Database Technology to Implement Real-time Data Architecture...
Leveraging NoSQL Database Technology to Implement Real-time Data Architecture...Leveraging NoSQL Database Technology to Implement Real-time Data Architecture...
Leveraging NoSQL Database Technology to Implement Real-time Data Architecture...
 
Maturity of Mobile Test Automation: Approaches and Future Trends- Impetus Web...
Maturity of Mobile Test Automation: Approaches and Future Trends- Impetus Web...Maturity of Mobile Test Automation: Approaches and Future Trends- Impetus Web...
Maturity of Mobile Test Automation: Approaches and Future Trends- Impetus Web...
 
Big Data Analytics with Storm, Spark and GraphLab
Big Data Analytics with Storm, Spark and GraphLabBig Data Analytics with Storm, Spark and GraphLab
Big Data Analytics with Storm, Spark and GraphLab
 
Webinar maturity of mobile test automation- approaches and future trends
Webinar  maturity of mobile test automation- approaches and future trendsWebinar  maturity of mobile test automation- approaches and future trends
Webinar maturity of mobile test automation- approaches and future trends
 
Next generation analytics with yarn, spark and graph lab
Next generation analytics with yarn, spark and graph labNext generation analytics with yarn, spark and graph lab
Next generation analytics with yarn, spark and graph lab
 
The Shared Elephant - Hadoop as a Shared Service for Multiple Departments – I...
The Shared Elephant - Hadoop as a Shared Service for Multiple Departments – I...The Shared Elephant - Hadoop as a Shared Service for Multiple Departments – I...
The Shared Elephant - Hadoop as a Shared Service for Multiple Departments – I...
 
Performance Testing of Big Data Applications - Impetus Webcast
Performance Testing of Big Data Applications - Impetus WebcastPerformance Testing of Big Data Applications - Impetus Webcast
Performance Testing of Big Data Applications - Impetus Webcast
 
Real-time Predictive Analytics in Manufacturing - Impetus Webinar
Real-time Predictive Analytics in Manufacturing - Impetus WebinarReal-time Predictive Analytics in Manufacturing - Impetus Webinar
Real-time Predictive Analytics in Manufacturing - Impetus Webinar
 
Webinar real-time predictive analytics in manufacturing
Webinar  real-time predictive analytics in manufacturingWebinar  real-time predictive analytics in manufacturing
Webinar real-time predictive analytics in manufacturing
 

Kürzlich hochgeladen

How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsJoaquim Jorge
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationRadu Cotescu
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationMichael W. Hawkins
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAndrey Devyatkin
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024The Digital Insurer
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...apidays
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93
 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilV3cube
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Scriptwesley chun
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer
 
Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessAdvantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessPixlogix Infotech
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 

Kürzlich hochgeladen (20)

How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of Brazil
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessAdvantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your Business
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 

Accelerating Hadoop Solution Lifecycle and Improving ROI- Impetus On-demand Webcast

  • 1. © 2014 Impetus Technologies1 July 25, 2014 Accelerating the Big Data Solution Lifecycle and Improving ROI
  • 2. © 2014 Impetus Technologies2 Agenda Big Data Analytics: Implementation patterns Challenges faced Jumbune – an open source lifecycle accelerator Enterprise solution lifecycle Ways to address the challenges Recorded version available at http://bit.ly/1nMw8nQ
  • 3. © 2014 Impetus Technologies3 Big Data Analytics Primary drive for performing analytics Rise of the enterprise data lake Utilization of analytical resources Recorded version available at http://bit.ly/1nMw8nQ
  • 4. © 2014 Impetus Technologies4 Primary Purposes of an Analytical Solution Optimize the business Reduce time taken by analytics Result in effective analytics Compete and win Recorded version available at http://bit.ly/1nMw8nQ
  • 5. © 2014 Impetus Technologies5 Rise of the Enterprise Data Lake BIG DATA Sources of Data: ETL from every source - RDBMS, flat files, queues, legacy off loading, logs Arrival of Data: Intermittent, bulk, incremental Theme : “Leave no Data unused” Recorded version available at http://bit.ly/1nMw8nQ
  • 6. © 2014 Impetus Technologies6 Utilization of Analytics Resources • Capitalize on all analytics resources (engines) available • Access data with a variety of processing engines – Storm, Spark, Yarn etc. • Model in data science analytical systems – R, Octave, SAS, etc. • Write complex logic in custom MapReduce • Reuse code as User Defined Functions (UDFs) • Create ad hoc queries using Hive and PIG • Customization of Mahout algorithms, machine learning libraries Recorded version available at http://bit.ly/1nMw8nQ
  • 7. © 2014 Impetus Technologies7 Enterprise Big Data Solution Trends • No more single purpose Hadoop clusters • Enterprise Data Lake: Data flowing from many sources • Integrated platforms using variety of analytical engines • Serving multiple business applications • Resource sharing is a must across applications and engines Recorded version available at http://bit.ly/1nMw8nQ
  • 8. © 2014 Impetus Technologies8 Enterprise Solution Lifecycle (High level view) Business Requirement Designing / Modelling Development and Testing Production and Monitoring Recorded version available at http://bit.ly/1nMw8nQ
  • 9. © 2014 Impetus Technologies9 Enterprise Solution Lifecycle (Ground level view) xxx xxx Business User Data Analyst Development Quality Test DevOpsData Lake Production and Monitoring Recorded version available at http://bit.ly/1nMw8nQ
  • 10. © 2014 Impetus Technologies10 Challenges in Enterprise Analytical Solutions No common platform to detect root causes Incremental imports may ingest bad data Cluster resources are shared and optimal utilization is the key Implementing models in custom MR without errors is like hitting the bull’s eye Bad logic or bad data Recorded version available at http://bit.ly/1nMw8nQ
  • 11. © 2014 Impetus Technologies11 Scenario: Digitization of Newspaper for Analyzing News xxx xxx Team: 5 Dev, 3 QA, 2 DevOps Simple Problem: ‘q’ was misread by OCR as 9 TIME • A single code fault on TB of data can consume 24 work hours total for 2 Developers + 1 QA COST • Additional hours by engineers + The cost of unproductive cloud instances, storage and resources Recorded version available at http://bit.ly/1nMw8nQ
  • 12. © 2014 Impetus Technologies12 Scenario: Hive Queries Interpreted as MapReduce Executions on a Hadoop Cluster xxx xxx Team: 2 Dev, 1 QA, 1 DevOps Simple Problem: Data imbalance across cluster, low performance by Hive queries. TIME • Development team were refactoring Hive queries for improving the performance COST • Additional hours by engineers + The cost of unproductive cloud instances, storage and resources Recorded version available at http://bit.ly/1nMw8nQ
  • 13. © 2014 Impetus Technologies13 Impact on ROI Delayed Analytics Increase in CostsProductivity Loss Defeats one of the prime purpose of analytics Defeats the purpose of business cost optimization Iterations reduce the productivity of dependent teams in the cycle Recorded version available at http://bit.ly/1nMw8nQ
  • 14. © 2014 Impetus Technologies14 Current Iterative Development Approach Local Debug/ Unit Tests HDFS Data Check Performance • Localized subset of data • Non parallel execution • Practically unfeasible • Error prone • Difficult to find bad code • Difficult to collaborate across environments Recorded version available at http://bit.ly/1nMw8nQ
  • 15. © 2014 Impetus Technologies15 A Complete Enterprise Platform Data Lake Enterprise Engines Solutions Governance Security Validate,Profile,DebugandMonitor Recorded version available at http://bit.ly/1nMw8nQ
  • 16. © 2014 Impetus Technologies16 Introducing Jumbune: An Open Source Solution “A catalyst to accelerate realization of Big Data Analytics solutions” Flow AnalyzerData Validation Cluster Monitor Job Profiler Recorded version available at http://bit.ly/1nMw8nQ
  • 17. © 2014 Impetus Technologies17
  • 18. © 2014 Impetus Technologies18
  • 19. © 2014 Impetus Technologies19
  • 20. © 2014 Impetus Technologies20
  • 21. © 2014 Impetus Technologies21 Full Lifecycle Support - Jumbune xxx xxx Development Quality DevOpsData Ingestion Recorded version available at http://bit.ly/1nMw8nQ
  • 22. © 2014 Impetus Technologies22 Jumbune - Key Features • In depth code level analysis of cluster wide flow • Record and field level data violation reports • No deployment on worker nodes - Ultra light agent installation on the gateway node • Ability to turn on/off cluster monitoring at will – reduces resource load • Customizable rack aware monitoring • Correlated profiling analysis of phases, throughput and resource consumption • Ability to work with all Hadoop distributions • Coming up support for Yarn, Spark, Mesos • Available as Open Source Recorded version available at http://bit.ly/1nMw8nQ
  • 23. © 2014 Impetus Technologies23 2 3 For general inquiries about other Impetus solutions and services reach us at bigdata@impetus.com Recorded version available at http://bit.ly/1nMw8nQ
  • 24. © 2014 Impetus Technologies24 Thank You! Website • http://jumbune.org Contribute • http://github.com/impetus-opensource/jumbune • http://jumbune.org/jira/JUM Social • Follow @jumbune Use #jumbune • Jumbune Group: http://linkd.in/1mUmcYm Forums • Users: users-subscribe@collaborate.jumbune.org • Dev: dev-subscribe@collaborate.jumbune.org • Issues: issues-subscribe@collaborate.jumbune.org Downloads • http://jumbune.org • https://bintray.com/jumbune/downloads/jumbune Recorded version available at http://bit.ly/1nMw8nQ