Big Data Architectures: Beyond Hadoop- Impetus Webinar

•

12 gefällt mir•1,184 views

Impetus Technologies

Impetus webcast ‘Big Data Architectures – Beyond the Elephant Ride’ available at http://lf1.me/PB/

Technologie Business

Impetus Technologies Inc.
1 © 2014 Impetus Technologies
Big Data Architectures
Beyond the Elephant Ride
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Outline
• The Hadoop ecosystem and challenges
• Big Data solutions beyond Hadoop
- How and where to use them?
• Some use cases
• Big Data Architecture Strategy
2 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Disclaimer
• Not advising to discard Hadoop
• Will discuss Big Data technologies that complement and
supplement Hadoop
3 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

What is Hadoop?
Scalable data processing engine
• DFS: Scalable fault-tolerant distributed file-system
• Map Reduce: Parallel processing programming model
4 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Hadoop Ecosystem
5 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Where to use Hadoop?
• Risk Analysis
– Intrusion detection, Credit scoring
• Recommendation
– Customers who purchased this also liked
6 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Where to use Hadoop?
• Sentiment Analysis
– Positive, Negative or Neutral sentiment in sentences
• Targeted Ads
– Display ads based on user behavior and preferences
7 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Where to use Hadoop?
• Machine Learning
– Spam vs. Not Spam
• And a lot of other areas…
8 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Challenges with Hadoop
9 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Limitations
• Data security
• Dependence on OS/Language
• MapReduce programming
• Batch processing only
10 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

11 © 2014 Impetus Technologies
Beyond Hadoop
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Faster Hadoop
• MapR
– Simple to manage (NFS)
– MapR Express Lane
– Handles real-time data flows
12 © 2014 Impetus Technologies
Dominant Players – HortonWorks,
Cloudera, Hadapt
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Transactional Systems
• E-commerce Websites
-ATM
-Traditional solutions - MySQL, Oracle, MSSQL
13 © 2014 Impetus Technologies
Go NewSQL - VoltDB, Clusterix
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Explaining VoltDB
14 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Real Time Computation
• Continuous computation
- Trending topics
• Stream processing
- Twitter Firehose
15 © 2014 Impetus Technologies
Try.. Storm, Esper, S4, CloudScale!
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Explaining Storm
Spouts- Data Source Bolts – Data Processors
16 © 2014 Impetus Technologies
Topologies – Combination of Spouts and Bots
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Real-time Traffic
17 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Doing it Right
18 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Graph Computation
• Page Rank
• Shortest Path
• “Friends of my friends’ friends”
19 © 2014 Impetus Technologies
We suggest – Giraph, Pregel
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

LinkedIn Degrees of Separation
20 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Fast Key-Value Access
• Show latest items listing in your homepage
• Caching
21 © 2014 Impetus Technologies
Explore NoSQL - Redis and Riak!
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

How it works?
Latest comments posted by user
Traditional Approach: Query in Runtime
SELECT * FROM foo WHERE ... ORDER BY time DESC LIMIT 10
Redis Live Cache Approach
FUNCTION get_latest_comments(start,num_items):
id_list = redis.lrange("latest.comments",start,start+num_items-1)
IF id_list.length < num_items
id_list = SQL_DB("SELECT ... ORDER BY time LIMIT ...")
END
RETURN id_list
END
22 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

How it works?
23 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

24 © 2014 Impetus Technologies
Good to Know
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Recap
Already Invested in Hadoop
25 © 2014 Impetus Technologies
Explore Faster Hadoop –
HortonWorks, Cloudera,
MapR, Hadapt
Alternatives to Hadoop HPCC, Disco
Complex business queries,
online transaction
processing
“New Gen” SQL
VoltDB, Clustrix, Hadapt
Real Time Analytics CloudScale, Storm, Esper
Fast Key-Value Access Redis, Riak
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Bleeding Edge A peek into the future
26 © 2014 Impetus Technologies
High performance super
computing Open MPI , BSP
Highly efficient, large scale
graph computing Pregel, Giraph
Low latency queries over very
large data sets
Dremel
Incremental updates on
massive datasets
Percolator (Caffeine)
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Architecture Strategy
27 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Recommendations
• Think Beyond the Warehouse
• Time for Real-time
• Not Only Hadoop
• Hadoop is an enabler for better data warehouse solutions,
not a replacement
• Back To SQL?
• SQL is not bad
• Hadoop and SQL complement each other
• Integrations & Visualizations
• Realtime
28 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Recommendations
• Big Data in upstream operational systems
• Forecasting Systems
• Supply Chains
• CRMs
29 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Our Architecture Strategy
30 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

31 © 2014 Impetus Technologies
About Impetus
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Our Expertise
• Strategic partners for software product engineering and
R&D
• Thought leaders in cutting-edge technologies
• Mature processes and practices that are methodical, yet
flexible
• Diverse domain expertise
32 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

33 © 2014 Impetus Technologies
Q & A
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

34 © 2014 Impetus Technologies
Thank You
Write to us at inquiry@impetus.com
Follow us on Twitter @impetustech
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=60

Weitere ähnliche Inhalte

Andere mochten auch

Choxi.com is the online shopping destination for quality, on-trend, branded and unbranded, in-demand goods at prices ranging from 50%-80% off retail. Choxi’s efficient business model cuts out the middleman to allow delivery of great products at the lowest possible cost to the consumer. Keith Bussey, VP of Technology at Choxi sought to move to AWS to achieve the promise of the cloud — the ability to instantly scale capacity and costs to meet seasonal business cycles. Choxi’s move to AWS was planned and executed in swift and efficient fashion. The relocation of the Clustrix database into the cloud infrastructure (AWS) was completed without issue in time for the recent holiday season and to avoid causing any downtime in Choxi’s 24×7 service. Join the webinar and learn first hand from Keith and Nick Lamb, Director of Consulting Services for Clustrix, how to benefit from tools like AWS and ClustrixDB to match the cycles and goals of your business.

Moving an E-commerce Site to AWS. A Case Study

Clustrix

If you run a Magento store that is impacted negatively by catalog updates or indexing/reindexing, listen in. Avoid catalog updates impacting your checkouts, site downtime and long page view load time with the ClustrixDB for Magento Bundle. Created exclusively for high-volume/complex-catalog retailers, this replacement backend is a proven upgrade for Magento sites. Let us show you how this all works. Recently, at this year’s Magento Imagine, we ran a LIVE demo of the ClustrixDB for Magento Bundle. Stats from the demo: System ran for 50 hours -- reindexing up to every 12 minutes. 2.6 million orders processed (14.6 per second) 147 million page views (816 per second) Average Response time of 267ms 0% error rate, 100% checkout uptime In this webinar, we’ll also cover: How you can enable catalog updates to process in the background without affecting the normal operations of your site with the Clustrix Shadow (re)Indexer A Magento-approved alternative database to MySQL that scales performance up/down as your add/subtract commodity nodes to the cluster. ClustrixDB has no read slaves, replication lag or sharding and Flexes Up and down to deliver exactly the right amount of performance and cost every month of the year.

Achieve new levels of performance for Magento e-commerce sites.

Clustrix

Db performance optimization with indexing

Rajeev Kumar

Clusterix at VDS 2016

Eamonn Maguire

Presto: Distributed sql query engine

kiran palaka

In this presentation, Paul introduces InfluxDB, a distributed time series database that he open sourced based on the backend infrastructure at Errplane. He talks about why you'd want a database specifically for time series and he covers the API and some of the key features of InfluxDB, including: • Stores metrics (like Graphite) and events (like page views, exceptions, deploys) • No external dependencies (self contained binary) • Fast. Handles many thousands of writes per second on a single node • HTTP API for reading and writing data • SQL-like query language • Distributed to scale out to many machines • Built in aggregate and statistics functions • Built in downsampling

Introduction to InfluxDB, an Open Source Distributed Time Series Database by ...

Hakka Labs

Andere mochten auch (6)

Moving an E-commerce Site to AWS. A Case Study

Achieve new levels of performance for Magento e-commerce sites.

Db performance optimization with indexing

Clusterix at VDS 2016

Presto: Distributed sql query engine

Introduction to InfluxDB, an Open Source Distributed Time Series Database by ...

Mehr von Impetus Technologies

Data Warehouse Modernization Webinar Series- Critical Trends, Implementation ...

Impetus Technologies

Future-Proof Your Streaming Analytics Architecture- StreamAnalytix Webinar

Impetus Technologies

Building Real-time Streaming Apps in Minutes- Impetus Webinar

Impetus Technologies

Smart Enterprise Big Data Bus for the Modern Responsive Enterprise- StreamAna...

Impetus Technologies

Impetus White Paper- Handling Data Corruption in Elasticsearch

Impetus Technologies

Real-world Applications of Streaming Analytics- StreamAnalytix Webinar

Impetus Technologies

Real-world Applications of Streaming Analytics- StreamAnalytix Webinar

Impetus Technologies

Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus...

Impetus Technologies

Accelerating Hadoop Solution Lifecycle and Improving ROI- Impetus On-demand W...

Impetus Technologies

Presentation on 'Deep Learning: Evolution of ML from Statistical to Brain-like Computing' Speaker- Dr. Vijay Srinivas Agneeswaran,Director, Big Data Labs, Impetus The main objective of the presentation is to give an overview of our cutting edge work on realizing distributed deep learning networks over GraphLab. The objectives can be summarized as below: - First-hand experience and insights into implementation of distributed deep learning networks. - Thorough view of GraphLab (including descriptions of code) and the extensions required to implement these networks. - Details of how the extensions were realized/implemented in GraphLab source – they have been submitted to the community for evaluation. - Arrhythmia detection use case as an application of the large scale distributed deep learning network.

Deep Learning: Evolution of ML from Statistical to Brain-like Computing- Data...

Impetus Technologies

SPARK SUMMIT SESSION - A majority of the electricity in the U.S. is traded in independent system operator (ISO) based wholesale markets. ISO-based markets typically function in a two-step settlement process with day-ahead (DA) financial settlements followed by physical real-time (spot) market settlements for electricity. In this work, we focus on obtaining equilibrium bidding strategies for electricity generators in DA markets. Electricity prices in DA markets are determined by the ISO, which matches competing supply offers from power generators with demand bids from load serving entities. Since there are multiple generators competing with one another to supply power, this can be modeled as a competitive Markov decision problem, which we solve using a reinforcement learning approach. For power networks of realistic sizes, the state-action space could explode, making the RL procedure computationally intensive. This has motivated us to solve the above problem over Spark. The talk provides the following takeaways: 1. Modeling the day-ahead market as a Markov decision process 2. Code sketches to show the markov decision process solution over Spark and Mahout over Apache Tez 3. Performance results comparing Mahout over Apache Tez and Spark.

SPARK USE CASE- Distributed Reinforcement Learning for Electricity Market Bi...

Impetus Technologies

Enterprise Ready Android and Manageability- Impetus Webcast

Impetus Technologies

Impetus webcast ‘Real-time Streaming Analytics: Business Value, Use Cases and Architectural Considerations’ available at http://bit.ly/1i6OrwR The webinar talks about- • How business value is preserved and enhanced using Real-time Streaming Analytics with numerous use-cases in different industry verticals • Technical considerations for IT leaders and implementation teams looking to integrate Real-time Streaming Analytics into enterprise architecture roadmap • Recommendations for making Real-time Streaming Analytics – real – in your enterprise • Impetus StreamAnalytix – an enterprise ready platform for Real-time Streaming Analytics

Real-time Streaming Analytics: Business Value, Use Cases and Architectural Co...

Impetus Technologies

Impetus webcast "Leveraging NoSQL Database Technology to Implement Real-time Data Architectures” available at http://bit.ly/1g6Eaj4 This webcast: • Presents trade-offs of using different approaches to achieve a real-time architecture • Closely examines an implementation of a NoSQL based real-time architecture • Shares specific capabilities offered by NoSQL Databases that enable cost and reliability advantages over other techniques

Leveraging NoSQL Database Technology to Implement Real-time Data Architecture...

Impetus Technologies

Maturity of Mobile Test Automation: Approaches and Future Trends- Impetus Web...

Impetus Technologies

Big Data Analytics with Storm, Spark and GraphLab

Impetus Technologies

Webinar maturity of mobile test automation- approaches and future trends

Impetus Technologies

Next generation analytics with yarn, spark and graph lab

Impetus Technologies

For Impetus’ White Papers archive, visit- http://lf1.me/drb/ This white paper talks about the design considerations for enterprises to run Hadoop as a shared service for multiple departments. As Hadoop becomes more mainstream and indispensable to enterprises, it is imperative that they build, operate and scale shared Hadoop clusters. The design considerations discussed in this paper will help enterprises accomplish the essential mission of running multi-tenant, multi-use Hadoop clusters at scale. The white paper talks about Identity, Security, Resource Sharing, Monitoring and Operations on the Central Service. For Impetus’ White Papers archive, visit- http://lf1.me/drb/

The Shared Elephant - Hadoop as a Shared Service for Multiple Departments – I...

Impetus Technologies

Performance Testing of Big Data Applications - Impetus Webcast

Impetus Technologies

Mehr von Impetus Technologies (20)