Big Data in Action – Real-World Solution Showcase

Grab some coffee and enjoy
the pre-show banter before
the top of the hour!

Big Data in Action: Real-World Solution Showcase
The Briefing Room

Twitter Tag: #briefr
The Briefing Room
Welcome
Host:
Eric Kavanagh
eric.kavanagh@bloorgroup.com

Mission
! Reveal the essential characteristics of enterprise software,
The Briefing Room
good and bad
! Provide a forum for detailed analysis of today’s innovative
technologies
! Give vendors a chance to explain their product to savvy
analysts
! Allow audience members to pose serious questions... and get
answers!

The Briefing Room
Topics
This Month: BIG DATA
March: CLOUD
April: BIG DATA
2014 Editorial Calendar at
www.insideanalysis.com/webcasts/the-briefing-room

The Briefing Room
Big Data

Analysts: Lindy Ryan and John O’Brien
Lindy Ryan is the Research Director for Radiant Advisor’s Data
Discovery and Visualization practice and leads research and analyst
activities in the confluence of data discovery, visualization, and data
science from a business needs perspective. She also retains the role
of Editor in Chief of RediscoveringBI Magazine. As Radiant Advisors’
Editor in Chief for three years, Lindy participated in in-depth
discussions and analysis with industry thought leaders and vendors
while maturing her position and perspectives in the BI industry.
John O’Brien is Principal and CEO of Radiant Advisors. With over 25
years of experience delivering value through data warehousing and BI
programs, John’s unique perspective comes from the combination of
his roles as a practitioner, consultant, and vendor in the BI industry.
His knowledge in designing, building, and growing enterprise BI
systems and teams brings real world insights to each role and phase
within a BI program. Today, through Radiant Advisors John provides
research and advisory services that guide companies in meeting the
demands of next generation information management, architecture,
and emerging technologies.
The Briefing Room

! IBM offers a full suite of Big Data solutions, including
InfoSphere Streams, InfoSphere BigInsights and InfoSphere
Data Explorer
! IBM also offers a series of products designed to leverage the
The Briefing Room
power of Hadoop
! Stream Integration is a Premier Business Partner with IBM
and focuses its consultancy exclusively on IBM products
IBM

The Briefing Room
Guests:
Eric Poulin
VP of Business Analytics,
Stream Integration
Paul Flach
VP of Enterprise Analytics,
Stream Integration

10
Big
Data
Performance
for
Analy3cs
Eric
Poulin
VP,
Analy3cs

Big
Data
Eric.poulin@streamintegra3on.com

11
11
Agenda
• Overview
of
Stream
Integra3on
• Big
Data
Performance
for
Analy3cs
• Modular
Analy3cs

12
Company
Overview
• Award Winning Information
Lifecycle Consultancy
• Founded in 2000
• IBM Premier Partner
• Exclusively focused on IBM
Information Management, Big Data
and Analytics
• Offices in North America,
Caribbean, and Europe
• Development and Support Centers
in India and China
Copyright
©
2014,
Stream
Integra3on
Inc.
All
rights
reserved.
12

LINKING
DATA
13
TO
THE
BUSINESS
REQUIREMENTS
TRANSACTIONAL

COLLABORATIVE
APPLICATIONS
MANAGE
CONTENT
ANALYZE
BIG
DATA
STRUCTURED
DATA
INTEGRATE
INFOSPHERE
MDM
GOVERN
DATA
BUSINESS
ANALYTICS
APPLICATIONS
STREAMS
EXTERNAL
INFORMATION
SOURCES
ww
QUALITY
LIFECYCLE
MANAGEMENT
SECURITY

PRIVACY
INFORMATION
SERVER
DESIGN
★
DEPLOY
★
OPERATE
★
MANAGE
★
EXTEND
BIG
INSIGHTS
TRADITIONAL
SOURCES
PUREDATA/NETEZZA
STREAMING
INFORMATION

14
Performance
for
the
Future
of
Analy3cs
Paul
Flach
Stream
Integra3on

15
Capabili3es
Required
for
Hadoop
Style
Workloads
Applica3on
Support
and
Development
Cluster
and
Workload
Management
Run3me
Visualiza3on

Discovery
Data
Ingest
Analy3cs
Engines
Data
Store
File
System
Tooling
Security
15

16
Big
SQL
provides
na3ve
SQL
for
Hadoop
ANSI
SQL
92+
support

17
Coordinator node
Map
Reduce
MPP
RunKme
n+2
User
Data
temp(s)
HDFS
Hadoop Data Node(s)
SQL sub-sections
Map
Reduce
MPP
RunKme
n+n
User
Data
temp(s)
HDFS
Head Node
Catalog
Host 2 Host n
Host 1
Cluster
network
Local
fs
(temps)
Local
fs
(catalog
tables)
Distributed
fs
sync
Map
Reduce
MPP
RunKme
n+1
User
Data
temp(s)
HDFS
Direct
Hadoop
data
access
sync
sync
Big
AcceleraKon
Query
OpKmizer
Common
SQL
BigInsights
–
DB2
–
Netezza
Oracle
–
Teradata
Next
Gen
Big
SQL
will
provide
first
MPP
query
engine
for
Hadoop

18
BigSheets
provides
business
users
with
access
to
data
without
programming
Spreadsheet-‐style
interface
Data
VisualizaKon

Graphs

19
Watson
Explorer
included
in
BigInsights
Faceted
Search,
NavigaKon

Discovery

20
AnalyKcs
Accelerators
provide
ability
to
extract
insights
more
quickly
Text
Social
Media
Machine
Data

21
App
Store
reduces
development
effort
and
enables
reusability
Combine
Hadoop
Apps

22
Open
Source
Hadoop
Components
Visualization Discovery Data Ingest
Open
Source
Analytics Engines
Cluster Optimization and Management
Nutch
Runtime
Data Store HBase
File System
MapReduce
HDFS
Application Support and Development Tooling
MapReduce
Pig
Hive
ZooKeeper
Sqoop
Security
HCatalog
Flume
Avro
Lucene
Oozie
Derby
22

23
BigInsights
Enterprise
Edi3on
Components
Visualization Discovery Data Ingest
Netezza
DB2
Analytics Engines
Cluster Optimization and Management Streams
Derby
Private
firewall
Open
Source
DataStage
Nutch
IBM
IBM InfoSphere BigInsights
Integrated
Installer
Runtime
Admin
Console
Data Store HBase
File System
MapReduce
HDFS
Text
Processing
Engine
and
Extractor
Library
(AQL+HIL)
JDBC
Application Support and Development Tooling
App
infrastructure
MapReduce
Pig
Hive
Splicable
Text
Compression
High
Availability
ZooKeeper
Sqoop
SystemML
Eclipse
Big
SQL
Security
HCatalog
R
Gnip
BoardReader
GPFS-‐FPO
LDAP
Guardium
Flume
Jaql
Avro
BigSheets
Dashboard
/
visualiza3on
Data
Explorer
Lucene
Oozie
PAM
Enhanced
Monitoring
Adap3ve
MapReduce
Teradata
23

Plagorm
Analy3c
Modules
Cloud
Computing
GIS Engine Forecasting
Engine
Routing Engine
Inventory Engine
Work Force
Engine
Solutions
Core Engine
Streams
BigInsights
IMDB
Column-‐Store
PureData
In-‐Flight
Data
Self-‐
Structured
Data
Frequently
Requested
Summaries
Low
Entropy
Data
Mixed
Workload
Requests
25 25

The Briefing Room
Perceptions Questions
Analysts:
Lindy Ryan and
John O’Brien

BIG DATA
IN ACTION
© Copyright 2014 Radiant Advisors. All Rights Reserved v1.10.000
28
Real-World Solution Showcase with Stream Integration
Inside Analysis – The Briefing Room, February 25, 2014
Lindy Ryan | Research Director, Data Discovery Visualization
@lindy_ryan lindy.ryan@radiantadvisors.com
John O’Brien | Principal Analyst, Modern Data Platforms
@obrienjw john.obrien@radiantadvisors.com

Big Data in Action: Real-World Solutions
MODERN DATA PLATFORM
Apache Hadoop
R programs
Map Reduce
Flexibility Class
Extending SQL Access to Big Data and Hadoop via Hive and other HDFS SQL engines
Highly Optimized for Analytics
MPP In-memory MOLAP
Hive SQL
Highly Specialized for Analytics
Document
Stores
PIG / Hive
29
Enterprise
Data
Warehouses
Master
Reference
Data
askdjfl
kasjdfl
iuyuiio
Graphs
Text
Analytics
Optimized Class Reference Class
Discovery, Scalable, Programs Discovery Analytics Oriented Stable, Context, SQL
Operational Systems, Big Data, Streams
HDFS
Columnar

SQL-ON-HADOOP
Apache
Hadoop v1
HCatalog
Apache
Hadoop v2
30
PIG
Hive-QL
MapReduce
Hadoop HDFS
Map
Reduce
PIG
Hive-QL
HCatalog
YARN
Hadoop HDFS
Hadoop
Distributions
and 3rd Party
PIG
Hive
Map
Reduce
HCatalog
YARN
Impala, HAWQ
InfiniDB, Presto
Hadoop HDFS
MPP Engine
Not all SQL-on-Hadoop is the same:
1. SQL capabilities (SQL-92, Analytic functions SQL-2003? SQL-2011? UDF?)
2. Scalability (not always the same as Hadoop scalability)
3. Speed (flat out performance response time without caching)
File types: ORCFILE, SEQPART, Parquet

TRADITIONAL FORMS OF DISCOVERY
31
Spreadsheets
• Most popular business “analytic” tool
• Having access to the data is the value
• Analysts can slice and dice data for insights
Basic Visualizations
• Provide visual representations of data
• Provide insights beyond plain text data
• Simplify complex information highlight trends

ANALYTIC FORMS OF DISCOVERY
32
Multi-Faceted, “Search Mode”
• Discovery within structured unstructured data
• Mine through various forms of data at once
• Google-like search to iterate and deep dive
Advanced Visualizations
• Visualize clusters of data and correlations
• Discover analytic models iteratively with data
• Visual cues and cognitive sciences UX

For more information
www.RadiantAdvisors.com
Twitter: @RadiantAdvisors #ModernBI #RediscoveringBI
RSS: feed://radiantadvisors.com/feed/
Email: info@RadiantAdvisors.com
LinkedIn: www.linkedin.com/company/radiant-advisors
Subscribe: Rediscovering BI quarterly e-magazine
www.radiantadvisors.com/rediscoveringbi
THANK YOU!
33

ANALYST QUESTIONS
34
1. How are you handling the performance or SQL
capabilities in Hive with Big SQL?
2. How do users define schema for Big SQL?
3. Can you explain user roles, security, and metadata in
the App Store? Who is the store administrator?

The Briefing Room

This Month: BIG DATA
March: CLOUD
April: BIG DATA
www.insideanalysis.com/webcasts/the-briefing-room
The Briefing Room
Upcoming Topics
2014 Editorial Calendar at
www.insideanalysis.com

THANK YOU
for your
ATTENTION!
The Briefing Room

Big Data in Action – Real-World Solution Showcase

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Big Data in Action – Real-World Solution Showcase

Similar to Big Data in Action – Real-World Solution Showcase (20)

More from Inside Analysis

More from Inside Analysis (20)

Recently uploaded

Recently uploaded (20)

Big Data in Action – Real-World Solution Showcase