This TDWI EU 2012 presentation looks at the various options for implementing a data store for analytical purposes and shows that there's no 'one size fits all' solution available

1. Database Shootout:
what's best for BI?

2. 2
The New Data Warehousing
Source: Timo Elliot, SAP

3. 3
Let's just buy Teradata...
Forrester Wave, as of April 2011
Gartner Group MQ, as of February 2012

4. 4
Or not...

5. 5
But...

6. ⇨
Back to basics: BI & DWHBack to basics: BI & DWH
⇨
The Need for SpeedThe Need for Speed
⇨
Database ArchitecturesDatabase Architectures
⇨
#BigData & the Hadoop Hoopla#BigData & the Hadoop Hoopla
⇨
The forgotten power ofThe forgotten power of Olap & MDXOlap & MDX
⇨
A Cloudy future?A Cloudy future?
⇨
Shootout: Evaluating alternativesShootout: Evaluating alternatives

7. 7
”
“
Business Intelligence (BI)
Process of identifying, collecting,
combining, analyzing, interpreting
and communicating internal and
external information to support
decision making processes
Concepts and methods to improveConcepts and methods to improve
business decision making by usingbusiness decision making by using
fact-based support systemsfact-based support systems
“
”
First definition of BI: 1958!

8. 8
Business Intelligence is....
Doing useful stuff with data, in order to…
Support the
Decision Making
process
So why not simply use
Decision Support Systems?

9. 9
How it all started in 1958...
⇨ Hans Peter Luhn (IBM) → A
Business Intelligence System
The notion of intelligence is also defined here, in a
more general sense, as the “ability to apprehend
the interrelationships of presented facts in such a
way as to guide action towards a desired goal.”
Full text on Timo Elliott's blog:
http://timoelliott.com/blog/2007/11/the_real_pioneer_of_busin
ess_i.html

10. 10
Luhn's Vision (1958!)
A Business Intelligence System
Abstract: An automatic system is being developed to disseminate
information to the various sections of any industrial, scientific or
government organization. This intelligence system will utilize data-
processing machines for auto-abstracting and auto-encoding of documents
and for creating interest profiles for each of the “action points” in an
organization. Both incoming and internally generated documents are
automatically abstracted, characterized by a word pattern, and sent
automatically to appropriate action points. This paper shows the flexibility
of such a system in identifying known information, in finding who needs to
know it and in disseminating it efficiently either in abstract form or as a
complete document.

11. 11
BI is dead; long live Analytics?

12. 12
12
Business Analytics
Evolution

13. 13
The Evolution of Enterprise Business
Intelligence
Enterprise Decision Management
Embracing all relevant data sources
BI injected into everyday business
processes
Master Data Management
Advanced Data Mining / Analytics
Business Activity Monitoring
Common Information &
Processes
BusinessValue
Disconnected Silos of
Information
Query/Reporting/Online Analytical Processing
Content Management/Data Warehousing
Search
2000 2005 2010 2015
Slide 13
Image courtesy Jim Fitzgerald, IBM research

14. 14
Evolving Business Intelligence Platform Requirements
Image courtesy Teradata

15. 15
Passive Monitoring: BI Starting Point
Source: Mark Madsen, Third Nature

16. 16
Supporting Better Analysis: Common Next Step
Source: Mark Madsen, Third Nature

17. 17
Active Monitoring
Source: Mark Madsen, Third Nature

18. 18
Active Monitoring and Feedback
Source: Mark Madsen, Third Nature

19. 19
Analysis
Source: Mark Madsen, Third Nature

20. 20
Prediction (passive)
Source: Mark Madsen, Third Nature

21. 21
Active Prediction
Source: Mark Madsen, Third Nature

22. 22
Prescription and Enterprise Decision Management
Source: Mark Madsen, Third Nature

24. 24
Watch out, the world is changing
750TB per week (compressed)
⇨ Sensor data
⇨ RFID
⇨ Sentiment Analysis
⇨
Text mining
⇨ Location data

25. 25
Machines generate most data

26. 26
Remember the Origins
•The general conception of a
separate architecture for BI has been
around longer, but this is the first
formal relational architecture and
definition published.
•One thing left out of most designs:
the box labeled business process
definitions.
“An architecture for a business and
information system”, B. A. Devlin, P. T.
Murphy, IBM Systems Journal, Vol.27,
No. 1, (1988)

27. 27
2012: we're still doing this!
Staging
Area
CSV
Files
ETL
ERP
DBMS
Sources ETL Process Data Warehouse EUL
DBMS
Files
ETL
Central DWH &
Data Marts
DBMS ETL
End User Layer,
in case you were
wondering ;-)

28. 28
We’ve (also) accumulated over 20 years of changes
Databases Documents Flat Files XML Queues ERP Applications
Source Environments
Data Consumers
Databases Dashboards OLAP Productivity BAM/BPM Reporting ETL Data Mining Applications
Warehouse
Database
ETL
Marts
ODS
EDR EII
Content
Store
EAI
Stream
processing
SQL Service API

29. 29
The assumption of the warehouse as a
database is gone
29
Traditional tabular
or structured data
Data at rest
Non-traditional
data (logs, audio,
documents)
Parallel
programming
platforms
Databases
Streaming
DBs/engines
Message
streams
Data in motion
Slide 29
Copyright Third Nature, Inc.

30. 30
The Need for Speed

31. 31
Why BI Projects Fail?
1. Query Performance Too Slow
(BI Survey 9)
70% of DWH's experience
performance constrained issues
of various types
(Gartner DWH MQ 2010)
Poor Query Performance
No 1 reason for replacing DWH
(TDWI Best Practices)

32. 32
Two dimensions of Speed
Companies wishing to maximize BI benefits
should focus on
1) support quality
2) implementation timeimplementation time
3) query response timequery response time and
4) breadth of deployment,
in that order.

33. 33
Minimize Implementation Time
Use 'RTF': vs
Off the shelf: Etc.
+ use Agile methods like Scrum or DSDM
+ look at Data Vault model & methodology

34. 34
Minimize Query Response Time
Source: TDWI Next generation Data Warehouse Platforms,
By Philip Russom
Why replace
a data
warehouse
solution?

35. 35
Solving Performance Problems
Replace every single thing before the database?
Migrating to an analytic database is twice as likely as to another row-store database.

36. 36
Applying “Laborware”: think twice...
⇨
Apply traditional optimization techniques:
⇨ Redesign solution
⇨ Add/optimize indexes
⇨ Horizontal partitioning
⇨ Add materialized views
⇨ Rewrite queries
⇨ Reorganize data
⇨ Offload old data
⇨ …
⇨ Costs will increase & recur!

37. “Hardware will change
the basic assumptions of
BI professionals about
what they can do”
Richard Hackathorn

38. 38
Numbers everyone should know
⇨
L1 cache reference 0.5 ns
⇨ Branch mispredict 5 ns
⇨
L2 cache reference 7 ns
⇨
Mutex lock/unlock 100 ns
⇨ Main memory reference 100 ns
⇨
Compress 1K bytes with Zippy 10,000 ns
⇨ Send 2K bytes over 1 Gbps network 20,000 ns
⇨
Read 1 MB sequentially from memory 250,000 ns
⇨ Round trip within same datacenter 500,000 ns
⇨
Disk seek 10,000,000 ns
⇨ Read 1 MB sequentially from network 10,000,000 ns
⇨
Read 1 MB sequentially from disk 30,000,000 ns
⇨ Send packet CA->Netherlands->CA 150,000,000 ns
Source: Jeff Dean, Google

39. 39
Trend: decreasing latency times
source:90’s versus 2010's

40. 40
The Good News: HW Cost Decline
⇨ < 2000:
⇨ tune software
⇨ 2012
⇨ hardware cheap
⇨ Mustang Index ~0.7
⇨ Cost per Gigaflop:
⇨ 1984: $ 15,000,000
⇨ 1997: $ 30,000
⇨ 2003: $ 82
⇨ 2011: $ 1.80

41. 41
Memory Cost Decline
41
We're still waiting!
⇨ 32 GB, May 2010:
⇨ 8 *4 GB = $1,200
⇨ 4* 8 GB = $2,000
⇨ 2*16GB = $2,400
⇨ 32 GB, May 2012:
⇨ 8 *4 GB = $ 280
⇨ 4* 8 GB = $ 350
⇨ 2*16GB = $ 500

42. 42
Intel keeps pushing the limits

43. 43
CPU: Moore's Law in action?
Same price!
Intel Xeon
E5-2680
635

44. 44
Storage costs keep going down
Year Size in GB US $/GB
1955 0.012 6,382,933.00
1960 0.01 3,686,400.00
1970 0.1 265,933.00
1980 2.5 16,000.00
1990 0.34 5,406.00
2000 40 7.17
2010 2,000 0.05
2012 3,000 0.07
“By the end of 2012, drives will have 100 times more
capacity at 1/100 of the cost per GB compared to 2000”

45. 45
Your next data warehouse?
The next-generation SDXC memory card specification,
released to members in April, 2009, dramatically
improves consumers digital lifestyles by increasing
storage capacity from more than 32 GB up to 2 TB and
increasing bus interface speed up to 104 MB per second
in 2009 with a road map to 300 MB per second.

46. 46
Architecture basics

47. 47
Choosing the right architecture is a trade off
FlexibilityFlexibility
AgilityAgility
Real-TimeReal-Time
ComplexityComplexity
IntegrationIntegration
AuditabilityAuditability
Data VolumeData Volume
Advanced
Analysis
Advanced
Analysis
PerformancePerformance
Low costLow cost
Skills &
Standards
Skills &
Standards
BI ArchitectureBI Architecture
Source:

48. 48
What this means…
• No ‘one size fits all’ solution
• Easy to over or under provision
• There are always exceptions
• Clueless analysts
• Tech savvy managers (even C-level)
• Excel Junkies

49. 49
Comparing Solutions
⇨ By Technology?
⇨
Columns, MPP, In-Memory, etc
⇨
By Storage Type?
⇨ Files, tables, OLAP
⇨ By Deployment type?
⇨ Appliance, Cloud, Saas
⇨ By Features/API?
⇨ SQL, MapReduce, R, etc.
⇨ By Speed?
⇨ TPC-H, Airline DB, Custom
⇨ By Licence type/price?
⇨ CPU, data size, memory usage

50. 50
SQL DB's for BI/Analytics

51. 51
Major BI Vendors have SQL DB's
IBM:
Microsoft:
Oracle:
SAP:
⇨ DB2, Netezza
⇨ SQL Server
⇨ MySQL, Oracle DB, Exalytics (TimesTen)
⇨ Sybase (IQ) & SAP Hana
..and all others are DB agnostic: Microstrategy, SAS, Tableau,
Tibco Spotfire, LogiXML, Pentaho, Jaspersoft, etc.

52. 52
Analytical DB's: What’s Different?
⇨ MPP: Massive Parallel Processing
⇨ Column based data organization
⇨ Data compression
⇨ Read optimization
⇨ In memory operation
⇨ Different disk configuration options
⇨ In DB analytics
⇨ Data mining
⇨ Statistics

53. 53
Architecture: SMP vs MPP
Different storage approaches:
● Shared Disk (clustering)
● Shared Nothing
Most DWH appliance & new software vendors
use Shared Nothing, MPP, Scale Out architecture

54. 54
Scaling Up and Out
Typical Workloads

55. 55
Blurring lines
⇨ 1 machine, up to:
⇨ 8 CPU/80 cores
⇨ 2 TB Ram
⇨ 24 SAS/SSD
⇨24*512 GB SSD = 12 TB
⇨24*900 GB SAS = 21 TB
“In terms of raw speed, nothing beats DASD”
Supermicro SuperServer 5086B-TRF

56. 56
Beware of SPOF's (or: why clustering?)

57. 57
Rows vs Columns
⇨ Nothing new about column storage: Taxir, 1969
⇨ Conceptual (and simplified) view:
Rows
Rows:
1,Smith,Joe,40000;2,Jones,Mary,
50000;3,Johnson,Cathy,44000;
1,2,3;Smith,Jones,Johnson;Joe,
Mary,Cathy;40000,50000,44000
EmpID Lastname Firstname Salary
1 Smith Joe 40000
2 Jones Mary 50000
3 Johnson Cathy 44000
Columns

58. 58
Rows vs Columns (2)
Source: Paraccel®
⇨ Columnar Challenges:
⇨ (fast) Loading
⇨ Updates

59. 59
Rows AND Columns!
⇨ Many vendors offer hybrid row/column options
⇨ Beware of differences between storage & indexing
⇨ Examples:
⇨ Teradata Aster
⇨ Greenplum
⇨ HP Vertica
⇨ Vectorwise
⇨ Microsoft
⇨ Oracle

60. 60
Data compression
Source:
⇨
Compression 50-90%
⇨
Some vendors claim > 95%
⇨
DB size < raw data size

61. 61
Read Optimization
⇨ OLTP: 90% write, 10% read
⇨ DWH: 10% write, 90% read
⇨ Common solution:
⇨ 'buffer' area (row oriented)
⇨ background process
updates/inserts to columns
⇨ Bulk loading = directSource:

62. 62
Memory Usage
⇨ Different approaches
⇨ Query (result) caching
⇨ Dynamic allocation
⇨ Explicit loading (e.g. dim)
⇨ Some products still disk
focused! (e.g. GP)
⇨ VectorWise: RAM as
secondary (!) storage

63. 63
Disk Usage/Configuration
⇨ 1. Disk/partition per CPU (core), e.g. Greenplum
⇨ 2. Software 'Raid' by DBMS, e.g. Paraccel
ADB

64. 64
Disk Usage/Configuration (2)
⇨ 3. Use standard devices, e.g. VectorWise, Vertica
ADB
⇨ 3 is easiest to set up (but some ADB's auto config)
⇨ Speed depends on other things too

65. 65
RAIS instead of RAID
⇨ 1. Failover Node (Hot Standby) ⇨ 2. Data Distribution
A
B
B
A
C
C
etc.
Hot Standby

66. 66
Mixed Storage Solution
⇨ SAN = SOR
⇨ Nodes = Persistent
subset
⇨ 'Blended Scan'
⇨ Patent Pending
Source: Paraccel®

67. 67
ILM: Software meets Hardware
⇨ Different approaches
⇨ Usage (e.g. TeraData)
⇨ Age (e.g. Oracle)
⇨ Partitions (e.g. Sybase IQ)
Burning
Hot
Warm
Cool
Cold
Sas
Sata
www.etre.com

68. 68
Beware of (Interconnect) Bottlenecks
Fast & Expensive SAN
Fast & Expensive Servers(s)
1Gb/s
1Gb/s shared
DWH
VM
ERP
VM
MAIL
VM
CRM
VM
Undersized Virtual DWH
You want:
* Dedicated hardware
* Infiniband QDR 12x: 96 Gb/s, or
* 100 Gb Ethernet: 100Gb/s
OR: Local storage (MPP w DASD)

69. 69
In Database Analytics
Source: Fuzzy Logix

70. 70
Everybody Loves R

71. 71
Inevitable In DB analytics
⇨ Fuzzy Logix
⇨ IBM/Netezza
⇨ IBM/Informix
⇨ SAP/Sybase
⇨ Paraccel
⇨ Microsoft
⇨ Asterdata/Teradata
⇨ SAS
⇨ IBM/Netezza
⇨ EMC Greenplum
⇨ TeraData
⇨ R
⇨ IBM/Netezza
⇨ AsterData/TeraData
⇨ Oracle
⇨ Greenplum
⇨ SAS

72. 72
The Hadoop Hoopla

73. 73
#BigData, the new frontier
Yes, these (and more) are all Open Source!

74. 74
#BigData?
Largest data set analyzed
KDNuggets poll 2012

75. 75
Putting #BigData into perspective*
Median
DWH size
*Idea by Glen Rabie, YellowFin BI

76. 76
*THIS* is Hadoop:
a Distributed File System
Data Distribution Data Retrieval using M/R

77. 77
#BigData & NoSQL: No Standards
“Each NoSQL DB has its own strengths/weaknesses;
most are not (directly) suited for typical BI workloads”

78. 78
The Great Divide(s)
⇨ Pure SQL DB's
⇨ All OS Column Stores
⇨ Paraccel, Kognitio
⇨ In Database Analytics
⇨ Map/Reduce (many)
⇨ R (GreenPlum)
⇨ SAS (TeraData)
⇨ Everything (Netezza iClass)
⇨ NoSQL Databases
⇨ Hive (Hadoop)
⇨ MongoDB
⇨ CouchDB
⇨ etc.

79. Worlds Colliding
⇨MapReduce (NoSQL)
⇨
Programming model
⇨
No DBMS/SQL required
⇨
Schema free
⇨
Exclusively <key,value>
⇨
Java, Python, C++, C,
etc.
⇨
Text/data mining
⇨
Eventually Consistent
⇨SQL (RDBMS)
⇨
Query language
⇨
DBMS required
⇨
Fixed schema
⇨
Complex structure
⇨
SQL
⇨
Not good at Text
⇨
ACID compliant

80. 80
What is MapReduce?
⇨ M/R is now patented by
Google (Patent
#7,650,331)
⇨ Used in many ADB's
⇨Hadoop, CouchDB
⇨AsterData
⇨GreenPlum
⇨Vertica
⇨...
MapReduce is a programming
model and an associated
implementation for processing and
generating large data sets.
Users specify a map function that
processes a key/value pair to
generate a set of intermediate
key/value pairs, and a reduce
function that merges all
intermediate values associated with
the same intermediate key

81. 81
MapReduce Explained
Source: http://blog.jteam.nl/2009/08/04/introduction-to-hadoop/
MR info: http://www.mapreduce.org (by Aster Data)

82. 82
M/R & SQL: How to get there
⇨ SQL on top of M/R
⇨ e.g. Hive-Hadoop
⇨ M/R invoking SQL
⇨ e.g. Greenplum
⇨ SQL invoking M/R
⇨ e.g. TeraData/Aster Data
⇨ Most ADB vendors implementing/investigating M/R
⇨ e.g. Vertica (Hadoop integration), Oracle, Netezza, etc.

83. 83

84. 84

85. 85

86. 86

87. 87

88. 88

89. 89

90. 90

91. 91

92. 92
(R/H/M)OLAP
⇨ OnLine Analytical Processing
⇨ Analyse multidimensional data
⇨ Basic architecture:
Data Warehouse
MDX
OLAP
engine/server
Analysis front end

93. 93
Stars and Cubes
⇨ Star schema
⇨ Dimension & fact tables
⇨ Best foundation for cubes
⇨ Cubes (logical/physical)
⇨ Dimensions
⇨Hierarchies
⇨ Levels
⇨Attributes
⇨ Measures

94. 94
The power of OLAP
Aggregates, positional calculations (prior vs current), range
calculations (ytd, mtd), level calculations (child to parent
contribution)

95. 95
MDX
⇨ Short for 'Multi Dimensional Expressions'
⇨ ~ SQL for OLAP:
⇨
SELECT
{set for column headers} ON COLUMNS,
{set for row headers} on ROWS
FROM [Cube Name]
WHERE {set for filtering}
⇨
SELECT:
{[Measures].[Unit Sales]} ON COLUMNS,
{[Product].[Drink], [Product].[Food]} ON ROWS
FROM [Sales]
WHERE [Time].[1997]

96. 96
The Power of MDX
Positional: [Measures].[Profit], [Time].PrevMember
Range: Aggregate(YTD(), [Measures].[Profit]
“MDX is far
more powerful
than SQL for
the typical BI
questions”

97. 97
Adding OLAP to the mix
⇨ Virtual Cubes, e.g.
⇨ Kognitio Pablo
⇨ Pentaho Mondrian
⇨ Microstrategy
⇨ Physical Cubes, e.g.
⇨ Microsoft Analysis Services
⇨ Oracle Essbase
⇨ Jedox Palo
Physical cubes allow 'write back': what
if, forecasting, budgetting & planning

98. 'New' kid on the block: SAP HANA

99. 99
The promises of the Cloud
⇨ “Utility computing”
⇨ Unlimited capacity
⇨ Pay as you go/by the sip
⇨ Lower costs
⇨ Always up to date
⇨ Invisible OS
⇨ Security
⇨ Safety

100. 100
Cloud still getting Hotter
Source: IBM CIO Survey 2011

101. 101
Types of Cloud Solutions
Virtualization
IaaS (Infrastructure)
PaaS (Platform)
SaaS (Software)
ValueAdded

102. 102
Cloud Cost Components
StorageStorage
BandwithBandwith
SLA/ServiceSLA/Service
CPU powerCPU power
MemoryMemory
Data transferData transfer
RequestsRequests

103. 103
BI&DWH aaS Scenarios

104. 104
The trouble with Cloud DWH
⇨ DWH aaS vendors:
⇨ e.g. 1010Data, Kognitio, Vertica, EMC/Greenplum
⇨
more will follow

105. 105
What about No Database at all?
Rick F. van der Lans
Key element: abstraction (de-coupling)

106. 106
Data Virtualization concept
Virtual DB
SQL SOAP REST FILE WS-*
Information Consumers

107. 107
© 2011 Composite Software, Inc. / Composite Proprietary
Example: Composite 6
Discovery
Active Cluster
Composite Information Server
XQuery, Java, WSDL, SCA
(Services Centric)
Front-end Applications
Security
Metadata Repository
Views, SQLScript
(Database Centric)
Security
Query Engine
Cost-based
Optimizer
Rules-based
Optimizer
Federation
Engine
Web Services
(HTTP, REST, SOAP, JSON, XQuery)
SQL
(ODBC, JDBC, ADO.NET)
Messaging
(JMS)
Java
(POJO)
Web Services
(HTTP, SOAP, JSON)
Messaging
(JMS)
Application
APIs
MF
Adapter
Java
(POJO)
Advanced Functions
Quality GovernanceCaching
SQL
(ODBC, JDBC)
URI
Monitor
Manager
Studio
Performance Plus
Adapters
Development
Environment
Runtime Server
Environment
Management
Environment
Applications, Big Data Stores, Excel, Flat Files, Mainframes, Messages, OLAP Cubes,
RDBMS, Web Services, XML Documents

108. 108
Meet

109. 109
A Unified Data Hub

110. 110
Virtual vs Physical trade offs
Source:Source:
Mark MadsenMark Madsen

111. 111
The Shootout!
⇨
Things to ask your (potential) vendor
⇨ References
⇨ Assist in a paid POC
⇨ License model & unit of cost: CPU, Core,
Server, (raw) Data volume, Memory used
⇨ Free dev/test editions (only pay for
production use)
⇨ Support options (updates only, mail/phone
support, etc)
⇨ If migrating: trade in discount
⇨ Opt out/de-integration options

112. 112
Does your DB cover the Basics?
⇨ Full SQL 2003 support?
⇨ Easy backup/restore features?
⇨ Scaling up or out?
⇨ Failover & persistency?
⇨ External (management) Tool integration?

113. 113
Which deployment types?
⇨ On Premise ⇨ Saas/Cloud
Software only
Appliance
Vendor/ISPCustomer

114. 114
Size/Workload/Complexity?
Source:
Bloor Group
Source:
Third Nature
Most
organizations
are here

115. 115
What's the question?
Source:

116. 116
Analytical Power?
Source: SAP (Sybase IQ 15.4)

117. 117
Beware of
Benchmarks !
⇨ Differences in
⇨# threads
⇨# cores
⇨# disks
⇨# nodes
⇨CPU generation/speed
1. Always use P.O.C. on your own
data & query workload
2. Don't trust the MQ's

118. ⇨
Ongoing Market Consolidation
⇨
More additional/alternative storage engines
⇨
Hybrid Row/Column solutions
⇨
Every db will get In DB analytics
⇨
Every db will get Hadoop/MR extensions
⇨
Everything in-memory

119. 119
So what's the best
database for BI?

120. 120

122. Web: www.tholis.com
Email: jos<at>tholis.com
Phone: +31-(0)6-51169606
Skype: tholis.jos
LinkedIn: jvdongen
Twitter: josvandongen
IRC: _grumpy
Jos van Dongen
In BI since 1991
Principal Consultant
Author/Speaker/Analyst
Proud member of #BBBT