1. Impact of Column-OrientedMain-Memory Databases on Enterprise Applications Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld Hasso Plattner Institute March 02, 2010

2. © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 2 Disclaimer This presentation outlines our general product direction and should not be relied on in making a purchase decision. This presentation is not subject to your license agreement or any other agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to develop or release any functionality mentioned in this presentation. This presentation and SAP's strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent.

3. Agenda The Hasso Plattner Institute Technical Foundation of Columnar In-Memory Databases Impact on Enterprise Applications © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 3

4. Agenda The Hasso Plattner Institute Technical Foundation of Columnar In-Memory Databases Impact on Enterprise Applications © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 4

5. Key Facts about the Hasso Plattner Institute Founded as a public private partnershipin 1998 in Potsdam near Berlin, Germany Institute belongs to theUniversity of Potsdam Ranked 1st in “CHE” 340 B.Sc. and M.Sc. students 10 professors, 91 PhD students Course of study: IT Systems Engineering © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 5

6. Prof. Dr. h.c. Hasso Plattner / Dr. Alexander Zeier Research focuses on the technical aspects of enterprise software anddesign of complex applications Memory-Based Data Management for Enterprise Applications Human-Centered Software Design and Engineering Maintenance and Evolution of Service-Oriented Enterprise Software Integration of RFID Technology in Enterprise Platforms Architecture-based Performance Simulation Research co-operations with Stanford, MIT, etc. Industry co-operations with SAP, Siemens, Audi, etc. Research GroupEnterprise Platform & Integration Concepts Partner of Stanford Center for Design Research Partner of MIT in Supply Chain Innovation © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 6

7. Agenda The Hasso Plattner Institute Technical Foundation of Columnar In-Memory Databases Impact on Enterprise Applications © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 7

8. Two separate worlds: OLTP and OLAP? © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 8

9. Two separate worlds: OLTP and OLAP? © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 9

10. Dominant Hardware Trends Multi-Core Technology Moore’s Law: “…number of transistors … doubling approximately CPU frequency hit limitin 2002, but Moore’s law holds today In-Memory Technology Increased size: up to 2TB of main-memory on one main board in 2010 Constantly dropping costs © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 10

11. 3 Aspects for a Hybrid Solution Columnar Storage New database layout accessing only needed portions of data Improve access for subsets of attributes In-Memory Fastest possible data access Spatial proximity Compression Reduce amount of data to fit in main memory Use cache and bus capacities more efficient © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 11

12. Row Store Column Store Storages: Row vs. Column © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 12

13. Columnar Storage: Architecture Claim: Columnar storage is suited for update-intensive applications © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 13

14. In-Memory: Aggregate Processing Time © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 14 The value of an attribute changes by calculation

15. Compression: Types © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 15

16. Dictionaries Compression:Advantages ofColumnar Storages © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 16

17. Scalability: Multiple CPU Cores Set processing is most frequent access type in EAs(scan is dominant pattern) Sequential column-wise scans show best bandwidth utilization between CPU cores and main memory Independence of tuples per column allows: easy partitioning, and parallel processing (see Hennessy [1]) Faster memory scans by improved memory bandwidth in next generation CPUs Neither materialized views nor aggregateseverything is calculated on-the-fly © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 17 [1] John L. Hennessy, David A. Patterson: Computer Architecture: A Quantitative Approach

19. Redundant dataobjectsareeliminiated

20. Neitherindicesnoraggregatesneed to bemaintained

21. Number of layersisminimized

22. No updates

23. Applicationlogicisadjacent to rawdata

24. No databaselocksrequired

25. Data movementsareminimzed

26. Sustainuse of existingresourcesApplication Cache DatabaseCache Pre-BuiltAggregates Raw Data + Stored Procedures + Mathematical Algorithms

28. Partitioning across hardware

29. Redundant-free data

30. Only few columns have high many different attribute values

32. Updates areperformed rare

33. OnlyveryfewcolumnsareaffectedbyupdatesFurtherinsightsavailable at SAP World Tour 2010 HPI booth 1.19. Insert Only

34. Agenda The Hasso Plattner Institute Technical Foundation of Columnar In-Memory Databases Impact on Enterprise Applications © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 21

35. Architecture of ExistingFinancials Systems © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 22

36. Architecture of Simplified Financials Systems © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 23 Only base tables, algorithms, and some indices

37. Analyzing Real Customer Data 1M records in BSEG ~ 1GB disk storage © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 24

38. Results:Distinct Values per Attribute Results on analyzing Financials Distinct values in accounting document headers (99 attributes) CPG Logistics Banking High Tech Discrete Manufacturing © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 25

39. Results:Accounting Document Updates Percentage of rows updated © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 26

40. Dunning © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 27

41. Available to Promise © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 28

42. Demand Planning © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 29

43. Insert Only Tuple visibility indicated by timestamps (POSTGRES-style time-travel [2]) Additional storage requirements can be neglected due to low update frequency Timestamp columns are not compressed to avoid additional merge costs Snapshot isolation Application-level locks Insert Only © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 30

44. Memory Consumption Experiments show a general factor 10 in compression (using dictionary compression and bit vector encoding) Additional storage savings by removing materialized aggregates, save ~2× Keep only the active partition of the data in memory (based on fiscal year), save ~5× Next generation blade servers will allow up to 512 GB RAM. Arrays of 100 blades already available 50 TB main memory would allow to cover the majority of SAP Business Suite customers © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 31

45. Impact on Application Development Formalized logic must be moved close to the engine Calculations must take place close to the data Reduction of application code OLTP queries must use minimal projections (SELECT * is not allowed) No caching necessary anymore © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 32

46. Conclusion Technology improvements allow re-thinking of how we build enterprise apps: A combined OLTP and OLAP system can share the same in-memory column store data base Our experiments with real applications and data prove it Open research challenges: Disaster recovery, extension for unstructured data, life cycle based data management © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 33

47. Further Information è SAP Public Web: EPIC@HPI: https://epic.hpi.uni-potsdam.de Hasso Plattner Institute: http://www.hpi-web.de © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 34

48. Thank you! Contact us! Hasso Plattner Institute EA²L / Enterprise Platform & Integration Concepts Matthieu-P. Schapranow August-Bebel-Str. 88 D-14482 Potsdam, Germany Matthieu-P. Schapranow matthieu.schapranow@hpi.uni-potsdam.de Responsible: Deputy Prof. of Prof. Hasso PlattnerDr. Alexander Zeierzeier@hpi.uni-potsdam.de © SAP 2008 / SAP TechEd 08 / <Session ID> Page 35

49. Feedback Please complete your session evaluation. Be courteous — deposit your trash, and do not take the handouts for the following session. Thank You ! © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 36