Iod 2010 1971_lohman_final

Technology Preview: Not your father’s database system Session Number 1971 Guy Lohman, PhD, IBM Keshava Murthy, IBM

[object Object],[object Object],[object Object],[object Object],[object Object],Disclaimer

Disclaimers ,[object Object],[object Object],[object Object]

IBM Smart Analytics Optimizer (ISAO) – Agenda ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Third Generation of Database Technology (is here) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Why Optimize Smart Analytics? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

What Is the IBM Smart Analytics Optimizer? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

What IBM Smart Analytics Optimizer is Designed For ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

IBM Smart Analytics Optimizer Configuration TCP/IP ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Bulk Loader SQL Queries (from apps) IBM Smart Analytics Optimizer Compressed DB partition Query Processor Data Warehouse Informix SQL (via DRDA) Query Router ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Results

IBM Smart Analytics Optimizer Architecture Informix . Accelerator Services DRDA Informix Database SQL Applications DS 5020 SAN GPFS GPFS GPFS GPFS GPFS GPFS 8 Core 48 GB Coord . 8 Core 48 GB Coord . 8 Core 48 GB Worker 8 Core 48 GB Worker 8 Core 48 GB Worker 8 Core 48 GB Worker Blade Center - S Flash Flash Flash Flash Flash Flash FC Module FC ... Ethernet Module IBM Smart Analytics Optimizer

IBM Partner testing their customer warehouse. ISAO Accelerates Most the Longest-Running Informix Queries Average Speed-up = 116x 1 hour

Initial Profiling of a data mart at a government agency. ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Getting Started: Loading Data into ISAO ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Define Transform Informix

Informix Client Application ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Informix Warehouse -- Now

Informix ISAO Appliance 2. Datamart Definition IBM Optim Data Studio 1. Identify the datamart to offload. 4. Create the metadata 5. Issue Off-load Datamart command Datamart and Data off-loading from Informix to ISAO 3. Return the SQL representation 6. Off-load the data 9. Return ACK 7. Distribute the data among blades 8. Compress the data

Informix Client Application ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],ISAO Appliance Step 3 offload SQL. DRDA over tcp/ip Step 4 Results DRDA over tcp/ip Query Processing – with Informix and ISAO 2. IDS query matching and redirection technology Local Execution

ISAO Supported Query Types ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Supported Queries in Percent per Workload 21,000 Queries from Various Workloads

IBM Smart Analytics Optimizer (ISAO) – Agenda ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

What’s the Big Deal? What’s so Disruptive? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Disruption 1 of 3 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],= 800 lbs.

Disruption 2 of 3 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Disruption 3 of 3 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Compression: Frequency Partitioning Top 64 traded goods – 6 bit code Rest Prod Origin Trade Info (volume, product, origin country) Histogram on Origin Histogram on Product Origin Product China USA GER, FRA, … Rest Table partitioned into Cells Column Partitions Vol ,[object Object],[object Object],[object Object],Cell 4 Cell 1 Cell 2 Cell 3 Cell 5 Cell 6 Common Values Rare values Number of Occurences

Query Processing ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Compressed and Partitioned Data Query Executor core + $ (HT) Dictionaries core + $ (HT) core + $ (HT) Cell 1 Cell 2 Cell 3

Fast Hash-based Grouping ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Joins ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],(1a) Dimension A scan(A) σ a IN-list of A.k 1 Hash-map: A.k 1  A.g (1b) Dimension B scan(B) σ b IN-list of B.k 2 , B.k 3 Hash-map: B.k 2 , B.k 3  B.h (2) Fact scan(F) σ f GROUP BY, Aggregation σ : F.fk 1 IN … σ : F.fk 2 , F.fk 3 IN … Look up values of g, h

IBM Smart Analytics Optimizer (ISAO) – Agenda ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Banks and Tuplets in IBM Smart Analytics Optimizer ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Bank β 1 (32 bits) Bank β 2 (32 bits) Bank β 3 (16 bits) Cell Block A 1 D 1 G 1 A 2 D 2 G 2 A 4 D 4 G 4 B 1 E 1 F 1 B 2 E 2 F 2 B 4 E 4 F 4 C 1 H 1 C 3 H 3 C 4 H 4 A 3 D 3 G 3 B 3 E 3 F 3 C 2 H 2

Register Stores Facilitate SIMD Parallelism ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],32 bits 32 bits 32 bits 32 bits 128 bits Vector Operation A 1 D 1 G 1 A 2 D 2 G 2 A 4 D 4 G 4 Bank β 1 (32 bits) A 3 D 3 G 3 B 1 E 1 F 1 B 2 E 2 F 2 B 4 E 4 F 4 C 1 H 1 C 3 H 3 C 4 H 4 Bank β 2 (32 bits) Bank β 3 (16 bits) Cell Block B 3 E 3 F 3 C 2 H 2 Result 1 Result 2 Result 3 Result 4 Operand Operand Operand Operand

Simultaneous Evaluation of Equality Predicates State==‘CA’ && Quarter == ‘Q4’ State==01001 && Quarter==1110 Translate value query to Code query Row Mask Selection result … … … … 01001 0 1110 0 == & ,[object Object],[object Object],[object Object],[object Object],State Quarter 11111 0 1111 0

IBM Smart Analytics Optimizer vs. a Column Store ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Evaluation Matches Hardware? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Updating ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Query Processing Multiple columns / word  less padding overhead Every column padded to word boundary  more padding/column  worse compression Compression IBM Smart Analytics Optimizer Column Store Aspect

Refereed Publications in Top 3 Professional Conferences ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Conclusions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Information and Analytics Communities ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Thank You! Your Feedback is Important to Us ,[object Object],[object Object],[object Object],[object Object],[object Object]

Thank You Merci Grazie Gracias Obrigado Danke Japanese English French Russian German Italian Spanish Brazilian Portuguese Arabic Traditional Chinese Simplified Chinese Hindi Tamil Thai Korean

Iod 2010 1971_lohman_final

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie Iod 2010 1971_lohman_final

Ähnlich wie Iod 2010 1971_lohman_final (20)

Mehr von Keshav Murthy

Mehr von Keshav Murthy (20)

Iod 2010 1971_lohman_final