Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Maximize Data Warehouse
Performance with
Parallel Queries
Presented by: Christo Kutrovsky
© 2015 Pythian – Presentation for DOAG
Who Am I
2
• Oracle ACE
• 15 years in Oracle field
• Joined Pythian 2003
• Part of ...
© 2015 Pythian – Presentation for DOAG3 © 2011 Pythian - Confidential
Why Companies Trust Pythian
• Recognized Leader:
• G...
© 2015 Pythian – Presentation for DOAG
Agenda
4
• Review Parallel Query concepts
• Review Exadata Data Flow capabilities
•...
5
Parallel Query Concepts
© 2015 Pythian – Presentation for DOAG
Oracle Parallel Query
6
• Producer/Consumer model
• A PQ slave is either a producer...
© 2015 Pythian – Presentation for DOAG
Oracle Parallel Query
7
• Producer/Consumer model
• A PQ slave is either a producer...
© 2015 Pythian – Presentation for DOAG
Oracle Parallel Query
8
• A PQ slave is either a producer or consumer in relation
t...
© 2015 Pythian – Presentation for DOAG
Oracle Parallel Query
9
• But logically, step-wise, when there are two sets it’s
fl...
© 2015 Pythian – Presentation for DOAG
Oracle Parallel Query
10
• QC is always flow-throught
PQ1
PQ2
PQ3
PQ4
PQ5
PQ6
PQ7
P...
© 2015 Pythian – Presentation for DOAG
Oracle Parallel Query
11
• select count(*) from table;
PQ1
PQ2
PQ3
PQ4
S
G
A
DB
ONE...
© 2015 Pythian – Presentation for DOAG
Oracle Parallel Query
12
• Producer/consumer model
• A slave is either producer and...
13
Exadata Data Flow
© 2015 Pythian – Presentation for DOAG
Multiple Tiers
14
• Every database has multiple tiers
• Memory
• SAN/NFS – network
...
© 2015 Pythian – Presentation for DOAG
Exadata X2-2 Data Flow (intra)
15
• Each DB node (compute) and CELL node (storage)
...
© 2015 Pythian – Presentation for DOAG
Exadata Data Flow (External)
16
• SQL Net Clients talk only to DB nodes
• Each DB n...
© 2015 Pythian – Presentation for DOAG
Exadata Data Flow (Half Rack)
DB1
INFINIBAND SWITCHES
DB2 DB3 DB4
CELL 01 CELL 02 C...
© 2015 Pythian – Presentation for DOAG
DISK: 2 GB/sec + FLASH: 4 GB/sec
= 6 GB/sec
X 7 cells = 42 GB/sec
Single Cell (Half...
© 2015 Pythian – Presentation for DOAG
Cell output (Half Rack)
DB1
INFINIBAND SWITCHES
DB2 DB3 DB4
CELL 01 CELL 02 CELL 03...
© 2015 Pythian – Presentation for DOAG
DB Node Consumption (Half Rack)
DB1
INFINIBAND SWITCHES
DB2 DB3 DB4
CELL 01 CELL 02...
© 2015 Pythian – Presentation for DOAG
Memory speed (Half Rack)
DB1
INFINIBAND SWITCHES
DB2 DB3 DB4
CELL 01 CELL 02 CELL 0...
© 2015 Pythian – Presentation for DOAG
DB Node to Client (Half Rack)
DB1
INFINIBAND SWITCHES
DB2 DB3 DB4
CELL 01 CELL 02 C...
© 2015 Pythian – Presentation for DOAG
Exadata Data Flow (External)
23
• Ethernet link aggregation requires
• Single conve...
© 2015 Pythian – Presentation for DOAG
Exadata Data Flow (Half Rack)
DB4 eth0
DB to THIS Client: 112 MB/sec (theoretical)
...
© 2015 Pythian – Presentation for DOAG
Exadata Data Flow (External)
25
• As per Exadata Documentation:
• Data Load Rate: U...
26
The Big Sort
© 2015 Pythian – Presentation for DOAG
Sample Data
27
• My favorite table – CKK
• 39 GiB of data
• 10,000,000 rows
• id, m...
© 2015 Pythian – Presentation for DOAG
Parallel Sort
28
select /*+ parallel(4) */ * from ckk c1 order by id;
-------------...
© 2015 Pythian – Presentation for DOAG
STORAGE FULL (OFFLOADED)
DB1
INFINIBAND SWITCHES
DB2 DB3 DB4
CELL 01 CELL 02 CELL 0...
© 2015 Pythian – Presentation for DOAG
Parallel Sort
30
select dfo_number "d", tq_id as "t", server_type, num_rows,rpad('x...
© 2015 Pythian – Presentation for DOAG
Parallel Sort
31
select /*+ parallel(4) */ * from ckk c1 order by id;
-------------...
© 2015 Pythian – Presentation for DOAG
The Ranger
32
• The Ranger Bug
• The RANGER uses a sample from the data to
determin...
© 2015 Pythian – Presentation for DOAG
Parallel Sort – random data
33
select /*+ parallel(4) */ id, mod5_id, mod5000_id, d...
© 2015 Pythian – Presentation for DOAG
Parallel Sort – random data
34
select /*+ parallel(32) */ id, mod5_id, mod5000_id, ...
© 2015 Pythian – Presentation for DOAG
Parallel Sort – random data
35
select /*+ parallel(32) */ id, mod5_id, mod5000_id, ...
© 2015 Pythian – Presentation for DOAG
The Ranger issue
36
• Distribution can be much worse
• When using analytics or othe...
© 2015 Pythian – Presentation for DOAG
Writing to disk – lack of buffering
37
• When a slave is writing to disk – it canno...
38
The Big Group By
© 2015 Pythian – Presentation for DOAG
Group By
39
select /*+ parallel(4) */ mod5_id, count(*) from ckk group by mod5_id;
...
© 2015 Pythian – Presentation for DOAG
Oracle Parallel Query
40
• But logically, step-wise, when there are two sets it’s
f...
© 2015 Pythian – Presentation for DOAG
Group by – small result set
41
select /*+ parallel(4) */ mod5_id, count(*) from ckk...
© 2015 Pythian – Presentation for DOAG
Group by – small result set
42
d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %
1 0 Produ...
© 2015 Pythian – Presentation for DOAG
Group by – small result set
43
d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %
1 0 Produ...
© 2015 Pythian – Presentation for DOAG
Group by – small result set
44
• When the result set from a group by is small – the...
© 2015 Pythian – Presentation for DOAG
select /*+ parallel(4) */ id, count(*) from ckk
group by id;
-- 4.8 sec
Group by – ...
© 2015 Pythian – Presentation for DOAG
select /*+ parallel(4) */ id, count(*) from ckk
group by id order by id;
-- 6.5 sec...
© 2015 Pythian – Presentation for DOAG
select /*+ parallel(32) */ id, count(*) from ckk
group by id;
-- 4.8 sec
Group by –...
© 2015 Pythian – Presentation for DOAG
Group by – large result set HASH
48
d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %
1 0 ...
© 2015 Pythian – Presentation for DOAG
Group by – large result set SORT
49
d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %
1 0 ...
© 2015 Pythian – Presentation for DOAG
Group by – summary
50
• Theoretical performance is reachable
• SORT based operation...
© 2015 Pythian – Presentation for DOAG
Group by – Partitioning
51
• What about partitioning?
create table ckk_p (id, mod5_...
© 2015 Pythian – Presentation for DOAG
Group by – with partitioning
52
select /*+ parallel(4) */ id,count(*) from ckk_p gr...
© 2015 Pythian – Presentation for DOAG
select /*+ parallel(64) */ * from ckk c1 order by id;
-----------------------------...
© 2015 Pythian – Presentation for DOAG
Analytics don’t speak parallel
54
select /*+ parallel(4) */ id, row_number() over (...
55
The Big Join
© 2015 Pythian – Presentation for DOAG
Test Data
56
• Copy of ckk – ckk2
• ckk_m5 – 5 rows dimension
create table ckk_m5 a...
© 2015 Pythian – Presentation for DOAG
Simple Hash Join
select /*+ parallel(4) */ * from ckk c1 join ckk_m5 m5 on (c1.mod5...
© 2015 Pythian – Presentation for DOAG
Simple Hash Join
58
PQ1
PQ2
PQ3
PQ4
S
G
A
STEP (1)
PQ5
PQ6
PQ7
PQ8
HASH
PGA
HASH
PG...
© 2015 Pythian – Presentation for DOAG
Simple Hash Join
59
PQ5
PQ6
PQ7
PQ8
HASH
PGA
HASH
PGA
HASH
PGA
HASH
PGA
S
G
A
STEP ...
© 2015 Pythian – Presentation for DOAG
Big Hash Join
select /*+ parallel(4) */ * from ckk c1join ckk2 c2 on (c1.id = c2.id...
© 2015 Pythian – Presentation for DOAG
Big Hash Join
61
PQ1
PQ2
PQ3
PQ4
S
G
A
STEP (1)
PQ5
PQ6
PQ7
PQ8
HASH
PGA
HASH
PGA
H...
© 2015 Pythian – Presentation for DOAG
Big Hash Join
62
PQ1
PQ2
PQ3
PQ4
S
G
A
STEP (2)
PQ5
PQ6
PQ7
PQ8
HASH
PGA
HASH
PGA
H...
© 2015 Pythian – Presentation for DOAG
Simple Hash Join
63
PQ5
PQ6
PQ7
PQ8
HASH
PGA
HASH
PGA
HASH
PGA
HASH
PGA
S
G
A
STEP ...
© 2015 Pythian – Presentation for DOAG
Hash Join summary
64
• HASH JOINS with small tables are extremely fast
• Uses BROAD...
© 2015 Pythian – Presentation for DOAG
Semi-Partitioned Hash Join
select /*+ parallel(32) */ * from ckk c1 join ckk_p c2 o...
© 2015 Pythian – Presentation for DOAG
Semi Partitioned Hash Join
66
PQ1
PQ2
PQ3
PQ4
S
G
A
STEP (1)
PQ5
PQ6
PQ7
PQ8
HASH
P...
© 2015 Pythian – Presentation for DOAG
Semi Partitioned Hash Join
67
PQ5
PQ6
PQ7
PQ8
HASH
PGA
HASH
PGA
HASH
PGA
HASH
PGA
S...
© 2015 Pythian – Presentation for DOAG
Semi Partitioned Hash Join
68
• Dramatically reduces HASH Area sizes
• Very likely ...
© 2015 Pythian – Presentation for DOAG
Partitioned Hash Join
select /*+ parallel(4) */ * from ckk_p c1 join ckk_p c2 on (c...
© 2015 Pythian – Presentation for DOAG
Partitioned Hash Join
select /*+ parallel(4) */ * from ckk_p c1 join ckk_p c2 on (c...
© 2015 Pythian – Presentation for DOAG
Switching places
select /*+ parallel(32) */ * from ckk_p c1 join ckk c2 on (c1.id =...
© 2015 Pythian – Presentation for DOAG
Outer Joins
select /*+ parallel(32) */ * from ckk_p c1 left join ckk c2 on (c1.id =...
73
The Big Join – with Index
© 2015 Pythian – Presentation for DOAG
Adding a few indexes
74
create index ckk$id on ckk(id)
parallel 32 tablespace ckbig...
© 2015 Pythian – Presentation for DOAG
Direct index access
select /*+ parallel(4) */ * from ckk where mod5000_id=100;
----...
© 2015 Pythian – Presentation for DOAG
Join with Index
select /*+ parallel(4) */ * from ckk c1 join ckk2 c2 on (c1.id = c2...
© 2015 Pythian – Presentation for DOAG
Indexes summary
77
• Parallel access on direct value queries only on
partitioned ta...
© 2015 Pythian – Presentation for DOAG
Final Thoughts
78
• Exadata allows Parallel Query to make sense
• High Bandwidth ev...
© 2009/2010 Pythian - Presentation for ABC Company79
The End
Thank You
Questions?
I blog at
http://www.pythian.com/news/au...
Nächste SlideShare
Wird geladen in …5
×

Christo Kutrovsky - Maximize Data Warehouse Performance with Parallel Queries

Oracle Data Warehouses are typically deployed on servers with very large number of cores, and increasingly on RAC. Making efficient use of all available cores when processing data warehouse workloads is therefore critical in achieving maximal performance. To make efficient use of all cores in a data warehouse system, skilled use of parallel queries is key.

  • Als Erste(r) kommentieren

Christo Kutrovsky - Maximize Data Warehouse Performance with Parallel Queries

  1. 1. Maximize Data Warehouse Performance with Parallel Queries Presented by: Christo Kutrovsky
  2. 2. © 2015 Pythian – Presentation for DOAG Who Am I 2 • Oracle ACE • 15 years in Oracle field • Joined Pythian 2003 • Part of Pythian Consulting Group • Special projects • Performance tuning • Critical services • Presenter at: IOUG, RMOUG, UKOUG, OpenWorld “oracle pinup”
  3. 3. © 2015 Pythian – Presentation for DOAG3 © 2011 Pythian - Confidential Why Companies Trust Pythian • Recognized Leader: • Global industry-leader in remote database administration services and consulting for Oracle, Oracle Applications, MySQL and SQL Server • Work with over 150 multinational companies such as Forbes.com, Fox Sports, Nordion and Western Union to help manage their complex IT deployments • Expertise: • One of the world’s largest concentrations of dedicated, full-time DBA expertise. Employ 7 Oracle ACEs/ACE Directors. • Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle GoldenGate & Oracle RAC. • Global Reach & Scalability: • 24/7/365 global remote support for DBA and consulting, systems administration, special projects or emergency response
  4. 4. © 2015 Pythian – Presentation for DOAG Agenda 4 • Review Parallel Query concepts • Review Exadata Data Flow capabilities • The Big Sort • The Big Group By • The Big Join • Final Thoughts
  5. 5. 5 Parallel Query Concepts
  6. 6. © 2015 Pythian – Presentation for DOAG Oracle Parallel Query 6 • Producer/Consumer model • A PQ slave is either a producer or consumer, never both Figure 8-2 Inter-operation Parallelism and Dynamic Partitioning Oracle® Database VLDB and Partitioning Guide
  7. 7. © 2015 Pythian – Presentation for DOAG Oracle Parallel Query 7 • Producer/Consumer model • A PQ slave is either a producer or consumer, never both Figure 8-2 Inter-operation Parallelism and Dynamic Partitioning Oracle® Database VLDB and Partitioning Guide
  8. 8. © 2015 Pythian – Presentation for DOAG Oracle Parallel Query 8 • A PQ slave is either a producer or consumer in relation to other PQ slaves PQ1 PQ2 PQ3 PQ4 PQ5 PQ6 PQ7 PQ8 S G A DB
  9. 9. © 2015 Pythian – Presentation for DOAG Oracle Parallel Query 9 • But logically, step-wise, when there are two sets it’s flow-through PQ1 PQ2 PQ3 PQ4 PQ5 PQ6 PQ7 PQ8 S G A DB ONE STEP (1)
  10. 10. © 2015 Pythian – Presentation for DOAG Oracle Parallel Query 10 • QC is always flow-throught PQ1 PQ2 PQ3 PQ4 PQ5 PQ6 PQ7 PQ8 S G A DB S G A QC SQLNET CLIENTONE STEP (2)
  11. 11. © 2015 Pythian – Presentation for DOAG Oracle Parallel Query 11 • select count(*) from table; PQ1 PQ2 PQ3 PQ4 S G A DB ONE STEP QC SQLNET CLIENT
  12. 12. © 2015 Pythian – Presentation for DOAG Oracle Parallel Query 12 • Producer/consumer model • A slave is either producer and consumer in relation to other slaves.
  13. 13. 13 Exadata Data Flow
  14. 14. © 2015 Pythian – Presentation for DOAG Multiple Tiers 14 • Every database has multiple tiers • Memory • SAN/NFS – network • SAN Performance • CPU Capacity • Client side connectivity • Each tier has it’s own nuances, properties and bandwidth limitations
  15. 15. © 2015 Pythian – Presentation for DOAG Exadata X2-2 Data Flow (intra) 15 • Each DB node (compute) and CELL node (storage) • Single dual port 40 Gbit infiniband (QDR) • 3,200 Mb/sec (full-duplex) • Each CELL node • Single dual port 40 Gbit infiniband (QDR) • 3,200 Mb/sec (full-duplex) • 12 x 4 TB SAS drives (7200 rpm) • Total bandwidth: 1,000 - 2,000 Mb/sec • 4 x Sun Flash Arrays • Total bandwidth: 4,100 Mb/sec
  16. 16. © 2015 Pythian – Presentation for DOAG Exadata Data Flow (External) 16 • SQL Net Clients talk only to DB nodes • Each DB node • 4 x 10 gigabit network ( 4 x 1.08 Gbytes/sec) • 2 x 10 Gigabit network (opt) ( 2 x 1.08 Gbytes/sec) • Alternatively, can use IP over infiniband • Exalogic • RMAN backups • Etc.
  17. 17. © 2015 Pythian – Presentation for DOAG Exadata Data Flow (Half Rack) DB1 INFINIBAND SWITCHES DB2 DB3 DB4 CELL 01 CELL 02 CELL 03 CELL 04 CELL 05 CELL 06 CELL 07 SGA SGA SGA SGA CLIENT RESULT SET
  18. 18. © 2015 Pythian – Presentation for DOAG DISK: 2 GB/sec + FLASH: 4 GB/sec = 6 GB/sec X 7 cells = 42 GB/sec Single Cell (Half Rack) DB1 INFINIBAND SWITCHES DB2 DB3 DB4 CELL 02 CELL 03 CELL 04 CELL 05 CELL 06 CELL 07 SGA SGA SGA SGA CLIENT RESULT SET CELL 01
  19. 19. © 2015 Pythian – Presentation for DOAG Cell output (Half Rack) DB1 INFINIBAND SWITCHES DB2 DB3 DB4 CELL 01 CELL 02 CELL 03 CELL 04 CELL 05 CELL 06 CELL 07 SGA SGA SGA SGA CLIENT RESULT SET CELL to SWITCH: 3.2 GB/sec X 7 cells = 22.4 GB/sec
  20. 20. © 2015 Pythian – Presentation for DOAG DB Node Consumption (Half Rack) DB1 INFINIBAND SWITCHES DB2 DB3 DB4 CELL 01 CELL 02 CELL 03 CELL 04 CELL 05 CELL 06 CELL 07 SGA SGA SGA SGA CLIENT RESULT SET SWITCH to DB NODE: 3.2 GB/sec X 4 nodes = 12.8 GB/sec
  21. 21. © 2015 Pythian – Presentation for DOAG Memory speed (Half Rack) DB1 INFINIBAND SWITCHES DB2 DB3 DB4 CELL 01 CELL 02 CELL 03 CELL 04 CELL 05 CELL 06 CELL 07 SGA SGA SGA SGA CLIENT RESULT SET DB to SGA (memory) 22 GB/sec (very CPU dependant) X 4 DB = 88 GB/sec
  22. 22. © 2015 Pythian – Presentation for DOAG DB Node to Client (Half Rack) DB1 INFINIBAND SWITCHES DB2 DB3 DB4 CELL 01 CELL 02 CELL 03 CELL 04 CELL 05 CELL 06 CELL 07 SGA SGA SGA SGA CLIENT RESULT SET DB to SINGLE SQLNET Client 112 MB/sec (1 GBIT) 1.08 GB/sec (10 GBIT)
  23. 23. © 2015 Pythian – Presentation for DOAG Exadata Data Flow (External) 23 • Ethernet link aggregation requires • Single conversation runs on a single path • Ethernet standard • One database sessions runs on single TCP connection, which must run on single path • Switches and network cards can implement different hashing algorithms to determine paths • Most are on IP level. Between 2 IPs the same NICs are used, regardless of number of connections
  24. 24. © 2015 Pythian – Presentation for DOAG Exadata Data Flow (Half Rack) DB4 eth0 DB to THIS Client: 112 MB/sec (theoretical) (18 MB/sec per session) IP: 10.31.33.1 Session Session Session Session Session Session eth1 eth2 eth3 eth0 eth1 eth2 eth3 S w i t c h
  25. 25. © 2015 Pythian – Presentation for DOAG Exadata Data Flow (External) 25 • As per Exadata Documentation: • Data Load Rate: Up to 10.5 TB/hour • 764 MB/sec per DB node • 3 GB/sec – can your other infrastructure provide such performance?
  26. 26. 26 The Big Sort
  27. 27. © 2015 Pythian – Presentation for DOAG Sample Data 27 • My favorite table – CKK • 39 GiB of data • 10,000,000 rows • id, mod5_id, mod5000_id, dt_dec, dt_pos etc. create table ckk /*parallel 8 */nologging tablespace ckbig as select rownum id, mod(rownum,5) mod5_id, mod(rownum,5000) mod5000_id, sysdate dt_fixed, sysdate - rownum/24/60 dt_dec, sysdate + rownum/24/60 dt_pos, sysdate + ora_hash(rownum,65,535)/24 dt_rand, sysdate+mod(rownum,10) dt_mod10, rpad('x',3500,'x') filler from (select rownum r from dual connect by level <= 10000) r1, (select rownum r from dual connect by level <= 1000) ;
  28. 28. © 2015 Pythian – Presentation for DOAG Parallel Sort 28 select /*+ parallel(4) */ * from ckk c1 order by id; ---------------------------------------------------------------------------------------------------------- | Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib | ---------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 10M| 33G| | | | | | 1 | PX COORDINATOR | | | | | | | | | 2 | PX SEND QC (ORDER) | :TQ10001 | 10M| 33G| | Q1,01 | P->S | QC (ORDER) | | 3 | SORT ORDER BY | | 10M| 33G| 38G| Q1,01 | PCWP | | | 4 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | | | 5 | PX SEND RANGE | :TQ10000 | 10M| 33G| | Q1,00 | P->P | RANGE | | 6 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | | | 7 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | | ---------------------------------------------------------------------------------------------------------- PQ1 PQ2 PQ3 PQ4 PQ5 – up ID=2.5 M PQ6 – up ID=5.0 M PQ7 – up ID=7.5 M PQ8 – up ID=10 M S G A DB
  29. 29. © 2015 Pythian – Presentation for DOAG STORAGE FULL (OFFLOADED) DB1 INFINIBAND SWITCHES DB2 DB3 DB4 CELL 01 CELL 02 CELL 03 CELL 04 CELL 05 CELL 06 CELL 07 SGA SGA SGA SGA CLIENT RESULT SET Decompression, Projection and Filter Happens on EVERY CELL NODE regardless of query parallelism. MPP mode.
  30. 30. © 2015 Pythian – Presentation for DOAG Parallel Sort 30 select dfo_number "d", tq_id as "t", server_type, num_rows,rpad('x',round(num_rows*10/nullif(max(num_rows) over (partition by dfo_number, tq_id, server_type),0)),'x') as "pr", round(bytes/1024/1024) mb, process, instance i,round(ratio_to_report (num_rows) over (partition by dfo_number, tq_id, server_type)*100) as "%", open_time, avg_latency, waits, timeouts,round(bytes/nullif(num_rows,0)) as "b/r” from v$pq_tqstat order by dfo_number, tq_id, server_type desc, process;; d t SERVER_TYPE NUM_ROWS pr MB PROCESS I % 1 0 Ranger 372 xxxxxxxxxx 2 QC 1 100 1 0 Producer 2,692,340 xxxxxxxxxx 9,161 P002 1 27 1 0 Producer 2,499,910 xxxxxxxxx 8,506 P002 2 25 1 0 Producer 2,692,340 xxxxxxxxxx 9,161 P003 2 27 1 0 Producer 2,115,410 xxxxxxxx 7,198 P003 1 21 1 0 Consumer 6,146,614 xxxxxxxxxx 20,915 P000 2 61 1 0 Consumer 781,680 x 2,660 P000 1 8 1 0 Consumer 1,968,128 xxx 6,697 P001 2 20 1 0 Consumer 1,103,578 xx 3,755 P001 1 11 1 1 Producer 13 x 0 P000 1 5 1 1 Producer 212 xxxxxxxxxx 1 P000 2 84 1 1 Producer 13 x 0 P001 2 5 1 1 Producer 13 x 0 P001 1 5 1 1 Consumer 200 xxxxxxxxxx 1 QC 1 100
  31. 31. © 2015 Pythian – Presentation for DOAG Parallel Sort 31 select /*+ parallel(4) */ * from ckk c1 order by id; ---------------------------------------------------------------------------------------------------------- | Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib | ---------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 10M| 33G| | | | | | 1 | PX COORDINATOR | | | | | | | | | 2 | PX SEND QC (ORDER) | :TQ10001 | 10M| 33G| | Q1,01 | P->S | QC (ORDER) | | 3 | SORT ORDER BY | | 10M| 33G| 38G| Q1,01 | PCWP | | | 4 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | | | 5 | PX SEND RANGE | :TQ10000 | 10M| 33G| | Q1,00 | P->P | RANGE | | 6 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | | | 7 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | | ---------------------------------------------------------------------------------------------------------- PQ1 PQ2 PQ3 PQ4 PQ5 – up to ID=6.1 M (6.1 M) PQ6 – up to ID=6.9 M (0.8 M) PQ7 – up to ID=8.8 M (1.9 M) PQ8 – up to ID=10.0 M (1.1M S G A DB
  32. 32. © 2015 Pythian – Presentation for DOAG The Ranger 32 • The Ranger Bug • The RANGER uses a sample from the data to determine the boundaries for PQ sort operations • Sometimes it’s off by a lot • It will ALWAYS miss when the input data is already somewhat sorted • Limit your scalability – no matter how big your machine is
  33. 33. © 2015 Pythian – Presentation for DOAG Parallel Sort – random data 33 select /*+ parallel(4) */ id, mod5_id, mod5000_id, dt_fixed, dt_dec, dt_pos, dt_rand, dt_mod10, substr(filler,1,2000) from ckk c1 order by dt_rand; --90 sec d t SERVER_TYPE NUM_ROWS pr MB PROCESS I % 1 0 Ranger 372 xxxxxxxxxx 1 QC 1 100 1 0 Producer 2,500,030 xxxxxxxxxx 4,925 P002 1 25 1 0 Producer 2,500,030 xxxxxxxxxx 4,925 P002 2 25 1 0 Producer 2,499,910 xxxxxxxxxx 4,925 P003 1 25 1 0 Producer 2,500,030 xxxxxxxxxx 4,925 P003 2 25 1 0 Consumer 2,424,554 xxxxxxxxx 4,776 P000 2 24 1 0 Consumer 2,576,164 xxxxxxxxx 5,075 P000 1 26 1 0 Consumer 2,272,136 xxxxxxxx 4,476 P001 2 23 1 0 Consumer 2,727,146 xxxxxxxxxx 5,373 P001 1 27 1 1 Producer 218 xxxxxxxxxx 0 P000 1 76 1 1 Producer 23 x 0 P000 2 8 1 1 Producer 23 x 0 P001 2 8 1 1 Producer 23 x 0 P001 1 8 1 1 Consumer 200 xxxxxxxxxx 0 QC 1 100
  34. 34. © 2015 Pythian – Presentation for DOAG Parallel Sort – random data 34 select /*+ parallel(32) */ id, mod5_id, mod5000_id, dt_fixed, dt_dec, dt_pos, dt_rand, dt_mod10, substr(filler,1,2000) from ckk c1 order by dt_rand; --10 sec d t SERVER_TYPE NUM_ROWS pr MB PROCESS I % 1 0 Ranger 3,872 xxxxxxxxxx 11 QC 1 100 1 0 Producer 312,520 xxxxxxxxxx 616 P016 1 3 1 0 Producer 312,520 xxxxxxxxxx 616 P016 2 3 1 0 Producer 312,520 xxxxxxxxxx 616 P017 1 3 1 0 Producer 312,520 xxxxxxxxxx 616 P017 2 3 ... 1 0 Consumer 151,262 xxx 298 P003 1 2 1 0 Consumer 454,696 xxxxxxxxxx 896 P003 2 5 1 0 Consumer 302,320 xxxxxxx 596 P004 1 3 1 0 Consumer 151,731 xxx 299 P004 2 2 1 0 Consumer 303,429 xxxxxxx 598 P005 1 3 1 0 Consumer 454,650 xxxxxxxxxx 896 P005 2 5 1 0 Consumer 303,259 xxxxxxx 597 P006 2 3 1 0 Consumer 455,172 xxxxxxxxxx 897 P006 1 5 1 0 Consumer 302,945 xxxxxxx 597 P012 1 3 1 0 Consumer 303,318 xxxxxxx 598 P012 2 3 1 0 Consumer 303,245 xxxxxxx 597 P013 1 3 1 0 Consumer 304,029 xxxxxxx 599 P013 2 3 1 0 Consumer 301,937 xxxxxxx 595 P014 2 3 1 0 Consumer 454,903 xxxxxxxxxx 896 P014 1 5 1 0 Consumer 151,887 xxx 299 P015 1 2 1 1 1 Consumer 200 xxxxxxxxxx 0 QC 1 100
  35. 35. © 2015 Pythian – Presentation for DOAG Parallel Sort – random data 35 select /*+ parallel(32) */ id, mod5_id, mod5000_id, dt_fixed, dt_dec, dt_pos, dt_rand, dt_mod10, substr(filler,1,2000) from ckk c1 order by id; --20 sec d t SERVER_TYPE NUM_ROWS pr MB PROCESS I % 1 0 Ranger 3,872 xxxxxxxxxx 11 QC 1 100 1 0 Producer 240,400 xxxxxx 474 P016 2 2 1 0 Producer 384,640 xxxxxxxxx 758 P016 1 4 1 0 Producer 216,360 xxxxx 426 P017 1 2 1 0 Producer 192,320 xxxx 379 P017 2 2 ... 1 0 Consumer 373,702 xxxx 736 P000 1 4 1 0 Consumer 372,530 xxxx 734 P000 2 4 1 0 Consumer 124,801 x 246 P001 2 1 1 0 Consumer 183,552 xx 361 P001 1 2 1 0 Consumer 734,398 xxxxxxx 1,446 P002 1 7 1 0 Consumer 22,402 44 P002 2 0 1 0 Consumer 191,810 xx 378 P003 1 2 1 0 Consumer 132,990 x 262 P003 2 1 1 0 Consumer 32,640 64 P004 1 0 1 0 Consumer 920,448 xxxxxxxxx 1,813 P004 2 9 1 0 Consumer 167,232 xx 329 P005 1 2 1 0 Consumer 110,466 x 218 P005 2 1 1 0 Consumer 1,055,669 xxxxxxxxxx 2,080 P014 2 11 1 0 Consumer 130,944 x 258 P014 1 1 1 0 Consumer 60,250 x 119 P015 2 1 1 0 Consumer 20,298 40 P015 1 0 ... 1 1 Consumer 200 xxxxxxxxxx 0 QC 1 100
  36. 36. © 2015 Pythian – Presentation for DOAG The Ranger issue 36 • Distribution can be much worse • When using analytics or other “input” into the sort operation that is pre-sorted • Often 90% goes to one slave, 10% to remaining • “Been worked on” – I am told • Problem made even worse due to lack of large buffers
  37. 37. © 2015 Pythian – Presentation for DOAG Writing to disk – lack of buffering 37 • When a slave is writing to disk – it cannot receive any messages or can receive a very small number of messages • Results in slowdowns when Oracle has to write to TEMP • Massive slowdown when only a few slaves have to write to TEMP • All producers have to wait • Significant slowdown when all slaves have to write to TEMP • Cannot parallelise sufficiently to leverage available disk bandwidth
  38. 38. 38 The Big Group By
  39. 39. © 2015 Pythian – Presentation for DOAG Group By 39 select /*+ parallel(4) */ mod5_id, count(*) from ckk group by mod5_id; --------------------------------------------------------------------------------------------------- | Id | Operation | Name | E-Rows |E-Bytes| TQ |IN-OUT| PQ Distrib | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 5 | 15 | | | | | 1 | PX COORDINATOR | | | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10001 | 5 | 15 | Q1,01 | P->S | QC (RAND) | | 3 | HASH GROUP BY | | 5 | 15 | Q1,01 | PCWP | | | 4 | PX RECEIVE | | 5 | 15 | Q1,01 | PCWP | | | 5 | PX SEND HASH | :TQ10000 | 5 | 15 | Q1,00 | P->P | HASH | | 6 | HASH GROUP BY | | 5 | 15 | Q1,00 | PCWP | | | 7 | PX BLOCK ITERATOR | | 10M| 28M| Q1,00 | PCWC | | | 8 | TABLE ACCESS STORAGE FULL| CKK | 10M| 28M| Q1,00 | PCWP | | --------------------------------------------------------------------------------------------------- select /*+ parallel(4) NO_GBY_PUSHDOWN*/ mod5_id, count(*) from ckk group by mod5_id; -------------------------------------------------------------------------------------------------- | Id | Operation | Name | E-Rows |E-Bytes| TQ |IN-OUT| PQ Distrib | -------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 5 | 15 | | | | | 1 | PX COORDINATOR | | | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10001 | 5 | 15 | Q1,01 | P->S | QC (RAND) | | 3 | HASH GROUP BY | | 5 | 15 | Q1,01 | PCWP | | | 4 | PX RECEIVE | | 10M| 28M| Q1,01 | PCWP | | | 5 | PX SEND HASH | :TQ10000 | 10M| 28M| Q1,00 | P->P | HASH | | 6 | PX BLOCK ITERATOR | | 10M| 28M| Q1,00 | PCWC | | | 7 | TABLE ACCESS STORAGE FULL| CKK | 10M| 28M| Q1,00 | PCWP | | --------------------------------------------------------------------------------------------------
  40. 40. © 2015 Pythian – Presentation for DOAG Oracle Parallel Query 40 • But logically, step-wise, when there are two sets it’s flow-through PQ1 PQ2 PQ3 PQ4 S G A DB STEP (1) GROUP PGA GROUP PGA GROUP PGA GROUP PGA PQ5 PQ6 PQ7 PQ8 GROUP PGA GROUP PGA GROUP PGA GROUP PGA S G A QCSTEP (2)
  41. 41. © 2015 Pythian – Presentation for DOAG Group by – small result set 41 select /*+ parallel(4) */ mod5_id, count(*) from ckk group by mod5_id; -- 4.8 sec d t SERVER_TYPE NUM_ROWS pr MB PROCESS I % 1 0 Producer 5 xxxxxxxxxx 0 P002 2 25 1 0 Producer 5 xxxxxxxxxx 0 P002 1 25 1 0 Producer 5 xxxxxxxxxx 0 P003 2 25 1 0 Producer 5 xxxxxxxxxx 0 P003 1 25 1 0 Consumer 4 xxxxx 0 P000 2 20 1 0 Consumer 8 xxxxxxxxxx 0 P000 1 40 1 0 Consumer 4 xxxxx 0 P001 2 20 1 0 Consumer 4 xxxxx 0 P001 1 20 1 1 Producer 2 xxxxxxxxxx 0 P000 1 40 1 1 Producer 1 xxxxx 0 P000 2 20 1 1 Producer 1 xxxxx 0 P001 1 20 1 1 Producer 1 xxxxx 0 P001 2 20 1 1 Consumer 5 xxxxxxxxxx 0 QC 1 100
  42. 42. © 2015 Pythian – Presentation for DOAG Group by – small result set 42 d t SERVER_TYPE NUM_ROWS pr MB PROCESS I % 1 0 Producer 2,692,340 xxxxxxxxxx 10 P004 2 27 1 0 Producer 2,307,600 xxxxxxxxx 8 P004 1 23 1 0 Producer 2,500,030 xxxxxxxxx 9 P005 2 25 1 0 Producer 2,500,030 xxxxxxxxx 9 P005 1 25 1 0 Consumer 4,000,000 xxxxxxxxxx 15 P002 2 40 1 0 Consumer 2,000,000 xxxxx 8 P002 1 20 1 0 Consumer 2,000,000 xxxxx 6 P003 2 20 1 0 Consumer 2,000,000 xxxxx 8 P003 1 20 1 1 Producer 1 xxxxx 0 P002 1 20 1 1 Producer 2 xxxxxxxxxx 0 P002 2 40 1 1 Producer 1 xxxxx 0 P003 1 20 1 1 Producer 1 xxxxx 0 P003 2 20 1 1 Consumer 5 xxxxxxxxxx 0 QC 1 100 select /*+ parallel(4) NO_GBY_PUSHDOWN*/ mod5_id, count(*) from ckk group by mod5_id; -- 4.8 sec
  43. 43. © 2015 Pythian – Presentation for DOAG Group by – small result set 43 d t SERVER_TYPE NUM_ROWS pr MB PROCESS I % 1 0 Producer 264,440 xxxxxx 1 P016 2 3 1 0 Producer 312,520 xxxxxxxx 1 P016 1 3 1 0 Producer 240,400 xxxxxx 1 P017 1 2 1 0 Producer 360,600 xxxxxxxxx 1 P017 2 4 ... 1 0 Consumer 0 0 P000 2 0 1 0 Consumer 2,000,000 xxxxxxxxxx 8 P000 1 20 1 0 Consumer 0 0 P001 2 0 ... 1 0 Consumer 0 0 P004 1 0 1 0 Consumer 0 0 P005 2 0 1 0 Consumer 2,000,000 xxxxxxxxxx 6 P005 1 20 1 0 Consumer 0 0 P006 2 0 1 0 Consumer 0 0 P006 1 0 1 0 Consumer 0 0 P007 2 0 1 0 Consumer 2,000,000 xxxxxxxxxx 8 P007 1 20 1 0 Consumer 2,000,000 xxxxxxxxxx 8 P008 1 20 1 0 Consumer 2,000,000 xxxxxxxxxx 8 P010 2 20 ... select /*+ parallel(32) NO_GBY_PUSHDOWN*/ mod5_id, count(*) from ckk group by mod5_id; -- 4.8 sec
  44. 44. © 2015 Pythian – Presentation for DOAG Group by – small result set 44 • When the result set from a group by is small – there are a lot of optimisations that can kick in • PUSHDOWN saves on PQ traffic • Receiving end has very little work to do • Very efficient to parallelise
  45. 45. © 2015 Pythian – Presentation for DOAG select /*+ parallel(4) */ id, count(*) from ckk group by id; -- 4.8 sec Group by – large result set HASH 45 d t SERVER_TYPE NUM_ROWS pr MB PROCESS I % 1 0 Producer 2,500,030 xxxxxxxx 17 P002 2 25 1 0 Producer 2,115,410 xxxxxx 14 P002 1 21 1 0 Producer 2,115,410 xxxxxx 14 P003 2 21 1 0 Producer 3,269,150 xxxxxxxxxx 21 P003 1 33 1 0 Consumer 2,499,213 xxxxxxxxxx 16 P000 1 25 1 0 Consumer 2,499,807 xxxxxxxxxx 16 P000 2 25 1 0 Consumer 2,500,981 xxxxxxxxxx 16 P001 1 25 1 0 Consumer 2,499,999 xxxxxxxxxx 16 P001 2 25 1 1 Producer 1,465 xxxxx 0 P000 2 17 1 1 Producer 2,929 xxxxxxxxxx 0 P000 1 33 1 1 Producer 2,928 xxxxxxxxxx 0 P001 1 33 1 1 Producer 1,464 xxxxx 0 P001 2 17 1 1 Consumer 200 xxxxxxxxxx 0 QC 1 100
  46. 46. © 2015 Pythian – Presentation for DOAG select /*+ parallel(4) */ id, count(*) from ckk group by id order by id; -- 6.5 sec Group by – large result set SORT 46 d t SERVER_TYPE NUM_ROWS pr MB PROCESS I % 1 0 Ranger 372 xxxxxxxxxx 0 QC 1 100 1 0 Producer 2,499,910 xxxxxxxxx 17 P002 1 25 1 0 Producer 2,692,340 xxxxxxxxxx 18 P002 2 27 1 0 Producer 2,500,030 xxxxxxxxx 17 P003 2 25 1 0 Producer 2,307,720 xxxxxxxxx 15 P003 1 23 1 0 Consumer 6,154,678 xxxxxxxxxx 40 P000 2 62 1 0 Consumer 793,712 x 5 P000 1 8 1 0 Consumer 1,095,514 xx 7 P001 1 11 1 0 Consumer 1,956,096 xxx 13 P001 2 20 1 1 Producer 4,391 xxxxxxxx 0 P000 1 24 1 1 Producer 5,379 xxxxxxxxxx 0 P000 2 29 1 1 Producer 4,391 xxxxxxxx 0 P001 2 24 1 1 Producer 4,391 xxxxxxxx 0 P001 1 24 1 1 Consumer 200 xxxxxxxxxx 0 QC 1 100
  47. 47. © 2015 Pythian – Presentation for DOAG select /*+ parallel(32) */ id, count(*) from ckk group by id; -- 4.8 sec Group by – large result set HASH 47 d t SERVER_TYPE NUM_ROWS pr MB PROCESS I % 1 0 Producer 360,600 xxxxxxxxx 2 P016 1 4 1 0 Producer 240,400 xxxxxx 2 P016 2 2 1 0 Producer 312,520 xxxxxxxx 2 P017 1 3 1 0 Producer 336,560 xxxxxxxxx 2 P017 2 3 ... 1 0 Consumer 312,829 xxxxxxxxxx 2 P000 2 3 1 0 Consumer 311,690 xxxxxxxxxx 2 P000 1 3 1 0 Consumer 312,371 xxxxxxxxxx 2 P001 1 3 1 0 Consumer 312,854 xxxxxxxxxx 2 P001 2 3 1 0 Consumer 312,255 xxxxxxxxxx 2 P002 2 3 1 0 Consumer 312,321 xxxxxxxxxx 2 P002 1 3 1 0 Consumer 313,241 xxxxxxxxxx 2 P003 2 3 1 0 Consumer 312,605 xxxxxxxxxx 2 P003 1 3 1 0 Consumer 312,202 xxxxxxxxxx 2 P004 1 3 1 0 Consumer 311,294 xxxxxxxxxx 2 P004 2 3 1 0 Consumer 312,338 xxxxxxxxxx 2 P005 1 3 1 0 Consumer 312,206 xxxxxxxxxx 2 P005 2 3 1 0 Consumer 312,898 xxxxxxxxxx 2 P006 1 3 1 0 Consumer 312,817 xxxxxxxxxx 2 P006 2 3 1 0 Consumer 312,291 xxxxxxxxxx 2 P007 1 3 1 0 Consumer 312,872 xxxxxxxxxx 2 P007 2 3 ... 1 1 Consumer 200 xxxxxxxxxx 0 QC 1 100
  48. 48. © 2015 Pythian – Presentation for DOAG Group by – large result set HASH 48 d t SERVER_TYPE NUM_ROWS pr MB PROCESS I % 1 0 Producer 312,520 xxxxxxxxxx 1,047 P016 1 3 1 0 Producer 312,520 xxxxxxxxxx 1,047 P016 2 3 1 0 Producer 312,520 xxxxxxxxxx 1,047 P017 1 3 ... 1 0 Consumer 311,603 xxxxxxxxxx 1,044 P000 1 3 1 0 Consumer 312,470 xxxxxxxxxx 1,047 P000 2 3 1 0 Consumer 312,484 xxxxxxxxxx 1,047 P001 2 3 1 0 Consumer 312,662 xxxxxxxxxx 1,048 P001 1 3 1 0 Consumer 312,341 xxxxxxxxxx 1,047 P002 2 3 1 0 Consumer 312,692 xxxxxxxxxx 1,048 P002 1 3 1 0 Consumer 313,114 xxxxxxxxxx 1,049 P003 2 3 1 0 Consumer 311,889 xxxxxxxxxx 1,045 P003 1 3 1 0 Consumer 312,323 xxxxxxxxxx 1,047 P004 1 3 1 0 Consumer 313,310 xxxxxxxxxx 1,050 P004 2 3 1 0 Consumer 312,715 xxxxxxxxxx 1,048 P005 2 3 1 0 Consumer 313,394 xxxxxxxxxx 1,050 P005 1 3 1 0 Consumer 312,718 xxxxxxxxxx 1,048 P006 1 3 1 0 Consumer 312,090 xxxxxxxxxx 1,046 P006 2 3 1 0 Consumer 312,720 xxxxxxxxxx 1,048 P007 1 3 1 0 Consumer 311,889 xxxxxxxxxx 1,045 P007 2 3 2 ... 1 1 Consumer 200 xxxxxxxxxx 1 QC 1 100 select /*+ parallel(32) */ id, filler, count(*) from ckk group by id,filler; -- 18 sec
  49. 49. © 2015 Pythian – Presentation for DOAG Group by – large result set SORT 49 d t SERVER_TYPE NUM_ROWS pr MB PROCESS I % 1 0 Ranger 3,872 xxxxxxxxxx 23 QC 1 100 1 0 Producer 288,480 xxxxxxx 967 P016 2 3 1 0 Producer 288,480 xxxxxxx 967 P016 1 3 1 0 Producer 336,560 xxxxxxxx 1,128 P017 2 3 1 0 Producer 240,400 xxxxxx 806 P017 1 2 ... 1 0 Consumer 722,115 xxxxxx 2,420 P002 2 7 1 0 Consumer 16,386 55 P002 1 0 1 0 Consumer 210,112 xx 704 P003 2 2 1 0 Consumer 112,510 x 377 P003 1 1 1 0 Consumer 940,928 xxxxxxxx 3,153 P004 1 9 1 0 Consumer 34,688 116 P004 2 0 1 0 Consumer 171,392 xx 574 P005 2 2 1 0 Consumer 110,466 x 370 P005 1 1 1 0 Consumer 163,658 x 548 P014 2 2 1 0 Consumer 1,113,523 xxxxxxxxxx 3,732 P014 1 11 1 0 Consumer 11,100 37 P015 1 0 1 0 Consumer 4,150 14 P015 2 0 ... 1 1 Consumer 200 xxxxxxxxxx 1 QC 1 100 select /*+ parallel(32) */ id, filler, count(*) from ckk group by id,filler order by id,filler; -- 41 sec
  50. 50. © 2015 Pythian – Presentation for DOAG Group by – summary 50 • Theoretical performance is reachable • SORT based operations have major issues • Especially with sorted inputs, such as from analytics • model clause can help – partition by is hash based • Partial disk staging causes major slowdowns • Balanced operation is critical
  51. 51. © 2015 Pythian – Presentation for DOAG Group by – Partitioning 51 • What about partitioning? create table ckk_p (id, mod5_id, mod5000_id, dt_fixed, dt_dec, dt_pos, dt_rand, dt_mod10, filler) partition by hash (id) partitions 32 tablespace ckbig nologging parallel 4 as select * from ckk;
  52. 52. © 2015 Pythian – Presentation for DOAG Group by – with partitioning 52 select /*+ parallel(4) */ id,count(*) from ckk_p group by id,filler; ------------------------------------------------------------------------------------------------------------------------ | Id | Operation | Name | E-Rows |E-Bytes|E-Temp | Pstart| Pstop | TQ |IN-OUT| PQ Distrib | ------------------------------------------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | 7071K| 23G| | | | | | | | 1 | PX COORDINATOR | | | | | | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10000 | 7071K| 23G| | | | Q1,00 | P->S | QC (RAND) | | 3 | PX PARTITION HASH ALL | | 7071K| 23G| | 1 | 32 | Q1,00 | PCWC | | | 4 | HASH GROUP BY | | 7071K| 23G| 38G| | | Q1,00 | PCWP | | | 5 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 32G| | 1 | 32 | Q1,00 | PCWP | | ------------------------------------------------------------------------------------------------------------------------ PQ1 PQ2 PQ3 PQ4 S G A STEP (1) GROUP PGA GROUP PGA GROUP PGA GROUP PGA QC LIST P1 P2 P3 P4 P5 P6 P7 P8
  53. 53. © 2015 Pythian – Presentation for DOAG select /*+ parallel(64) */ * from ckk c1 order by id; ---------------------------------------------------------------------------------------------------------- | Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib | ---------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 7071K| 23G| | | | | | 1 | PX COORDINATOR | | | | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10001 | 7071K| 23G| | Q1,01 | P->S | QC (RAND) | | 3 | HASH GROUP BY | | 7071K| 23G| 38G| Q1,01 | PCWP | | | 4 | PX RECEIVE | | 10M| 32G| | Q1,01 | PCWP | | | 5 | PX SEND HASH | :TQ10000 | 10M| 32G| | Q1,00 | P->P | HASH | | 6 | PX BLOCK ITERATOR | | 10M| 32G| | Q1,00 | PCWC | | | 7 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 32G| | Q1,00 | PCWP | | ---------------------------------------------------------------------------------------------------------- Group by – enough partitions 53 PQ1 PQ2 PQ3 PQ4 S G A STEP (1) GROUP PGA GROUP PGA GROUP PGA GROUP PGA QC LIST P1 P2 P3 P4 P5 P6 P7 P8
  54. 54. © 2015 Pythian – Presentation for DOAG Analytics don’t speak parallel 54 select /*+ parallel(4) */ id, row_number() over (partition by id,filler order by null) r from ckk_p ; ---------------------------------------------------------------------------------------------------------- | Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib | ---------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 10M| 32G| | | | | | 1 | PX COORDINATOR | | | | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 32G| | Q1,01 | P->S | QC (RAND) | | 3 | WINDOW SORT | | 10M| 32G| 38G| Q1,01 | PCWP | | | 4 | PX RECEIVE | | 10M| 32G| | Q1,01 | PCWP | | | 5 | PX SEND HASH | :TQ10000 | 10M| 32G| | Q1,00 | P->P | HASH | | 6 | PX BLOCK ITERATOR | | 10M| 32G| | Q1,00 | PCWC | | | 7 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 32G| | Q1,00 | PCWP | | ---------------------------------------------------------------------------------------------------------- select /*+ NO_parallel */ id, row_number() over (partition by id,filler order by null) r from ckk_p ; -------------------------------------------------------------------------------------- | Id | Operation | Name | E-Rows |E-Bytes|E-Temp | Pstart| Pstop | -------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 10M| 32G| | | | | 1 | PARTITION HASH ALL | | 10M| 32G| | 1 | 32 | | 2 | WINDOW SORT | | 10M| 32G| 38G| | | | 3 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 32G| | 1 | 32 | --------------------------------------------------------------------------------------
  55. 55. 55 The Big Join
  56. 56. © 2015 Pythian – Presentation for DOAG Test Data 56 • Copy of ckk – ckk2 • ckk_m5 – 5 rows dimension create table ckk_m5 as select rownum mod5_id, rpad('x',1000,'x') filler from dual connect by level <=5; alter table ckk_m5 modify (mod5_id not null);
  57. 57. © 2015 Pythian – Presentation for DOAG Simple Hash Join select /*+ parallel(4) */ * from ckk c1 join ckk_m5 m5 on (c1.mod5_id = m5.mod5_id); -------------------------------------------------------------------------------------------------- | Id | Operation | Name | E-Rows |E-Bytes| TQ |IN-OUT| PQ Distrib | -------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 10M| 42G| | | | | 1 | PX COORDINATOR | | | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 42G| Q1,01 | P->S | QC (RAND) | |* 3 | HASH JOIN | | 10M| 42G| Q1,01 | PCWP | | | 4 | PX RECEIVE | | 5 | 5020 | Q1,01 | PCWP | | | 5 | PX SEND BROADCAST | :TQ10000 | 5 | 5020 | Q1,00 | P->P | BROADCAST | | 6 | PX BLOCK ITERATOR | | 5 | 5020 | Q1,00 | PCWC | | | 7 | TABLE ACCESS STORAGE FULL| CKK_M5 | 5 | 5020 | Q1,00 | PCWP | | | 8 | PX BLOCK ITERATOR | | 10M| 33G| Q1,01 | PCWC | | | 9 | TABLE ACCESS STORAGE FULL | CKK | 10M| 33G| Q1,01 | PCWP | | --------------------------------------------------------------------------------------------------
  58. 58. © 2015 Pythian – Presentation for DOAG Simple Hash Join 58 PQ1 PQ2 PQ3 PQ4 S G A STEP (1) PQ5 PQ6 PQ7 PQ8 HASH PGA HASH PGA HASH PGA HASH PGA S G A QC CKK _M5 0 GiB All Hash Areas: 0 GiB
  59. 59. © 2015 Pythian – Presentation for DOAG Simple Hash Join 59 PQ5 PQ6 PQ7 PQ8 HASH PGA HASH PGA HASH PGA HASH PGA S G A STEP (2) QC CKK (39 GiB)
  60. 60. © 2015 Pythian – Presentation for DOAG Big Hash Join select /*+ parallel(4) */ * from ckk c1join ckk2 c2 on (c1.id = c2.id) ------------------------------------------------------------------------------------------------- | Id | Operation | Name | E-Rows |E-Bytes|E-Temp |IN-OUT| PQ Distrib | ------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 10M| 66G| | | | | 1 | PX COORDINATOR | | | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10002 | 10M| 66G| | P->S | QC (RAND) | |* 3 | HASH JOIN BUFFERED | | 10M| 66G| 8502M| PCWP | | | 4 | PX RECEIVE | | 10M| 33G| | PCWP | | | 5 | PX SEND HASH | :TQ10000 | 10M| 33G| | P->P | HASH | | 6 | PX BLOCK ITERATOR | | 10M| 33G| | PCWC | | | 7 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | PCWP | | | 8 | PX RECEIVE | | 10M| 33G| | PCWP | | | 9 | PX SEND HASH | :TQ10001 | 10M| 33G| | P->P | HASH | | 10 | PX BLOCK ITERATOR | | 10M| 33G| | PCWC | | | 11 | TABLE ACCESS STORAGE FULL| CKK2 | 10M| 33G| | PCWP | | -------------------------------------------------------------------------------------------------
  61. 61. © 2015 Pythian – Presentation for DOAG Big Hash Join 61 PQ1 PQ2 PQ3 PQ4 S G A STEP (1) PQ5 PQ6 PQ7 PQ8 HASH PGA HASH PGA HASH PGA HASH PGA S G A QC CKK 39 GiB All Hash Areas: 39 GiB
  62. 62. © 2015 Pythian – Presentation for DOAG Big Hash Join 62 PQ1 PQ2 PQ3 PQ4 S G A STEP (2) PQ5 PQ6 PQ7 PQ8 HASH PGA HASH PGA HASH PGA HASH PGA S G A QC CKK 2 39 GiB All Hash Areas: 78 GiB
  63. 63. © 2015 Pythian – Presentation for DOAG Simple Hash Join 63 PQ5 PQ6 PQ7 PQ8 HASH PGA HASH PGA HASH PGA HASH PGA S G A STEP (3) QC
  64. 64. © 2015 Pythian – Presentation for DOAG Hash Join summary 64 • HASH JOINS with small tables are extremely fast • Uses BROADCAST to send copy of table to every slave • As much PGA work area as parallel degree * small table size • HASH JOINS with big tables can be extremely expensive • Almost entire result set written to temp • Unless – partitioning
  65. 65. © 2015 Pythian – Presentation for DOAG Semi-Partitioned Hash Join select /*+ parallel(32) */ * from ckk c1 join ckk_p c2 on (c1.id = c2.id) ----------------------------------------------------------------------------------------------------------- | Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib | ----------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 10M| 66G| | | | | | 1 | PX COORDINATOR | | | | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 66G| | Q1,01 | P->S | QC (RAND) | |* 3 | HASH JOIN | | 10M| 66G| 1062M| Q1,01 | PCWP | | | 4 | PART JOIN FILTER CREATE | :BF0000 | 10M| 33G| | Q1,01 | PCWP | | | 5 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | | | 6 | PX SEND PARTITION (KEY) | :TQ10000 | 10M| 33G| | Q1,00 | P->P | PART (KEY) | | 7 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | | | 8 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | | | 9 | PX PARTITION HASH JOIN-FILTER| | 10M| 33G| | Q1,01 | PCWC | | | 10 | TABLE ACCESS STORAGE FULL | CKK_P | 10M| 33G| | Q1,01 | PCWP | | -----------------------------------------------------------------------------------------------------------
  66. 66. © 2015 Pythian – Presentation for DOAG Semi Partitioned Hash Join 66 PQ1 PQ2 PQ3 PQ4 S G A STEP (1) PQ5 PQ6 PQ7 PQ8 HASH PGA HASH PGA HASH PGA HASH PGA S G A QC CKK 39 GiB All Hash Areas: 39 GiB
  67. 67. © 2015 Pythian – Presentation for DOAG Semi Partitioned Hash Join 67 PQ5 PQ6 PQ7 PQ8 HASH PGA HASH PGA HASH PGA HASH PGA S G A STEP (2) QC CKK_P (39 GiB) LIST P1 P2 P3 P4 P5 P6 P7 P8
  68. 68. © 2015 Pythian – Presentation for DOAG Semi Partitioned Hash Join 68 • Dramatically reduces HASH Area sizes • Very likely to fit in RAM – no disk • Reduces from 3 steps to 2 steps • Always reads the non-partitioned table into HASH areas • Each slave reads it’s own partition
  69. 69. © 2015 Pythian – Presentation for DOAG Partitioned Hash Join select /*+ parallel(4) */ * from ckk_p c1 join ckk_p c2 on (c1.id = c2.id) -------------------------------------------------------------------------------------------------------- | Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib | -------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 10M| 66G| | | | | | 1 | PX COORDINATOR | | | | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10000 | 10M| 66G| | Q1,00 | P->S | QC (RAND) | | 3 | PX PARTITION HASH ALL | | 10M| 66G| | Q1,00 | PCWC | | |* 4 | HASH JOIN | | 10M| 66G| 8502M| Q1,00 | PCWP | | | 5 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 33G| | Q1,00 | PCWP | | | 6 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 33G| | Q1,00 | PCWP | | --------------------------------------------------------------------------------------------------------
  70. 70. © 2015 Pythian – Presentation for DOAG Partitioned Hash Join select /*+ parallel(4) */ * from ckk_p c1 join ckk_p c2 on (c1.id = c2.id) -------------------------------------------------------------------------------------------------------- | Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib | -------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 10M| 66G| | | | | | 1 | PX COORDINATOR | | | | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10000 | 10M| 66G| | Q1,00 | P->S | QC (RAND) | | 3 | PX PARTITION HASH ALL | | 10M| 66G| | Q1,00 | PCWC | | |* 4 | HASH JOIN | | 10M| 66G| 8502M| Q1,00 | PCWP | | | 5 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 33G| | Q1,00 | PCWP | | | 6 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 33G| | Q1,00 | PCWP | | --------------------------------------------------------------------------------------------------------
  71. 71. © 2015 Pythian – Presentation for DOAG Switching places select /*+ parallel(32) */ * from ckk_p c1 join ckk c2 on (c1.id = c2.id); ---------------------------------------------------------------------------------------------------------- | Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib | ---------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 10M| 66G| | | | | | 1 | PX COORDINATOR | | | | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 66G| | Q1,01 | P->S | QC (RAND) | |* 3 | HASH JOIN BUFFERED | | 10M| 66G| 1062M| Q1,01 | PCWP | | | 4 | PX PARTITION HASH ALL | | 10M| 33G| | Q1,01 | PCWC | | | 5 | TABLE ACCESS STORAGE FULL | CKK_P | 10M| 33G| | Q1,01 | PCWP | | | 6 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | | | 7 | PX SEND PARTITION (KEY) | :TQ10000 | 10M| 33G| | Q1,00 | P->P | PART (KEY) | | 8 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | | | 9 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | | ---------------------------------------------------------------------------------------------------------- select /*+ parallel(32) LEADING(c2) */ * from ckk_p c1 join ckk c2 on (c1.id = c2.id); ----------------------------------------------------------------------------------------------------------- | Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib | ----------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 10M| 66G| | | | | | 1 | PX COORDINATOR | | | | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 66G| | Q1,01 | P->S | QC (RAND) | |* 3 | HASH JOIN | | 10M| 66G| 1062M| Q1,01 | PCWP | | | 4 | PART JOIN FILTER CREATE | :BF0000 | 10M| 33G| | Q1,01 | PCWP | | | 5 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | | | 6 | PX SEND PARTITION (KEY) | :TQ10000 | 10M| 33G| | Q1,00 | P->P | PART (KEY) | | 7 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | | | 8 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | | | 9 | PX PARTITION HASH JOIN-FILTER| | 10M| 33G| | Q1,01 | PCWC | | | 10 | TABLE ACCESS STORAGE FULL | CKK_P | 10M| 33G| | Q1,01 | PCWP | | -----------------------------------------------------------------------------------------------------------
  72. 72. © 2015 Pythian – Presentation for DOAG Outer Joins select /*+ parallel(32) */ * from ckk_p c1 left join ckk c2 on (c1.id = c2.id); ---------------------------------------------------------------------------------------------------------- | Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib | ---------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 10M| 66G| | | | | | 1 | PX COORDINATOR | | | | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 66G| | Q1,01 | P->S | QC (RAND) | |* 3 | HASH JOIN OUTER BUFFERED | | 10M| 66G| 1062M| Q1,01 | PCWP | | | 4 | PX PARTITION HASH ALL | | 10M| 33G| | Q1,01 | PCWC | | | 5 | TABLE ACCESS STORAGE FULL | CKK_P | 10M| 33G| | Q1,01 | PCWP | | | 6 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | | | 7 | PX SEND PARTITION (KEY) | :TQ10000 | 10M| 33G| | Q1,00 | P->P | PART (KEY) | | 8 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | | | 9 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | | ---------------------------------------------------------------------------------------------------------- select /*+ parallel(32) SWAP_JOIN_INPUTS(c2)*/ * from ckk_p c1 left join ckk c2 on (c1.id = c2.id); ---------------------------------------------------------------------------------------------------------- | Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib | ---------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 10M| 66G| | | | | | 1 | PX COORDINATOR | | | | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 66G| | Q1,01 | P->S | QC (RAND) | |* 3 | HASH JOIN RIGHT OUTER | | 10M| 66G| 1062M| Q1,01 | PCWP | | | 4 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | | | 5 | PX SEND PARTITION (KEY) | :TQ10000 | 10M| 33G| | Q1,00 | P->P | PART (KEY) | | 6 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | | | 7 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | | | 8 | PX PARTITION HASH ALL | | 10M| 33G| | Q1,01 | PCWC | | | 9 | TABLE ACCESS STORAGE FULL | CKK_P | 10M| 33G| | Q1,01 | PCWP | | ----------------------------------------------------------------------------------------------------------
  73. 73. 73 The Big Join – with Index
  74. 74. © 2015 Pythian – Presentation for DOAG Adding a few indexes 74 create index ckk$id on ckk(id) parallel 32 tablespace ckbig nologging; --6.5 sec create index ckk$mod5000_id on ckk(mod5000_id) parallel 32 tablespace ckbig nologging compress; --5.8 sec create index ckk_p$mod5000_id on ckk_p(mod5000_id) parallel 32 tablespace ckbig nologging compress local; --16 sec
  75. 75. © 2015 Pythian – Presentation for DOAG Direct index access select /*+ parallel(4) */ * from ckk where mod5000_id=100; ----------------------------------------------------------------------- | Id | Operation | Name | E-Rows |E-Bytes| ----------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 2000 | 6941K| | 1 | TABLE ACCESS BY INDEX ROWID| CKK | 2000 | 6941K| |* 2 | INDEX RANGE SCAN | CKK$MOD5000_ID | 2000 | | ----------------------------------------------------------------------- select /*+ parallel(4) */ * from ckk_p where mod5000_id=100; -------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | E-Rows |E-Bytes| TQ |IN-OUT| PQ Distrib | -------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 2000 | 6941K| | | | | 1 | PX COORDINATOR | | | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10000 | 2000 | 6941K| Q1,00 | P->S | QC (RAND) | | 3 | PX PARTITION HASH ALL | | 2000 | 6941K| Q1,00 | PCWC | | | 4 | TABLE ACCESS BY LOCAL INDEX ROWID| CKK_P | 2000 | 6941K| Q1,00 | PCWP | | |* 5 | INDEX RANGE SCAN | CKK_P$MOD5000_ID | 2000 | | Q1,00 | PCWP | | --------------------------------------------------------------------------------------------------------------
  76. 76. © 2015 Pythian – Presentation for DOAG Join with Index select /*+ parallel(4) */ * from ckk c1 join ckk2 c2 on (c1.id = c2.id) where c2.mod5000_id=5; ------------------------------------------------------------------------------------------------- | Id | Operation | Name | E-Rows |E-Bytes| TQ |IN-OUT| PQ Distrib | ------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 2017 | 13M| | | | | 1 | PX COORDINATOR | | | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10000 | | | Q1,00 | P->S | QC (RAND) | | 3 | NESTED LOOPS | | | | Q1,00 | PCWP | | | 4 | NESTED LOOPS | | 2017 | 13M| Q1,00 | PCWP | | | 5 | PX BLOCK ITERATOR | | | | Q1,00 | PCWC | | |* 6 | TABLE ACCESS STORAGE FULL| CKK2 | 2000 | 6941K| Q1,00 | PCWP | | |* 7 | INDEX RANGE SCAN | CKK$ID | 1 | | Q1,00 | PCWP | | | 8 | TABLE ACCESS BY INDEX ROWID| CKK | 1 | 3554 | Q1,00 | PCWP | | -------------------------------------------------------------------------------------------------
  77. 77. © 2015 Pythian – Presentation for DOAG Indexes summary 77 • Parallel access on direct value queries only on partitioned tables and on local indexes • Parallel access on global or non-partitioned only from joins (multiple different values)
  78. 78. © 2015 Pythian – Presentation for DOAG Final Thoughts 78 • Exadata allows Parallel Query to make sense • High Bandwidth everywhere • Very efficient table scans • Optimiser sometimes picks up wrong access plan • There are deficiencies in SORT operations – balancing the workloads • There are deficiencies in handling disk write operations in PQ Slaves • Can be extremely fast under the right circumstances
  79. 79. © 2009/2010 Pythian - Presentation for ABC Company79 The End Thank You Questions? I blog at http://www.pythian.com/news/author/kutrovsky/

×