SlideShare ist ein Scribd-Unternehmen logo
1 von 31
Basic Query Tuning Primer Pg West 2009 2009-10-17
Basic tool kit for query tuning For Diagnosis: ,[object Object]
Query pg_class
Query pg_stats
SET enable_[nestloop|hashagg|...] = [on|off]
SET [random_page|...]_cost = [n]
RESET all
Postgres server logs ,[object Object],[object Object]
Occasionally helpful: gdb, dtrace, sar, iostat -x, etc. For Remedy: ,[object Object]
ALTER TABLE [table] ALTER COLUMN [column] SET STATISTICS [n]
SET default_statistics_target = [n], work_mem = [n]
CREATE INDEX
Reorganize the SQL (e.g. filter large tables directly, avoid joins, refactor to temp tables or WITH clause, etc.)
Rarely appropriate: SET [gucs], table-level autovac storage-params, denormalize columns, partition large table
System tuning is not query tuning Relevance: System tuning affects all queries, but it optimizes for an aggregate workload, not an individual query. The overall performance of a system is the product of many factors, including but not limited to: ,[object Object]
System settings: kernel cache page size, tcp buffer size, dirty page flush policy, io scheduler, etc.
Filesystem settings: type, readahead, direct/buffered io, extent size, journaling policy, internal/external journal, atime/mtime maint., existence of snapshots, etc.
Postgres settings: version, GUCS (default_statistics_target, shared_buffers, temp_buffers, work_mem, wal_buffers, *_cost, enable_*, effective_cache_size, fsync), WAL on separate disks, etc.
Total workload: cumulative effect of all tasks running at a particular point in time, all of which compete for resources (e.g. cpu, memory, cache, disk I/O, locks).
Accumulated state: contents of caches (Pg, kernel, controller), physical file fragmentation on disk, optimizer statistics (specific to cluster; not portable via pg_dump)
What is an execution plan? To see a query's execution plan, use the  EXPLAIN  command before the query. To also run the query and capture how much time was spent in each step of the plan, use  EXPLAIN ANALYZE . Original SQL: pg841=# select count(*) from orders where created_ts >= '2009-09-01'::timestamp ; count  ------- 65472 (1 row) Execution plan with  estimated   costs and row-counts: pg841=#  explain  select count(*) from orders where created_ts >= '2009-09-01'::timestamp ; QUERY PLAN  ------------------------------------------------------------------------------------ Aggregate  (cost=2100.85..2100.86 rows=1 width=0) ->  Seq Scan on orders  (cost=0.00..1937.00 rows= 65538  width=0) Filter: (created_ts >= '2009-09-01 00:00:00'::timestamp without time zone) (3 rows) Execution plan with  actual   runtimes  and row-counts: pg841=#  explain analyze  select count(*) from orders where created_ts >= '2009-09-01'::timestamp ; QUERY PLAN  ------------------------------------------------------------------------------------------------------------------- Aggregate  (cost=2100.85..2100.86 rows=1 width=0)   (actual time=251.210..251.212 rows=1 loops=1) ->  Seq Scan on orders  (cost=0.00..1937.00 rows= 65538  width=0)   (actual time=0.018..140.495 rows= 65472  loops=1) Filter: (created_ts >= '2009-09-01 00:00:00'::timestamp without time zone) Total runtime: 251.269 ms (4 rows)
Each step in the execution plan is a node in a tree hierarchy.  The leftmost (“top”) node pulls data from its children, which do the same for their children. Nodes with no children start working first, since they gather the data from tables or indexes to be processed by the rest of the plan nodes. Run Order  QUERY PLAN  -------   --------------------------------------------------------------------------------------------------------------- 4th  HashAggregate  (cost= 1119.28 ..1120.75 rows=98 width=12) 3rd   ->  Nested Loop  (cost= 0.00 ..1118.79 rows=98 width=12) 1st   ->  Index Scan using idx_order_details_product_id on order_details  (cost= 0.00 ..397.99 rows=98 width=8) Index Cond: (product_id = 1000) 2nd   ->  Index Scan using orders_pkey on orders  (cost= 0.00 ..7.34 rows=1 width=12) Index Cond: (orders.order_id = order_details.order_id) Filter: (orders.created_ts >= '2009-09-01 00:00:00'::timestamp without time zone) (7 rows) Reading an execution plan: In what order do the nodes run?
The two numbers for “cost” and “actual time” represent when the 1st and last rows will be output by that node. Different kinds of nodes have different amounts of lag between when it receives its first input and when it produces its first output.  This “startup cost” is implied by the difference in the first cost value of the  node  vs. its  slowest-starting child . Here, the  Hash  node has high startup cost – it cannot feed data to its parent ( Hash Join ) until it receives and processes all of the data from its child  Seq Scan  node.  So the  Seq Scan  node's final cost ( 1937.00 ) becomes the  Hash  node's initial cost ( 1937.00 ).  In practice, the child's actual completion time ( 151.703 ) is a lower bound for the  Hash  node's first-output time ( 288.962 ). QUERY PLAN  ------------------------------------------------------------------------------------------------------------------------- Hash Join  ( cost= 3013.22 .. 45974.93   rows=651270 width=12) ( actual time= 289.152 .. 6297.282   rows=654720 loops=1) Hash Cond: (order_details.order_id = orders.order_id) ->  Seq Scan  on order_details  ( cost= 0.00 .. 14902.00   rows=1000000 width=12) ( actual time= 0.037 .. 2026.802  rows=1000000 loops=1) ->  Hash   ( cost= 1937.00 .. 1937.00  rows=65538 width=8) ( actual time= 288.962 .. 288.962  rows=65472 loops=1) ->  Seq Scan  on orders  ( cost= 0.00 .. 1937.00  rows=65538 width=8) ( actual time= 0.014 .. 151.703  rows=65472 loops=1) Filter: (created_ts >= '2009-09-01 00:00:00'::timestamp without time zone) Total runtime: 7349.662 ms (7 rows) Reading an execution plan: What do the numbers mean for "cost" and "actual time"?
Access methods  are used by leaf nodes to pull data from tables/indexes. Join methods  specify which type of algorithm will be used to implement each of the query's joins. QUERY PLAN  ------------------------------------------------------------------------------------------------------ Sort  (cost=248.54..248.79 rows=99 width=8) Sort Key: o.order_id ->  Nested Loop  (cost=4.34..245.26 rows=99 width=8) ->  Nested Loop  (cost=4.34..234.94 rows=10 width=4) ->  Seq Scan  on customers c  (cost=0.00..194.00 rows=1 width=4) Filter: (name = 'JOHN DOE'::text) ->  Bitmap Heap Scan  on orders o  (cost=4.34..40.82 rows=10 width=8) Recheck Cond: (o.cust_id = c.cust_id) ->  Bitmap Index Scan  on idx_orders_cust_id  (cost=0.00..4.34 rows=10 width=0) Index Cond: (o.cust_id = c.cust_id) ->  Index Scan  using order_details_pk on order_details od  (cost=0.00..0.91 rows=10 width=8) Index Cond: (od.order_id = o.order_id) (12 rows) QUERY PLAN  ---------------------------------------------------------------------------------------------------------------------- Aggregate   (cost=18850.37..18850.38 rows=1 width=8) ->  Hash Join   (cost=53.18..18801.97 rows=9679 width=8) Hash Cond: (od.order_id = o.order_id) ->  Seq Scan  on order_details od  (cost=0.00..14902.00 rows=1000000 width=8) ->  Hash  (cost=41.00..41.00 rows=974 width=4) ->  Index Scan  using pidx_orders_order_id_not_shipped on orders o  (cost=0.00..41.00 rows=974 width=4) Filter: is_paid (7 rows) Reading an execution plan: What are access methods and join methods?
Create and ANALYZE a small toy table. pg841=#  create temp table  my_tab1  as select  * from orders limit 100 ; SELECT pg841=#  analyze  my_tab1  ; ANALYZE EXPLAIN a simple query against it. pg841=#  explain   select * from my_tab1 where created_ts >  '2009-10-14 08:14:26'::timestamp - '1 day'::interval  ; QUERY PLAN  ------------------------------------------------------------------------------------ Seq Scan  on my_tab1  (cost=0.00.. 2.25  rows=17 width=25) Filter: (created_ts >  '2009-10-13 08:14:26' ::timestamp without time zone) (2 rows) The planner was able to combine the filter's two literals.  The table is accessed by SeqScan, since there are no indexes yet on this temp table.  Add an index, force the planner not to SeqScan, and compare the same query's cost estimates for IndexScan versus SeqScan. pg841=#  create index  idx_my_tab1_test on my_tab1 ( created_ts ) ; CREATE INDEX pg841=#  set enable_seqscan = off   ; SET pg841=#  explain  select * from my_tab1 where created_ts > '2009-10-14 08:14:26'::timestamp - '1 day'::interval ; QUERY PLAN  ---------------------------------------------------------------------------------------- Index Scan  using idx_my_tab1_test on my_tab1  ( cost =0.00.. 8.55  rows=17 width=25) Index Cond: (created_ts > '2009-10-13 08:14:26'::timestamp without time zone) (2 rows) pg841=#  reset all ; RESET The optimizer prefers (assigns a lower cost) to SeqScan this table because it is so tiny (1 block).  An IndexScan would require reading a 2nd block (the index itself) and doing extra comparisons. Play with small toy queries.
Run your query with EXPLAIN (or if practical, EXPLAIN ANALYZE), and look for nodes where the cost, row-count, or actual_time significantly increases compared to its children. In this example, the SQL is missing its join criteria.  The estimated  cost  and  row-count  skyrocket in the  Nested Loop   node, because it is returning the cross-product of all rows from both its input nodes. pg841=# explain pg841-# select pg841-#  customers.cust_id as customer_id, pg841-#  max(customers.name) as customer_name, pg841-#  count(distinct orders.order_id) as num_orders, pg841-#  max(orders.shipped_ts) as latest_shipment_datetime pg841-# from orders, customers , products  /* THIS JOIN TO “products” IS SPURIOUS */ pg841-# where pg841-#  orders.cust_id = customers.cust_id pg841-#  and orders.created_ts >= now() - '30 days'::interval pg841-# group by pg841-#  customers.cust_id pg841-# order by num_orders desc pg841-# limit 10 pg841-# ; QUERY PLAN  ----------------------------------------------------------------------------------------------------------------------- Limit  ( cost =15307415.27.. 15307415.29  rows=10 width=29) ->  Sort  ( cost =15307415.27.. 15307440.27  rows=10000 width=29) Sort Key: (count(DISTINCT orders.order_id)) ->  GroupAggregate  ( cost =185.03.. 15307199.17  rows=10000 width=29) ->  Nested Loop  ( cost =185.03.. 10207124.17  rows=509990000  width=29) ->  Merge Join  ( cost =0.03.. 7139.17  rows=50999  width=29) Merge Cond: (customers.cust_id = orders.cust_id) ->  Index Scan using customers_pkey on customers  ( cost =0.00.. 318.48  rows=10000 width=17) ->  Index Scan using idx_orders_cust_id on orders  ( cost =0.00.. 6158.23  rows=50999 width=16) Filter: (orders.created_ts >= (now() - '30 days'::interval)) ->  Materialize  ( cost =185.00.. 285.00  rows=10000  width=0) ->  Seq Scan on products  ( cost =0.00.. 175.00  rows=10000 width=0) (12 rows) What does a "bad plan" look like?  Does it imply possible tune-ups?
Rerun the modified query with EXPLAIN to confirm that it looks better.  The  final cost estimate  is much lower, and there are no more huge jumps in cost from one step to the next. pg841=# explain pg841-# select pg841-#  customers.cust_id as customer_id, pg841-#  max(customers.name) as customer_name, pg841-#  count(distinct orders.order_id) as num_orders, pg841-#  max(orders.shipped_ts) as latest_shipment_datetime pg841-# from orders, customers pg841-# where pg841-#  orders.cust_id = customers.cust_id pg841-#  and orders.created_ts >= now() - '30 days'::interval pg841-# group by pg841-#  customers.cust_id pg841-# order by num_orders desc pg841-# limit 10 pg841-# ; QUERY PLAN  ---------------------------------------------------------------------------------------------------- Limit  ( cost =10243.47.. 10243.50  rows=10 width=29) ->  Sort  (cost=10243.47..10268.47 rows=10000 width=29) Sort Key: (count(DISTINCT orders.order_id)) ->  GroupAggregate  (cost=9214.91..10027.37 rows=10000 width=29) ->  Sort  (cost=9214.91..9342.40 rows=50997 width=29) Sort Key: customers.cust_id ->  Hash Join  (cost=294.00..4005.92 rows=50997 width=29) Hash Cond: (orders.cust_id = customers.cust_id) ->  Seq Scan on orders  (cost=0.00..2437.00 rows=50997 width=16) Filter: (created_ts >= (now() - '30 days'::interval)) ->  Hash  (cost=169.00..169.00 rows=10000 width=17) ->  Seq Scan on customers  (cost=0.00..169.00 rows=10000 width=17) (12 rows) After previewing the new plan with EXPLAIN, run EXPLAIN ANALYZE to confirm runtime is better. Retest after rewriting the query to remove the spurious join...
Here's a trivial query joining 2 tables. pg841=# explain select count(*) from my_tab1  inner join  my_tab2  using (key)  ; QUERY PLAN  --------------------------------------------------------------------------------------- Aggregate  (cost=42.26..42.27 rows=1 width=0) ->  Nested Loop  (cost=0.00..42.01 rows=100 width=0) ->  Seq Scan on my_tab1  (cost=0.00..2.00 rows=100 width=4) ->  Index Scan using idx_my_tab2 on my_tab2  (cost=0.00..0.39 rows=1 width=4) Index Cond: (my_tab2.key = my_tab1.key) (5 rows) Notice that the  Nested Loop  node does not specify any join conditions, even though the SQL does.  The  Nested Loop  is effectively cross-joining its 2 inputs.  Why would it do that, when the SQL says to join on column  key ? Because the join condition has been pushed down into child #2's index filter.  So for each row from child #1 ( Seq Scan ), the parent ( Nested Loop ) calls child #2 ( Index Scan ), passing it info from child #1's row.  If this session forbids the use of indexes, the join filter won't be pushed down. pg841=#  set enable_indexscan = off ; SET pg841=#  set enable_bitmapscan = off ; SET pg841=# explain select count(*) from my_tab1  inner join  my_tab2  using (key)  ; QUERY PLAN  --------------------------------------------------------------------------- Aggregate  (cost=229.35..229.36 rows=1 width=0) ->  Nested Loop  (cost=2.10..229.10 rows=100 width=0) Join Filter: (my_tab1.key = my_tab2.key) ->  Seq Scan on my_tab1  (cost=0.00..2.00 rows=100 width=4) ->  Materialize  (cost=2.10..3.10 rows=100 width=4) ->  Seq Scan on my_tab2  (cost=0.00..2.00 rows=100 width=4) (6 rows) pg841=# reset all ; RESET Join Conditions can sometimes be implemented by a child node.
Stale or missing statistics: When was my table last analyzed? Review the last time when each table was analyzed (either manually or by autovacuum).  Use your knowledge of how and when your data changes to decide if the statistics are likely to be out of date. pg841=# select pg841-#  schemaname, pg841-#  relname, pg841-#  last_analyze  as last_manual, pg841-#  last_autoanalyze   as last_auto, pg841-#  greatest (last_analyze, last_autoanalyze) pg841-# from pg841-#  pg_stat_user_tables pg841-# where pg841-#  relname = 'my_tab' pg841-# ; -[ RECORD 1 ]------------------------------ schemaname  | public relname  | my_tab last_manual | 2009-10-03 18:45:57.627593-07 last_auto  | 2009-10-03 23:08:32.914092-07 greatest  | 2009-10-03 23:08:32.914092-07
Stale or missing statistics: How many rows does the optimizer think my table has? How are my columns' values distributed? Many bad plans are due to poor cardinality estimates for one or more nodes.  Sometimes this is due to stale or missing statistics.  For example, if a column was added or a significant percentage of rows were inserted, deleted, or modified, then the optimizer statistics should be refreshed. You can view the table-level optimizer statistics in pg_class: pg841=# select  reltuples , relpages from  pg_class  where relname = 'my_tab' ; reltuples | relpages  -----------+---------- 1000   |  5 (1 row) And the more detailed column-level optimizer statistics are shown in pg_stats: pg841=# select * from  pg_stats  where tablename = 'my_tab' and attname = 'bar' ; -[ RECORD 1 ]-----+--------------------------- schemaname  | public tablename  | my_tab attname  | bar null_frac  | 0 avg_width  | 4 n_distinct  | 13 most_common_vals  | {1,2} most_common_freqs  | {0.707,0.207} histogram_bounds  | {0,3,4,5,6,7,8,9,10,11,12} correlation  | 0.876659
A new index is usually helpful if it greatly reduces the number of rows the query must visit. This table has an index for all possible combinations of its columns. pg841=#  my_tab  Table "public.my_tab" Column |  Type  | Modifiers  --------+---------+----------- foo  | integer |  bar  | integer |  Indexes: "idx_my_tab_bar" btree  (bar) "idx_my_tab_bar_foo" btree  (bar, foo) "idx_my_tab_foo" btree  (foo) "idx_my_tab_foo_bar" btree  (foo, bar) At least 1 is unnecessary, and up to 2 could be dropped without forcing any query to use a Seq Scan. pg841=# explain select * from my_tab where foo = 10 and bar = 10 ; QUERY PLAN  --------------------------------------------------------------------------------- Index Scan using idx_my_tab_bar_foo on my_tab  (cost=0.00..8.27 rows=1 width=8) Index Cond: ((bar = 10) AND (foo = 10)) (2 rows) pg841=# drop index idx_my_tab_bar_foo ; DROP INDEX pg841=# drop index idx_my_tab_foo_bar ; DROP INDEX pg841=# explain select * from my_tab where foo = 10 and bar = 10 ; QUERY PLAN  ----------------------------------------------------------------------------- Index Scan using idx_my_tab_foo on my_tab  (cost=0.00..8.27 rows=1 width=8) Index Cond: (foo = 10) Filter: (bar = 10) (3 rows) When will adding a new index improve query performance?

Weitere ähnliche Inhalte

Was ist angesagt?

PostgreSQL Procedural Languages: Tips, Tricks and Gotchas
PostgreSQL Procedural Languages: Tips, Tricks and GotchasPostgreSQL Procedural Languages: Tips, Tricks and Gotchas
PostgreSQL Procedural Languages: Tips, Tricks and GotchasJim Mlodgenski
 
Same plan different performance
Same plan different performanceSame plan different performance
Same plan different performanceMauro Pagano
 
Great performance at scale~次期PostgreSQL12のパーティショニング性能の実力に迫る~
Great performance at scale~次期PostgreSQL12のパーティショニング性能の実力に迫る~Great performance at scale~次期PostgreSQL12のパーティショニング性能の実力に迫る~
Great performance at scale~次期PostgreSQL12のパーティショニング性能の実力に迫る~Insight Technology, Inc.
 
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스PgDay.Seoul
 
collectd & PostgreSQL
collectd & PostgreSQLcollectd & PostgreSQL
collectd & PostgreSQLMark Wong
 
PGConf APAC 2018 - Lightening Talk #2 - Centralizing Authorization in PostgreSQL
PGConf APAC 2018 - Lightening Talk #2 - Centralizing Authorization in PostgreSQLPGConf APAC 2018 - Lightening Talk #2 - Centralizing Authorization in PostgreSQL
PGConf APAC 2018 - Lightening Talk #2 - Centralizing Authorization in PostgreSQLPGConf APAC
 
PostgreSQL query planner's internals
PostgreSQL query planner's internalsPostgreSQL query planner's internals
PostgreSQL query planner's internalsAlexey Ermakov
 
Tests unitaires pour PostgreSQL avec pgTap
Tests unitaires pour PostgreSQL avec pgTapTests unitaires pour PostgreSQL avec pgTap
Tests unitaires pour PostgreSQL avec pgTapRodolphe Quiédeville
 
Using histograms to get better performance
Using histograms to get better performanceUsing histograms to get better performance
Using histograms to get better performanceSergey Petrunya
 
Accelerating Local Search with PostgreSQL (KNN-Search)
Accelerating Local Search with PostgreSQL (KNN-Search)Accelerating Local Search with PostgreSQL (KNN-Search)
Accelerating Local Search with PostgreSQL (KNN-Search)Jonathan Katz
 
クラウドDWHとしても進化を続けるPivotal Greenplumご紹介
クラウドDWHとしても進化を続けるPivotal Greenplumご紹介クラウドDWHとしても進化を続けるPivotal Greenplumご紹介
クラウドDWHとしても進化を続けるPivotal Greenplumご紹介Masayuki Matsushita
 
Top 10 Mistakes When Migrating From Oracle to PostgreSQL
Top 10 Mistakes When Migrating From Oracle to PostgreSQLTop 10 Mistakes When Migrating From Oracle to PostgreSQL
Top 10 Mistakes When Migrating From Oracle to PostgreSQLJim Mlodgenski
 
Deep dive into PostgreSQL statistics.
Deep dive into PostgreSQL statistics.Deep dive into PostgreSQL statistics.
Deep dive into PostgreSQL statistics.Alexey Lesovsky
 
Histograms in 12c era
Histograms in 12c eraHistograms in 12c era
Histograms in 12c eraMauro Pagano
 
An Introduction To PostgreSQL Triggers
An Introduction To PostgreSQL TriggersAn Introduction To PostgreSQL Triggers
An Introduction To PostgreSQL TriggersJim Mlodgenski
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLMark Wong
 

Was ist angesagt? (20)

Strategic autovacuum
Strategic autovacuumStrategic autovacuum
Strategic autovacuum
 
PostgreSQL Procedural Languages: Tips, Tricks and Gotchas
PostgreSQL Procedural Languages: Tips, Tricks and GotchasPostgreSQL Procedural Languages: Tips, Tricks and Gotchas
PostgreSQL Procedural Languages: Tips, Tricks and Gotchas
 
Spark_Documentation_Template1
Spark_Documentation_Template1Spark_Documentation_Template1
Spark_Documentation_Template1
 
Same plan different performance
Same plan different performanceSame plan different performance
Same plan different performance
 
Great performance at scale~次期PostgreSQL12のパーティショニング性能の実力に迫る~
Great performance at scale~次期PostgreSQL12のパーティショニング性能の実力に迫る~Great performance at scale~次期PostgreSQL12のパーティショニング性能の実力に迫る~
Great performance at scale~次期PostgreSQL12のパーティショニング性能の実力に迫る~
 
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스
 
collectd & PostgreSQL
collectd & PostgreSQLcollectd & PostgreSQL
collectd & PostgreSQL
 
PGConf APAC 2018 - Lightening Talk #2 - Centralizing Authorization in PostgreSQL
PGConf APAC 2018 - Lightening Talk #2 - Centralizing Authorization in PostgreSQLPGConf APAC 2018 - Lightening Talk #2 - Centralizing Authorization in PostgreSQL
PGConf APAC 2018 - Lightening Talk #2 - Centralizing Authorization in PostgreSQL
 
PostgreSQL query planner's internals
PostgreSQL query planner's internalsPostgreSQL query planner's internals
PostgreSQL query planner's internals
 
Tests unitaires pour PostgreSQL avec pgTap
Tests unitaires pour PostgreSQL avec pgTapTests unitaires pour PostgreSQL avec pgTap
Tests unitaires pour PostgreSQL avec pgTap
 
Using histograms to get better performance
Using histograms to get better performanceUsing histograms to get better performance
Using histograms to get better performance
 
Accelerating Local Search with PostgreSQL (KNN-Search)
Accelerating Local Search with PostgreSQL (KNN-Search)Accelerating Local Search with PostgreSQL (KNN-Search)
Accelerating Local Search with PostgreSQL (KNN-Search)
 
クラウドDWHとしても進化を続けるPivotal Greenplumご紹介
クラウドDWHとしても進化を続けるPivotal Greenplumご紹介クラウドDWHとしても進化を続けるPivotal Greenplumご紹介
クラウドDWHとしても進化を続けるPivotal Greenplumご紹介
 
Full Text Search in PostgreSQL
Full Text Search in PostgreSQLFull Text Search in PostgreSQL
Full Text Search in PostgreSQL
 
Top 10 Mistakes When Migrating From Oracle to PostgreSQL
Top 10 Mistakes When Migrating From Oracle to PostgreSQLTop 10 Mistakes When Migrating From Oracle to PostgreSQL
Top 10 Mistakes When Migrating From Oracle to PostgreSQL
 
Aspects of 10 Tuning
Aspects of 10 TuningAspects of 10 Tuning
Aspects of 10 Tuning
 
Deep dive into PostgreSQL statistics.
Deep dive into PostgreSQL statistics.Deep dive into PostgreSQL statistics.
Deep dive into PostgreSQL statistics.
 
Histograms in 12c era
Histograms in 12c eraHistograms in 12c era
Histograms in 12c era
 
An Introduction To PostgreSQL Triggers
An Introduction To PostgreSQL TriggersAn Introduction To PostgreSQL Triggers
An Introduction To PostgreSQL Triggers
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQL
 

Andere mochten auch

Andere mochten auch (6)

Mac Kinnon Cig2005 Presentation
Mac Kinnon Cig2005 PresentationMac Kinnon Cig2005 Presentation
Mac Kinnon Cig2005 Presentation
 
Martin Rideg
Martin RidegMartin Rideg
Martin Rideg
 
Mark Majoros
Mark MajorosMark Majoros
Mark Majoros
 
SIR_SocialMedia_Syntens
SIR_SocialMedia_SyntensSIR_SocialMedia_Syntens
SIR_SocialMedia_Syntens
 
Startup Weekend & LocomotiveCMS (In Chinese)
Startup Weekend & LocomotiveCMS (In Chinese)Startup Weekend & LocomotiveCMS (In Chinese)
Startup Weekend & LocomotiveCMS (In Chinese)
 
Bmx I Skateparks
Bmx I SkateparksBmx I Skateparks
Bmx I Skateparks
 

Ähnlich wie Basic Query Tuning Primer - Pg West 2009

Checking clustering factor to detect row migration
Checking clustering factor to detect row migrationChecking clustering factor to detect row migration
Checking clustering factor to detect row migrationHeribertus Bramundito
 
Explaining Explain
Explaining ExplainExplaining Explain
Explaining ExplainRobert Treat
 
Postgres performance for humans
Postgres performance for humansPostgres performance for humans
Postgres performance for humansCraig Kerstiens
 
Tony jambu (obscure) tools of the trade for tuning oracle sq ls
Tony jambu   (obscure) tools of the trade for tuning oracle sq lsTony jambu   (obscure) tools of the trade for tuning oracle sq ls
Tony jambu (obscure) tools of the trade for tuning oracle sq lsInSync Conference
 
Histograms : Pre-12c and Now
Histograms : Pre-12c and NowHistograms : Pre-12c and Now
Histograms : Pre-12c and NowAnju Garg
 
Common Performance Pitfalls in Odoo apps
Common Performance Pitfalls in Odoo appsCommon Performance Pitfalls in Odoo apps
Common Performance Pitfalls in Odoo appsOdoo
 
Writing efficient sql
Writing efficient sqlWriting efficient sql
Writing efficient sqlj9soto
 
Введение в современную PostgreSQL. Часть 2
Введение в современную PostgreSQL. Часть 2Введение в современную PostgreSQL. Часть 2
Введение в современную PostgreSQL. Часть 2Dzianis Pirshtuk
 
Using PostgreSQL statistics to optimize performance
Using PostgreSQL statistics to optimize performance Using PostgreSQL statistics to optimize performance
Using PostgreSQL statistics to optimize performance Alexey Ermakov
 
Introduction to Parallel Execution
Introduction to Parallel ExecutionIntroduction to Parallel Execution
Introduction to Parallel ExecutionDoug Burns
 
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and AdvisorsYour tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and AdvisorsJohn Kanagaraj
 
Materialized views in PostgreSQL
Materialized views in PostgreSQLMaterialized views in PostgreSQL
Materialized views in PostgreSQLAshutosh Bapat
 
11 Things About11g
11 Things About11g11 Things About11g
11 Things About11gfcamachob
 
11thingsabout11g 12659705398222 Phpapp01
11thingsabout11g 12659705398222 Phpapp0111thingsabout11g 12659705398222 Phpapp01
11thingsabout11g 12659705398222 Phpapp01Karam Abuataya
 
Top 10 Oracle SQL tuning tips
Top 10 Oracle SQL tuning tipsTop 10 Oracle SQL tuning tips
Top 10 Oracle SQL tuning tipsNirav Shah
 
John Melesky - Federating Queries Using Postgres FDW @ Postgres Open
John Melesky - Federating Queries Using Postgres FDW @ Postgres OpenJohn Melesky - Federating Queries Using Postgres FDW @ Postgres Open
John Melesky - Federating Queries Using Postgres FDW @ Postgres OpenPostgresOpen
 
Adaptive Query Optimization
Adaptive Query OptimizationAdaptive Query Optimization
Adaptive Query OptimizationAnju Garg
 
query-optimization-techniques_talk.pdf
query-optimization-techniques_talk.pdfquery-optimization-techniques_talk.pdf
query-optimization-techniques_talk.pdfgaros1
 

Ähnlich wie Basic Query Tuning Primer - Pg West 2009 (20)

query_tuning.pdf
query_tuning.pdfquery_tuning.pdf
query_tuning.pdf
 
Checking clustering factor to detect row migration
Checking clustering factor to detect row migrationChecking clustering factor to detect row migration
Checking clustering factor to detect row migration
 
Explaining Explain
Explaining ExplainExplaining Explain
Explaining Explain
 
Postgres performance for humans
Postgres performance for humansPostgres performance for humans
Postgres performance for humans
 
Do You Know The 11g Plan?
Do You Know The 11g Plan?Do You Know The 11g Plan?
Do You Know The 11g Plan?
 
Tony jambu (obscure) tools of the trade for tuning oracle sq ls
Tony jambu   (obscure) tools of the trade for tuning oracle sq lsTony jambu   (obscure) tools of the trade for tuning oracle sq ls
Tony jambu (obscure) tools of the trade for tuning oracle sq ls
 
Histograms : Pre-12c and Now
Histograms : Pre-12c and NowHistograms : Pre-12c and Now
Histograms : Pre-12c and Now
 
Common Performance Pitfalls in Odoo apps
Common Performance Pitfalls in Odoo appsCommon Performance Pitfalls in Odoo apps
Common Performance Pitfalls in Odoo apps
 
Writing efficient sql
Writing efficient sqlWriting efficient sql
Writing efficient sql
 
Введение в современную PostgreSQL. Часть 2
Введение в современную PostgreSQL. Часть 2Введение в современную PostgreSQL. Часть 2
Введение в современную PostgreSQL. Часть 2
 
Using PostgreSQL statistics to optimize performance
Using PostgreSQL statistics to optimize performance Using PostgreSQL statistics to optimize performance
Using PostgreSQL statistics to optimize performance
 
Introduction to Parallel Execution
Introduction to Parallel ExecutionIntroduction to Parallel Execution
Introduction to Parallel Execution
 
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and AdvisorsYour tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
 
Materialized views in PostgreSQL
Materialized views in PostgreSQLMaterialized views in PostgreSQL
Materialized views in PostgreSQL
 
11 Things About11g
11 Things About11g11 Things About11g
11 Things About11g
 
11thingsabout11g 12659705398222 Phpapp01
11thingsabout11g 12659705398222 Phpapp0111thingsabout11g 12659705398222 Phpapp01
11thingsabout11g 12659705398222 Phpapp01
 
Top 10 Oracle SQL tuning tips
Top 10 Oracle SQL tuning tipsTop 10 Oracle SQL tuning tips
Top 10 Oracle SQL tuning tips
 
John Melesky - Federating Queries Using Postgres FDW @ Postgres Open
John Melesky - Federating Queries Using Postgres FDW @ Postgres OpenJohn Melesky - Federating Queries Using Postgres FDW @ Postgres Open
John Melesky - Federating Queries Using Postgres FDW @ Postgres Open
 
Adaptive Query Optimization
Adaptive Query OptimizationAdaptive Query Optimization
Adaptive Query Optimization
 
query-optimization-techniques_talk.pdf
query-optimization-techniques_talk.pdfquery-optimization-techniques_talk.pdf
query-optimization-techniques_talk.pdf
 

Kürzlich hochgeladen

Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxLoriGlavin3
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteDianaGray10
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxNavinnSomaal
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningLars Bell
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett
 

Kürzlich hochgeladen (20)

Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
 
DMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special EditionDMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special Edition
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test Suite
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptx
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
 

Basic Query Tuning Primer - Pg West 2009

  • 1. Basic Query Tuning Primer Pg West 2009 2009-10-17
  • 2.
  • 8.
  • 9.
  • 10. ALTER TABLE [table] ALTER COLUMN [column] SET STATISTICS [n]
  • 11. SET default_statistics_target = [n], work_mem = [n]
  • 13. Reorganize the SQL (e.g. filter large tables directly, avoid joins, refactor to temp tables or WITH clause, etc.)
  • 14. Rarely appropriate: SET [gucs], table-level autovac storage-params, denormalize columns, partition large table
  • 15.
  • 16. System settings: kernel cache page size, tcp buffer size, dirty page flush policy, io scheduler, etc.
  • 17. Filesystem settings: type, readahead, direct/buffered io, extent size, journaling policy, internal/external journal, atime/mtime maint., existence of snapshots, etc.
  • 18. Postgres settings: version, GUCS (default_statistics_target, shared_buffers, temp_buffers, work_mem, wal_buffers, *_cost, enable_*, effective_cache_size, fsync), WAL on separate disks, etc.
  • 19. Total workload: cumulative effect of all tasks running at a particular point in time, all of which compete for resources (e.g. cpu, memory, cache, disk I/O, locks).
  • 20. Accumulated state: contents of caches (Pg, kernel, controller), physical file fragmentation on disk, optimizer statistics (specific to cluster; not portable via pg_dump)
  • 21. What is an execution plan? To see a query's execution plan, use the EXPLAIN command before the query. To also run the query and capture how much time was spent in each step of the plan, use EXPLAIN ANALYZE . Original SQL: pg841=# select count(*) from orders where created_ts >= '2009-09-01'::timestamp ; count ------- 65472 (1 row) Execution plan with estimated costs and row-counts: pg841=# explain select count(*) from orders where created_ts >= '2009-09-01'::timestamp ; QUERY PLAN ------------------------------------------------------------------------------------ Aggregate (cost=2100.85..2100.86 rows=1 width=0) -> Seq Scan on orders (cost=0.00..1937.00 rows= 65538 width=0) Filter: (created_ts >= '2009-09-01 00:00:00'::timestamp without time zone) (3 rows) Execution plan with actual runtimes and row-counts: pg841=# explain analyze select count(*) from orders where created_ts >= '2009-09-01'::timestamp ; QUERY PLAN ------------------------------------------------------------------------------------------------------------------- Aggregate (cost=2100.85..2100.86 rows=1 width=0) (actual time=251.210..251.212 rows=1 loops=1) -> Seq Scan on orders (cost=0.00..1937.00 rows= 65538 width=0) (actual time=0.018..140.495 rows= 65472 loops=1) Filter: (created_ts >= '2009-09-01 00:00:00'::timestamp without time zone) Total runtime: 251.269 ms (4 rows)
  • 22. Each step in the execution plan is a node in a tree hierarchy. The leftmost (“top”) node pulls data from its children, which do the same for their children. Nodes with no children start working first, since they gather the data from tables or indexes to be processed by the rest of the plan nodes. Run Order QUERY PLAN ------- --------------------------------------------------------------------------------------------------------------- 4th HashAggregate (cost= 1119.28 ..1120.75 rows=98 width=12) 3rd -> Nested Loop (cost= 0.00 ..1118.79 rows=98 width=12) 1st -> Index Scan using idx_order_details_product_id on order_details (cost= 0.00 ..397.99 rows=98 width=8) Index Cond: (product_id = 1000) 2nd -> Index Scan using orders_pkey on orders (cost= 0.00 ..7.34 rows=1 width=12) Index Cond: (orders.order_id = order_details.order_id) Filter: (orders.created_ts >= '2009-09-01 00:00:00'::timestamp without time zone) (7 rows) Reading an execution plan: In what order do the nodes run?
  • 23. The two numbers for “cost” and “actual time” represent when the 1st and last rows will be output by that node. Different kinds of nodes have different amounts of lag between when it receives its first input and when it produces its first output. This “startup cost” is implied by the difference in the first cost value of the node vs. its slowest-starting child . Here, the Hash node has high startup cost – it cannot feed data to its parent ( Hash Join ) until it receives and processes all of the data from its child Seq Scan node. So the Seq Scan node's final cost ( 1937.00 ) becomes the Hash node's initial cost ( 1937.00 ). In practice, the child's actual completion time ( 151.703 ) is a lower bound for the Hash node's first-output time ( 288.962 ). QUERY PLAN ------------------------------------------------------------------------------------------------------------------------- Hash Join ( cost= 3013.22 .. 45974.93 rows=651270 width=12) ( actual time= 289.152 .. 6297.282 rows=654720 loops=1) Hash Cond: (order_details.order_id = orders.order_id) -> Seq Scan on order_details ( cost= 0.00 .. 14902.00 rows=1000000 width=12) ( actual time= 0.037 .. 2026.802 rows=1000000 loops=1) -> Hash ( cost= 1937.00 .. 1937.00 rows=65538 width=8) ( actual time= 288.962 .. 288.962 rows=65472 loops=1) -> Seq Scan on orders ( cost= 0.00 .. 1937.00 rows=65538 width=8) ( actual time= 0.014 .. 151.703 rows=65472 loops=1) Filter: (created_ts >= '2009-09-01 00:00:00'::timestamp without time zone) Total runtime: 7349.662 ms (7 rows) Reading an execution plan: What do the numbers mean for "cost" and "actual time"?
  • 24. Access methods are used by leaf nodes to pull data from tables/indexes. Join methods specify which type of algorithm will be used to implement each of the query's joins. QUERY PLAN ------------------------------------------------------------------------------------------------------ Sort (cost=248.54..248.79 rows=99 width=8) Sort Key: o.order_id -> Nested Loop (cost=4.34..245.26 rows=99 width=8) -> Nested Loop (cost=4.34..234.94 rows=10 width=4) -> Seq Scan on customers c (cost=0.00..194.00 rows=1 width=4) Filter: (name = 'JOHN DOE'::text) -> Bitmap Heap Scan on orders o (cost=4.34..40.82 rows=10 width=8) Recheck Cond: (o.cust_id = c.cust_id) -> Bitmap Index Scan on idx_orders_cust_id (cost=0.00..4.34 rows=10 width=0) Index Cond: (o.cust_id = c.cust_id) -> Index Scan using order_details_pk on order_details od (cost=0.00..0.91 rows=10 width=8) Index Cond: (od.order_id = o.order_id) (12 rows) QUERY PLAN ---------------------------------------------------------------------------------------------------------------------- Aggregate (cost=18850.37..18850.38 rows=1 width=8) -> Hash Join (cost=53.18..18801.97 rows=9679 width=8) Hash Cond: (od.order_id = o.order_id) -> Seq Scan on order_details od (cost=0.00..14902.00 rows=1000000 width=8) -> Hash (cost=41.00..41.00 rows=974 width=4) -> Index Scan using pidx_orders_order_id_not_shipped on orders o (cost=0.00..41.00 rows=974 width=4) Filter: is_paid (7 rows) Reading an execution plan: What are access methods and join methods?
  • 25. Create and ANALYZE a small toy table. pg841=# create temp table my_tab1 as select * from orders limit 100 ; SELECT pg841=# analyze my_tab1 ; ANALYZE EXPLAIN a simple query against it. pg841=# explain select * from my_tab1 where created_ts > '2009-10-14 08:14:26'::timestamp - '1 day'::interval ; QUERY PLAN ------------------------------------------------------------------------------------ Seq Scan on my_tab1 (cost=0.00.. 2.25 rows=17 width=25) Filter: (created_ts > '2009-10-13 08:14:26' ::timestamp without time zone) (2 rows) The planner was able to combine the filter's two literals. The table is accessed by SeqScan, since there are no indexes yet on this temp table. Add an index, force the planner not to SeqScan, and compare the same query's cost estimates for IndexScan versus SeqScan. pg841=# create index idx_my_tab1_test on my_tab1 ( created_ts ) ; CREATE INDEX pg841=# set enable_seqscan = off ; SET pg841=# explain select * from my_tab1 where created_ts > '2009-10-14 08:14:26'::timestamp - '1 day'::interval ; QUERY PLAN ---------------------------------------------------------------------------------------- Index Scan using idx_my_tab1_test on my_tab1 ( cost =0.00.. 8.55 rows=17 width=25) Index Cond: (created_ts > '2009-10-13 08:14:26'::timestamp without time zone) (2 rows) pg841=# reset all ; RESET The optimizer prefers (assigns a lower cost) to SeqScan this table because it is so tiny (1 block). An IndexScan would require reading a 2nd block (the index itself) and doing extra comparisons. Play with small toy queries.
  • 26. Run your query with EXPLAIN (or if practical, EXPLAIN ANALYZE), and look for nodes where the cost, row-count, or actual_time significantly increases compared to its children. In this example, the SQL is missing its join criteria. The estimated cost and row-count skyrocket in the Nested Loop node, because it is returning the cross-product of all rows from both its input nodes. pg841=# explain pg841-# select pg841-# customers.cust_id as customer_id, pg841-# max(customers.name) as customer_name, pg841-# count(distinct orders.order_id) as num_orders, pg841-# max(orders.shipped_ts) as latest_shipment_datetime pg841-# from orders, customers , products /* THIS JOIN TO “products” IS SPURIOUS */ pg841-# where pg841-# orders.cust_id = customers.cust_id pg841-# and orders.created_ts >= now() - '30 days'::interval pg841-# group by pg841-# customers.cust_id pg841-# order by num_orders desc pg841-# limit 10 pg841-# ; QUERY PLAN ----------------------------------------------------------------------------------------------------------------------- Limit ( cost =15307415.27.. 15307415.29 rows=10 width=29) -> Sort ( cost =15307415.27.. 15307440.27 rows=10000 width=29) Sort Key: (count(DISTINCT orders.order_id)) -> GroupAggregate ( cost =185.03.. 15307199.17 rows=10000 width=29) -> Nested Loop ( cost =185.03.. 10207124.17 rows=509990000 width=29) -> Merge Join ( cost =0.03.. 7139.17 rows=50999 width=29) Merge Cond: (customers.cust_id = orders.cust_id) -> Index Scan using customers_pkey on customers ( cost =0.00.. 318.48 rows=10000 width=17) -> Index Scan using idx_orders_cust_id on orders ( cost =0.00.. 6158.23 rows=50999 width=16) Filter: (orders.created_ts >= (now() - '30 days'::interval)) -> Materialize ( cost =185.00.. 285.00 rows=10000 width=0) -> Seq Scan on products ( cost =0.00.. 175.00 rows=10000 width=0) (12 rows) What does a "bad plan" look like? Does it imply possible tune-ups?
  • 27. Rerun the modified query with EXPLAIN to confirm that it looks better. The final cost estimate is much lower, and there are no more huge jumps in cost from one step to the next. pg841=# explain pg841-# select pg841-# customers.cust_id as customer_id, pg841-# max(customers.name) as customer_name, pg841-# count(distinct orders.order_id) as num_orders, pg841-# max(orders.shipped_ts) as latest_shipment_datetime pg841-# from orders, customers pg841-# where pg841-# orders.cust_id = customers.cust_id pg841-# and orders.created_ts >= now() - '30 days'::interval pg841-# group by pg841-# customers.cust_id pg841-# order by num_orders desc pg841-# limit 10 pg841-# ; QUERY PLAN ---------------------------------------------------------------------------------------------------- Limit ( cost =10243.47.. 10243.50 rows=10 width=29) -> Sort (cost=10243.47..10268.47 rows=10000 width=29) Sort Key: (count(DISTINCT orders.order_id)) -> GroupAggregate (cost=9214.91..10027.37 rows=10000 width=29) -> Sort (cost=9214.91..9342.40 rows=50997 width=29) Sort Key: customers.cust_id -> Hash Join (cost=294.00..4005.92 rows=50997 width=29) Hash Cond: (orders.cust_id = customers.cust_id) -> Seq Scan on orders (cost=0.00..2437.00 rows=50997 width=16) Filter: (created_ts >= (now() - '30 days'::interval)) -> Hash (cost=169.00..169.00 rows=10000 width=17) -> Seq Scan on customers (cost=0.00..169.00 rows=10000 width=17) (12 rows) After previewing the new plan with EXPLAIN, run EXPLAIN ANALYZE to confirm runtime is better. Retest after rewriting the query to remove the spurious join...
  • 28. Here's a trivial query joining 2 tables. pg841=# explain select count(*) from my_tab1 inner join my_tab2 using (key) ; QUERY PLAN --------------------------------------------------------------------------------------- Aggregate (cost=42.26..42.27 rows=1 width=0) -> Nested Loop (cost=0.00..42.01 rows=100 width=0) -> Seq Scan on my_tab1 (cost=0.00..2.00 rows=100 width=4) -> Index Scan using idx_my_tab2 on my_tab2 (cost=0.00..0.39 rows=1 width=4) Index Cond: (my_tab2.key = my_tab1.key) (5 rows) Notice that the Nested Loop node does not specify any join conditions, even though the SQL does. The Nested Loop is effectively cross-joining its 2 inputs. Why would it do that, when the SQL says to join on column key ? Because the join condition has been pushed down into child #2's index filter. So for each row from child #1 ( Seq Scan ), the parent ( Nested Loop ) calls child #2 ( Index Scan ), passing it info from child #1's row. If this session forbids the use of indexes, the join filter won't be pushed down. pg841=# set enable_indexscan = off ; SET pg841=# set enable_bitmapscan = off ; SET pg841=# explain select count(*) from my_tab1 inner join my_tab2 using (key) ; QUERY PLAN --------------------------------------------------------------------------- Aggregate (cost=229.35..229.36 rows=1 width=0) -> Nested Loop (cost=2.10..229.10 rows=100 width=0) Join Filter: (my_tab1.key = my_tab2.key) -> Seq Scan on my_tab1 (cost=0.00..2.00 rows=100 width=4) -> Materialize (cost=2.10..3.10 rows=100 width=4) -> Seq Scan on my_tab2 (cost=0.00..2.00 rows=100 width=4) (6 rows) pg841=# reset all ; RESET Join Conditions can sometimes be implemented by a child node.
  • 29. Stale or missing statistics: When was my table last analyzed? Review the last time when each table was analyzed (either manually or by autovacuum). Use your knowledge of how and when your data changes to decide if the statistics are likely to be out of date. pg841=# select pg841-# schemaname, pg841-# relname, pg841-# last_analyze as last_manual, pg841-# last_autoanalyze as last_auto, pg841-# greatest (last_analyze, last_autoanalyze) pg841-# from pg841-# pg_stat_user_tables pg841-# where pg841-# relname = 'my_tab' pg841-# ; -[ RECORD 1 ]------------------------------ schemaname | public relname | my_tab last_manual | 2009-10-03 18:45:57.627593-07 last_auto | 2009-10-03 23:08:32.914092-07 greatest | 2009-10-03 23:08:32.914092-07
  • 30. Stale or missing statistics: How many rows does the optimizer think my table has? How are my columns' values distributed? Many bad plans are due to poor cardinality estimates for one or more nodes. Sometimes this is due to stale or missing statistics. For example, if a column was added or a significant percentage of rows were inserted, deleted, or modified, then the optimizer statistics should be refreshed. You can view the table-level optimizer statistics in pg_class: pg841=# select reltuples , relpages from pg_class where relname = 'my_tab' ; reltuples | relpages -----------+---------- 1000 | 5 (1 row) And the more detailed column-level optimizer statistics are shown in pg_stats: pg841=# select * from pg_stats where tablename = 'my_tab' and attname = 'bar' ; -[ RECORD 1 ]-----+--------------------------- schemaname | public tablename | my_tab attname | bar null_frac | 0 avg_width | 4 n_distinct | 13 most_common_vals | {1,2} most_common_freqs | {0.707,0.207} histogram_bounds | {0,3,4,5,6,7,8,9,10,11,12} correlation | 0.876659
  • 31. A new index is usually helpful if it greatly reduces the number of rows the query must visit. This table has an index for all possible combinations of its columns. pg841=# my_tab Table "public.my_tab" Column | Type | Modifiers --------+---------+----------- foo | integer | bar | integer | Indexes: "idx_my_tab_bar" btree (bar) "idx_my_tab_bar_foo" btree (bar, foo) "idx_my_tab_foo" btree (foo) "idx_my_tab_foo_bar" btree (foo, bar) At least 1 is unnecessary, and up to 2 could be dropped without forcing any query to use a Seq Scan. pg841=# explain select * from my_tab where foo = 10 and bar = 10 ; QUERY PLAN --------------------------------------------------------------------------------- Index Scan using idx_my_tab_bar_foo on my_tab (cost=0.00..8.27 rows=1 width=8) Index Cond: ((bar = 10) AND (foo = 10)) (2 rows) pg841=# drop index idx_my_tab_bar_foo ; DROP INDEX pg841=# drop index idx_my_tab_foo_bar ; DROP INDEX pg841=# explain select * from my_tab where foo = 10 and bar = 10 ; QUERY PLAN ----------------------------------------------------------------------------- Index Scan using idx_my_tab_foo on my_tab (cost=0.00..8.27 rows=1 width=8) Index Cond: (foo = 10) Filter: (bar = 10) (3 rows) When will adding a new index improve query performance?
  • 32.
  • 33.
  • 34. Thanks for coming! Questions?
  • 36. Which other tables also have a column named "cust_id"? Relevance: Find join candidates, missing foreign key constraints, or otherwise related tables in a large/complex schema. pg841=# select pg841-# attrelid::regclass as table_name, pg841-# attname as column_name pg841-# from pg_attribute pg841-# where attname ~ 'cust_id' pg841-# ; table_name | column_name ------------------------+------------- customers | cust_id other_namespace.orders | cust_id (2 rows)
  • 37.
  • 38. idx_tup_read = How many rows was this index used to read from the table?
  • 39. idx_tup_fetch = How many of those table rows were valid "live" rows? pg841=# select pg841-# indexrelname, pg841-# idx_scan , pg841-# idx_tup_read , pg841-# idx_tup_fetch pg841-# from pg_stat_user_indexes pg841-# where relname = 'my_tab' pg841-# order by indexrelname pg841-# ; indexrelname | idx_scan | idx_tup_read | idx_tup_fetch --------------------+----------+--------------+--------------- idx_my_tab_bar | 2 | 20 | 17 idx_my_tab_foo | 1 | 3 | 3 idx_my_tab_foo_bar | 1 | 1 | 1 (3 rows)
  • 40.
  • 41. If an index was used more often in the past than recently, its counter may still be high.
  • 42. Newly created indexes have had less time to accumulate hits.
  • 43.
  • 44. If no other index can be used, the query will instead SeqScan the table. This may be more or less efficient, depending on index selectivity, caching of table/index blocks, and relative speed of sequential vs. scattered I/O for uncached blocks.
  • 45. What functions/aggregates are available? Relevance: Know what your database can do for you. In Pg 8.3 and earlier : Normal functions are shown by f . It includes built-in functions (from pg_catalog) as well as user-defined functions (in your search_path, e.g. “public”): pg838=# f List of functions Schema | Name | Result data type | Argument data types ------------+---------+------------------+--------------------- pg_catalog | abs | bigint | bigint ... public | my_func | integer | integer ... (1853 rows) Aggregate functions are similarly shown by a . pg838=# a List of aggregate functions Schema | Name | Result data type | Argument data types | Description ------------+------+------------------+---------------------+------------- pg_catalog | avg | double precision | real | ... (117 rows)
  • 46.
  • 47. includes aggregates, in addition to normal functions pg841=# f List of functions Schema | Name | Result data type | Argument data types | Type --------+---------+------------------+---------------------+-------- public | my_aggr | anyarray | anyelement | agg public | my_func | integer | integer | normal (2 rows) To list the built-in functions/aggregates in Pg 8.4, use either of these: pg841=# fS pg841=# f pg_catalog. ... (2208 rows)
  • 48. What functions/aggregates are available? (cont.) Wild cards (*?) are allowed, as with most psql metacommands: pg841=# fS bool* List of functions Schema | Name | Result data type | Argument data types | Type ------------+-------------------+------------------+---------------------+-------- pg_catalog | bool | boolean | integer | normal pg_catalog | bool_and | boolean | boolean | agg ... (16 rows) For more options, see psql's built-in help: pg841=# ... Informational (options: S = show system objects, + = additional detail) ... f[antw][S+] [PATRN] list [only agg/normal/trigger/window] functions ...
  • 49. How is function “my_func” implemented? Relevance: If a function-call is a query's bottleneck, seeing its code may suggest a tune-up. View the source code for your user-defined functions written in an interpreted language (e.g. SQL, plpgsql, plperl, etc.): pg841=# f+ my_func List of functions -[ RECORD 1 ]-------+---------------- Schema | public Name | my_func Result data type | integer Argument data types | integer Type | normal Volatility | immutable Owner | mss Language | sql Source code | select $1 + 1 Description | Or (in Pg 8.4) open the function in an editor. (Be careful what you change!) pg841=# f my_func(int) Alternately, query the catalog directly: pg841=# select prosrc from pg_proc where oid = 'my_func(int)' ::regprocedure ; prosrc ----------------- select $1 + 1
  • 50. What are the other concurrent sessions doing? Relevance: Know your system's workload – other concurrent sessions' queries compete for system resources (disk I/O, CPU time, cache, work mem, locks, network bandwidth, temp space, etc.). The pg_stat_activity table gives 1 row per session: pg841=# select * pg841-# from pg_stat_activity pg841-# where procpid != pg_backend_pid() pg841-# order by xact_start pg841-# ; -[ RECORD 1 ]-+------------------------------------------------------ datid | 16480 datname | pg841 procpid | 2646 usesysid | 10 usename | mss current_query | select count(1) from orders o1 cross join orders o2 ; waiting | f xact_start | 2009-10-04 15:35:25.637107-07 query_start | 2009-10-04 15:35:41.666956-07 backend_start | 2009-10-04 15:34:08.653364-07 client_addr | 127.0.0.1 client_port | 59733 The meaning of each column is listed in the docs: http://www.postgresql.org/docs/8.4/static/monitoring-stats.html
  • 51. Is lock contention affecting my session? Relevance: While waiting for a lock, a query is doing no useful work. First we'll create some lock contention: Session #1: create index idx_customers_foo on customers ( foo ) ; Session #2: analyze customers ; Session #3: vacuum customers ; Diagnosis, the easy way: “ pg_stat_activity” reports which sessions are waiting due to lock contention, but it gives few details. pg841=# select procpid, usename, xact_start, query_start, current_query pg841-# from pg_stat_activity pg841-# where waiting pg841-# order by query_start pg841-# ; procpid | usename | xact_start | query_start | current_query ---------+---------+---------------------+---------------------+--------------------- 7165 | mss | 2009-10-10 12:58:08 | 2009-10-10 12:58:14 | analyze customers ; 7189 | mss | 2009-10-10 12:59:01 | 2009-10-10 12:59:01 | vacuum customers ; (2 rows) Sometimes the stalled queries are easy to read (as above), so you can guess what lock is needed. Whenever it is not obvious, you can ask the database, by looking in “pg_locks”.
  • 52. Is lock contention affecting my session? (cont.) Diagnosis, the meticulous way: Look in “pg_locks” for more details about which lock each of the stalled sessions needs next but cannot get. pg841=# select pg841-# pid , pg841-# locktype, pg841-# mode , pg841-# relation::regclass as relname , pg841-# transactionid as target_xid, pg841-# virtualxid as target_vxid , pg841-# virtualtransaction as my_vxid , pg841-# granted pg841-# from pg_locks pg841-# where pid in (7189, 7165) pg841-# order by pid, locktype, mode, relation pg841-# ; pid | locktype | mode | relname | target_xid | target_vxid | my_vxid | granted ------+------------+--------------------------+----------------------------+------------+-------------+---------+--------- 7165 | relation | AccessShareLock | pg_class | | | 3/2 | t 7165 | relation | AccessShareLock | pg_namespace | | | 3/2 | t 7165 | relation | AccessShareLock | pg_class_oid_index | | | 3/2 | t 7165 | relation | AccessShareLock | pg_class_relname_nsp_index | | | 3/2 | t 7165 | relation | AccessShareLock | pg_namespace_nspname_index | | | 3/2 | t 7165 | relation | AccessShareLock | pg_namespace_oid_index | | | 3/2 | t 7165 | relation | ShareUpdateExclusiveLock | customers | | | 3/2 | f 7165 | virtualxid | ExclusiveLock | | | 3/2 | 3/2 | t 7189 | relation | ShareUpdateExclusiveLock | customers | | | 4/4 | f 7189 | virtualxid | ExclusiveLock | | | 4/4 | 4/4 | t (10 rows)
  • 53. Who is blocking my query? Relevance: To prevent recurring lock contention you must identify both the waiters and the blocker. May be obvious from examining the waiters' stalled “current_query”. If not, narrow the list of suspected blockers by: (1) Find which resource is contended (e.g. which table or row). (2) Find which other transactions hold a conflicting lock on that resource. The first step we did in the previous slides. The second step again visits “pg_locks”. pg841=# select pg841-# pid, pg841-# locktype, pg841-# mode, pg841-# relation::regclass as relname, pg841-# transactionid as target_xid, pg841-# virtualxid as target_vxid, pg841-# virtualtransaction as my_vxid, pg841-# granted pg841-# from pg_locks pg841-# where relation = ' customers '::regclass pg841-# order by pid, locktype, mode, relation pg841-# ; pid | locktype | mode | relname | target_xid | target_vxid | my_vxid | granted ------+----------+--------------------------+-----------+------------+-------------+---------+--------- 7141 | relation | AccessShareLock | customers | | | 2/4 | t 7141 | relation | ShareLock | customers | | | 2/4 | t 7165 | relation | ShareUpdateExclusiveLock | customers | | | 3/2 | f 7189 | relation | ShareUpdateExclusiveLock | customers | | | 4/4 | f (4 rows)
  • 54. Which locking modes conflict? Lock modes Which lock modes conflict? Examples of commands that set these modes on each table. Locktype “transactionid” and “virtualxid” use lock modes “Exclusive” and “Share”. Exclusive is used to lock your own transactionid, and Share is used to wait for a different transaction (e.g. when waiting for a row-level lock).
  • 55. This example supposes that an important query only cares about a predefined set of rare values in a column (i.e. find which orders are not yet paid for). A normal index would suffice, but a partial index is even more efficient. A partial index is only usable if its own WHERE clause is included in the query's. pg841=# select is_paid, count(1) from orders group by is_paid ; is_paid | count ---------+------- f | 945 t | 99055 (2 rows) pg841=# explain analyze select count(*) from orders where not is_paid ; QUERY PLAN -------------------------------------------------------------------------------------------------------------- Aggregate (cost=1689.46..1689.47 rows=1 width=0) (actual time=32.861..32.863 rows=1 loops=1) -> Seq Scan on orders (cost=0.00.. 1687.00 rows=983 width=0) (actual time=0.036.. 31.221 rows=945 loops=1) Filter: (NOT is_paid) Total runtime: 32.921 ms (4 rows) pg841=# create index pidx_orders on orders ( order_id ) where not is_paid ; CREATE INDEX pg841=# explain analyze select count(*) from orders where not is_paid ; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------- Aggregate (cost=47.27..47.28 rows=1 width=0) (actual time=5.282..5.284 rows=1 loops=1) -> Index Scan using pidx_orders on orders (cost=0.00.. 44.81 rows=983 width=0) (actual time=0.097.. 3.569 rows=945 loops=1) Total runtime: 5.352 ms (3 rows) When will adding a new index improve query performance? Partial index example
  • 56. Narration. Query and plan. Title