All things vacuum in Postgresql

1. VACUUM in
PostgreSQL
Zalando SE
Rafia Sabih
26-09-2018
1

2. 2
Outline
● Why VACUUM
● Things around VACUUM
● VACUUM utility
● Inside VACUUM
● Proposals around VACUUM
2

3. Why VACUUM?
3

4. 4
Update Tuples in PostgreSQL
T1 T2 T3
T4
T1
Old
T2
T3
T4
Update T2
New
T2
Page Page
Both versions of the
tuples exist in the page
4

5. 5
Delete tuples in PostgreSQL
T1 T2 T3
T4
T1 T2 T3
T4
Delete T3
Page Page
Tuple is marked deleted but
remains in the page and its
space remains unused
5

6. 6
Why VACUUM
● An UPDATE or DELETE of a row does not immediately removes the old version
○ VACUUM removes dead row versions in tables and indexes
○ Does not return the space to the operating system
■ except in the special case where one or more pages at the end of a table
become entirely free and an exclusive table lock can be easily obtained
○ VACUUM FULL minimizes the size of the table by writing a complete new
version of the table
● To overcome this, a new storage engine namely zheap based on undo logs is
proposed
6

7. 7
Why VACUUM: Transaction wraparound
● The value of current txid reaches the maximum allowed value
○ The system remains blocked until old xacts are frozen
○ Frozen_txid
■ To avoid the problem of transaction wraparound
■ Reserved value is 2, meaning always visible
○ Freeze process rewrites the value of Xmin to frozen_txid for the tuples with
too old Xmin
○ The xids are checked against a cutoff -- vacuum_freeze_min_age
7

8. Around VACUUM!
8

9. Around VACUUM: CLOG and MultiXacts
● Commit log(CLOG) is a logical array for the status of the xacts
● Stored in shared memory and used for transaction lifecycle
● At every checkpoint or shutdown, flushed to disk in pg_xact directory
● Multixacts store information about row-level locks, updates etc. when multiple
transactions are involved
Committed Committed Aborted In-progress In-progress In-progress
1440 1441 1442 1443 1444 1445XIDs
status
9

10. Around VACUUM: VisibilityMap
● Introduced to help VACUUM process in identifying which pages of the relation
has dead tuples
● Provides information on if the tuples in the page are frozen
● Can be found in data directory with the suffix _vm
● Up-to date VMs speed-ups index-only scans
○ No need to visit corresponding heap pages
● Check extension pg_visibility for more information
10

11. Around VACUUM: FreeSpaceMap
● Tells about the free space available in the pages of the relation
○ Associated with every relation whether normal or index
● While inserting, it is required to check through FSM to find out which page has
enough space
● Pg_freespacemap extension can be used for better visualization
11

12. Around VACUUM: HeapOnlyTuple
● A precursor to the VACUUM utility
○ Performs the defragmentation of a page
● Reuse the space of the index entries of the deleted or obsolete tuples
● Only for the case when updated tuple does not change any indexed
columns
Col1 Col2 Col3 Col4
With HOT
A new tuple with unchanged indexed column
creates no new index entry
Without HOT
Every version of a row has its own index
entries
12

13. VACUUM utility
13

14. Tasks of VACUUM
● Prevent the issues of transaction or multi-xact id wraparound
■ Freeze xids when required
● To reclaim the space of dead tuples and indexes
■ Remove clog entries that are not required
■ Defragment pages with live tuples
● Update statistics of tables, it helps in better query plans
● Update visibility and freespace maps
● Remove CLOGs and multixacts
14

15. What can be vacuumed?
● Tuples or indexes that are not visible to any running transactions
○ Avoid long-running transactions
● Streaming replication with hot_standby_feedback enabled might also limit the
vacuum on the master
● On an internal level vacuum requires special type of lock on the buffer --
BufferLockForCleanup which locks the buffer and checks that no other process
has pinned that buffer
○ Long running select queries can also be blocking
15

16. VACUUM options
● Standard vacuum
● VACUUM FULL
○ Rewrites the table and indexes
○ Moderately-frequent standard VACUUM is better than infrequent VACUUM
FULL runs for maintaining heavily-updated tables
○ Alternative: pg_repack
● VACUUM FREEZE
○ Aggressively freezes tuples
● Auto-vacuum
○ To run vacuum and analyze commands in the background
● Vacuumdb
○ For multiple databases in a cluster
16

17. AUTOVACUUM
● Comprises of a launcher and workers
● Priority for each database
○ XID freeze
○ MXID freeze
○ Least recently auto-vacuumed
○ These priorities are not within database
● Multiple workers can work at the same time on a database
● Auto-vacuum can get terminated if other process needs access to tables
○ Workers tries to get lock, if not available skip the table for now
● Auto-vacuum running for xid-wraparound are more aggressive and can not be
canceled
● Temporary tables are ignored 17

18. Tuning VACUUM
● For autovacuum
○ Autovacuum_max_workers
○ Autovacuum_naptime
○ Autovacuum_vacuum_threshold
○ Autovacuum_vacuum_scale_factor
■ # dead_tup > (reltuples * scale_factor) + threshold
○ Autovacuum_work_mem
■ Default value -1 means take the value of maintenance_work_mem
● Vacuum_cost_delay
18

19. Take-away
● VACUUM
○ Removes dead tuples and indexes
○ Prevents the problem of transaction wraparound
○ No memory is returned to OS!
○ Autovacuum is your friend
● Obstacles for VACUUM
○ Long running transactions
○ Low autovacuum_work_mem
● Bloated indexes are more difficult than bloated relations
○ Reindex!
● pg_repack!
19

20. VACUUM internals
20

21. At a glance
● For each target relation
○ Acquire ShareUpdateExclusiveLock
○ Scan all pages to get all dead tuples, and freeze old tuples if necessary
■ The list of dead tuples and the index tuples pointing to dead tuples is
limited by maintenance_work_mem
■ If the scan is not complete and amount of memory is consumed, it
goes ahead to perform the next work for the listed tuples and
continues the scan later
21

22. At a glance
○ Remove the index tuples that point to the respective dead tuples if exists
○ For each page of the table
■ Remove the dead tuples and reallocate the live tuples in the page
■ Update FSM and VM
22

23. At a glance
● Truncate the last page if possible
○ Update both the statistics and system catalogs of the target table
○ Release ShareUpdateExclusiveLock
● Remove both unnecessary files and pages of the clog if possible
23

24. Important routines
● Vacuum
○ Entry point for vacuum and analyze commands
○ Pgstat_vacuum_stat : gives information about the dead objects - db and
relations
● Vacuum_rel
○ Checks ownership of the table underway, skips if not allowed
○ Checks if the relation is vacuum-able - ! matview, !partition_table,
!temp_table of other backends, etc.
○ Vacuums toast tables if present
○ Calls cluster (for vacuum full) or heap_vacuum_rel as required
24

25. Important routines
● Heap_vacuum_rel
○ Scans the heap and indexes
○ Calls lazy_truncate_heap as required
■ Deletes the last pages of the relation if empty
○ Freeze xid or mxid if required
● Lazy_scan_heap
○ Skip all-visible pages
■ But only when the number of pages to be skipped passes
SKIP_PAGES_THRESHOLD
■ Skipping a page also means not updating relfrozenxid!
■ In aggressive mode (freeze), skip only if all-frozen
25

26. Important routines
● Lazy_vacuum_indexes
○ Deletes index entries corresponding to dead tuples
● Lazy_cleanup_index
○ Calls index specific cleanup method
○ Update the index stats
● Lazy_vacuum_heap
○ Deletes the tuples only after corresponding index entries are deleted
○ Actually removes dead tuples and compacts the space
26

27. Important routines
● Vac_update_datfrozenxid
○ Updates datfrozenxid and datminmxid for the database
○ If one of them got updated, truncate pg_xact and/or pg_multixact
○ Vac_truncate_clog
■ Truncates the clog files and pages
● Analyze_rel
○ Collect the statistics of updated tables
○ Update the pg_statistics, pg_class
○ Collect all unique columns and indexes to be analysed
○ Lower bound for sample size is 100 rows
27

28. More to come...
● Zheap
○ In-place updates
○ Older versions of tuples are kept in a separate entity - undo logs
● Block level parallel vacuum (and autovacuum)
○ Parallelize vacuum of a table among workers
● Resume vacuum or autovacuum after interruption
○ Save the blocks that were vacuumed previously and resume from there
● Reduce duplicates in BTree indexes
○ For each key that have duplicates, a list is created which contains the
info about the duplicate tuples
○ Improves index bloat
28