9.6_Course Material-Postgresql_002.pdf

PostgreSQL - Advanced Administration Training
© Copyright EnterpriseDB Corporation, 2015. All rights reserved. 1
Course Agenda
• Introduction
• System Architecture
• Installation and Database
Clusters
• Configuration
• Creating and Managing
Databases
• User Tools - CUI and GUI

• Security
• SQL Primer
• Backup, Recovery and PITR
• Routine Maintenance Tasks
• Data Dictionary
• Loading and Moving Data
• SQL Tuning
• Performance Tuning •
Streaming Replication •
Connection Pooling using
pgpool-II
• Table Partitioning
• Extensions
• Monitoring
• Upgrading Best Practices

Introduction
Objectives
• In this module we will cover:
− PostgreSQL
− Features of PostgreSQL
− General Database Limits
− Common Database Object Names

Facts about PostgreSQL
• The world‟s most advanced open source
database • Designed for extensibility and
customization • ANSI/ISO compliant SQL support
• Actively developed for more than 20 years

− University Postgres (1986-1993)
− Postgres95 (1994-1995)
− PostgreSQL (1996-current)
PostgreSQL Community
• Community mailing lists
www.Postgresql.org/community/lists
• Support Forums
http://forums.enterprisedb.com/forums/list.page
• Commercial SLAs

www.enterprisedb.com/products-services-training/subscriptions
PostgreSQL Lineage

Major Features of PostgreSQL

• Portable:
− Written in ANSI C
− Supports Windows, Linux, Mac OS/X and major UNIX platforms • Reliable:
− ACID Compliant
− Supports Transactions
− Supports Save points
− Uses Write Ahead Logging (WAL)
• Scalable:
− Uses Multi-version Concurrency Control
− Supports Table Partitioning
− Supports Tablespaces
Major Features of PostgreSQL (continued)

• Secure:
− Employs Host-Based Access Control
− Provides Object-Level Permissions
− Supports Logging
− SSL 36
• Recovery and Availability:
− Streaming Replication
− Replication Slots
− Supports Hot-Backup, pg_basebackup
− Point-in-Time Recovery
• Advanced:
− Supports Triggers and Functions
− Supports Custom Procedural Languages PL/pgSQL, PL/Perl, PL/TCL,
PL/PHP, PL/Java
− Upgrade using pg_upgrade
− Unlogged Tables
− Materialized Views
Write Ahead Logging : REDO and Archiving

• PostgreSQL uses Write-Ahead Logging (WAL) as its approach to
transaction logging.
• WAL's central concept is that changes to data files (where tables and
indexes reside) must be written only after those changes have been logged,
that is, when log records describing the changes have been flushed to
permanent storage.
• No need to flush data pages to disk on every transaction commit.
• In the event of a crash we will be able to recover the database using the
logs
• This is roll-forward recovery, also

known as REDO
Rollback or Undo
• Dynamic allocation of disk space is used for the storage and processing of records within
tables.
• The files that represent the table grow as the table grows.
• It also grows with transactions that are performed against it.
• The data is stored within the file that represents the table.
• When deletes and updates are performed on a table, the file that represents the object
will contain the previous data

• This space gets reused, but need to force recovery of used space. (Using Vaccum)
General Database Limits
Limit
Maximum Database Size
Value
Unlimited
Maximum Table Size 32 TB
Maximum Row Size 1.6 TB
Maximum Field Size 1 GB
Maximum Rows per Table Unlimited
Maximum Columns per Table 250-1600 (Depending on Column types)
Maximum Indexes per Table Unlimited

Common Database Object Names
Industry Term
Table or Index
PostgreSQL Term
Relation
Row Tuple
Column Attribute
Data Block Page (when block is on disk)
Page Buffer (when block is in memory)
Summary

• In this module we covered:
− PostgreSQL
− Features of PostgreSQL
− General Database Limits
− Common Database Object Names

System Architecture
Module Objectives
• Architectural Summary
• Process and Memory Architecture
• Utility Processes
• Connection Request-Response
• Disk Read Buffering Disk •
Write Buffering
• Background Writer Cleaning Scan
• Commit and Checkpoint
• Statement Processing
• Physical Database Architecture
• Data Directory Layout
• Installation Directory Layout
• Page Layout

Architectural Summary
• PostgreSQL uses processes, not threads •
Postmaster process acts as supervisor • Several
utility processes perform background work
− postmaster starts them, restarts them if they die
• One backend process per user session −
postmaster listens for new connections

Process and Memory Architecture

Postmaster Shared Memory
Shared Buffers WAL Buffers
Process Array
BGWRITER
CHECKPOINTER
AUTO --
VACUUM
WAL Writer
STATS
CO
LL
EC
TO
R
AR
CH
IV
ER
LOG WRITER
Data
Data
Files
Files
Error Log
Files
WAL
WAL
Segment
s Segments
Archived
Archive
d WAL
WAL

18
Utility Processes
• Background writer
− Writes updated data blocks to disk
• WAL writer
− Flushes write-ahead log to disk
• Checkpointer process
− Automatically performs a checkpoint based on config parameters
• Autovacuum launcher

− Starts Autovacuum workers as needed
• Autovacuum workers
− Recover free space for reuse
More Utility Process
• Logging collector
− Routes log messages to syslog, eventlog, or log files
• Stats collector
− Collects usage statistics by relation and block
• Archiver
− Archives write-ahead log files

Postmaster as Listener
• Postmaster is the master process called postgres
• Listens on 1-and-only-1 tcp port • Receives client connection request

21
Client request a
connection
Postmaster

Shared Memory
Disk Read Buffering
• PostgreSQL buffer cache
(shared_buffers) reduces
physical reads from
storage.
• Read the block once, then
examine it many times in
cache.
postgres
postgres
postgres

Shared (data) Buffers
Stable Database
Disk Write Buffering
• Blocks are written to disk
only when needed:
− To make room for new
blocks
− At checkpoint time

postgres
postgres
postgres Shared (data) Buffers
CHECKPOINT
Stable Database

Background Writer Cleaning Scan
• Background writer scan
attempts to ensure an
adequate supply of clean
buffers.
• bgwriter writes dirty blocks to
storage as needed.
postgres db2 get db cfg for sample |
grep -i log postgres
postgres Shared
(data) Buffers
BGWRITER
Stable Database

Write Ahead Logging
• Backend process that writes
data to Write Ahead Logging
(WAL) buffers.
postgres
postgres
postgres
• Flush WAL buffers
periodically (WAL
writer), on commit, or
when buffers are full.
• Group commit.

Shared (data) Buffers
Stable
Database
WAL
Buffer
Transaction
Log
Transaction Log Archiving
• Archiver spawns a task to

copy pg_xlog log files to
archive location when full.
postgres
postgres
postgres
WAL
ARCHIVE COMMAND
Shared (data)
Buffers

Stable Database
Buffer
Transaction
Log
Commit and Checkpoint

• Before Commit
− Uncommitted updates are in memory.
• After Commit
− Committed updates written from shared memory to WAL log file (on disk).
• After Checkpoint
− Modified data pages are written from shared memory to the data files.
Statement Processing

• Check Syntax
• Call Traffic Cop
• Identify Query Type
Optimize
• Execute Query based on
• Command Processor if
needed
• Break Query in Tokens

Pars e
• Planner generate Plan •
Uses Database Statistics •
Query Cost Calculation •
Choose best plan
query plan
Execute
Database Cluster (Instance) Data Directory

Layout Data
Global Base pg_tblsc pg_xlog pg_log
Status
Directories
Configuration
FilesPostmaster Info Files
Cluster wide database
objects
Contains Databases
Symbolic link to tablespaces
Write ahead logs
Error logs
pg_clog, pg_multiexact, pg_snapshots, pg_stat, pg_subtrans,pg_notify,
pg_serial, pg_replslot, pg_logical, pg_dynshmem
postgresql.conf, pg_hba.conf, pg_ident.conf

Installation Directory Layout
• bin – Binary files
• include – Header files
• lib – Library files
• doc – Help files for different components, contrib modules and extensions
• share – Extensions and sample config files
• scripts – Some useful script files
• stackbuilder – installation directory for Stackbuilder
/opt/PostgresPlus/9.4AS
bin
include

lib
doc
share
scripts
stackbuilderplus
pg_env.sh,
uninstall-* files 30
Data File Storage Internals
• File-per-table, file-per-index.

• A table-space is a directory.
• Each database that uses that table-space gets a subdirectory.
• Each relation using that tablespace/database combination gets one or
more files, in 1GB chunks.
• Additional files used to hold auxiliary information (free space map, visibility
map).
• Each file name is a number (see pg_class.relfilenode).
Free Space and Visibility Map Files
•Each Relation has a free space map

− Stores information about free space available in the relation − File named as
filenode number plus the suffix _fsm
•Tables also have a visibility map
− Track which pages are known to have no dead tuples − Stored in a fork with the
suffix _vm
32
Sample - Data Directory
Layout 14307
Database OID

Base
14312
Data
pg_tblsc
16650
/storage
Tablespace OID
14297 14300
14405 14498

14300 14301
/pg_tab
14307 16651
16700 16701
Lab Exercise

1. Write a command to display a list of all PostgreSQL related processes on
your operating system
2. Write a SQL statement to find the data file name and location for
„customers‟ table
Module Summary

• Architectural Summary
• Shared Memory
• Inter-processes Communication
• Statement Processing
• Utility Processes
• Disk Read Buffering Disk
• Write Buffering
• Background Writer Cleaning Scan
• Commit and Checkpoint •
Physical Database Architecture •
Data Directory Layout
• Installation Directory Layout
• Page Layout

Installation
Module Objectives
• OS User and Permissions
• Installation Options
• Installation of PostgreSQL
• StackBuilder
• Setting Environmental Variables

OS User and Permissions
• PostgreSQL runs as a daemon (Unix / Linux) or service (Windows).
• Installation requires superuser/admin access.
• All PostgreSQL processes and data files must be owned by a user in the
OS.
− OS user is un-related to database user accounts.

− For security reasons, the OS user must not be root or an administrative account.
• SELinux Permissions
− SELinux must be set to permissive mode on systems with SELinux.
Supported Platforms
• PostgreSQL is available for Windows, Linux and Solaris systems
• Windows:
− Windows Server 2008 R1 (32 and 64 bit)
− Windows Server 2008 R2 (64 bit)
− Windows 2012 (64-bit)
• Linux platforms:

− CentOS 6.x and 7.x (64-bit)
− Red Hat Enterprise Linux 6.x and 7.x (64-bit)
− Ubuntu 14.04 LTS
− SLES 11.x
Supported Locales
• PostgreSQL has been tested and certified for the following locales:
− en_US United States English
− zh_HK Traditional Chinese with Hong Kong SCS
− zh_TW Traditional Chinese for Taiwan
− zh_CN Simplified Chinese
− ja_JP Japanese
− ko_KR Korean

Text Mode Installation
• Run the installer in text mode:
# ./postgresql.9.4-linux.run –mode text
•Choose the language using a number[1-5]
Please select the installation language

[1] English - English
[2] Japanese - 日本語
[3] Simplified Chinese - 简体中文
[4] Traditional Chinese - 繁体中文
[5] Korean - 한국어
Please choose an option [1] :
Text Mode Installation Steps
• Installer will prompt for different steps:
− License Agreement
− User Authentication
− Installation Directory
− Components
− Data and WAL Directory

− Configuration Mode
− Database superuser password and Locale
StackBuilder
• Simplifies the process of downloading and installing modules
• StackBuilder requires Internet access
• Run stackbuilder using the StackBuilder menu option from the Postgresql
9.4 menu

Setting Environmental Variables
• Setting environment variables is very important for trouble free
startup/shutdown of the database server.
− PATH –should point correct bin directory.

− PGDATA –should point correct data cluster directory.
− PGPORT –should point correct port on which database cluster is running.
− Edit .bash_profile to set the variables
− In Windows set these variables using my computer properties page.
Module Summary
• OS User and Permissions
• Installation Options

• Installation of PostgreSQL
• StackBuilder
• Setting Environmental Variables

Module 4
Database Clusters
© 2015 EnterpriseDB Corporation. All rights reserved. 46
Module Objectives
• Database Clusters
• Creating a Database Cluster
• Starting and Stopping the Server (pg_ctl)
• Connect to the Server Using psql

Database Clusters
• A Database Cluster is a collection of databases managed by a single
server instance.
• Database Clusters are comprised of:
− Data directory
− Port

• Default databases are created named:
− template0
− template1
− postgres
Creating a Database Cluster
• Choose the data directory location for new cluster.
• Initialize the database cluster storage area (data directory) using initdb
utility.
• initdb will create the data directory if it doesn‟t exist.

• You must have permissions on the parent directory so that initdb can
create the data directory.
• Data directory can be created manually using superuser access and
ownership can be given to postgres user.
Example - initdb
• In

the above example database system will be owned by user “postgres.” •
postgres OS user must also own the server process.
• postgres user is the database superuser.
• The default server config file will be created in /edbstore named
postgresql.conf.
• -W is used to force initdb to prompt for the superuser password.
Starting a Database Cluster
• Choose a unique port for postmaster

• Change the port in postgresql.conf
• pg_ctl utility can be used to start a cluster
• Syntax
− pg_ctl start [options]
− Options
-D location of the database storage area
-l write (or append) server log to FILENAME
-w wait until operation completes
-t seconds to wait when using -w option
Connecting to a Database Cluster
• The psql and pgAdmin-III clients can be used for connections

Reload a Database Cluster
• Some configuration parameter changes do not require a restart
• Changes can be reloaded using the pg_ctl utility • Changes
can also be reloaded using pg_reload_conf() • Syntax
pg_ctl reload [options]

-D location of the database storage area
-s only print errors, no informational messages
Stopping a Database Cluster
• pg_ctl can be used to stop a database cluster
• pg_ctl supports three modes of shutdown
− smartquit after all clients have disconnected
− Fast (default)quit directly, with proper shutdown (default)
− immediatequit without complete shutdown; will lead to recovery • Syntax

− pg_ctl stop [-W] [-t SECS] [-D DATADIR] [-s] [-m SHUTDOWN-MODE]
View Cluster Control Information
• pg_controldata can be used to view the control information for a
database cluster
• Syntax:
− pg_controldata [DATADIR]

Module Summary
• Database Clusters
• Creating a Database Cluster
• Starting and Stopping the Server (pg_ctl)

• Connect to the Server Using psql
Lab Exercise - 1
1. A new website is to be developed for an online music store. −
Create a new cluster edbdata with ownership of postgres user.
− Start your edbdata cluster
− Stop your edbdata cluster with fast mode.

− Reload your cluster with pg_ctl utility and using
pg_reload_conf() function.

Configuration
Objectives
− Server Configuration File
− Setting Server Parameters
− Connection Settings
− Security and Authentication Settings
− Memory Settings
− Query Planner Settings -> Performance tuning day4
− Log Management
− Background Writer Settings
− Statement Behavior -> rountine maintenance day3 −

Autovacuum Settings -> rountine maintenance day3
Server Configuration File
• There are many configuration parameters that effect the behavior of the
database system.
• Server config file postgresql.conf stores all the server
parameters. • All parameter names are case-insensitive.
• Some parameters require restart.
• Query to list of all parameters.
− SELECT name,setting FROM pg_settings;
• Query to list all parameters requiring server restart.
− SELECT name FROM pg_settings WHERE context = 'postmaster'; •

One way to set these parameters is to edit the file postgresql.conf,
which is normally kept in the data directory.
Setting Server Parameters
• Some parameters can be changed per session using the SET command.
• Some parameters can be changed at the user level using ALTER USER.
• Some parameters can be changed at the database level using ALTER
DATABASE.
• The SHOW command can be used to see settings.
• pg_settings catalog table lists settings information.

PostgreSQL ALTER SYSTEM
• ALTER SYSTEM command:
− Available starting with PostgreSQL 9.4
− Edits cluster settings without editing postgresql.conf
− Writes the setting to a file called postgresql.auto.conf
• Example:
− ALTER SYSTEM SET work_mem=20480;

• postgresql.auto.conf is always read last during server reload/restarts
• Reset a parameter change -
− ALTER SYSTEM SET work_mem = DEFAULT;
Connection Settings
• The following parameters in the postgresql.conf file control the connection
settings:
− listen_addresses (string)
− Specifies the TCP/IP address(es) on which the server is to listen for connections − port (integer)
− The TCP port the server listens on; 5432 by default
− max_connections (integer)
− Determines the maximum number of concurrent connections to the database server −

superuser_reserved_connections (integer)
− Determines the number of connection “slots” that are reserved for connections by Advanced Server
superusers
63
Memory Settings
• The following parameters in postgresql.conf control the memory settings:
− shared_buffers (integer)
− Sets the amount of memory the database server uses for shared memory buffers
− temp_buffers (integer)
− Sets the maximum number of temporary buffers used by each database session
− work_mem (default 4MB)
− Specifies the amount of memory to be used by internal sort operations and hash tables before
switching to temporary disk files
− autovacuum_work_mem
− Specifies the maximum amount of memory to be used by each autovacuum worker process

− maintenance_work_mem (default 64MB)
− Specifies the maximum amount of memory to be used in maintenance operations, such as
CREATE INDEX, and ALTER TABLE ADD FOREIGN KEY
Write Ahead Logs Settings
• wal_level (default: minimal): Determines how much information is written
to the WAL. Other values are archive, hot_standby and logical. • fsync
(default on): Turn this off to make your database much faster – and silently
cause arbitrary corruption in case of a system crash – limited applicability.
• wal_buffers (default: -1, autotune): The amount of memory used in shared
memory for WAL data. The default setting of -1 selects a size equal to
1/32nd (about 3%) of shared_buffers.

• checkpoint_segments (default 3): Maximum number of 16MB WAL file
segments between checkpoints. Default is too small!
• checkpoint_timeout (default 5 minutes): Maximum time between
checkpoints.
Logging – Where to Log
• log_destination: Valid values are combinations of stderr, csvlog,
syslog, and eventlog.
• logging_collector: Enables advanced logging features. csvlog requires
logging_collector.
• log_directory: Directory where log files are written. Default pg_log.

• log_filename: Format of log file name (e.g. postgresql-%Y-%m-
%d_%H%M%S.log).
• log_rotation_age: Automatically rotate logs after this much time.
Requires logging_collector.
• log_rotation_size: Automatically rotate logs when they get this big.
Requires logging_collector.
66
Logging - When To Log
• client_min_messages (default NOTICE). Messages of this severity level
or above are sent to the client.
• log_min_messages (default WARNING). Messages of this severity level
or above are sent to the server.

• log_min_error_statement (default ERROR). When a message of this
severity or higher is written to the server log, the statement that caused it is
logged along with it.
• log_min_duration_statement (default -1, disabled): When a statement
runs for at least this long, it is written to the server log, with its duration.
Logging - What To Log
• log_connections (default off): Log successful connections to the server log.
• log_disconnections (default off): Log some information each time a session disconnects,
including the duration of the session.
• log_error_verbosity (default “default”): Can also select “terse” or “verbose”.

• log_duration (default off): Log duration of each statement.
• log_line_prefix: Additional details to log with each line.
• log_statement (default none): Legal values are none, ddl, mod (DDL and all other data
modifying statements), or all.
• log_temp_files (default -1): Log temporary files of this size or larger, in kilobytes.
• log_checkpoints (default off): Causes checkpoints and restart points to be logged in the
server log.
Lab Exercise
1. Open psql and write a statement to change work_mem to 10MB. This
change must persist across server restarts
2. Open psql and write a statement to change work_mem to 20MB for the
current session

3. Open psql and write a statement to change work_mem to 1 MB for the
postgres user
4. Write a query to list all parameters requiring a server restart
5. Take a backup of the postgresql.conf file
Lab Exercise
6. Open the configuration file for your Postgres database cluster and make
the following changes
− Maximum allowed connections to 50
− Authentication time to 10 mins

− Shared buffers to 256 MB
− work_mem to 10 MB
− wal_buffers to 8MB
7. Restart the server and verify the changes made in previous step
Summary
• In this module we covered:
− Setting PostgresPlus Advanced Server Parameters
− Access Control
− Connection Settings

− Security and Authentication Settings
− Memory Settings
− Query Planner Settings
− Log Management
− Background Writer Settings
− Statement Behaviour
− Autovacuum Settings
Module 6

PSQL
Objectives
− Client Interface psql
− Connecting to a Database
− psql Meta Commands
− Executing SQL Commands
− Advance Features

PSQL
• psql is the name of PostgreSQL PSQL's executable.
• It enables you to type in queries interactively, issue them to PostgreSQL,
and see the query results.
− psql [option...] [dbname [username]]

Connecting to a Database
• In order to connect to a database you need to know the name of your
target database, the host name and port number of the server and what
user name you want to connect as.
• Command line options, namely -d, -h, -p, and -U respectively.
• Environment variables PGDATABASE, PGHOST,

PGPORT and PGUSER to appropriate values.
Conventions
• psql has it's own set of commands, all of which start with a backslash ().
These are in no way related to SQL commands, which operate in the
server. psql commands only affect psql.
• Some commands accept a pattern. This pattern is a modified regex. Key
points:

− * and ? are wildcards
− Double-quotes are used to specify an exact name, ignoring all special characters and
preserving case
On Startup...
• psql will execute commands from $HOME/.psqlrc, unless option -X is
specified.
• -f FILENAME will execute the commands in FILENAME, then exit • -c
COMMAND will execute COMMAND (SQL or internal) and then exit •

--help will display all the startup options, then exit
• --version will display version info and then exit
Entering Commands
• psql uses the command line editing capabilities that are available in the
native OS. Generally, this means
• Up and Down arrows cycle through command history
− on UNIX, there is tab completion for various things, such as SQL commands and to a more
limited degree, table and field names
− disabled with -n

• s will show the command history
• s FILENAME will save the command history
• e will edit the query buffer and then execute it
• e FILENAME will edit FILENAME and then execute it
• w FILENAME will save the query buffer to FILENAME
Controlling Output
• There are numerous ways to control output.
• The most important are:
− -o FILENAME or o FILENAME will send query output
(excluding STDERR) to FILENAME (which may be a

pipe) − g FILENAME executes the query buffer,
sending output to FILENAME (may be a pipe)
− -q runs quietly.
Advance Features: Variables
• psql provides variable substitution •
Variables are simply name/value pairs • Use
set meta command to set a variable
postgres=# set city Edmonton

postgres =# echo :city
Edmonton
• Use unset to delete a variable
postgres =# unset city

9.6_Course Material-Postgresql_002.pdf

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie 9.6_Course Material-Postgresql_002.pdf

Ähnlich wie 9.6_Course Material-Postgresql_002.pdf (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

9.6_Course Material-Postgresql_002.pdf