Weitere ähnliche Inhalte
Ähnlich wie Basic MySQL Troubleshooting for Oracle DBAs (20)
Mehr von Sveta Smirnova (20)
Kürzlich hochgeladen (20)
Basic MySQL Troubleshooting for Oracle DBAs
- 2. Basic MySQL
Troubleshooting for Oracle
DBAs
Sveta Smirnova
Principal Technical Support Engineer
2Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Picture Here
- 3. Program Agenda
Introduction
Basic single-client issues
Concurrency issues
High availability solutions
Tools
3Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 7. Log files
General query log
Slow query log
Binary log
Relay log
Error log
InnoDB logs (and, probably, created by other storage engines)
7Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 10. Storage engines
From troubleshooting point of view
●
Own data/index format
●
Own locking model
●
Own diagnostic
●
Own log files
●
CHECK TABLE
10Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 12. MySQL Access Privilege System
No roles by default, limited user limits
All records are in the mysql database (schema)
Pluggable authentication since version 5.5
Connections: TCP/IP with login-password, socket (Unix), named pipe
(Windows)
SELECT user, host FROM mysql.user
SELECT USER(), CURRENT_USER()
SHOW GRANTS
12Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 13. Error handling
Errors vs warnings
mysql> select max (f1) from t1;
ERROR 1630 (42000): FUNCTION test.max does not exist. Check the
'Function Name Parsing and Resolution' section in the Reference Manual
mysql> select * from t1 where "f1"=1;
Empty set, 1 warning (0.05 sec)
mysql> show warnings;
++++
| Level | Code | Message |
++++
| Warning | 1292 | Truncated incorrect DOUBLE value: 'f1' |
++++
1 row in set (0.00 sec)
13Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 14. Error handling
Application (C API)
Error information
●
mysql_errno
●
mysql_error
Warnings
●
mysql_info
●
mysql_sqlstate
●
mysql_warning_count
14Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 16. Error handling
in stored routines
GET DIAGNOSTICS
●
GET DIAGNOSTICS rows = ROW_COUNT, conditions = NUMBER;
●
GET DIAGNOSTICS CONDITION 1 code = RETURNED_SQLSTATE,
msg = MESSAGE_TEXT;
http://dev.mysql.com/doc/refman/5.6/en/diagnostics-area.html
16Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 17. Options
Have global and session scope
Can be set
●
In configuration files
●
Command line parameter
●
SET [GLOBAL] var_name = NEW_VAL
●
SHOW [GLOBAL] VARIABLES
Only two built-in levels of write access
●
SUPER for GLOBAL and limited SESSION variables
●
Non-privileged user can change any session option!
17Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 18. Options
Allocated at
●
Server startup
●
User connection creation
●
For certain operations
To watch status
●
SHOW [GLOBAL] STATUS
http://dev.mysql.com/doc/refman/5.6/en/mysqld-option-tables.html
18Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 20. Optimizer
EXPLAIN is less powerful if compare with Oracle
●
It is improved in version 5.7.3
●
Graphic EXPLAIN in MySQL Workbench 6.0
●
MySQL’s EXPLAIN Command New Features [HOL9734, passed]
EXPLAIN EXTENDED
INFORMATION_SCHEMA.OPTIMIZER_TRACE
Status variables 'Handler_%'
20Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 23. EXPLAIN in MySQL
EXPLAIN EXTENDED
●
Follow by SHOW WARNINGS
EXPLAIN PARTITIONS
EXPLAIN FORMAT=JSON
http://www.slideshare.net/SvetaSmirnova/troubleshooting-my-sqlperformanceaddonsen
http://education.oracle.com/pls/web_prod-plq-dad/db_pages.getCourseDesc?dc=D79908_1879034
http://dev.mysql.com/doc/refman/5.6/en/explain-output.html
23Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 24. Storage engine specifics
They care about physical data, so all data information are on their level
●
Corruption
●
Index statistics
CHECK TABLE to check for errors
24Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 25. Storage engine specifics
MyISAM
Stores data in files
●
*frm – table definition, common for all storage engines
●
*MYD – data file
●
*MYI – index file
myisamchk
myisam_recovery_options to check each time you open a table
●
First access – table is opened
●
Second, third, .. access – existent descriptor used
●
FLUSH TABLES – closes opened descriptors
25Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 26. Storage engine specifics
InnoDB
Transactional storage engine
Physical layout
●
*frm file – table definition
●
Shared tablespace
●
*ibd file – tablespace for individual table
●
Optional: innodb_file_per_table
●
Recommended
Redo log files
OPTIMIZE TABLE = ALTER + ANALYZE
Automatic startup check
●
26Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 28. Lock types
MDL locks
Table locks
Row locks
Read locks
●
Block writes
Write locks
●
Block both read and writes
28Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 29. Transactions and Their Relationship With Locks
Server-level
●
MDL locks
Engine level
●
Table locks
●
Row locks
AUTOCOMMIT
●
Supported
29Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 30. Diagnostic Tools
SHOW [FULL] PROCESSLIST
●
Universal tool which lists all currently running connections
SHOW ENGINE INNODB STATUS
INFORMATION SCHEMA
●
PROCESSLIST
●
InnoDB tables
PERFORMANCE SCHEMA
●
MDL locks
Unlocking MySQL: Dealing with Locking in MySQL [CON4038, passed]
30Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 33. SHOW PROCESSLIST and Transactions
mysql> show processlist;
+++++++++
| Id | User | Host | db | Command | Time | State | Info |
+++++++++
| 28 | root | localhost | NULL | Query | 0 | NULL | show processlist |
| 31 | sveta | localhost | test | Query | 7 | Updating | update tlog set eid=2345 where ln=1 |
| 32 | sveta | localhost | test | Sleep | 18 | | NULL |
+++++++++
3 rows in set (0.00 sec)
Not much info!
33Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
No lock information!
- 34. Storage engine specifics
Transactions and engine-level locks are done at the engine level
Storage engines have own diagnostic tools
Use them when hit an issue with such a lock
34Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 35. Storage engine specifics
InnoDB
InnoDB Monitors
●
SHOW ENGINE INNODB STATUS
●
Lock Monitor
INFORMATION_SCHEMA tables
●
INNODB_TRX
●
INNODB_LOCKS
●
INNODB_LOCK_WAITS
35Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 36. Diagnostics: P_S
Monitors internal operations
●
Events
●
Waits
●
Mutexes
●
Statements
●
Stages
Similar to Oracle wait interface
Improving Performance with MySQL Performance Schema [HOL9733, Sunday,
1:00 PM]
Making the Performance Schema Easier to Use [CON5282, Sunday, 10:00 AM]
Performance Schema and ps_helper [CON4077, passed]
36Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 37. MDL Locks
Waiting thread only
SHOW PROCESSLIST, INFORMATION_SCHEMA.PROCESSLIST
●
Waiting for table metadata lock
Performance Schema
●
MUTEX_INSTANCES
●
EVENTS_WAITS_CURRENT
●
THREADS
37Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 38. Performance Schema MDL queries
mysql> SELECT * FROM mutex_instances WHERE
LOCKED_BY_THREAD_ID
IS NOT NULLG
*********************** 1. row ***********************
NAME:
wait/synch/mutex/sql/MDL_wait::LOCK_wait_status
OBJECT_INSTANCE_BEGIN: 37104744
LOCKED_BY_THREAD_ID: 18
1 row in set (0.01 sec)
38Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 39. Performance Schema MDL queries
mysql> SELECT THREAD_ID, EVENT_ID, EVENT_NAME, SOURCE, TIMER_START,
OBJECT_INSTANCE_BEGIN, OPERATION FROM events_waits_current WHERE
THREAD_ID IN(SELECT LOCKED_BY_THREAD_ID FROM mutex_instances WHERE
LOCKED_BY_THREAD_ID IS NOT NULL)G
*************************** 1. row ***************************
THREAD_ID: 18
EVENT_ID: 461
EVENT_NAME: wait/synch/cond/sql/MDL_context::COND_wait_status
SOURCE: mdl.cc:1210
TIMER_START: 97623253523306
OBJECT_INSTANCE_BEGIN: 0
OPERATION: timed_wait
1 row in set (0.00 sec)
39Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 41. MySQL Cluster
Special storage engine: NDB
Stores data on two or more physical machines
Two or more copies
General troubleshooting techniques
●
Applicable
Specific NDB storage engine techniques
41Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 42. Replication
Overview
Always available, but you must setup it before use
Asynchronous master-slave
Master
●
Keeps all updates in separate file: binary log
Slave
●
IO thread read updates from master and stores in relay log file
●
SQL thread executes updates
42Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 45. For developers
MySQL Workbench
●
SQL editor
●
Database designer
●
Introduction to MySQL Database Development with MySQL Workbench
[CON3967, Sunday, 5:30 PM]
MySQL Command line client
●
Command-line interface for SQL
●
SELECT, INSERT, UPDATE, DELETE, etc.
45Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 46. For DBA
MySQL Enterprise Monitor
●
Real-time MySQL performance and availability monitoring
●
Error information
●
Advisors
●
Forcasts
46Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 47. For DBA
MySQL Workbench
●
Server administration
●
Backups
●
Using the New MySQL Workbench Tools for Performance Tuning
[HOL9786, passed]
MySQL command line client
●
Command-line interface for SQL
●
GRANT, CREATE, OPTIMIZE, etc.
mysqladmin
47Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 48. Command line
They are not
Set of tools to do various administrative job
Same!
●
MySQL Workbench Utilities
●
http://dev.mysql.com/downloads/tools/utilities/
●
●
Development of Fault-Tolerant Failover Tools with MySQL Utilities
[CON4276, Sunday, 2:30 PM]
Percona Toolkit
●
http://www.percona.com/software/percona-toolkit
48Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 49. Command line
MySQL Sandbox
●
Sandbox for your tests
●
https://launchpad.net/mysql-sandbox
PS_HELPER view
●
www.markleith.co.uk/ps_helper/
●
Performance Schema and ps_helper [CON4077, passed]
49Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 50. References
MySQL User Reference Manual
●
http://dev.mysql.com/doc/refman/5.6/en/index.html
Knowledge Management Database
Forums
●
http://forums.mysql.com
Bug trackers
●
http://bugs.mysql.com
●
Oracle Internal Bugs database
My Oracle Support
●
https://support.oracle.com
50Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 55. The preceding is intended to outline our general product
direction. It is intended for information purposes only, and may
not be incorporated into any contract.
It is not a commitment to deliver any material, code, or
functionality, and should not be relied upon in making
purchasing decisions. The development, release, and timing
of any features or functionality described for Oracle’s products
remains at the sole discretion of Oracle.
55Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 56. MySQL Troubleshooting for
Oracle DBAs
Part II: details
Sveta Smirnova
Principal Technical Support Engineer
56Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Picture Here
- 58. Replication IO thread:
communication issues
Access error
●
Check slave error log
●
SHOW SLAVE STATUS
●
Try to connect using normal MySQL client using slave credentials
●
SHOW GRANTS
●
●
Fix privileges on master
Restart slave
58Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 59. Replication SQL thread:
typical issues
Simple master-slave
●
Data is different on master and slave
●
Replication event can not be applied
Different errors on master and slave
●
Slave lags far behind the master
Circular replication or other writes in addition to slave SQL thread
●
Data is different on master and slave
●
59Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 60. Replication SQL thread:
Data is different on master and slave
Was the table modified besides the SQL thread?
●
How?
●
Can it affect content of the table in the wrong way?
Are the table definitions same on master and slave?
Is it possible that master events was applied in wrong order?
●
Use mysqlbinlog to find queries which caused the issue
●
Check master's application to find what is wrong
60Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 61. Replication SQL thread:
Events from master were applied in wrong order
Lock issues
●
InnoDB
Triggers
●
SET GLOBAL slave_skip_counter
●
Synchronize tables!
Different options
●
Start slave with master's options, then check
61Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 62. Replication SQL thread:
Slave lags far behind the master
Threads
●
Master runs in multiple update threads
●
Slave uses single
Seconds_behind_master is growing
Tune slave performance
●
Buffers
●
Indexes (for statement)
62Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 68. Optimizer
INFORMATION_SCHEMA.OPTIMIZER_TRACE
Turn tracing on (it's off by default):
SET optimizer_trace="enabled=on";
SELECT <your query here>;
SELECT * FROM INFORMATION_SCHEMA.OPTIMIZER_TRACE;
More queries...
When done with tracing, disable it:
SET optimizer_trace="enabled=off";
http://dev.mysql.com/doc/internals/en/optimizer-tracing.html
68Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 72. Options
Scope
Global
●
Control parameters, necessary for all server processes
●
Location of server files: datadir etc.
●
Shared buffers
●
More
Session
●
Control connection-specific parameters
http://dev.mysql.com/doc/refman/5.6/en/mysqld-option-tables.html
72Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 74. Options
Who can change
Global options and few session options
●
A user with privilege SUPER
Session options
●
Anybody
There is no limits!
74Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 75. Options
When allocated
Those which control behavior of whole server
●
Once at server startup
●
Can start with low values, then grow to specified
Connection options
●
For every connection when connection opens
Operation-specific
●
For every operation when needed
75Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 79. Options
Troubleshooting best practices
Record currently used variables
●
SHOW [GLOBAL] VARIABLES
Make change dynamically if possible
●
SET [GLOBAL] var_name=NEW_VAL
Test in one session first
Then change global variable
Change configuration file after you are happy with results
79Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 80. Options
When affecting option is not known
Record currently used variables
●
SHOW [GLOBAL] VARIABLES
Start mysqld with option –nodefaults
●
This option must be first one!
Check if problem is solved
Change variable values one-by-one until you find one which leads to the
problem
80Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 82. SHOW ENGINE INNODB STATUS
TRANSACTIONS
Trx id counter 1B1C
Purge done for trx's n:o < 1B19 undo n:o < 0
History list length 189
LIST OF TRANSACTIONS FOR EACH SESSION:
TRANSACTION 0, not started
MySQL thread id 28, OS thread handle 0x7fa9301a0700, query
id 8653 localhost root
show engine innodb status
82Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 83. SHOW ENGINE INNODB STATUS
TRANSACTION 1B1B, ACTIVE 6 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 31, OS thread handle 0x7fa9301e1700, query id 8652 localhost sveta Updating
update tbllog set `Employee ID`=2345 where `Log Number`=1
TRX HAS BEEN WAITING 6 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 81967 n bits 72 index `PRIMARY` of table `test`.`tlog`
trx id 1B1B lock_mode X locks rec but not gap waiting
Record lock, heap no 6 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 4; hex 80000001; asc ;;
1: len 6; hex 000000001b19; asc ;;
2: len 7; hex 14000140340110; asc @4 ;;
3: len 4; hex 80003039; asc 09;;
4: SQL NULL;
83Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 84. SHOW ENGINE INNODB STATUS
TRANSACTION 1B19, ACTIVE 787 sec
2 lock struct(s), heap size 376, 1 row lock(s), undo log
entries 1
MySQL thread id 32, OS thread handle 0x7fa930222700, query
id 8647 localhost sveta
No lock information
For the transaction
Which holds the lock!
84Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 85. InnoDB Monitors
Pseudo-tables
Turn periodical logging into error log
●
CREATE TABLE innodb_monitor(f1 int) ENGINE=INNODB;
Lock monitor
●
Changes output format of InnoDB Monitor
●
CREATE TABLE innodb_lock_monitor(f1 int) ENGINE=INNODB;
85Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 87. InnoDB Information Schema tables
INNODB_TRX
INNODB_LOCKS
INNODB_LOCK_WAITS
87Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 88. InnoDB Information Schema tables
mysql> SELECT * FROM INNODB_LOCK_WAITS;
+++++
|requesting_trx_id|requested_lock_id|blocking_trx_id|blocking_lock_id|
+++++
| 1B1F | 1B1F:0:81967:6 | 1B19 | 1B19:0:81967:6 |
+++++
1 row in set (0.00 sec)
88Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 89. InnoDB Information Schema tables
mysql> select trx_mysql_thread_id, trx_id from innodb_trx;
+++
| trx_mysql_thread_id | trx_id |
+++
| 31 | 1B1F |
| 32 | 1B19 |
+++
2 rows in set (0.00 sec)
89Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 91. MySQL Access Privilege System
Overview
No roles by default, limited user limits
All records are in the mysql database (schema)
Pluggable authentication since version 5.5
Connections
●
TCP/IP with login-password
●
Socket (Unix)
●
Named pipe (Windows)
91Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 92. MySQL Access Privilege System
How to handle connection issues
SELECT user, host FROM mysql.user
Most descriptive host first, then wildcard
Socket connection by default on Unix
92Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 93. MySQL Access Privilege System
Sort order
mysql> select user, host from mysql.user order by user desc, host
desc;
+++
| user | host |
+++
| root | localhost |
| root | delly |
| root | ::1 |
| root | 127.0.0.1 |
| foo | % |
| | localhost |
+++
6 rows in set (0.00 sec)
93Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 94. MySQL Access Privilege System
Wrong access
SHOW GRANTS [FOR user@host]
Grant tables
●
mysql.db
●
mysql.tables_priv
●
mysql.columns_priv
●
mysql.procs_priv
●
mysql.proxies_priv
SELECT USER(), CURRENT_USER()
94Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 95. MySQL Access Privilege System
Grants
mysql> show grants;
++
| Grants for root@localhost |
++
| GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION |
| GRANT PROXY ON ''@'' TO 'root'@'localhost' WITH GRANT OPTION |
++
2 rows in set (0.02 sec)
95Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 96. MySQL Access Privilege System
Grants
mysql> show grants for foo@'%';;
++
| Grants for foo@% |
++
| GRANT USAGE ON *.* TO 'foo'@'%' |
| GRANT ALL PRIVILEGES ON `test`.* TO 'foo'@'%' |
++
2 rows in set (0.00 sec)
96Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
- 99. The preceding is intended to outline our general product
direction. It is intended for information purposes only, and may
not be incorporated into any contract.
It is not a commitment to deliver any material, code, or
functionality, and should not be relied upon in making
purchasing decisions. The development, release, and timing
of any features or functionality described for Oracle’s products
remains at the sole discretion of Oracle.
99Copyright © 2013, Oracle and/or its affiliates. All rights reserved.