This talk will give an overview of the different available Java MySQL connectors (Connector/J, MariaDB Java Connector) and connection pools (Commons-DBCP, C3P0, ...).
One of the things with the default configuration of these solutions is that they are very chatty. This can have a noticeable impact on application performance and database load. I've seen many environments where over 50% of the total amount of queries are caused by such behavior. This behavior will be explained through examples seen on production systems as well as recommendations on optimization will be made.
Load balancing and transparent failover solutions will be described for both MySQL's traditional asynchronous replication and Galera based replication (Percona XtraDB Cluster or MariaDB Galera Cluster).
14. Connector/J - Creating Connection
SHOW VARIABLES WHERE
Variable_name ='language' ORâŚ
SELECT
@@session.auto_increment_increment;
SET NAMES latin1;
SET character_set_results = NULL;
SET autocommit=1;
SET sql_mode=
'NO_ENGINE_SUBSTITUTION,STRICT_TRAN
S_TABLES';
14
15. MariaDB - Creating Connection
set autocommit=1;
USE employees;
show variables like 'sql_mode';
15
17. Connector/J & MariaDB Java Client -
Verbosity
⢠Connector/J is more verbose when starting a
connection
⢠Usually not a problem:
â connection pools are commonly usedâ¨
(more coming soonâŚ)
â connections are reused
â Actually I like but not too much.
17
23. Client or Server Side Prepared Statements
⢠Server side Prepared statements:
â reduce network traffic
â query is already optimized on server.
⢠Support:
â MariaDB Java client only supports client
side
â Connector/J default in client side
23
24. Server Side Prepared Statements
PREPARE stmt1 FROM
select * from employees
where emp_no = ?;
EXECUTE # API CALL
select * from employees
where emp_no = 20000;
DEALLOCATE PREPARE stmt1;
24
25. Connector/J: Server Side Prepared
Statements
⢠useServerPrepStmts = false
â disabled by default
⢠Mind looking at:
â cachePrepStmts = false
⢠do PREPARE, EXECUTE, DEALLOCATE
every timeâŚ, 3 round trips?
â prepStmtCacheSize = 25
â prepStmtCacheSqlLimit = 256
⢠low defaults
25
https://bugs.mysql.com/bug.php?id=74932
26. Benchmark: Prepared Statements
26
QPS
900
1800
2700
3600
MariaDB CONN/J CONN/J SRVCONN/J SRV+Cache
3.506QPS
2.047QPS
3.342QPS3.400QPS
select * from employees_alotofindexes
where first_name='moss' and birth_date > "1954-06-14"
and gender="M" and hire_date > "1998-01-01"G
51. WHY DOES IT HAVE TO DO
THAT?
Because of
âapplication bugsâ
51
52. Connection Pools - Why Chattiness
Examples
⢠*maybe* forgot to COMMIT / ROLLBACK
⢠wanting AUTOCOMMIT=1â¨
but a previous TRX set it to 0
⢠Changing TRX Isolation Level
⢠Is connection still working?
52
53. MYSQL CONNECTORS
CONNECTION POOLS
Connection Pools
Issues
Resetting
Environment
Testing Connectivity
Pool Sizing
Lingering Transactions
Analysis
Examples
Graceful Failover
Conn/J Extra Features
53
54. Connection Pool - Resetting Status
54
JDBC-Pool C3P0 DBCP2 HikariCP
Rollback rollbackOnReturn=false autoCommitOnClose=false rollbackOnReturnâ¨
=true
Commit commitOnReturn=false autoCommitOnClose=false n/a n/a
Avoid see above forceIgnoreUnresolvedTransa
ctions=false
see above
Auto
Commit
Driver Driver enableAutoCommitâ¨
OnReturn=true
Driver
55. MYSQL CONNECTORS
CONNECTION POOLS
Connection Pools
Issues
Resetting Environment
Testing Connectivity
Pool Sizing
Lingering Transactions
Analysis
Examples
Graceful Failover
Conn/J Extra Features
55
56. Connection Pool - Testing
⢠Making sure the connection is still active
⢠If not, maybe reopen a connection
⢠Not recommended as DB
⢠However, applications:
⢠do not like errors
⢠do not retry gracefully
56
57. Connection Pool - Testing
⢠If connections REALLY need to be testedâŚ
⢠do not specify test query like:
⢠SELECT 1
⢠SELECT * FROM DUAL
⢠Leave default, all of the connection pools
use:â¨
JDBC4 isValid();
57
58. Connection Pool - Testing
58
JDBC-Pool C3P0 DBCP2 HikariCP
Test Before testOnBorrow=false testConnectionOnCheckOutâ¨
=false
testOnBorrow=false n/a
Test After testOnReturn=false testConnectionOnCheckInâ¨
=false
testOnReturn=false n/a
Test While Idle testWhileIdle=false idleConnectionTestPeriodâ¨
=0
testWhileIdle=false n/a
JDBC4
isValid()
default default default jdbc4ConnectionTestâ¨
=true (default)
Query validationQueryâ¨
(isValid)
preferredTestQuery=null validationQueryâ¨
(isValid)
connectionTestQueryâ¨
=none
Interval? validationInterval=30000 n/a n/a n/a
59. Connection Pool - Testing
⢠JDBC: validationInterval=30sâ¨
WHY? It defeats the whole purpose!
59
60. MYSQL CONNECTORS
CONNECTION POOLS
Connection Pools
Issues
Resetting Environment
Testing Connectivity
Pool Sizing
Lingering Transactions
Analysis
Examples
Graceful Failover
Conn/J Extra Features
60
61. Connection Pool - Pool Sizing
⢠Funnelling on Application Level, is good
⢠Smaller Number is Better
⢠+- * CPUâs on DB
⢠maybe a bit more (waiting on IOâŚ)
⢠all application servers combined
⢠Response Time vs Throughput
61
62. Connection Pool - Pool Sizing
62
JDBC-Pool C3P0 DBCP2 HikariCP
Amount of
Connections
maxActive=100 maxPoolSize=15 maxTotal=8 maximumPoolSize=10
Maximum Idle
Connections
maxIdle=100 maxIdleTime=0** maxIdle=8 n/a
Minimum Idle
Connections
minIdle=10 minPoolSize=3 minIdle=0 minimumIdle=max
Startup Size initialSize=10 initialPoolSize=3 initialSize=0 minimumIdle
63. MYSQL CONNECTORS
CONNECTION POOLS
Connection Pools
Issues
Resetting Environment
Testing Connectivity
Pool Sizing
Lingering
Transactions
Analysis
Examples
Graceful Failover
Conn/J Extra Features
63
64. Connection Pool - Avoid Lingering
Transactions
⢠Application forgets to return the connection
⢠Statements that take longer than âŚ
⢠Avoid this!
⢠Fix Application
64
66. MYSQL CONNECTORS
CONNECTION POOLS
Connection Pools
Issues
Resetting Environment
Testing Connectivity
Pool Sizing
Lingering Transactions
Analysis
Examples
Graceful Failover
Conn/J Extra Features
66
67. Connection Pools - How To Look At
Workload?
⢠Slow Query Log
⢠tcpdump
⢠pt-query-digest
⢠Percona Cloud Tools
67
68. MYSQL CONNECTORS
CONNECTION POOLS
Connection Pools
Issues
Resetting Environment
Testing Connectivity
Pool Sizing
Lingering Transactions
Analysis
Examples
Graceful Failover
Conn/J Extra Features
68
69. Connection Pool - Example Transaction
Connection con = ds.getConnection();
con.setTransactionIsolation
(Connection.TRANSACTION_READ_COMMITTED);
con.setAutoCommit(false);
PreparedStatement stmt =
con.prepareStatement("select * from
employees where emp_no = ?");
stmt.setInt(1, 20000);
ResultSet rs = stmt.executeQuery();
stmt.close();
rs.close();
con.commit();
con.close();
69
70. For Connectors - RECAP
⢠MySQL Connector/J
⢠useConfigs=maxPerformance
⢠useServerPrepStmts=true
⢠MariaDB Java Client
⢠HikariCP: Built in
⢠JDBC-Pool:
jdbcInterceptors=âConnectionState"
⢠Other Pools: UNKNOWN
70
71. Connection Pool - TRX JDBC
select * from employees
where emp_no = 20000;
commit;
71
200
72. Connection Pool - TRX C3P0
SET SESSION TRANSACTION
ISOLATION LEVEL READ COMMITTED;
SET autocommit=0;
select * from employees
where emp_no = 20000;
commit;
SET autocommit=1;
SET SESSION TRANSACTION
ISOLATION LEVEL REPEATABLE READ;
72
600
73. Connection Pool - TRX C3P0
mysql> set global
tx_isolation=âREAD-COMMITTEDâ;
forceIgnoreUnresolvedTransactions=true
73
200
74. Connection Pool - TRX DBCP
SET autocommit=1;
# administrator command: Ping;
SET autocommit=0;
select * from employees
where emp_no = 20000;
commit;
rollback;
SET autocommit=1;
74
700
76. Connection Pool - TRX HikariCP
SET SESSION TRANSACTION
ISOLATION LEVEL READ COMMITTED;
SET autocommit=0;
select * from employees
where emp_no = 20000;
commit;
SET autocommit=1;
SET SESSION TRANSACTION
ISOLATION LEVEL REPEATABLE READ;
76
600
77. Connection Pool - TRX HikariCP
mysql> set global
tx_isolation=âREAD-COMMITTEDâ;
autoCommit=false
77
200
84. Connection Pools - Graceful Failover
⢠During âmaintenanceâ, what do we do?
⢠KILL old connections?
⢠Wait until connections are closed? (Define
lifetimes?)
⢠Ignore it?
84
85. Connection Pools - Graceful Failover
⢠Some connection pools can close
connections gracefully, when idle.
⢠For âsynchronousâ replication systems
⢠using JMX
⢠No Application Errors!
85
Method
JDBC-Pool purgeOnReturn()
C3P0 softResetAllUsers()
DBCP n/a
HikariCP softEvictConnections(),
suspendPool(), resumePool() <â- ASYNC