The presentation will be focused on dealing with an unexpected high traffic event as it is happening. We will address the impact a traffic spike can have on your database – from poor query response time to replication lag issues – and provide actionable tips and tricks. We will talk about how to bring your database under control, earn performance gains, and delight your customers.

1. © 2020 Percona 1

2. Focus on the Traffic
Spike
Different causes of slowdown
have different approaches, we
focus on Traffic Spike in this
Presentation
© 2020 Percona 3

3. Something you
can do NOW
Focus on Changes which you
can implement in Hours or
Days, not weeks or Months
© 2020 Percona 4

4. Companion
Article
© 2020 Percona5
https://per.co.na/18Things

5. #1 Scale your
Cloud Instance
Size
© 2020 Percona6
https://aws.amazon.com/ec2/instance-types/

6. Scaling Instance
Size when
Money are not
concern
© 2020 Percona7

7. Best Practice
© 2020 Percona 8
Scaling by the Credit Card
Use As Temporary Solution
While you find more cost effective options
Or until Traffic Spike Passes

8. #2 Deploy More
MySQL Slaves
© 2020 Percona9

9. Read Intensive
Applications
© 2020 Percona 10
Works best for Read Intensive Applications
Still needs massive amount of data copies (you
pay for)
Caching not very efficient due to many copies in
memory
Changing Application to get advantage of Replicas
may be hard

10. #3 Deploy
ProxySQL
© 2020 Percona 11

11. ProxySQL
Considerations
© 2020 Percona 12
Great for Connection Pooling
Can assist with Read-Write Splitting
Supports TTL (Time to Live) based Query Caching
Helps to improve High Availability with Failover
Handling

12. Why NOT
ProxySQL
© 2020 Percona 13
ProxySQL is NOT
100% MySQL
Compatible
Some application
Adjustments may
need be required
ThreadPool in
Percona Server for
MySQL, MariaDB
Connection Pool in
your Application
Server (Java etc)

13. #4 Disable Heavy
Load Application
Features
© 2020 Percona 14
Great Application Design Practice –
allow disabling features easier
What Are high load and low business
value features ?
Can be applied to specific users too

14. #5 Check for
Resource
Bottlenecks
© 2020 Percona15
https://per.co.na/PMM

15. Primary Raw
Resources of a
System
© 2020 Percona16

16. #6 Get More CPU
Cores, or Faster
Cores
© 2020 Percona17

17. Look at Run
Queue Latency
© 2020 Percona18
Tool from BCC Tools Collection https://github.com/iovisor/bcc

18. Beware of
Stealing
© 2020 Percona19

19. “Compute
Optimized”
© 2020 Percona20

20. #7 Get More
Memory
© 2020 Percona21

21. Do not Run out of Virtual Memory
© 2020 Percona 22

22. Beware Serious Swapping
© 2020 Percona 23

23. True Memory
Usage by Process
© 2020 Percona24
Look at Resident + Swap

24. #8 Move to
Faster Storage
© 2020 Percona 25
Memory is faster than fastest storage
Fast Storage for Write Performance
Fast Storage for Read Performance,
when data set is so large fitting in
memory is impractical

25. Amount and Type of IO Activity
© 2020 Percona 26

26. How Latency Compares to the Historical
© 2020 Percona 27

27. “Fsync” Is Special kind of IO
© 2020 Percona 28

28. Biolatency: Block
Device Latency
© 2020 Percona29
Tool from BCC Tools Collection https://github.com/iovisor/bcc

29. #9 Check your
Network
© 2020 Percona 30
Bandwidth
Latency
Packet Loss

30. How Much Network Traffic do you Have ?
© 2020 Percona 31

31. External
Measurmenets
© 2020 Percona32

32. Retransmits – The silent Killer
© 2020 Percona 33

33. Where
Retransmits
come from
© 2020 Percona34
Tool from BCC Tools Collection https://github.com/iovisor/bcc

34. #10 Locate and
Optimize Queries
which cause the
load
© 2020 Percona35

35. Query Execution
Details
© 2020 Percona36

36. Check Explain
© 2020 Percona 37

37. #11 Add Missing
Indexes
© 2020 Percona 38
Missing Index can cause query to run
1000s times slower than it should
Most Applications Have Some Missing
Indexes
Index Purpose: Lookup Data, Sorting,
Serving The Data

38. #12 Drop
Unneeded
Indexes
© 2020 Percona 39
Indexes are Expensive both for Writes and for
Reads
You only should have indexes you need – Drop
Unused Indexes
Drop Duplicate Indexes
Assess Redundant Indexes for Dropping

39. #13 Configure
MySQL Server
Properly
MySQL Defaults are not designed for High Performance Servers or
High Intensity Workloads
© 2020 Percona40

40. Top MySQL Settings To Change
max_connections=N
table_definition_cache=M
table_open_cache=X
innodb_buffer_pool_size= 16G (70% of RAM)
innodb_flush_method=O_DIRECT
innodb_log_file_size=2G
© 2020 Percona 41

41. Operating
System Limits
Limit of Open Files is the most important for MySQL
© 2020 Percona42

42. #14 Purge
Unneeded Data
© 2020 Percona 43
Is old data needed in the same
operational database ?
Move to different Server
Move to Archival Storage

43. Tool to Consider
pt-archiver
© 2020 Percona44

44. #15 Perform
Database
Maintenance
• Database Stats getting out of sync – ANALYZE TABLE
• Database Fragmentation and Bloat – OPTIMIZE TABLE
© 2020 Percona45

45. Optimizing All
Tables mysqlcheck –optimize -A
© 2020 Percona46

46. Word of Caution Database Maintenance makes database faster but takes a lot of
resources to run - not a good thing to do during peak load
© 2020 Percona 47

47. #16 Check your
Background Jobs
© 2020 Percona 48
Can this Background Jobs postponed beyond Peak Times
Can this job be run on the replica (or other data jobs)
Am I avoiding batch job overlaps ?
Is this background job overdue for optimization ?
Can resources this background job uses be limited ?
(throttling)

48. #17 Check for
Data Hotspots
Data Hotspots – data objects (typically row) which have read,
write or locking contention
© 2020 Percona49

49. Can cause excessive Errors due to Rollbacks
© 2020 Percona 50

50. Can see specific queries having issues
© 2020 Percona 51

51. #18 Check your
Application
Server
Configuration
© 2020 Percona 52
Re-Establishing
Connections
Unnecessary, Especially
with SSL/TLS
Poor Connection Pool
Settings
Poor Performance
Settings (especially for
JDBC)
Use of Prepared
Statements
Caching Settings (if
Available)

52. Summary
© 2020 Percona 53
Facing Traffic
spike with
MySQL – Do not
Panic
There are many
tools available
for you
Do not try
applying all of
them, pick most
appropriate
ones
Average system
can candle at
least 3x times
traffic using only
changing “low
hanging fruits”

54. Thank you, Let’s Connect!
https://www.linkedin.com/in/peterzaitsev/
https://twitter.com/PeterZaitsev