Fix and Find Production Issues (high level, no SQL or code)

1. Finding and Fixing
Production Issues
Dave Stokes
@stoker
David.Stokes@oracle.com

2. Gorilla in cancer slide
Recently an image of a gorilla
was 'added' to slides of cancer
cells and reviewed by a group of
radiologists.

3. Gorilla in cancer slide
Recently an image of a gorilla
was 'added' to slides of cancer
cells and reviewed by a group of
radiologists.
The gorilla was recognized only
by 17% of the radiologists!

4. The gorilla responsed

5. Who? Me worry?
● If you relied on the work of the
radiologists, would you be
concerned?
● Lets assume form here on your
LAMP stack is tuned, cached,
tweaked optimally.

6. Who? Me worry?
● If you relied on the work of the
radiologists, would you be
concerned?

7. Do not forget
The output from one woman
pregnant for nines months is
much different than that of nine
women pregnant for one month
– Werner Von Braun

8. Problem Solvers

9. What are your trying
to maximize/minimize
● What is the goal?
● Work systemically
● Optimization needs to be a
continuous process
● Be able to back out changes!!!!
● Really think

10. Who said?
"What is a system? A system is a network of
interdependent components that work together to
try to accomplish the aim of the system. A system
must have an aim. Without an aim, there is no
system. The aim of the system must be clear to
everyone in the system. The aim must include
plans for the future. The aim is a value judgment.
(We are of course talking here about a man-
made system.)”

11. Must have an aim
W. Edwards ● What is the
Demming 'aim' of your
LAMP stack?
● Hopefully not
'ready, fire,
aim'

12. Yogi Berra
If you don't know where you are going, you might
wind up someplace else.

13. Define
● What are your 'norms'
– Avg/Min/Max times
– End to end times or per section
– Reasonable variances
– Know your service levels

14. H. L. Mencken
For every complex problem there is an answer
that is clear, simple, and wrong.

15. Costs
● What is the ● What is the
cost of a cost of
query? Time, storage/
resources, retrieval?
and/or $ ● Growth deltas

16. Kano Model or you
are constantly slipping

17. Finding problems
● Outside of ● Re-norm-ing
norms – New and/or
– Change improved
causes – New
– Regression benchmark

18. Pipeline Pig
● Pigging in the context of
pipelines refers to the practice
of using pipeline inspection
gauges or 'pigs' to perform
various maintenance
operations on a pipeline. This
is done without stopping the
flow of the product in the
pipeline.
– How could you adapt this
idea for your environment?

19. Audit
● Random
● Every N
– Start to finish
– Section

20. Tools
● Testing suites ● Monitoring
– $$ tools
– No P-N-P
● Utilities, general system
– Systat, iostat, ps, show user

21. MEM

22. Similar products
● Solarwinds, ManageEngine,
MySQL Monitor Tool
● MySQL Workbench, Percona
tool kit

23. Re-Reengineer
● Service tiers
● Shard, shard, shard
● Pre-cache, move cache forward
● Scale vertically, diagonally
● Pray Moore's law is really true!

24. Virtual Developer Day
● March 12
– beginners and advanced tracks
– Free and Online
– Hands-on labs
– https://oracle.6connex.com/porta
l/mysql/login/?langR=en_US&mcc=O
TNblog

25. A&Q
● David.Stokes@oracle.com
● @stoker
● NorthTexasMySQL.org
● Slideshare.net/davestokes