There are many systems where Database plays central role for business operations. More often than not performance and maintainability of your “Main” database is key factor of success of your business. In order to deploy DevOps culture in your organization you need to Trust your developers and give them access to Production Database. Developers could find great insights in database metrics so you would have opportunity to improve both performance and maintainability dramatically for moderate costs. I am going to talk about catching and solving problems in Production Databases – how to increase performance and decrease costs in long time perspective. The talk is based on experience we’ve received in Production.

1. A Database story by DevOps
exploring SQL Database in Production.

2. Anton Martynenko
Dev & QA & OPS = DevOps
Head of Technical Operations at Youscan
.Net, Windows Azure, SqlServer, NoSql…
Twitter: @aamartynenko
am@youscan.biz

3. YouScanis a leader of the emerging social media monitoring & analytics market in Russia and Ukraine.
Our products:
http://youscan.ru
http://leadscanner.ru

4. The Reality:
Time is limited
Talk covers a lot of stuff
No time to go into details
If you have a question –make notes and ask later

5. The Production
Main DB
WWW
Android app
Service
iOsapp

6. The Production with more databases
Main DB
Clients
Tools
WWW
Android app
Service
iOsapp

7. The Production
Main DB
WWW
Android app
Service
iOsapp

8. Database as a Black Box
WWW
Android app
Service
iOsapp

9. Database as a source of problems
Main DB
WWW
Android app
Service
iOsapp

11. Main DB
Firewall for production DB
WWW
Android app
Service
iOsapp

12. DevOps–Developer with access to Prod
Main DB
WWW
Android app
Service
iOsapp

13. Typical Database headaches
1.Performance
2.Whatever-SQL mess in DB
3.Maintenance
Main DB

14. Typical Database headaches
1.Performance
2.Whatever-SQL mess in DB
3.Maintenance
Main DB

15. Performance issues
Memory
Disk I/O bottlenecks
High CPU usage

16. Know your Production hardware!
-Memory
-Disk I/O
-CPU usage

17. Understand your load

18. Latencies are important

19. Understand Database “Math”
Tables
Indexes
Queries

20. Understand Transaction Isolation
Serializable
Repeatable reads
Read committed
Read uncommitted

21. Performance: User Experience
WWW
“Place order”
“Success”
Main DB

22. User’s Transaction is complex
WWW
“Place order”
“Success”
Main DB

23. User’s Transaction is complex
WWW
“Place order”
“Success”
Main DB
So Slow!

24. Performance: TOP I/O Queries

25. Performance: TOP CPU Queries

26. Google to find TOP IO/CPU Queries

27. Catch the bottleneck
CPU Score
Query
13543541
SELECT * FROM ProductsWHERE …
7545314
SELECT * FROM Userswhere …
567541
INSERT INTO Reviews …
IO Score
Query
68743513
SELECT * FROM Authors INNER JOIN...
1475214
INSERT INTO OrderQueue…
1246876
INSERT INTO SystemLogs…

28. Analyze Top IO Queries
Analyze Top CPU Queries

29. Understand your query

30. Understand indexes used by query

31. Understand execution plan

32. Slow “SELECT”
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)

33. Slow “INSERT”, “UPDATE”
Too many Indexes
Suboptimal Query
Low disk IOPS (or too many inserts)

34. Understand the query

35. Catch!

36. Where is the problem?
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
Too many Indexes

37. Root cause: missing index
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
Too many Indexes
Review existing indexes
Add new index
Or change existing index

38. Root cause: query
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
Too many Indexes
Understand execution plan
Use Sqlquery parameters
Do select/update minimal field set

39. Root cause: low memory
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
Too many Indexes
Review fields in index
Archive old data
Check DB Engine config
Add more memory

40. Root cause: low IOPS
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
Too many Indexes
Use partitioning
Go to SSD
Get rid of queries

41. Root cause: too many indexes
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
Too many Indexes
Remove unused
Remove features

42. Many indexes = many features
Remove uselessfeatures

43. Analyze user behavior

44. Big Table

45. Table Partitioning
Jan 2014
Feb 2014
Mar 2014
Apr 2014
May 2014
June 2014
Table Orders, All Data
Partitioned table Orders,
Partition = Month

46. Database Sharding
Multiple instances,
Shard per country
Main DB
GB Main DB
DE Main DB
FR Main DB
Single instance,
All Data

47. Multitenancy
Multiple instances,
Shard per client
Main DB
Coca Cola
Main DB
BMW
Main DB
Shell
Main DB
Single instance,
All Data
Nestle
Main DB
Macdonald’s
Main DB
Siemens
Main DB

48. DB License very expensive
Coca Cola
Main DB
BMW
Main DB
Shell
Main DB
Nestle
Main DB
Macdonald’s
Main DB
Siemens
Main DB

49. Typical Database headaches
1.Performance
2.Whatever-SQL mess in DB
3.Maintenance
Main DB

50. Databases do a few things really well:
•They store and retrieve data
•They enforce relationships between data entities
•They provide the means to query the data for answers

51. T-SQL, PL-SQL, Whatever-SQL are poor as programming languages

52. SQL programming tools sux

53. Debugging and testing is not easy…
Typical T-SQL or PL-SQL mess in Database

54. What about Refucktoringyour SQL code?
Typical T-SQL or PL-SQL mess in Database

55. Java/C#/etc. with modern tools
Typical T-SQL or PL-SQL mess in Database

56. WTF your code is doing in Database?

57. There are no reasonsto put your Business Logicin Database

58. Data -> Database
Data access -> DAL
Transaction -> (not sql)Code

59. Deadlocks

60. Complex code -->Deadlocks

61. ORM
ORM
Main DB
Application

62. In Legacy systems more often than notORM adds complexity to the system

63. ORM
ORM is magic
?
Main DB
Application

64. Main DB
Lightweight ORM
Application

65. Typical Database headaches
1.Performance
2.Whatever-SQL mess in DB
3.Maintenance
Main DB

67. Manage schema changes

68. Change = Script

69. Two-staged schema changes

70. Antipattern: Configuration in DB
WWW
Android app
Service
iOsapp

71. Antipattern: Lookup tables in DB

72. Antipattern: Queuing in Database

73. Keep Schema clean and simple
Get rid of configuration in DB
Get rid of lookup tables in DB
Get rid of write contention
Long term goals:

74. Database size

75. Restore/backup
10 minutes
1 Hour
12 Hours
X3 days

76. Do Archive old data

77. Trustis foundation for everything

78. Thanks! Questions?

79. Me:
https://www.linkedin.com/pub/anton-martynenko/10/289/9b5
YouScan-http://youscan.ru/
LeadScanner-http://leadscanner.ru/
Monitoring & Metrics:
Zabbix-http://www.zabbix.com/
KissMetrics-https://www.kissmetrics.com/
Kibana-http://www.elasticsearch.org/overview/kibana/