Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Ā
Sql server operational best practices notes from the field - charley hanania - lausanne - 2011-06-09
1. Notes from the Field
Presented to the Swiss PASS Chapter - Zurich.
Hotel Continental - Lausanne, Switzerland
15:30 ā 19:00, June 9th 2011
Charley Hanania, QS2 AG
B.Sc (Computing), MCP, MCDBA, MCITP, MCTS, MCT, Microsoft MVP: SQL Server
Senior Database Specialist
2. My Background
ā¢ Now:
ā¢ Microsoft MVP: SQL Server
ā¢ Database Consultant at QS2 AG
ā¢ Formerly:
ā¢ Production Product Owner of MS SQL Server Platform at UBS Investment Bank
ā¢ Technical Team Lead
ā¢ Consultant etcā¦
ā¢ ITIL v3 Certified
ā¢ SQL Server Certified since 1998
ā¢ On SQL Server since 1995
ā¢ Version 4 on OS/2
ā¢ IT Professional since 1992
ā¢ PASS
ā¢ Chapter Leader ā Switzerland
ā¢ Regional Mentor ā Europe
ā¢ 24 Hours of PASS Team Member
ā¢ Event Speaker
3
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
4. Agenda
ā¢ Best Practices Defined
ā¢ āSomeā Operational Best Practices
ā¢ Best Practices in this Presentation
1. Obfuscation
2. Rubber Stamping your installation
3. Using Projects and Solutions
4. Governance
5. SLAās, SLEās, SLOās
6. Monitoring Appropriately
7. Diagnostics Reports
ā¢ Recap
ā¢ Questions
5
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
5. Definition :: Best Practices
ā¢ Focus SQL Server / Operations
ā¢ Guidelines, methods, procedures & policies that have been used
with numerous customers & installations and have assisted with:
ā¢ Root Cause Analysis
ā¢ Clarity of Design
ā¢ Avoidance of Issues
ā¢ Performance
ā¢ Security
ā¢ Disaster
ā¢ Speed of implementation
ā¢ System Longevity
6
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
6. āSomeā Operational Best Practices
7
Rename SA
Rename Administrator
Disable SA
SQL IO
64KB Cluster Size
Multiple Files/Filegroups
Multiple TempDB Files
Partitioning
Filtered Indexes
Development Standards
Projects and Solutions
Scripting with SQLCMD
Scripting and Automating with Powershell
Multiple development Environments
Singleton Development Environments
Principle of Least Privilege
Governance through PBM
Security Engineering
Performance Engineering
Resilience Engineering
Backup/Recovery
Compression
Solid State Disks
Recovery Planning
Availability Planning
Disaster Planning
Monitoring
Service Level Agreements
Service Level Expectations
Service Level Objectives
Templated Installations
Capacity Planning
Check Sheets
Run Books
Encryption
Authentication Certificates
Code Reuse
Plan Reuse
indexing for Read
Indexing for write
Asynchronous logging & messaging
Virtualisation
Maintenance Checking
Consolidation
Baselines
Instances
Port Changes
SQLServerOperationalBestPractices-NotesfromtheField
Page File
Instant Initialisation
Memory
7. Best Practices in this Presentationā¦
8
Rename SA
Rename Administrator
Disable SA
SQL IO
64KB Cluster Size
Multiple Files/Filegroups
Multiple TempDB Files
Partitioning
Filtered Indexes
Development Standards
Projects and Solutions
Scripting with SQLCMD
Scripting and Automating with Powershell
Multiple development Environments
Singleton Development Environments
Principle of Least Privilege
Governance through PBM
Security Engineering
Performance Engineering
Resilience Engineering
Backup/Recovery
Compression
Solid State Disks
Recovery Planning
Availability Planning
Disaster Planning
Monitoring
Service Level Agreements
Service Level Expectations
Service Level Objectives
Templated Installations
Capacity Planning
Check Sheets
Run Books
Encryption
Authentication Certificates
Code Reuse
Plan Reuse
indexing for Read
Indexing for write
Asynchronous logging & messaging
Virtualisation
Maintenance Checking
Consolidation
Baselines
Instances
Port Changes 07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
Page File
Instant Initialisation
Memory
9. Obfuscation
ā¢ Why?
ā¢ Securityā¦
ā¢ If you canāt find the lock you canāt try your key
ā¢ If you canāt find the door knob, you canāt open the door
ā¢ If you canāt find the door you canāt get in nor out.
10
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
10. Obfuscation
ā¢ Customer Scenarios
ā¢ Requirement:
ā¢ Build me a Laptop Demo copy of our offering to Military Security Standards
ā¢ If lost / stolen its useless
ā¢ Valid Intellectual property
ā¢ Full suite of software, middleware, back-end
ā¢ Fully self-contained
11
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
11. Obfuscation
ā¢ Customer Scenarios
ā¢ IP Conscious Customer:
ā¢ Make sure our data and databases are completely secure.
ā¢ When youāre done, weāll contract a Russian hacker to break in.
12
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
14. Rubber Stamping your Installation 15
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
15. Rubber Stamping your installation
ā¢ Why?
ā¢ Closer to the single version of the truth.
ā¢ Know the parameters your system operates under.
ā¢ Rebuild, reset, replace.
ā¢ Speed of deployment.
16
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
16. Rubber Stamping your installation
ā¢ Customer Scenarios
ā¢ Development / Test / Production Environments were out of Sync
ā¢ Collations
ā¢ Service Accounts
ā¢ File locations
ā¢ Other Environmental Issues
ā¢ Changes developed and tested failed or performed badly in Production
17
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
19. Using Projects and Solutions 20
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
20. Using Projects and Solutions
ā¢ Why?
ā¢ Gives order to implementations
ā¢ Grouping of items for a solution
ā¢ Search and reuse
ā¢ Assists in Team-Oriented approaches
21
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
21. Using Projects and Solutions
ā¢ Customer Scenarios
ā¢ Teams of 3+ developers
ā¢ Overwriting each otherās code
ā¢ Loss of scripts and object definitions
ā¢ Propagation of the wrong objects to test/production
ā¢ Unclear path to implementation
22
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
22. Using Projects and Solutions
ā¢ Solutions:
ā¢ Use Ticketing system for functionality requests
ā¢ Agree on scope of work
ā¢ Develop with clear, sequenced file naming ā even scratch files
ā¢ When ready for promotion into communal development environment,
group files into a āRelease xxxā project.
ā¢ Deploy, test etcā¦
23
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
25. SLAās, SLEās, SLOās
ā¢ What?
ā¢ SLA: āService Level Agreementā
ā¢ Agreement with Business / Management on what the
applications/services/systems will provide to the āend customerā
ā¢ SLE: āService Level Expectationā
ā¢ Agreement of mutual understanding with Business / Management on what the
applications/services/systems will provide to the āend customerā
ā¢ SLO: āService Level Objectiveā
ā¢ Agreement internally on what the applications/services/systems will provide to
the āend customerā
ā¢ Focus:
ā¢ Performance
ā¢ Availability
ā¢ Recoverability
ā¢ [Not Functionality]
26
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
26. SLAās, SLEās, SLOās
ā¢ Why?
ā¢ Without agreed, contracted and planned service levels, management
and customers (users) have nothing to track the:
ā¢ Availability of Service;
ā¢ Performance of Systems; and
ā¢ Quality of Implementation.
ā¢ Management can assign funds, plan and allocate relevant resources to
meet the Agreements
ā¢ Business understands the true process, business and technical
dependencies of customer facing systems.
ā¢ Change is entered into warily and with full support of all associated
parties.
27
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
27. SLAās, SLEās, SLOās
ā¢ Customer Scenarios
ā¢ 1 Resource :: Developer/DBA/DB{system} Admin for a customer critical
system across APAC and Europe.
ā¢ Management were happy:
ā¢ Simple Change Management
ā¢ Quick Feature implementation
ā¢ Low Costs
ā¢ High Revenues
ā¢ Resource was happy:
ā¢ Job Stability
ā¢ Developer Artistic Licence
ā¢ Integral Part of the Team
28
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
28. SLAās, SLEās, SLOās
ā¢ Customer Scenarios
ā¢ 1 Resource :: Developer/DBA/DB{system} Admin for a customer critical
system across APAC and Europe.
ā¢ Problem:
ā¢ Management and Developer had an argument on strategy
ā¢ Productivity ebbed
ā¢ Quality degraded
ā¢ Motivation decreased
ā¢ Resource was āsickā more often
ā¢ All Application and system IP was with the 1 Resource!
29
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
29. SLAās, SLEās, SLOās
ā¢ Customer Scenarios
ā¢ Very technical team
ā¢ Eat and Breath Technology
ā¢ Communication skills ā average
ā¢ Management adapted to Technical Team style
ā¢ Unmanaged changes
ā¢ Periods of instability
ā¢ Overall - best efforts support
ā¢ Operations Expanded
ā¢ Customer base increased
ā¢ Internal and external customers became highly dependant on the
technology base. 30
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
30. SLAās, SLEās, SLOās
ā¢ Customer Scenarios
ā¢ Very technical team
ā¢ Eat and Breath Technology
ā¢ Communication skills ā average
ā¢ Periods of system unresponsiveness
ā¢ Outages affecting the whole organisation
ā¢ etc
ā¢ Periods when users āfeltā like there were issues with the application.
31
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
31. SLAās, SLEās, SLOās
ā¢ Solutions:
1. Define which applications, systems and services are in production
2. Define and categorise them by important to business
ā¢ Including cost & revenue
3. Define metrics focussed on business processes such as:
ā¢ Customer Creation
ā¢ Business Transaction completion
ā¢ Business Transaction retrieval
4. Define Capacity / Availability needs such as:
ā¢ X New Customers per day
ā¢ Y Business Transactions per hour
ā¢ Available during core business hours (8x5) etc
32
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
32. SLAās, SLEās, SLOās
ā¢ Solutions:
5. Review Platform abilities based on Definitions
ā¢ Measure Current as a baseline
6. Review Human Resource needs based on defined System Availability
and Capacity needs.
7. Define a plan for when/if system can approach those levels
ā¢ Redevelopment / refactoring
ā¢ Personnel Acquisition / Acquiescence
8. Measure and Monitor Progress
ā¢ Stops the fire-fighting
ā¢ Allows partners and customers to see progress
ā¢ Gives agility in approach, focussed on importance and need.
33
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
34. Monitoring Appropriately
ā¢ Why?
ā¢ If you donāt know, thenā¦ You donāt knowā¦
ā¢ Availability
ā¢ Capacity
ā¢ Performance
ā¢ Recoverability
ā¢ Reduces Fire-fighting
ā¢ Leaves your fingers on the Pulse.
35
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
35. Monitoring Appropriately
ā¢ Customer Scenarios
ā¢ Too Little:
ā¢ āWhy is my system slowā¦?ā
ā¢ āWas the system up last night? I tried toā¦ā
ā¢ How long would it take us to rebuild
ā¢ Too much:
ā¢ āWe missed the performance issue because we had thousands of Printer
Job Completion entries to closeā¦ā
ā¢ āWe canāt add any more SQL Counters as thereās no more capacity in the
DWā
ā¢ āWhyās the Network so slow? Oh, its all the traffic being sent to the
Monitoring and alerts DWā¦ā
36
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
36. Monitoring Appropriately
ā¢ Solutions:
ā¢ Centralised āSystem Monitorā-Type Consoles
ā¢ Environment at a glance
ā¢ Standard interfaces
ā¢ Alerts and trending
ā¢ Agent or Poll methodologies
ā¢ Technologies:
ā¢ Systems Centre Operations Manager
ā¢ Nagios etc
ā¢ System Monitor
ā¢ Management Data Warehouse
ā¢ DMVStats
ā¢ SQL Server 2008 Management Studio
ā¢ Activity Monitor/Performance Dashboard Reports
ā¢ SQL Server Health and History Tool
ā¢ RML Utilities for SQL Server
etc
37
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
38. Governance
ā¢ Why?
ā¢ Governance is simply the execution of standards policies
ā¢ For control
ā¢ For consistency
ā¢ For Reporting
ā¢ For ease of Support and Maintenance
39
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
39. Governance
ā¢ Customer Scenarios
ā¢ Too much:
ā¢ Large Organisation
ā¢ Such strict policies that everything was either unsupported or an exception.
ā¢ Reduced Productivity
ā¢ No policy on items that werenāt covered by the policies.
ā¢ Too little:
ā¢ Large Organisation
ā¢ Do whatās needed to fulfil the requirements
ā¢ Support nightmare
ā¢ Great for consultants! ļ
ā¢ No Standards / documentation either
ā¢ SLAās ??? 40
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
40. Governance
ā¢ Solutions:
ā¢ Various frameworks eg. ITIL
ā¢ For SQL Server Auditing / Control:
ā¢ DDL Triggers
ā¢ Policy Based Management
ā¢ SQL Server Auditing Framework
41
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
44. Diagnostics Reports
ā¢ Why?
ā¢ As a tool to help when investigating issues
ā¢ Is a cursory finger on the pulse
ā¢ Should be 2-pronged
1. Business focussed
2. System focussed
58
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
45. Diagnostics Reports
ā¢ Customer Scenarios
ā¢ System with thousands of devices reporting on short intervals and
bursts (status etc) asynchronously
ā¢ Business processes triggered on certain criteria
ā¢ Customer facing interfaces experience varying loads
ā¢ Database is core to the offering
ā¢ When looking at issues the general same approach is taken
1. Are the feeds running?
2. Are they coming through with the expected throughput/latency?
3. Are major process components locking / blocking?
4. Are the External Interfaces open and serving?
59
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
46. Diagnostics Reports
ā¢ Solutions:
1. Use scripting languages to provide a āprinter test pageā-type report
2. Use Reporting Services to provide a consolidated view of several
systems
3. Use SQL Server Management Studioās Custom Reports to get instance
focussed info or to report out from a warehouse.
60
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
47. Recap
ā¢ Best Practices Defined
ā¢ āSomeā Operational Best Practices
ā¢ Best Practices in this Presentation
1. Obfuscation
2. Rubber Stamping your installation
3. Using Projects and Solutions
4. Governance
5. SLAās, SLEās, SLOās
6. Monitoring Appropriately
7. Diagnostics Reports
ā¢ Recap
ā¢ Questions
61
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField