SlideShare ist ein Scribd-Unternehmen logo
1 von 34
Conquering the Giants:Very Large Databases For audio, call the phone number in your meeting invite. Audio will not come through the computer speakers.
SQL Server Expertfor Quest Software Editor-in-Chief at SQLServerPedia.com Former SQL DBA Managed data warehouses,>80tb SAN storage for $7b company
Conquering the Giants: Agenda What’s a VLDB? Ginormous Growth Big Backups Tremendous Tables Slow SQL Statements Your Wingman: Quest Resources Photo Licensed with Creative Commons from http://www.flickr.com/photos/pagedooley/2172001078/
What Makes a VLDB? 1 terabyte Billion rows But Not:User Qty,Criticality,Transactions Per Second 4 Photo Licensed with Creative Commons from http://www.flickr.com/photos/tcmhitchhiker/1053095394
VLDB Space Challenges 5
Default File Growth Settings 6
Starts Out Small
 7
Ginormous Growths 8
Instant File Init in Secpol.msc 9
VLDBs = Big Backups Time Problem Nightly Load Windows Time to Recover Dev, QA, Test, DR 10
Giant Monstrous Backups 11
Cutting Them Down to Size 12
Next Weapon: SAN Snapshots 13
Post-Snapshot 14
Making the Snap Available 15
Weapon: Backup Compression 16 Photo Licensed with Creative Commons from http://www.flickr.com/photos/tcmhitchhiker/1053045338/
Or – Compress Everything! 17
VLDB Problem: Giant Tables 18
Hunt Down and Kill Unused Indexes 19
Breaking Giants Into Pieces 20
The New Way: Partitioning 21
Partitioning Basics Create filegroups Create files in the filegroups Design partition function Design partition scheme Alter table’s clustered index 22
Sliding Window Loads Partition by day Create newtable similarto Sales table,on same filegroup Load one day’sdata into empty table Swap out a partition for that table 23
Giant Drawbacks Not easy to change Requires lots of experimentation Partition elimination doesn’t always work Requires familiarity with queries, data Storage skills help Hardware-dependent 24 Photo Licensed with Creative Commons from http://www.flickr.com/photos/cellphonesusie/3224335140/
SQLCAT MAXDOP FAQ 25
The New-New Way: 2008 R2 26
Slow SQL Statements 27
Your Shovel: Resource Governor Slows zombie queries down Throttle CPU, memory Can throttle jobs, backups too Doesn’t throttle storage (yet) 28 Photo Licensed with Creative Commons from http://www.flickr.com/photos/archiemcphee/3612278108/
Implementing The Governator Workload groups: ETL processes Reports AdHoc Resource pool: ETL Reports Classification function based on: Login name Application name 29 Photo Licensed with Creative Commons from http://www.flickr.com/photos/shaunwong/2467881648/
End Results in SSMS 2008 30
Resource Governor Caveats Ask the network team about QoS Get manager consensus on SLAs Keep it absolutely transparent Avoid false alarms about performance problems Teach managers how to “see” it working 31
Your Wingman: Quest Software Capacity Manager v3.0 Capacity Planning Index Defragmentation Partitioning GUI LiteSpeed v5.1 SmartDiffs Throttled Compression 32
Showing the Giant Who’s Boss Use Instant File Initialization Consider Differential Backups Compress Data Drop Baggage Partitioning FTW Resource Governator Check Out QuestCapacity Manager Photo Licensed with Creative Commons from http://www.flickr.com/photos/keeg/1661401660/
VLDB Resources Presentation Resources:http://sqlserverpedia.com/wiki/VLDB Quest Software:http://www.quest.com/sql-server/ Upcoming Webcasts:http://www.quest.com/events

Weitere Àhnliche Inhalte

Andere mochten auch

VLDB Administration Strategies
VLDB Administration StrategiesVLDB Administration Strategies
VLDB Administration StrategiesMurilo Miranda
 
Flashy prefetching for high performance flash drives
Flashy prefetching for high performance flash drivesFlashy prefetching for high performance flash drives
Flashy prefetching for high performance flash drivesPratik Bhat
 
Vldb 2010 event processing tutorial
Vldb 2010 event processing tutorialVldb 2010 event processing tutorial
Vldb 2010 event processing tutorialOpher Etzion
 
Big data processing systems research
Big data processing systems researchBig data processing systems research
Big data processing systems researchVasia Kalavri
 
SQL Server 2008 R2 - Implementing High Availabilitty
SQL Server 2008 R2 - Implementing High AvailabilittySQL Server 2008 R2 - Implementing High Availabilitty
SQL Server 2008 R2 - Implementing High AvailabilittyRishu Mehra
 
Daniel Abadi: VLDB 2009 Panel
Daniel Abadi: VLDB 2009 PanelDaniel Abadi: VLDB 2009 Panel
Daniel Abadi: VLDB 2009 PanelDaniel Abadi
 
2016 VLDB - Messing Up with Bart: Error Generation for Evaluating Data-Cleani...
2016 VLDB - Messing Up with Bart: Error Generation for Evaluating Data-Cleani...2016 VLDB - Messing Up with Bart: Error Generation for Evaluating Data-Cleani...
2016 VLDB - Messing Up with Bart: Error Generation for Evaluating Data-Cleani...Boris Glavic
 
Performance Issues on Hadoop Clusters
Performance Issues on Hadoop ClustersPerformance Issues on Hadoop Clusters
Performance Issues on Hadoop ClustersXiao Qin
 
Hadoop admin
Hadoop adminHadoop admin
Hadoop adminBalaji Rajan
 

Andere mochten auch (11)

VLDB Administration Strategies
VLDB Administration StrategiesVLDB Administration Strategies
VLDB Administration Strategies
 
Vldb Yaron Moshe
Vldb  Yaron MosheVldb  Yaron Moshe
Vldb Yaron Moshe
 
Ś Ś•ŚąŚ
Ś Ś•ŚąŚŚ Ś•ŚąŚ
Ś Ś•ŚąŚ
 
Flashy prefetching for high performance flash drives
Flashy prefetching for high performance flash drivesFlashy prefetching for high performance flash drives
Flashy prefetching for high performance flash drives
 
Vldb 2010 event processing tutorial
Vldb 2010 event processing tutorialVldb 2010 event processing tutorial
Vldb 2010 event processing tutorial
 
Big data processing systems research
Big data processing systems researchBig data processing systems research
Big data processing systems research
 
SQL Server 2008 R2 - Implementing High Availabilitty
SQL Server 2008 R2 - Implementing High AvailabilittySQL Server 2008 R2 - Implementing High Availabilitty
SQL Server 2008 R2 - Implementing High Availabilitty
 
Daniel Abadi: VLDB 2009 Panel
Daniel Abadi: VLDB 2009 PanelDaniel Abadi: VLDB 2009 Panel
Daniel Abadi: VLDB 2009 Panel
 
2016 VLDB - Messing Up with Bart: Error Generation for Evaluating Data-Cleani...
2016 VLDB - Messing Up with Bart: Error Generation for Evaluating Data-Cleani...2016 VLDB - Messing Up with Bart: Error Generation for Evaluating Data-Cleani...
2016 VLDB - Messing Up with Bart: Error Generation for Evaluating Data-Cleani...
 
Performance Issues on Hadoop Clusters
Performance Issues on Hadoop ClustersPerformance Issues on Hadoop Clusters
Performance Issues on Hadoop Clusters
 
Hadoop admin
Hadoop adminHadoop admin
Hadoop admin
 

Ähnlich wie Vld bs conquering the giant

Case Study with Answers.com on Scaling with Memcached and MySQL
Case Study with Answers.com on Scaling with Memcached and MySQLCase Study with Answers.com on Scaling with Memcached and MySQL
Case Study with Answers.com on Scaling with Memcached and MySQLanswers
 
Kscope 14 Presentation : Virtual Data Platform
Kscope 14 Presentation : Virtual Data PlatformKscope 14 Presentation : Virtual Data Platform
Kscope 14 Presentation : Virtual Data PlatformKyle Hailey
 
DevOps, Databases and The Phoenix Project UGF4042 from OOW14
DevOps, Databases and The Phoenix Project UGF4042 from OOW14DevOps, Databases and The Phoenix Project UGF4042 from OOW14
DevOps, Databases and The Phoenix Project UGF4042 from OOW14Kyle Hailey
 
Db As Behaving Badly... Worst Practices For Database Administrators Rod Colledge
Db As Behaving Badly... Worst Practices For Database Administrators Rod ColledgeDb As Behaving Badly... Worst Practices For Database Administrators Rod Colledge
Db As Behaving Badly... Worst Practices For Database Administrators Rod Colledgesqlserver.co.il
 
Data Virtualization: Revolutionizing data cloning
Data Virtualization: Revolutionizing data cloningData Virtualization: Revolutionizing data cloning
Data Virtualization: Revolutionizing data cloning Kyle Hailey
 
SQL Server On SANs
SQL Server On SANsSQL Server On SANs
SQL Server On SANsQuest Software
 
BGOUG "Agile Data: revolutionizing database cloning'
BGOUG  "Agile Data: revolutionizing database cloning'BGOUG  "Agile Data: revolutionizing database cloning'
BGOUG "Agile Data: revolutionizing database cloning'Kyle Hailey
 
Apache Storm
Apache StormApache Storm
Apache StormEdureka!
 
Data Vault 2.0 Demystified: East Coast Tour
Data Vault 2.0 Demystified: East Coast TourData Vault 2.0 Demystified: East Coast Tour
Data Vault 2.0 Demystified: East Coast TourWhereScape
 
Kscope 2013 delphix
Kscope 2013 delphixKscope 2013 delphix
Kscope 2013 delphixKyle Hailey
 
6212883126866262792 performance testing_cloud
6212883126866262792 performance testing_cloud6212883126866262792 performance testing_cloud
6212883126866262792 performance testing_cloudLocuto Riorama
 
Kubernetes Failure Stories - KubeCon Europe Barcelona
Kubernetes Failure Stories - KubeCon Europe BarcelonaKubernetes Failure Stories - KubeCon Europe Barcelona
Kubernetes Failure Stories - KubeCon Europe BarcelonaHenning Jacobs
 
VMware vFabric Data Director for DB as a Service
VMware vFabric Data Director for DB as a ServiceVMware vFabric Data Director for DB as a Service
VMware vFabric Data Director for DB as a ServiceVMware vFabric
 
Data as a Service
Data as a Service Data as a Service
Data as a Service Kyle Hailey
 
Business_Continuity_Planning_with_SQL_Server_HADR_options_TechEd_Bangalore_20...
Business_Continuity_Planning_with_SQL_Server_HADR_options_TechEd_Bangalore_20...Business_Continuity_Planning_with_SQL_Server_HADR_options_TechEd_Bangalore_20...
Business_Continuity_Planning_with_SQL_Server_HADR_options_TechEd_Bangalore_20...LarryZaman
 
DOs and DONTs on the way to 10M users
DOs and DONTs on the way to 10M usersDOs and DONTs on the way to 10M users
DOs and DONTs on the way to 10M usersYoav Avrahami
 
Denver SQL Saturday The Next Frontier
Denver SQL Saturday The Next FrontierDenver SQL Saturday The Next Frontier
Denver SQL Saturday The Next FrontierKellyn Pot'Vin-Gorman
 
Virtualization and SAN Basics for DBAs
Virtualization and SAN Basics for DBAsVirtualization and SAN Basics for DBAs
Virtualization and SAN Basics for DBAsQuest Software
 
Stating the obvious - 121 Test Automation Day, Dublin, 2018
Stating the obvious - 121 Test Automation Day, Dublin, 2018Stating the obvious - 121 Test Automation Day, Dublin, 2018
Stating the obvious - 121 Test Automation Day, Dublin, 2018Giulio Vian
 
Things learned from OpenWorld 2013
Things learned from OpenWorld 2013Things learned from OpenWorld 2013
Things learned from OpenWorld 2013Connor McDonald
 

Ähnlich wie Vld bs conquering the giant (20)

Case Study with Answers.com on Scaling with Memcached and MySQL
Case Study with Answers.com on Scaling with Memcached and MySQLCase Study with Answers.com on Scaling with Memcached and MySQL
Case Study with Answers.com on Scaling with Memcached and MySQL
 
Kscope 14 Presentation : Virtual Data Platform
Kscope 14 Presentation : Virtual Data PlatformKscope 14 Presentation : Virtual Data Platform
Kscope 14 Presentation : Virtual Data Platform
 
DevOps, Databases and The Phoenix Project UGF4042 from OOW14
DevOps, Databases and The Phoenix Project UGF4042 from OOW14DevOps, Databases and The Phoenix Project UGF4042 from OOW14
DevOps, Databases and The Phoenix Project UGF4042 from OOW14
 
Db As Behaving Badly... Worst Practices For Database Administrators Rod Colledge
Db As Behaving Badly... Worst Practices For Database Administrators Rod ColledgeDb As Behaving Badly... Worst Practices For Database Administrators Rod Colledge
Db As Behaving Badly... Worst Practices For Database Administrators Rod Colledge
 
Data Virtualization: Revolutionizing data cloning
Data Virtualization: Revolutionizing data cloningData Virtualization: Revolutionizing data cloning
Data Virtualization: Revolutionizing data cloning
 
SQL Server On SANs
SQL Server On SANsSQL Server On SANs
SQL Server On SANs
 
BGOUG "Agile Data: revolutionizing database cloning'
BGOUG  "Agile Data: revolutionizing database cloning'BGOUG  "Agile Data: revolutionizing database cloning'
BGOUG "Agile Data: revolutionizing database cloning'
 
Apache Storm
Apache StormApache Storm
Apache Storm
 
Data Vault 2.0 Demystified: East Coast Tour
Data Vault 2.0 Demystified: East Coast TourData Vault 2.0 Demystified: East Coast Tour
Data Vault 2.0 Demystified: East Coast Tour
 
Kscope 2013 delphix
Kscope 2013 delphixKscope 2013 delphix
Kscope 2013 delphix
 
6212883126866262792 performance testing_cloud
6212883126866262792 performance testing_cloud6212883126866262792 performance testing_cloud
6212883126866262792 performance testing_cloud
 
Kubernetes Failure Stories - KubeCon Europe Barcelona
Kubernetes Failure Stories - KubeCon Europe BarcelonaKubernetes Failure Stories - KubeCon Europe Barcelona
Kubernetes Failure Stories - KubeCon Europe Barcelona
 
VMware vFabric Data Director for DB as a Service
VMware vFabric Data Director for DB as a ServiceVMware vFabric Data Director for DB as a Service
VMware vFabric Data Director for DB as a Service
 
Data as a Service
Data as a Service Data as a Service
Data as a Service
 
Business_Continuity_Planning_with_SQL_Server_HADR_options_TechEd_Bangalore_20...
Business_Continuity_Planning_with_SQL_Server_HADR_options_TechEd_Bangalore_20...Business_Continuity_Planning_with_SQL_Server_HADR_options_TechEd_Bangalore_20...
Business_Continuity_Planning_with_SQL_Server_HADR_options_TechEd_Bangalore_20...
 
DOs and DONTs on the way to 10M users
DOs and DONTs on the way to 10M usersDOs and DONTs on the way to 10M users
DOs and DONTs on the way to 10M users
 
Denver SQL Saturday The Next Frontier
Denver SQL Saturday The Next FrontierDenver SQL Saturday The Next Frontier
Denver SQL Saturday The Next Frontier
 
Virtualization and SAN Basics for DBAs
Virtualization and SAN Basics for DBAsVirtualization and SAN Basics for DBAs
Virtualization and SAN Basics for DBAs
 
Stating the obvious - 121 Test Automation Day, Dublin, 2018
Stating the obvious - 121 Test Automation Day, Dublin, 2018Stating the obvious - 121 Test Automation Day, Dublin, 2018
Stating the obvious - 121 Test Automation Day, Dublin, 2018
 
Things learned from OpenWorld 2013
Things learned from OpenWorld 2013Things learned from OpenWorld 2013
Things learned from OpenWorld 2013
 

KĂŒrzlich hochgeladen

Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024The Digital Insurer
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘RTylerCroy
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAndrey Devyatkin
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businesspanagenda
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherRemote DBA Services
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodJuan lago vĂĄzquez
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024The Digital Insurer
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobeapidays
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...DianaGray10
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel AraĂșjo
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 

KĂŒrzlich hochgeladen (20)

Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
 

Vld bs conquering the giant

  • 1. Conquering the Giants:Very Large Databases For audio, call the phone number in your meeting invite. Audio will not come through the computer speakers.
  • 2. SQL Server Expertfor Quest Software Editor-in-Chief at SQLServerPedia.com Former SQL DBA Managed data warehouses,>80tb SAN storage for $7b company
  • 3. Conquering the Giants: Agenda What’s a VLDB? Ginormous Growth Big Backups Tremendous Tables Slow SQL Statements Your Wingman: Quest Resources Photo Licensed with Creative Commons from http://www.flickr.com/photos/pagedooley/2172001078/
  • 4. What Makes a VLDB? 1 terabyte Billion rows But Not:User Qty,Criticality,Transactions Per Second 4 Photo Licensed with Creative Commons from http://www.flickr.com/photos/tcmhitchhiker/1053095394
  • 6. Default File Growth Settings 6
  • 9. Instant File Init in Secpol.msc 9
  • 10. VLDBs = Big Backups Time Problem Nightly Load Windows Time to Recover Dev, QA, Test, DR 10
  • 12. Cutting Them Down to Size 12
  • 13. Next Weapon: SAN Snapshots 13
  • 15. Making the Snap Available 15
  • 16. Weapon: Backup Compression 16 Photo Licensed with Creative Commons from http://www.flickr.com/photos/tcmhitchhiker/1053045338/
  • 17. Or – Compress Everything! 17
  • 18. VLDB Problem: Giant Tables 18
  • 19. Hunt Down and Kill Unused Indexes 19
  • 20. Breaking Giants Into Pieces 20
  • 21. The New Way: Partitioning 21
  • 22. Partitioning Basics Create filegroups Create files in the filegroups Design partition function Design partition scheme Alter table’s clustered index 22
  • 23. Sliding Window Loads Partition by day Create newtable similarto Sales table,on same filegroup Load one day’sdata into empty table Swap out a partition for that table 23
  • 24. Giant Drawbacks Not easy to change Requires lots of experimentation Partition elimination doesn’t always work Requires familiarity with queries, data Storage skills help Hardware-dependent 24 Photo Licensed with Creative Commons from http://www.flickr.com/photos/cellphonesusie/3224335140/
  • 26. The New-New Way: 2008 R2 26
  • 28. Your Shovel: Resource Governor Slows zombie queries down Throttle CPU, memory Can throttle jobs, backups too Doesn’t throttle storage (yet) 28 Photo Licensed with Creative Commons from http://www.flickr.com/photos/archiemcphee/3612278108/
  • 29. Implementing The Governator Workload groups: ETL processes Reports AdHoc Resource pool: ETL Reports Classification function based on: Login name Application name 29 Photo Licensed with Creative Commons from http://www.flickr.com/photos/shaunwong/2467881648/
  • 30. End Results in SSMS 2008 30
  • 31. Resource Governor Caveats Ask the network team about QoS Get manager consensus on SLAs Keep it absolutely transparent Avoid false alarms about performance problems Teach managers how to “see” it working 31
  • 32. Your Wingman: Quest Software Capacity Manager v3.0 Capacity Planning Index Defragmentation Partitioning GUI LiteSpeed v5.1 SmartDiffs Throttled Compression 32
  • 33. Showing the Giant Who’s Boss Use Instant File Initialization Consider Differential Backups Compress Data Drop Baggage Partitioning FTW Resource Governator Check Out QuestCapacity Manager Photo Licensed with Creative Commons from http://www.flickr.com/photos/keeg/1661401660/
  • 34. VLDB Resources Presentation Resources:http://sqlserverpedia.com/wiki/VLDB Quest Software:http://www.quest.com/sql-server/ Upcoming Webcasts:http://www.quest.com/events

Hinweis der Redaktion

  1. Production, Dev, QA, DRWhen files are this big, even the simplest thing is time-consuming – think growing a file.
  2. Runs faster tooFrees up nightly windowDoesn’t ease recovery time, though. Makes recovery time LONGER.Let’s look at a solution that can ease recovery time too.
  3. Do your backups off the snapshotHold multiple snapshots over timeNot everybody can afford this level of recoverability though.Let’s look at something that Microsoft included with SQL 2008 Enterprise Edition.
  4. New in SQL 2008Enterprise Edition onlyInclude DR, development boxesNot configurable
  5. New in SQL 2008Enterprise Edition onlyInclude DR, development boxesNot inherited – tables, indexesScript to make it easier at the SSP article
  6. Note index size on 2nd rowEvery row has to be backed up, DBCC’d. Harder to get DBCC time on VLDBs.Every index has to be backed up and reorganized
  7. Dropping indexes gives us less of them, but they’re still huge.How can we break them into smaller pieces?
  8. Easier because it’s transparent to the application.
  9. Happens nearly instantaneouslyOnly a metadata change
  10. Zombie ad-hoc queries run forever. User closes web browser, but the query runs on headlessly