Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Denver devops : enabling DevOps with data virtualization

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Wird geladen in …3
×

Hier ansehen

1 von 124 Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Andere mochten auch (20)

Anzeige

Ähnlich wie Denver devops : enabling DevOps with data virtualization (20)

Anzeige

Aktuellste (20)

Denver devops : enabling DevOps with data virtualization

  1. 1. Accelerating DevOps with Virtual Data 1 http://kylehailey.com kyle@delphix.com Tim Gorman tim@delphix.com
  2. 2. Accelerating the tempo of application development Fortune 1 Retailer #1 Social Network #1 Financial Services #1 Network Equipment #1 Cable #1 Wholesale #1 Food Service Co. #1 Office Supplies #1 Apparel & Footwear #1 Chip Manufacturing #1 Pharma #1 Auction Marketplace #1 Total Healthcare #1 Aerospace #1 Computer Access #1 CPG #1 ETL Software #1 Availability Service #1 Mutual Life Ins. #1 Satellite TV #1 State Gov #1 Cruise Line #1 Retirement Fund #1 IT Services #1 Game Software © 2014 Delphix. All Rights Reserved Private and confidential 2
  3. 3. Are you too busy to Innovate? Inertia
  4. 4. What is DevOps = tools + culture • Culture : – Empathy – Collaboration – Bridging silos, avoid blame • Tools : – Automation – Measurement – Self-service 4
  5. 5. Note: DevOps > Tools + Culture DevOps= optimizing flow from Dev to Ops to Pro 5 “Do not seek to follow in the footsteps of the wise. Seek what they sought” - Matsuo Bashō Goal = company’s bottom line
  6. 6. The Goal : Theory of Constraints Improvement not made at the constraint is an illusion factory floor optimization
  7. 7. Factory floor
  8. 8. Factory floor constraint Not a relay race
  9. 9. Tune before constraint constraint Tuning here Stock piling
  10. 10. Tune after constraint constraint Tuning here Starvation
  11. 11. Factory floor : straight forward constraint Goal: find constraint optimize it
  12. 12. Theory of Constraints work for IT ? • Goals Clarify • Metrics Define • Constraints Identify • Priorities Set • Iterations Fast • CI • Cloud • Agile • Kanban • Kata “IT is the factory floor of this century”
  13. 13. The Phoenix Project What is the constraint in IT ?
  14. 14. What are the top 5 constraints in IT? 1. Dev environments setup 2. QA setup 3. Code Architecture 4. Development 5. Product management “One of the most powerful things that organizations can do is to enable development and testing to get environment they need when they need it“ - Gene Kim
  15. 15. Data is the constraint CIO Magazine Survey: 60% Projects Over Schedule 85% delayed waiting for data Data is the Constraint only getting worse Gartner: Data Doomsday, by 2017 1/3rd IT in crisis
  16. 16. In this presentation : • Data Constraint • Solution • Use Cases
  17. 17. • Data Constraint • Solution • Use Cases
  18. 18. moving data is hard – Storage & Systems – Personnel – Time
  19. 19. Typical Architecture Production Instance Database File system
  20. 20. Typical Architecture Production Instance Backup Database File system Database File system
  21. 21. Typical Architecture Production Instance Reporting Backup Database File system Instance Database File system Database File system
  22. 22. Typical Architecture Production Instance Database File system Triple Tax Dev, QA, UAT Reporting Backup Instance Instance Instance Instance Database Database File system Database File system File system Database File system Database File system
  23. 23. Typical Architecture Production Instance Database File system Instance Instance Instance Instance Database Database File system Database File system File system Database File system Database File system
  24. 24. Data floods infrastructure 92% of the cost of business, in financial services business , is “data” www.wsta.org/resources/industry-articles Most companies average 5% IT spending , ½ on “data” http://uclue.com/?xq=1133
  25. 25. Four Areas hit by data constraint 1. IT Capital resources $ 2. IT Operations personnel $ 3. Application Development $$$ 4. Business $$$$$$$
  26. 26. 1. Hardware – copies take up space –Servers –Storage –Network –Data center floor space, power, cooling
  27. 27. $ Never enough environments
  28. 28. $ IT Operations – copying data takes people time • People 1000s hours per year just for DBAs – DBAs – SYS Admin – Storage Admin – Backup Admin – Network Admin • $100s Millions for data center modernizations
  29. 29. $ Application Development – wait for copies • Inefficient QA: Higher costs of QA • QA Delays : Greater re-work of code • Sharing DB Environments : Bottlenecks • Using DB Subsets: More bugs in Prod • Slow Environment Builds: Delays
  30. 30. $ Business – decisions depend on data access Ability to capture revenue • Business Applications – Delays cause lost revenue • Business Intelligence – Old data = less intelligence
  31. 31. companies unaware
  32. 32. companies unaware Boss, Storage Admin, DBA Developer or Analyst
  33. 33. companies unaware Metrics – Time – Old Data – Storage Other – Analysts –Audits
  34. 34. What Problems does Data Constraint Cause 1. Bottlenecks 2. Waiting for environments 3. Waiting to check in code 4. Production Bugs 5. Expensive Slow QA
  35. 35. Your Project Available Resources
  36. 36. Development : bottlenecks Frustration Waiting
  37. 37. Development : Bugs Old Unrepresentative Data
  38. 38. Development : subsets False Negatives False Positives Bugs in Production
  39. 39. Production Wall 40
  40. 40. Development : silos
  41. 41. QA : Long Build times X Bug 70 60 50 40 30 20 10 0 1 2 3 4 5 6 7 Delay in Fixing the bug Cost To Correct Software Engineering Economics – Barry Boehm (1981)
  42. 42. DevOps and Data : Impossible? Life with Waterfall Dream of Agile & CI
  43. 43. Waterfall vs Agile 44
  44. 44. Missed ! Goal Agile & CI vsWaterfall Agile & CI Achieved !
  45. 45. bugs time Missed ! Goal Agile & CI Achieved ! Bugs
  46. 46. profit time Missed ! Goal Agile & CI Achieved ! Profit
  47. 47. Missed ! Goal Cost per Deployment Agile & CI Achieved ! Cost Per Deployment time
  48. 48. DevOps and Data : Impossible? • Data & DevOps : Impossible ? • 20 copies of production a day for CI • Each copy is like
  49. 49. In this presentation : • Data Constraint • Solution • Use Cases
  50. 50. 99% of blocks are identical Clone 1 Clone 2 Clone 3
  51. 51. Solution
  52. 52. Thin Clone Clone 1 Clone 2 Clone 3
  53. 53. Technology Core : file system snapshots • EMC – 16 snapshots on Symmetrix – Write performance impact – No snapshots of snapshots • Netapp – 255 snapshots • ZFS – Compression – Unlimited snapshots – Snapshots of Snapshots • DxFS – “” – Storage agnostic – Shared cache in memory Also check out new SSD storage such as: Pure Storage, EMC XtremIO
  54. 54. Fuel not equal car Challenges 1. Technical 2. Bureaucracy
  55. 55. Bureaucracy Developer Asks for DB Get Access Manager approves DBA Request system Setup DB System Admin Request storage Setup machine Storage Admin Allocate storage (take snapshot)
  56. 56. 1hour 9 days 1 day Why are hand offs so expensive? Bureaucracy
  57. 57. Technical Challenge Production Filer Database Luns Target A Target B Target C snapshot clones InsIntsatannccee InInssttaanncece InInssttaannccee InInssttaannccee Instance Source
  58. 58. Development Filer Production Filer clones Database LUNs snapshot Technical Challenge Instance Target A InInssttaannccee Target B InInssttaanncece Target C InInssttaannccee Instance
  59. 59. Technical Challenge Production Copy Time Flow Purge Storage Development File System Instance 1 2 3 Clone (snapshot) Compress Share Cache Provision Mount, recover, rename Self Service, Roles & Security Instance
  60. 60. How to get a Data Virtualization? 2 1 – EMC + SRDF – Netapp 2 + SMO 1 – Oracle EM 12c + data guard + Netapp /ZFS – Actifio - hardware – Delphix - software 3 1 2 Source sync Deploy automation Storage snapshots 1 2 3
  61. 61. Goal : virtualize, govern, deliver 62 • Masking: Masking • Security: Chain of custody • Self Service: Logins • Developer: Versioning , branching • Audit: Live Archive Data Supply Chain Data Virtualization Thin Cloning Snap Shots
  62. 62. Dev Production Time Flow Prod 2.6 Dev finishes a sprint or point release and QA forks off a clone virtual database from Dev database
  63. 63. Dev QA Production Time Flow Prod 2.6 Continuous integration Nightly or hourly regressions
  64. 64. Dev QA Production Time Flow Prod 2.6 Dev finishes a sprint or point release and QA forks off a clone virtual database from Dev database
  65. 65. Dev QA Production Time Flow Prod 2.6 Dev finishes a sprint or point release and QA forks off a clone virtual database from Dev database UAT
  66. 66. Prod Dev 2.7 QA UAT Production Time Flow UAT QA Dev 2.6
  67. 67. Intel hardware DB2 Data File Systems Binaries Install Delphix on x86 hardware
  68. 68. Allocate Any Storage to Delphix Allocate Storage Any type Pure Storage + Delphix Better Performance for 1/10 the cost
  69. 69. One time backup of source database Production InsIIntnsasttanannccceee Database File system
  70. 70. DxFS (Delphix) Compress Data Production InsIIntnsasttanannccceee Database Data is compressed typically 1/3 size File system
  71. 71. Incremental forever change collection Production Database File system Changes Time Window • Collected incrementally forever • Old data purged InsIIntnsasttanannccceee
  72. 72. Snapshot 1 – full backup once only at link time Jonathan Lewis © 2013 Virtual DB 73 / 30 a b c d e f g h i We start with a full backup - analogous to a level 0 rman backup. Includes the archived redo log files needed for recovery. Run in archivelog mode.
  73. 73. Snapshot 2 (from SCN) a b c d e f g h i b' c' The "backup from SCN" is analogous to a level 1 incremental backup (which includes the relevant archived redo logs). Sensible to enable BCT. Jonathan Lewis © 2013 Delphix executes standard rman scripts
  74. 74. Apply Snapshot 2 a b b' c c' d e f g h i The Delphix appliance unpacks the rman backup and "overwrites" the initial backup with the changed blocks - but DxFS makes new copies of the blocks Jonathan Lewis © 2013
  75. 75. Drop Snapshot 1 a b' c' d e f g h i The call to rman leaves us with a new level 0 backup, waiting for recovery. But we can pick the snapshot root block. We have EVERY level 0 backup Jonathan Lewis © 2013
  76. 76. Creating a vDB a b' c' d e f g h i The first step in creating a vDB is to take a snapshot of the filesystem as at the backup you want (then roll it forward) Jonathan Lewis © 2013 My vDB (filesystem) Your vDB (filesystem)
  77. 77. Creating a vDB a b' c' d e f g h i The first step in creating a vDB is to take a snapshot of the filesystem as at the backup you want (then roll it forward) Jonathan Lewis © 2013 My vDB (filesystem) Your vDB (filesystem) b'' c'' ff ii i’
  78. 78. Database Virtualization
  79. 79. Three Physical Copies Three Virtual Copies Data Virtualization Appliance
  80. 80. Before Virtual Data Production Dev, QA, UAT Instance Reporting Backup Database File system Instance Instance Instance Instance Database Database File system Database File system File system Database File system Database File system “triple data tax”
  81. 81. With Virtual Data Production Instance Dev & QA Instance InInssttaannccee InInssttaannccee Database Reporting Instance Database Backup Database Instance Instance Instance Database Database Database File system Data Virtualization Appliance
  82. 82. • Problem in the Industry • Solution • Use Cases
  83. 83. Use Cases 1. Development and QA 2. Production Support 3. Business
  84. 84. Use Cases 1. Development and QA 2. Production Support 3. Business
  85. 85. Development: Virtual Data • Unlimited • Full size • Self Service Development
  86. 86. Virtual Data: Easy Instance Instance Instance Instance Source DVA
  87. 87. Development Virtual Data: Parallelize gif by Steve Karam
  88. 88. Development Virtual Data: Full size
  89. 89. Development Virtual Data: Self Service
  90. 90. QA : Virtual Data • Fast • Parallel • Rollback • A/B testing
  91. 91. Dev QA QA Virtual Data : Fast Prod Instance DVA • Low Resource • Find bugs Fast Production Time Flow
  92. 92. QA with Virtual Data: Rewind Instance QA Prod Production Time Flow
  93. 93. QA with Virtual Data: A/B Instance Instance Instance Index 1 Index 2 Production Time Flow
  94. 94. Data Version Control Dev QA 2.1 Dev QA 2.2 DVA Production Time Flow 2.1 2.2 Prod Instance 11/11/2014 95
  95. 95. Use Cases 1. Development and QA 2. Production Support 3. Business
  96. 96. • Backups • Recovery • Forensics • Migration • Consolidation Recovery
  97. 97. 9TB database 1TB change day 30 day backups storage requirements 98 70 60 50 40 30 20 10 0 week 1 week 2 week 3 week 4 original Oracle Delphix
  98. 98. Recovery Source Instance Recover VDB Instance Drop DVA Production Time Flow
  99. 99. Forensics Instance Development DVA Source Production Time Flow
  100. 100. Development (the new production) Instance Development DVA Source Development Prod & VDB Time Flow X
  101. 101. Migration
  102. 102. Consolidation
  103. 103. Use Cases 1. Development and QA 2. Production Support 3. Business Intelligence
  104. 104. Business Intelligence • ETL • Temporal • Confidence Testing • Federated Databases • Audits
  105. 105. Business Intelligence: ETL and Refresh Windows 1pm 10pm 8am noon
  106. 106. Business Intelligence: batch taking too long 1pm 10pm 8am noon 2011 2012 2013 2014 2015
  107. 107. 6am 8am 10pm 10pm 8am noon 9pm 1pm 10pm 8am noon 2011 2012 2013 2014 2015
  108. 108. Business Intelligence: ETL and DW Refreshes Prod Instance DW & BI Instance
  109. 109. Virtual Data: Fast Refreshes • Collect only Changes • Refresh in minutes Prod Instance BI and DW ETL 24x7 DVA Production Time Flow
  110. 110. Temporal Data
  111. 111. Confidence testing
  112. 112. Modernization: Federated Source1 Instance Source2 Instance DVA Production Time Flow 1 Production Time Flow 2
  113. 113. Modernization: Federated
  114. 114. Modernization: Federated “I looked like a hero” Tony Young, CIO Informatica
  115. 115. Live Archive Production Time Flow Audit Prod Instance DVA 11/11/2014 116
  116. 116. Use Case Summary 1. Development & QA 2. Production Support 3. Business
  117. 117. How expensive is the Data Constraint? DVA at Fortune 500 : Dev throughput increase by 2x
  118. 118. How expensive is the Data Constraint? Faster • Financial Close • BI refreshes • Surgical recovery • Projects
  119. 119. Virtual Data Quotes • Projects “12 months to 6 months.” – New York Life • Insurance product “about 50 days ... to about 23 days” – Presbyterian Health • “Can't imagine working without it” – State of California
  120. 120. Summary • Problem: Data is the constraint • Solution: Virtualize Data • Results: • Half the time for projects • Higher quality • Increase revenue
  121. 121. Thank you! • Kyle Hailey| Oracle ACE and Technical Evangelist, Delphix – Kyle@delphix.com – kylehailey.com – slideshare.net/khailey
  122. 122. 124

×