Optimizing IT Costs & Services With Big Data (Little Effort!) - Case Studies - #Pink13
•Als PPTX, PDF herunterladen•
0 gefällt mir•594 views
IT organizations have a wealth of Service Management and Service Delivery tools, processes and metrics that typically exist in relative isolation. This session will present detailed real-life examples of how existing tools and metrics can be brought together using big data techniques to optimize costs and performance of IT environments.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (6)
Ähnlich wie Optimizing IT Costs & Services With Big Data (Little Effort!) - Case Studies - #Pink13
Ähnlich wie Optimizing IT Costs & Services With Big Data (Little Effort!) - Case Studies - #Pink13 (20)
Mehr von TeamQuest Corporation
Mehr von TeamQuest Corporation (10)
Optimizing IT Costs & Services With Big Data (Little Effort!) - Case Studies - #Pink13
- 1. Optimizing IT Costs & Services with Big Data, Little Effort… David Wagner TeamQuest Advocate TeamQuest and the TeamQuest logo are registered trademarks in the US, EU and elsewhere. All other trademarks and service marks are the property of their respective owners. Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
- 2. Agenda • Why? • Big Data: conceptual overview • 2013 Capacity Management 101: – History – Goals – Obstacles • New “Big Data” approaches – Concepts – Case Study Value Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
- 3. Why does TeamQuest Exist? • We passionately believe always having and using the right amount of resources is a societal imperative – Anything less is failure – Anything more is wasteful • 20+ years sole focus – ensuring our customers can continuously and automatically perform at their utmost level of efficiency – ensuring business service performance, conserving scarce resources, saving money and improving productivity • We call this: IT Service Optimization Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
- 4. What this Presentation is… and is not! • IS: – Applying Big data approaches to Capacity Management • Faster and larger value • More scalable – New ways to think about optimization beyond ITIL Capacity Management • Include ITIL Service Management and Delivery • Not just technology anymore • Is NOT! – A Primer on Big Data or a Big Data “how to” Presentation • Hadoop ecosystem deep dive, etc… Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
- 5. Big Data at 50,000 feet… • Big Data is about: data actionable information – Plethora of existing sources • Technology • Business (Sales, Marketing, …) • Service (Transactions, SLAs, …) – Learning new insights from “old” data – Key is Analytics • Deep • Wide • Adaptable • But… Optimizing costs with Capacity Management? Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
- 6. Technology Approaches • Data Access and Aggregation – Build huge “data marts” (aka: Data Warehousing) – Integrate with multiple different data sources • Technology (e.g. Server, Network, Storage, etc.) • Service (Catalog, Metrics, Tickets, etc.) • Business (KPIs, Plans, Transactions, etc.) • Implement Analytics against/across – Flexible and adaptive – Turn data within, into actionable information across • But… Capacity Management??? Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
- 7. 2013 Capacity Management 101 - History • Answering “what if” questions… – Change in technology, demand, etc… impact? – Focus on Optimizing Server Cost versus Performance • Extremely Technology-centric – Servers, Mainframes – Occasionally Storage or Network – in isolation • Big Value and Return, but also effort – Highly trained staff – Required building a central, massive datamart (CMIS) – Scalability of Staff, Tools, …, Politics Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
- 8. 2013 Capacity Management – Goals: What • Maintain traditional value, and add – Optimize – Amplify – Accelerate • Increase Business relevance – Valuable predictive analytics in business and service context – Optimize Efficiency • Virtualization and Cloud Scale to everything – Many to many inter-relationships; Capacity critical Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
- 9. 2013 Capacity Management 101 – Goals: How • Integrate and Analyze across multiple sources – Technology (e.g. Server, Network, Storage, etc.) – Service (Catalog, Metrics, Tickets, etc.) – Business (KPIs, Plans, Transactions, etc.) • Single pane of “Analytic Glass” – Ability to tie together, correlate, and operate across • Tear down the wall! – Don’t force reinvention… or data duplication! – Flexible and adaptive – Turn data within, into actionable information across Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
- 10. 2013 Capacity Management 101 - Obstacles • Data Access and Aggregation – Building huge “data marts” (fka: Data Warehousing) • Complexity = (data ETL) x (# sources) x (maintenance effort) • Compliance: Data duplication, privacy, audit, etc… • Costly and time consuming • Implementing Analytics against/across – General purpose BI Analytics for Capacity? – Traditional Performance/Capacity for General Purpose? • “Big Data” + ITIL = Optimized Capacity Management? Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
- 11. Capacity Management with ITIL 2011 • Service Strategy – Financial management • Service Design – Service Level and Availability management • Service Transition – Asset, Change and Configuration Management • Service Operations – Service Desk – Application and IT operations – Event, Incident, Problem • Or, in simpler terms… – Integrate Capacity across ITIL V2: Service Support and Service Delivery! Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
- 12. Optimized Capacity Management • Leverage the data (and tools) you have! – Don’t reinvent or reimplement • Quickly and easily with True Federation – Use existing data/tools already in place – Don’t force data duplication, ETL – Capacity Analysis across data sources • Key ITIL discipline metrics amplify Capacity Management Value – Strategy factor financials – Design factor Service Levels, technology performance – Transition track business and technology changes – Operations factor Service risks, multiple technologies Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
- 13. Integrated Case Study Walkthrough • ITIL: Strategy – Capacity Management integrated with Financial costing/reporting • ITIL: Design – Capacity Management integrated with Risk Registry • ITIL: Transition – Includes integration with Asset and Configuration Management • ITIL: Operations – Integration with Service Desk – Operations factor Service risks, multiple technologies Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
- 14. Very Large Bank As an IT Shop: • Operate tens of thousands of servers • Every server platform under the sun • Manage dozens of data centers • Huge mainframe with many thousands of MIPS • Thousands of VMs • Thousands of VDIs & Citrix • Many Petabytes of storage Copyright © 2012 TeamQuest Corporation. All Rights Reserved. 14
- 15. Seamless data integration & analysis 1. All capacity/performance data 2. All platforms, OS’s, … 3. Configuration data 4. Change records 5. Risk registry Copyright © 2012 TeamQuest Corporation. All Rights Reserved. 15
- 16. Deliverable: Fully Automated Application Report We need: 1. Risk detection and tracking 2. Risk reporting 3. Actionable information Reporting has to be: • Automated • Repeatable • Human-readable – financials, business terms, not “speeds and feeds” Copyright © 2012 TeamQuest Corporation. All Rights Reserved. 16
- 17. Analysis Overview Application and Configuration from Service Catalog and CMDB Copyright © 2012 TeamQuest Corporation. All Rights Reserved. 17
- 18. Usage Patterns Time Series data from Performance and Event Management Copyright © 2012 TeamQuest Corporation. All Rights Reserved. 18
- 19. Service Desk and Risk management Copyright © 2012 TeamQuest Corporation. All Rights Reserved. 19
- 20. Existing Capacity Issues Scaleably ID Possible Copyright © 2012 TeamQuest Corporation. All Rights Reserved. 20
- 21. ID Possible Future Capacity Risks Copyright © 2012 TeamQuest Corporation. All Rights Reserved. 21
- 22. Fixed Costs / Variable Costs - Method Variable Costs Source: wikipedia.org Fixed Costs Copyright © 2012 TeamQuest Corporation. All Rights Reserved. 22
- 23. Capacity Management + Strategy (Financials) Fixed/Variable Cost server0009b01a - Excess Capacity Report Produced by the Server Capacity & Performance Management (SCPM) Team Analysis Period: August 01 2010 to August 31 2010 Run Time: 4:09 PM September 27 2010 (8 seconds) Purpose: To analyze the system's current resource consumption and compute the available headroom based on a fixed/variable costs methodology and our rules-of-thumb. This report also attempts to determine the nearest bottlenecks, from a consumption perspective. server0009b01a: Maximum Growth Capability by Resource server0009b01a: Top 10 PIDs Name Growth Vaule NAME PIDGROWTH SLOPE MINCPU AVGCPU MAXCPU CPU RunQ Length Growth 2.15 System:4 17.54 0.00 0.06 0.09 5.18 Disk - 0 4.41 NTRtScan:1660 29.88 -0.00 0.00 0.02 3.01 Memory Utilization Growth 5.48 beasvc:1080 47.72 -0.00 0.00 0.14 1.89 FS - C: 10.61 svchost:840 184.41 -0.00 0.03 0.07 0.52 Virtual Memory Growth 20.27 svchost:872 213.22 0.00 0.05 0.08 0.47 CPU Growth 38.10 TmListen:2160 763.86 -0.00 0.00 0.01 0.12 Net In 100MB - NIC1 260.82 python:1788 848.86 0.00 0.00 0.03 0.11 Net Out 100MB - NIC1 349.39 wmiprvse:268 987.95 -0.00 0.00 0.04 0.09 Net In 1GB - NIC1 2608.18 wmiprvse:2228 1322.98 0.00 0.02 0.04 0.09 Net Out 1GB - NIC1 3493.92 wmiprvse:2044 1328.88 -0.00 0.02 0.05 0.09 23 Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
- 24. Capacity Management + Strategy (Financials) Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
- 25. Capacity Management + Strategy (Financials) Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
- 26. VM Optimization Analysis • Thousands of VMs • Some too small • Some too big • Some idle • Which ones? • What size should they be? Copyright © 2012 TeamQuest Corporation. All Rights Reserved. 26
- 27. Physical to Virtual Analysis Copyright © 2012 TeamQuest Corporation. All Rights Reserved. 27
- 28. Capacity Optimization Candidates Total Virtual Machines Idle Virtual Machines Oversized Virtual Machines 22 3 13 Idle Virtual Machines Recommende Avg % Max Max % Average Avg % Max % CPU Avg % Total Recommende d SSO vCPUs Max % CPU CPU Memory Memory Memory Memory Ready CPU Memory d SSO vCPUs Memory in Ready Used Util Used Util GB CLUSTER0019V019 4 0 0 0 0 4096 0 0 0 0 1 2 CLUSTER0019V024 2 0 0 0 0 2000 0 0 0 0 1 2 CLUSTER0019V029-OLD_DO_NOT_USE 4 0 0 0 0 4096 0 0 0 0 1 2 Oversized Virtual Machines Avg % Max Max % Average Avg % Recommend Recommended Max % CPU Avg % Total vCPUs Max % CPU CPU Memory Memory Memory Memory ed SSO SSO Memory in Ready CPU Memory Ready Used Util Used Util vCPUs GB CLUSTER0019V001 2 40 5 9 1 2044 1921 94 1729 85 1 4 CLUSTER0019V003 2 30 10 7 2 2048 1895 93 1737 85 1 4 CLUSTER0019V004 2 45 51 3 3 2048 1914 93 1333 65 2 4 CLUSTER0019V005 2 41 17 8 2 2048 1955 95 1716 84 2 4 CLUSTER0019V006 2 45 41 2 2 2048 1963 96 1510 74 2 4 CLUSTER0019V008 2 27 23 2 2 2048 1860 91 1232 60 1 4 CLUSTER0019V013 2 30 40 3 3 2048 1845 90 1326 65 1 4 CLUSTER0019V014 2 30 36 3 2 2048 1834 90 1286 63 1 4 CLUSTER0019V018 2 32 30 3 2 2000 1843 92 1581 79 1 4 CLUSTER0019V029-REAL 4 42 30 5 4 4096 3612 88 2951 72 4 8 CLUSTER0019V030 2 47 23 2 2 2048 1881 92 1387 68 2 4 CLUSTER0019v009 2 43 14 2 1 4096 3117 76 2258 55 2 4 CLUSTER0019v010 2 30 18 2 1 4096 3102 76 2151 53 1 4 Copyright © 2012 TeamQuest Corporation. All Rights Reserved. 28
- 29. Delivered: • Repeatable processes • Quicker analysis • Powerful • Flexible Copyright © 2012 TeamQuest Corporation. All Rights Reserved. 29
Hinweis der Redaktion
- Our IT shop is huge. We have tens of thousands of servers, all platforms, all types, all kinds. We manage dozens of data centers and have one of the largest mainframe installations in North America. We manage thousands upon thousands of VMs, VDIs, and Citrix. Of course we have more storage than seems humanely possible – PetaBytes of storage.
- We are huge and complex. We have huge amounts of data and don’t want to duplicate it. We need all of this data integrated into a single analytical tool. Performance Surveyor provides us with seamless Capacity Management across all of our data collection toolsets and platforms. We have many tools for configuration information, change records, trouble ticketing, etc. We also need to track our capacity risks from discovery through remediation.Performance Surveyor helps us pull that information together into a single, analytical report.Some highlights:We used Performance Surveyor to make the transition between two different collection tools without changing a single report.As you could guess, we have terabytes and terabytes of performance/capacity data. We cannot afford to duplicate this data, the SAN storage and management costs would be astronomical. Performance Surveyor easily integrates this data into our reports without data duplication.We have integrated many tools and data sets into Performance Surveyor. Things like BMC Performance AssuranceHP OpenViewVMWare Virtual CenterEMC Control CenterRemedyand our own capacity risk registryPerformance Surveyor understands all of these things. It is smart enough to adjust the reporting to properly reflect the differences between platforms, data types, and metrics. For example, the same report that analyzes a physical server’s CPU activity can also analyze a VMWare virtual server’s CPU activity. Performance Surveyor knows the difference, and the reports can make use of it.How about an example?
- We all need enterprise-wide reporting. We want it automated and application specific. When we built ours, we asked ourselves three questions:How can we detect and track capacity/performance risks for an application?How can we report these risks to the application support teams?How can we reduce the report to only include actionable information?Our Performance Surveyor monthly application report automatically identifies capacity risks for the application. The automation makes it highly repeatable while reducing errors. It even makes our junior staff effective by removing guess work and their reliance on senior staff. The report contains only the actionable information, not pages and pages of unnecessary details.Let me show you an example.
- This is our Monthly Health Check Report for the Brokers Data Warehouse application. Notice that the Introduction section of the report includes the What, Why, and How. This helps our customers fully understand what we are doing and why.The most important part of the report is the “Ratings For Month” section. This section is for the analyst to summarize the health of this application. For this report, the analyst is concerned about the predicted lack of memory due to a single-node failure and have rated all servers as YELLOW in the dashboard. Please note that this is the only manual entry in the report. Just these two paragraphs. No other editing is required.
- Our Monthly Health Check also includes Usage Patterns for each tier of the application. A business hours only chart is on the left, and the critical batch window is on the right. Notice that this application’s usage pattern is highly repeatable except for the week of October 24th on the left. This chart helped us understand the effect of a business event on this application and was used as the “before picture” for tuning efforts. These types of period-over-period charts have become very valuable in our daily analysis tasks. And the good news is that they are extremely easy to create in Performance Surveyor.
- Our Monthly Health Check report easily contains Asset information alongside our risk registry items. This particular server’s history contains three previous capacity issues for CPU, Memory, and file system space. Our homegrown risk registry is used to track these items from identification through remediation. We track the date opened, why it was opened, our notes, and the closure reason. Notice that two of the issues were closed based on feedback from the Application Owner. While the memory issue was resolved by tuning Oracle’s SGA. This history is invaluable for our analysis as well as providing historical context for the application owner. We have too many applications and servers to track this by hand. We had to have a tracking tool, and it had to be integrated into our reporting tool. This was easily done with Performance Surveyor.
- Our Monthly Health Check also analyzes the resource consumption of the application to detect possible capacity issues. In this report, Performance Surveyor detected a possible CPU problem. Notice that only the relevant charts / tables are produced! No historical charts/tables for CPU and RunQ are displayed because there are no historical issues with CPU and RunQ. The report only included the chart for the broken rule (System Wait time), a CPU Utilization chart for context, and detailed charts to help understand what processes may have caused the problem.Why waste paper on irrelevant information? Why distract the reader? Thankfully, Performance Surveyor automatically removes the extraneous information so I don’t have to pay someone to do it.I only want actionable information in my reports.
- Our Monthly Health Check does attempt to detect future capacity risks based on historical data. In this report, Performance Surveyor detected a possible disk space problem during the next 30 days. It also detected possible disk space and CPU problems over the next 180 days. Notice that a summary table is created that includes the projected problem date and metric value. Again, only the actionable information is included only the relevant charts for each risk are included. When there are no detected problems, we make sure to say so. Notice the automatically generated green “No Projected Processor Problems” message on the left.
- In Business terms:Fixed Costs are business expenses that are not dependent on the level of goods or services produced by the business. These are overhead costs like salaries and real estate.Variable Costs are expenses that change in proportion to the activity of a business. These costs are based on volume, such as hamburger buns, shipping costs, or the number of cashiers.In technology terms:Fixed costs are defined as the resource utilization required to keep the system running regardless of changes to the user/application load. System monitoring is an example of fixed costs.Variable costs are defined as the resource utilization consumed as application load changes. An example would be "each bond trading transaction requires 3 seconds of CPU time; 100 bond trading transactions needs 300 seconds".For our purposes:fixed costs are analogous to the minimum utilization during business hours.Variable costs are analogous to the difference between the peak and minimum utilization during business hours.Notice that for our OnLine Transaction Processing system, the Transactions Per Second provides a curve very similar to the CPU utilization curve. This is usually the case during business hours.We also know that batch processing is very different than OLTP. To accommodate this, we apply the method twice: once during the OLTP window and once during the BATCH window.
- We have a very large VMware installation. We have many thousands of Virtual Machines. We know that many of them are over provisioned and some are too small. How do we sift through all of that data and find them? How do we determine what the configuration should be?
- We have developed a P2V worksheet. We took our rules and methods, and plugged them in. Now we can quickly and easily:test a server for virtual candidacyunderstand its resource consumptionAnd know what size VM is requiredAll of this is done auto-magically. Just pick a server and a timeframe, and then press the Go button.
- We used Performance Surveyor to analyze our VMWare Clusters. The analysis determines which servers are idle, too big, and too small. It also determines what the VM’s size should be.In this report, we identified three idle virtual machines. We would target these VMs for shutdown.We also found thirteen VMs that were too big. The report recommends that the first VM be reduced to one virtual CPU and increased to four gigabytes of RAM.All of this is done automatically based on a set of rules embedded in the report. These rules are easily modified and could be changed based on needs and environment. For example, the rules may be very different for Virtual Desktops versus Virtual Servers. We have found that we like our rules better than being trapped into a vendor’s proprietary rules.In 2011, because of this report, we optimized a tenth of our VMs to reclaim over two thousand vCPUs. We have already identified thousands of more vCPUs to reclaim in 2012, and we will add memory and storage reclamation.Before we had this capability, we were constantly building VMware clusters. Because of this report and our reclamation activities, we have not built a cluster in quite a while and don’t have plans to build one soon. As you know, over provisioning in the Vmware space can affect performance. Due to reclamation, our applications’ performance increased and our trouble tickets decreased.This was a huge win on so many fronts!
- With Performance Surveyor’s assistance, my team has successfully delivered on our mission to reduce, optimize, consolidate, and virtualize. Performance Surveyor allowed us to build repeatable, powerful processes. It helped us do our analysis faster and easier. It provides a flexible capability that accepts our analysis methods, seamlessly integrates our data sources, and delivers human-readable reports.We are pleased with what we have accomplished, and have plans for much, much more.