Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

AWS Melbourne Cost Mgt. and Opti. Meetup - 20181109 - v2.2

229 Aufrufe

Veröffentlicht am

Updated slides from AWS Melbourne - Cost Mgt. and Optimisation Meetup. (CloudWatch). Agenda:
2:00pm - Setup
2:10pm - Kick off, welcome, and intro
2:20pm - Jason Gorringe: How to get the most out of your AWS usage via pillars of Allocation, Avoidance, Accountability and Transparency
2:50pm - Discussion and Q&A
3:00pm - Peter Shi: Developing a Cost Management Dashboard that provides high speed to insight
4:30pm - Discussion, Q&A, and networking over drinks and snacks
5:30pm - Event Concludes

Veröffentlicht in: Technologie
  • Loggen Sie sich ein, um Kommentare anzuzeigen.

AWS Melbourne Cost Mgt. and Opti. Meetup - 20181109 - v2.2

  1. 1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Cost Management & Optimisation Interest Group Melbourne Usergroup Meetup Nov 2018
  2. 2. Agenda • 2:00pm - Setup • 2:10pm - Kick off, welcome, and intro • 2:20pm - Jason Gorringe, Australia Post: How to get the most out of your AWS usage • 2:50pm - Discussion and Q&A • 3:00pm - Peter Shi, AWS: Developing a Cost Management Dashboard that provides high speed to insight • 4:30pm - Discussion, Q&A, and networking over drinks and snacks • 5:30pm - Event Concludes
  3. 3. Learning & Accelerating Asking questions Sharing stories Voice to target product asks Contribute & open source Respect NDAs Taking off the sales hat Fully realise the benefits of AWS
  4. 4. Why should we care about Cost Optimisation? Example non-prod workload checklist Can this run on CentOS/Linux? $1000 Turn off outside of work hours? Right size down by 1 size? $118 $236 $787 Can this run on EC2 Spot? $30 Starting non-prod workload
  5. 5. Agenda • 2:00pm - Setup • 2:10pm - Kick off, welcome, and intro • 2:20pm - Jason Gorringe, Australia Post: How to get the most out of your AWS usage • 2:50pm - Discussion and Q&A • 3:00pm - Peter Shi, AWS: Developing a Cost Management Dashboard that provides high speed to insight • 4:30pm - Discussion, Q&A, and networking over drinks and snacks • 5:30pm - Event Concludes
  6. 6. Cloud Cost Optimization HOW TO GET THE MOST OUT OF YOUR AWS USAGE
  7. 7. History • About Post • About Me • My Journey with Cloud Services • Cost Optimisation principles • Where to from here?
  8. 8. Beginning the journey Communicate • Constant two-way communication between business, IT and the billing team is vital Educate • An understanding of the intricacies of AWS billing Empower • Give control to users of the platform to manage their own costs
  9. 9. Cost Allocation ASSIGNING COSTS TO THOSE RESPONSIBLE
  10. 10. • Tagging (keys and values) • Policy/standards • Tagging (mapping) • Transition • Data sources • Did we mention tagging?
  11. 11. Cost Avoidance BETTER USAGE OF YOUR PRODUCTS
  12. 12. • Cost control • Know your products • Usage • Multi vendor
  13. 13. Cost Accountability THE TOOLS AND WHO SHOULD USE THEM
  14. 14. • Spot instances • Rightsizing • Reserved instances • Billing tools
  15. 15. Cost Transparency COMMUNICATION
  16. 16. • Showback and Chargeback • Monitoring and reporting • Analysis and trending • Collaboration and communication
  17. 17. Where to from here? • Automated tagging • Product/team account based strategy • Review of services consumed • Tagging (integration and central management)
  18. 18. Questions
  19. 19. Agenda • 2:00pm - Setup • 2:10pm - Kick off, welcome, and intro • 2:20pm - Jason Gorringe, Australia Post: How to get the most out of your AWS usage • 2:50pm - Discussion and Q&A • 3:00pm - Peter Shi, AWS: Developing a Cost Management Dashboard that provides high speed to insight • 4:30pm - Discussion, Q&A, and networking over drinks and snacks • 5:30pm - Event Concludes
  20. 20. Agenda • 2:00pm - Setup • 2:10pm - Kick off, welcome, and intro • 2:20pm - Jason Gorringe, Australia Post: How to get the most out of your AWS usage • 2:50pm - Discussion and Q&A • 3:00pm - Peter Shi, AWS: Developing a Cost Management Dashboard that provides high speed to insight • 4:30pm - Discussion, Q&A, and networking over drinks and snacks • 5:30pm - Event Concludes
  21. 21. Contents • Why should I build my own dashboard? • AWS Data Sources • Data Pipelines into Athena • Gaining Speed to Insight in Quicksight (incl. visualization tips and KPIs)
  22. 22. Contents • Why should I build my own dashboard? • AWS Data Sources • Data Pipelines into Athena • Gaining Speed to Insight in Quicksight (incl. visualization tips and KPIs)
  23. 23. Speed, scale, complexity, and value at stake drives the need for visibility and speed to insight
  24. 24. Pick the tool that provides the cost visibility and speed to insight that you need Simple, Static, Small environment Complex, Dynamic, Large environment 1. Monthly AWS Invoice 2. AWS Billing console 3. AWS Cost Explorer and AWS Budgets 4. AWS Billing File Analysis, DIY dashboards, and 3rd party tools
  25. 25. 100% Agility Engineering bias 100% Control Finance bias Balanced approach Speed to Insight will help you take a balanced approach
  26. 26. Contents • Why should I build my own dashboard? • AWS Data Sources • Data Pipelines into Athena • Gaining Speed to Insight in Quicksight (incl. visualization tips and KPIs)
  27. 27. AWS Data Sources • AWS Cost and Usage Report (CUR) • Hourly billing data for each service + more info such as RI usage • Can have a very large number of rows and columns • AWS CloudWatch data • Resource Utilization data • DIY budget and revenue data • Flat .csv file of how much you’ve budgeted to spend + revenue generated associated with AWS spend • AWS CloudTrail
  28. 28. Step 1.1: Generate the AWS Cost and Usage Report (CUR) for your account @ Payer acct. level • CUR is the data source of Cost Explorer • Enable the CUR (5 minute exercise) • https://docs.aws.amazon.com/ awsaccountbilling/latest/about v2/billing-reports- gettingstarted- turnonreports.html
  29. 29. Step 1.2: Create some DIY budget and revenue data • Create CSV file with 4 columns: • Account id • Month • Budget (example business constraint) • Revenue (example business metric) Other examples: • minutes on website • number of devs
  30. 30. Step 1.3: Save CloudWatch data across relevant accounts to S3 • Amazon CloudWatch is a monitoring and management service that collects and reports resource metrics • Metrics that indicate effective EC2 use (for many workloads) includes: • CPU % utilisation • Memory % utilisation • Network IO (to internet and to EBS) • For the purposes of today’s exercise we’ll use only CPU %
  31. 31. Step 1.3: Save CloudWatch data across relevant accounts to S3 • Open source multi-account example of Cost Optimization: EC2 Right Sizing solution which collects CloudWatch data across multiple accounts https://github.com/saltysoup/cost-optimization-multi
  32. 32. Contents • Why should I build my own dashboard? • AWS Data Sources • Data Pipelines into Athena • Gaining Speed to Insight in Quicksight (incl. visualization tips and KPIs)
  33. 33. What’s a data pipeline and what’s Athena? • Data pipelines get your data in a format and to a location where you want it to be, typically in an automated way. Also known as ETL (Extract, Transform and Load) • The “Load” portion of this will get our data into Amazon Athena, our serverless interactive query service that can query data directly from S3 (Simple Storage Service). • $5 per TB of data scanned. For the typical $100k p.m. biller analysing billing data via Athena should cost approx. $5 per month. However it’s always smart to set an AWS Budget warning to catch rogue scripts.
  34. 34. Step 2.1: Create an automated way to get CUR billing data into Athena • Option 1: Use an open source tool https://bitbucket.org/atlassian/squeegee/wiki/Home • Option 2: To be updated in next version of slides
  35. 35. Step 2.2: Get your DIY Budget and Revenue data into Athena • Option 1: Manual via Athena SQL script • Option 2: Automated method 1: S3 Event -> call Lambda -> run Athena SQL https://docs.aws.amazon.com/lambda/latest/dg/with-s3- example.html • Option 3: Automated method 2: AWS Glue
  36. 36. Step 2.2: Get your DIY Budget and Revenue data into Athena – Example SQL CREATE EXTERNAL TABLE IF NOT EXISTS dbname.budget_and_rev_data ( `accountid` string, `month_` string, `budget` string, `revenue` string ) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' WITH SERDEPROPERTIES ( 'separatorChar' = ',', 'quoteChar' = '"', 'escapeChar' = '' ) LOCATION 's3://bucketname/budgetandrevenue/' TBLPROPERTIES ('has_encrypted_data'='false', "skip.header.line.count"="1")
  37. 37. Step 2.3: Get your CloudWatch data into Athena • Option 1: Manual via Athena SQL script • Option 2: Automated method 1: S3 Event -> call Lambda -> run Athena SQL https://docs.aws.amazon.com/lambda/latest/dg/with-s3- example.html • Option 3: Automated method 2: AWS Glue
  38. 38. Step 2.3: Get your CloudWatch data into Athena – Example SQL CREATE EXTERNAL TABLE IF NOT EXISTS dbname.cw_data ( `humanReadableTimestamp` string, `timestamp` string, `accountId` string, `az` string, `instanceId` string, `instanceType` string, `instanceTags` string, `ebsBacked` string, `volumeIds` string, `instanceLaunchTime` string, `humanReadableInstanceLaunchTime` string, `CPUUtilization` string, `NetworkIn` string, `NetworkOut` string, `DiskReadOps` string, `DiskWriteOps` string ) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSe rde' WITH SERDEPROPERTIES ( 'separatorChar' = ',', 'quoteChar' = '"', 'escapeChar' = '' ) LOCATION 's3://bucketname/cw/' TBLPROPERTIES ('has_encrypted_data'='false', "skip.header.line.count"="1")
  39. 39. Contents • Why should I build my own dashboard? • AWS Data Sources • Data Pipelines into Athena • Gaining Speed to Insight in Quicksight (incl. visualization tips and KPIs)
  40. 40. What’s QuickSight? • Amazon QuickSight is a fast, cloud-powered BI service that makes it easy to build visualizations, perform ad-hoc analysis, and quickly get business insights from your data. • Accessed from any browser or mobile device. • First BI service to offer pay-per-session pricing so no upfront costs, no annual commitments, and no charges for inactive users!
  41. 41. How much does QuickSight cost?
  42. 42. A part of Speed to Insight is getting data to where you are familiar with consuming it from (e.g. email). QuickSight can send dashboards to you via email. https://aws.amazon.com/blogs/big-data/amazon-quicksight-now-supports- email-reports-and-data-labels/
  43. 43. Step 3.1: Set up a new Athena data source in QuickSight • S3 permissions • Data types • Date format • Decimal format
  44. 44. Step 3.1: Set up a new Athena data source in QuickSight – Example SQL for date formatting SELECT substring(cast(from_iso8601_timestamp(bill_billingperiodstartdate) AS varchar), 1, 19) AS billingperiodstartdate, substring(cast(from_iso8601_timestamp(bill_billingperiodenddate) AS varchar), 1, 19) AS billingperiodenddate, substring(cast(from_iso8601_timestamp(lineitem_usagestartdate) AS varchar), 1, 19) AS usagestartdate, substring(cast(from_iso8601_timestamp(lineitem_usageenddate) AS varchar), 1, 19) AS usageenddate, * FROM dbname.cost_usage_report
  45. 45. Step 3.1: Set up a new Athena data source in QuickSight – Example Date format syntax for QuickSight yyyy-MM-dd HH:mm:ss
  46. 46. Step 3.2: Visualize spend by account and month • This helps us see • Largest spends • Changes in spend • But what do we mean by spend?
  47. 47. Step 3.2: Visualize spend by account and month • What do we mean by spend? Cost View CUR Column Description Blended • lineItem/UnblendedCost Cost based on a common rate across all accounts Unblended • lineItem/UnblendedCost Cost based on a common rate across all accounts Amortized • reservation/AmortizedUpfront FeeForBillingPeriod • reservation/UnusedAmortized UpfrontFeeForBillingPeriod Amortised value of upfront RI spend Public On Demand Cost • pricing/ publicOnDemandCost True reflection of usage. Price that would have been paid if run on-demand and if no-free tier
  48. 48. Step 3.2: Visualize spend by account and month • What would give better insight? • Spend by account by week • Having account names instead of IDs • Having a granular account structure • Tagging for visibility into apps, teams, cost centers
  49. 49. Step 3.3: Now lets visualize: “what is my % spend against budget?” • Create a view in Athena that joins budget and billing data • Visualise the view in QuickSight A “view” in database-world is a saved query that does not store any data but runs that query each time you ask. This allows the query to be designed to always retrieve the latest data. The query can source data from one or more data sources
  50. 50. Step 3.4: Why is account X over budget? Lets see spend by service/product for that account • Create spend by product for all accounts • Add a parameterised filter for linked account • Which service/product significantly increased during April?
  51. 51. Step 3.5: Great to see which service, but which team and app drove this change? • Which team significantly increased spend during April? • Which app significantly increased spend during April?
  52. 52. Step 3.6: Search for optimization opportunity in resource sizing via EC2 instance utilization • Join CloudWatch data with billing data in Athena (showing peak CPU over 14 days by instance and tag) and visualize in QuickSight • Create a join view in Athena • Visualise the view in QuickSight
  53. 53. Step 3.6: Search for optimization opportunity in resource sizing via EC2 instance utilization - SQL SELECT lineitem_resourceid , month , lineitem_usageaccountid , instancetags , max_cpu , sum(CAST(unblendedcost_withoutri AS DOUBLE)) AS unblendedcost_withoutri , sum(CAST(lineitem_unblendedcost AS DOUBLE)) AS lineitem_unblendedcost FROM "dbname"."cost_usage_report" INNER JOIN (SELECT instanceId, instancetags, max(cpuutilization) AS max_cpu FROM "dbname"."cw_data" GROUP BY instanceId, instancetags) cw ON lineitem_resourceid = cw.instanceId GROUP BY lineitem_resourceid , month , lineitem_usageaccountid , instancetags , max_cpu
  54. 54. Step 3.7: Establish our first KPI • What is the cost per revenue change month on month for the account with a business value metric? • Use the budget and revenue view created earlier • Visualise the view in QuickSight via the KPI visual type
  55. 55. Step 3.7: Establish our first KPI – Example SQL SELECT lineitem_resourceid , month , lineitem_usageaccountid , instancetags , max_cpu , sum(CAST(unblendedcost_withoutri AS DOUBLE)) AS unblendedcost_withoutri , sum(CAST(lineitem_unblendedcost AS DOUBLE)) AS lineitem_unblendedcost FROM "dbname"."cost_usage_report" INNER JOIN (SELECT instanceId, instancetags, max(cpuutilization) AS max_cpu FROM "dbname"."cw_data" GROUP BY instanceId, instancetags) cw ON lineitem_resourceid = cw.instanceId GROUP BY lineitem_resourceid , month , lineitem_usageaccountid , instancetags , max_cpu
  56. 56. Contents • Why should I build my own dashboard? • AWS Data Sources • Data Pipelines into Athena • Gaining Speed to Insight in Quicksight (incl. visualization tips and KPIs)
  57. 57. REAGroup has driven cost governance and good cost behaviour through Finance working with Engineering A talk about their story is here: http://bit.ly/FinOpsAtREA
  58. 58. DevOps drives agility, applying FinOps principles enhances control whilst maintaining agility
  59. 59. Next step options (if useful) • Have a chat with your TAM about this • If you need help we have AWS ProServe who can help, Let your account manager know and ask to CC Peter • Try this yourself
  60. 60. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you!
  61. 61. Agenda • 2:00pm - Setup • 2:10pm - Kick off, welcome, and intro • 2:20pm - Jason Gorringe, Australia Post: How to get the most out of your AWS usage • 2:50pm - Discussion and Q&A • 3:00pm - Peter Shi, AWS: Developing a Cost Management Dashboard that provides high speed to insight • 4:30pm - Survey, Discussion, Q&A, and networking over drinks and snacks • 5:30pm - Event Concludes

×