AWS Melbourne Cost Mgt. and Opti. Meetup - 20181109 - v2.2

2. Agenda • 2:00pm - Setup • 2:10pm - Kick off, welcome, and intro • 2:20pm - Jason Gorringe, Australia Post: How to get the most out of your AWS usage • 2:50pm - Discussion and Q&A • 3:00pm - Peter Shi, AWS: Developing a Cost Management Dashboard that provides high speed to insight • 4:30pm - Discussion, Q&A, and networking over drinks and snacks • 5:30pm - Event Concludes

3. Learning & Accelerating Asking questions Sharing stories Voice to target product asks Contribute & open source Respect NDAs Taking off the sales hat Fully realise the benefits of AWS

4. Why should we care about Cost Optimisation? Example non-prod workload checklist Can this run on CentOS/Linux? $1000 Turn off outside of work hours? Right size down by 1 size? $118 $236 $787 Can this run on EC2 Spot? $30 Starting non-prod workload

6. Cloud Cost Optimization HOW TO GET THE MOST OUT OF YOUR AWS USAGE

7. History • About Post • About Me • My Journey with Cloud Services • Cost Optimisation principles • Where to from here?

8. Beginning the journey Communicate • Constant two-way communication between business, IT and the billing team is vital Educate • An understanding of the intricacies of AWS billing Empower • Give control to users of the platform to manage their own costs

9. Cost Allocation ASSIGNING COSTS TO THOSE RESPONSIBLE

10. • Tagging (keys and values) • Policy/standards • Tagging (mapping) • Transition • Data sources • Did we mention tagging?

11. Cost Avoidance BETTER USAGE OF YOUR PRODUCTS

12. • Cost control • Know your products • Usage • Multi vendor

13. Cost Accountability THE TOOLS AND WHO SHOULD USE THEM

14. • Spot instances • Rightsizing • Reserved instances • Billing tools

15. Cost Transparency COMMUNICATION

16. • Showback and Chargeback • Monitoring and reporting • Analysis and trending • Collaboration and communication

17. Where to from here? • Automated tagging • Product/team account based strategy • Review of services consumed • Tagging (integration and central management)

18. Questions

21. Contents • Why should I build my own dashboard? • AWS Data Sources • Data Pipelines into Athena • Gaining Speed to Insight in Quicksight (incl. visualization tips and KPIs)

23. Speed, scale, complexity, and value at stake drives the need for visibility and speed to insight

24. Pick the tool that provides the cost visibility and speed to insight that you need Simple, Static, Small environment Complex, Dynamic, Large environment 1. Monthly AWS Invoice 2. AWS Billing console 3. AWS Cost Explorer and AWS Budgets 4. AWS Billing File Analysis, DIY dashboards, and 3rd party tools

25. 100% Agility Engineering bias 100% Control Finance bias Balanced approach Speed to Insight will help you take a balanced approach

27. AWS Data Sources • AWS Cost and Usage Report (CUR) • Hourly billing data for each service + more info such as RI usage • Can have a very large number of rows and columns • AWS CloudWatch data • Resource Utilization data • DIY budget and revenue data • Flat .csv file of how much you’ve budgeted to spend + revenue generated associated with AWS spend • AWS CloudTrail

28. Step 1.1: Generate the AWS Cost and Usage Report (CUR) for your account @ Payer acct. level • CUR is the data source of Cost Explorer • Enable the CUR (5 minute exercise) • https://docs.aws.amazon.com/ awsaccountbilling/latest/about v2/billing-reports- gettingstarted- turnonreports.html

29. Step 1.2: Create some DIY budget and revenue data • Create CSV file with 4 columns: • Account id • Month • Budget (example business constraint) • Revenue (example business metric) Other examples: • minutes on website • number of devs

30. Step 1.3: Save CloudWatch data across relevant accounts to S3 • Amazon CloudWatch is a monitoring and management service that collects and reports resource metrics • Metrics that indicate effective EC2 use (for many workloads) includes: • CPU % utilisation • Memory % utilisation • Network IO (to internet and to EBS) • For the purposes of today’s exercise we’ll use only CPU %

31. Step 1.3: Save CloudWatch data across relevant accounts to S3 • Open source multi-account example of Cost Optimization: EC2 Right Sizing solution which collects CloudWatch data across multiple accounts https://github.com/saltysoup/cost-optimization-multi

33. What’s a data pipeline and what’s Athena? • Data pipelines get your data in a format and to a location where you want it to be, typically in an automated way. Also known as ETL (Extract, Transform and Load) • The “Load” portion of this will get our data into Amazon Athena, our serverless interactive query service that can query data directly from S3 (Simple Storage Service). • $5 per TB of data scanned. For the typical $100k p.m. biller analysing billing data via Athena should cost approx. $5 per month. However it’s always smart to set an AWS Budget warning to catch rogue scripts.

34. Step 2.1: Create an automated way to get CUR billing data into Athena • Option 1: Use an open source tool https://bitbucket.org/atlassian/squeegee/wiki/Home • Option 2: To be updated in next version of slides

35. Step 2.2: Get your DIY Budget and Revenue data into Athena • Option 1: Manual via Athena SQL script • Option 2: Automated method 1: S3 Event -> call Lambda -> run Athena SQL https://docs.aws.amazon.com/lambda/latest/dg/with-s3- example.html • Option 3: Automated method 2: AWS Glue

36. Step 2.2: Get your DIY Budget and Revenue data into Athena – Example SQL CREATE EXTERNAL TABLE IF NOT EXISTS dbname.budget_and_rev_data ( `accountid` string, `month_` string, `budget` string, `revenue` string ) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' WITH SERDEPROPERTIES ( 'separatorChar' = ',', 'quoteChar' = '"', 'escapeChar' = '' ) LOCATION 's3://bucketname/budgetandrevenue/' TBLPROPERTIES ('has_encrypted_data'='false', "skip.header.line.count"="1")

37. Step 2.3: Get your CloudWatch data into Athena • Option 1: Manual via Athena SQL script • Option 2: Automated method 1: S3 Event -> call Lambda -> run Athena SQL https://docs.aws.amazon.com/lambda/latest/dg/with-s3- example.html • Option 3: Automated method 2: AWS Glue

38. Step 2.3: Get your CloudWatch data into Athena – Example SQL CREATE EXTERNAL TABLE IF NOT EXISTS dbname.cw_data ( `humanReadableTimestamp` string, `timestamp` string, àccountId` string, àz` string, ìnstanceId` string, ìnstanceType` string, ìnstanceTags` string, èbsBacked` string, `volumeIds` string, ìnstanceLaunchTime` string, `humanReadableInstanceLaunchTime` string, `CPUUtilization` string, `NetworkIn` string, `NetworkOut` string, `DiskReadOps` string, `DiskWriteOps` string ) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSe rde' WITH SERDEPROPERTIES ( 'separatorChar' = ',', 'quoteChar' = '"', 'escapeChar' = '' ) LOCATION 's3://bucketname/cw/' TBLPROPERTIES ('has_encrypted_data'='false', "skip.header.line.count"="1")

40. What’s QuickSight? • Amazon QuickSight is a fast, cloud-powered BI service that makes it easy to build visualizations, perform ad-hoc analysis, and quickly get business insights from your data. • Accessed from any browser or mobile device. • First BI service to offer pay-per-session pricing so no upfront costs, no annual commitments, and no charges for inactive users!

41. How much does QuickSight cost?

42. A part of Speed to Insight is getting data to where you are familiar with consuming it from (e.g. email). QuickSight can send dashboards to you via email. https://aws.amazon.com/blogs/big-data/amazon-quicksight-now-supports- email-reports-and-data-labels/

43. Step 3.1: Set up a new Athena data source in QuickSight • S3 permissions • Data types • Date format • Decimal format

44. Step 3.1: Set up a new Athena data source in QuickSight – Example SQL for date formatting SELECT substring(cast(from_iso8601_timestamp(bill_billingperiodstartdate) AS varchar), 1, 19) AS billingperiodstartdate, substring(cast(from_iso8601_timestamp(bill_billingperiodenddate) AS varchar), 1, 19) AS billingperiodenddate, substring(cast(from_iso8601_timestamp(lineitem_usagestartdate) AS varchar), 1, 19) AS usagestartdate, substring(cast(from_iso8601_timestamp(lineitem_usageenddate) AS varchar), 1, 19) AS usageenddate, * FROM dbname.cost_usage_report

45. Step 3.1: Set up a new Athena data source in QuickSight – Example Date format syntax for QuickSight yyyy-MM-dd HH:mm:ss

46. Step 3.2: Visualize spend by account and month • This helps us see • Largest spends • Changes in spend • But what do we mean by spend?

47. Step 3.2: Visualize spend by account and month • What do we mean by spend? Cost View CUR Column Description Blended • lineItem/UnblendedCost Cost based on a common rate across all accounts Unblended • lineItem/UnblendedCost Cost based on a common rate across all accounts Amortized • reservation/AmortizedUpfront FeeForBillingPeriod • reservation/UnusedAmortized UpfrontFeeForBillingPeriod Amortised value of upfront RI spend Public On Demand Cost • pricing/ publicOnDemandCost True reflection of usage. Price that would have been paid if run on-demand and if no-free tier

48. Step 3.2: Visualize spend by account and month • What would give better insight? • Spend by account by week • Having account names instead of IDs • Having a granular account structure • Tagging for visibility into apps, teams, cost centers

49. Step 3.3: Now lets visualize: “what is my % spend against budget?” • Create a view in Athena that joins budget and billing data • Visualise the view in QuickSight A “view” in database-world is a saved query that does not store any data but runs that query each time you ask. This allows the query to be designed to always retrieve the latest data. The query can source data from one or more data sources

50. Step 3.4: Why is account X over budget? Lets see spend by service/product for that account • Create spend by product for all accounts • Add a parameterised filter for linked account • Which service/product significantly increased during April?

51. Step 3.5: Great to see which service, but which team and app drove this change? • Which team significantly increased spend during April? • Which app significantly increased spend during April?

52. Step 3.6: Search for optimization opportunity in resource sizing via EC2 instance utilization • Join CloudWatch data with billing data in Athena (showing peak CPU over 14 days by instance and tag) and visualize in QuickSight • Create a join view in Athena • Visualise the view in QuickSight

53. Step 3.6: Search for optimization opportunity in resource sizing via EC2 instance utilization - SQL SELECT lineitem_resourceid , month , lineitem_usageaccountid , instancetags , max_cpu , sum(CAST(unblendedcost_withoutri AS DOUBLE)) AS unblendedcost_withoutri , sum(CAST(lineitem_unblendedcost AS DOUBLE)) AS lineitem_unblendedcost FROM "dbname"."cost_usage_report" INNER JOIN (SELECT instanceId, instancetags, max(cpuutilization) AS max_cpu FROM "dbname"."cw_data" GROUP BY instanceId, instancetags) cw ON lineitem_resourceid = cw.instanceId GROUP BY lineitem_resourceid , month , lineitem_usageaccountid , instancetags , max_cpu

54. Step 3.7: Establish our first KPI • What is the cost per revenue change month on month for the account with a business value metric? • Use the budget and revenue view created earlier • Visualise the view in QuickSight via the KPI visual type

55. Step 3.7: Establish our first KPI – Example SQL SELECT lineitem_resourceid , month , lineitem_usageaccountid , instancetags , max_cpu , sum(CAST(unblendedcost_withoutri AS DOUBLE)) AS unblendedcost_withoutri , sum(CAST(lineitem_unblendedcost AS DOUBLE)) AS lineitem_unblendedcost FROM "dbname"."cost_usage_report" INNER JOIN (SELECT instanceId, instancetags, max(cpuutilization) AS max_cpu FROM "dbname"."cw_data" GROUP BY instanceId, instancetags) cw ON lineitem_resourceid = cw.instanceId GROUP BY lineitem_resourceid , month , lineitem_usageaccountid , instancetags , max_cpu

57. REAGroup has driven cost governance and good cost behaviour through Finance working with Engineering A talk about their story is here: http://bit.ly/FinOpsAtREA

58. DevOps drives agility, applying FinOps principles enhances control whilst maintaining agility

59. Next step options (if useful) • Have a chat with your TAM about this • If you need help we have AWS ProServe who can help, Let your account manager know and ask to CC Peter • Try this yourself

61. Agenda • 2:00pm - Setup • 2:10pm - Kick off, welcome, and intro • 2:20pm - Jason Gorringe, Australia Post: How to get the most out of your AWS usage • 2:50pm - Discussion and Q&A • 3:00pm - Peter Shi, AWS: Developing a Cost Management Dashboard that provides high speed to insight • 4:30pm - Survey, Discussion, Q&A, and networking over drinks and snacks • 5:30pm - Event Concludes