AWS Summit Berlin 2013 - Optimizing your AWS applications and usage to reduce costs
 

Like this? Share it with your network

Share

AWS Summit Berlin 2013 - Optimizing your AWS applications and usage to reduce costs

am

  • 876 Views

Many customers choose AWS because they need a highly reliable, scalable, and low-cost platform on which to run their applications. Low “pay only for what you use” pricing and frequent price ...

Many customers choose AWS because they need a highly reliable, scalable, and low-cost platform on which to run their applications. Low “pay only for what you use” pricing and frequent price decreases are just the beginning of how AWS can help you optimize your usage and achieve lower costs. In this session, you will learn about a few simple tools for monitoring and managing your AWS resource usage that you can start using right away, as well as some innovative features that can help you operate at lower costs programmatically. Cost allocation reporting, detailed usage reports, billing alerts, EC2 Auto Scaling, Spot and Reserved Instances, and idle resource detection are just a few of the tools and features we will cover.

Statistiken

Views

Gesamtviews
876
Views auf SlideShare
876
Views einbetten
0

Actions

Gefällt mir
1
Downloads
29
Kommentare
0

0 Einbettungen 0

No embeds

Zugänglichkeit

Kategorien

Details hochladen

Uploaded via as Adobe PDF

Benutzerrechte

© Alle Rechte vorbehalten

Report content

Als unangemessen gemeldet Als unangemessen melden
Als unangemessen melden

Wählen Sie Ihren Grund, warum Sie diese Präsentation als unangemessen melden.

Löschen
  • Full Name Full Name Comment goes here.
    Sind Sie sicher, dass Sie...
    Ihre Nachricht erscheint hier
    Processing...
Kommentar posten
Kommentar bearbeiten

AWS Summit Berlin 2013 - Optimizing your AWS applications and usage to reduce costs Presentation Transcript

  • 1. Optimizing Your AWS Applications and Usageto Reduce CostsSteffen KrauseTechnical Evangelist@AWS_Aktuellskrause@amazon.de
  • 2. Agenda• Objective– Options to save money on AWS• Fit the cloud to your product and business model– Use Only What You Need• pay only for what you use– Measure and Manage– Scale Opportunistically
  • 3. Use Only What You NeedAnd pay only for what you use!
  • 4. Scale on demandRigid On-Premise ResourcesWasteCustomerDissatisfactionActual demandPredicted DemandCapacityTimeElastic Cloud ResourcesActual demandResources scaled to demandCapacityTimeVS.
  • 5. Use only what you needAWS cost savings opportunities• Right-size– Select appropriate resources– Scale up and down as appropriate– Turn off unused resources• Payment models– Flexibility vs. predictability– Mixing payment models• Measure and manage– Monitor for saving options
  • 6. Right-size your EC2 instances• An instance type forevery purpose• Assess your memory& CPU requirements– Fit yourapplication to theresource– Fit the resource toyour application• Only use a largerinstance whenneeded
  • 7. StandardHigh-CPUHigh-MemoryMicroCluster ComputeCluster GPUHigh I/OHigh StorageHigh Cluster MemoryMost Apps, Low-cost,App Server / WebServerDatabases, DatabasesDatabases…Compute + NetworkThroughputScale-out Compute,Batch ProcessingFor Starters, Lowthroughout, WebsitesParallel ProcessingOLAP, Hadoop,File SystemsNoSQL, Best forRandom IOPSIn-memory Appsand DBs. Best$/RAMEC2 Instance Family use cases
  • 8. Optimize your storage choice too: S3 & Glacier• S3 and Glacier are both:– Secure– Flexible– Low-cost– Scalable: over 1.3 trillion customer objects– Durable: 99.999999999% (11 “9”s)AmazonGlacier
  • 9. Choosing between S3 and Glacier• Amazon Simple Storage Service (S3)– Designed to serve static content• high volumes, low latency, frequent access– From 5.5¢/GB/Month: 11 9’s Durability– From 3.7¢/GB/Month: 4 9’s Durability (reducedredundancy)• Amazon Glacier– Designed for long-term cold storage/archiving• infrequent access, long retrieval times (3-5 hrs)– From 1¢ /GB/Month• But retrieving data is slower and more expensive than on S3
  • 10. S3 and Glacier tips• Optimize access– Reduce payload size– # of accesses (e.g., consolidated logs)• Monitor for unexpected access/growth patterns– Misconfigured log archiving• Set Lifecycle Policies– Object expiration dates– Auto-move S3 files to GlacierIllumina, the leading provider of DNAsequencing instruments, uses Glacier to storelarge blocks of genomic data all over the world
  • 11. Use only what you needAWS cost savings opportunities• Right-size– Select appropriate resources– Scale up and down as appropriate– Turn off unused resources• Payment models– Flexibility vs. predictability– Mixing payment models• Measure and manage– Monitor for saving options
  • 12. Fit your payment model to your business modelEC2 pricing plansOn-DemandInstancesReservedInstancesSpotInstancesPay as you go for computingpowerFlat hourly rate, no up-frontcommitmentsPay an up-front fee for acapacity reservation and alower hourly rate (up to 72%savings)1-year or 3-year termsRI Marketplace: sell RIs you nolonger need; buy RIs at adiscountPay what you want for spareEC2 capacity: your instances runif your bid exceeds the SpotpricePotential for large scale at lowcost: When they’re available,take advantage of 1,000s ofSpot Instances at up to 90%savings10:0010:0510:1010:15
  • 13. Use a spectrum of payment modelsExample:FrontendOn-Demand/Reserved Instances+BackendSpot Instances* e.g., batch video transcoding
  • 14. Measure and Manage“If you cannot measure it,you cannot improve it.”- Lord Kelvin
  • 15. AWS Monitoring and Management Services• Detailed cloud monitoring and management– Consolidated Billing (in “Account Activity”)– CloudWatch (in AWS ManagementConsole)– Billing Alerts (in “Account Activity”)– Trusted Advisor (in “Support Center”)– Other APIs: tags, programmatic access, etc.• + Third-party services
  • 16. Consolidated BillingSingle payer for a group of accounts• One Bill for multiple accounts• Easy Tracking of accountcharges (e.g., download CSV ofcost data)• Group Activities by PayingAccount (e.g., Dev, Stage, Test,Prod)• Volume Discounts can bereached faster with combinedusage• Reserved Instances are sharedacross accounts (including RDSReserved DBs)• AWS Credits are combined tominimize your bill
  • 17. Consolidated Billing Demo (1/3)• Get an overall summary total for all your users and accounts
  • 18. Consolidated Billing Demo (2/3)• From your payment account login, view details of each linked account
  • 19. Consolidated Billing Demo (3/3)• Drill down into detail’s of each account• Download a CSV file for line item details –> Excel
  • 20. Amazon CloudWatch• Monitoring for AWS cloud resources and applications– EC2, RDS, EBS, ELB, SQS, SNS, DynamoDB, EMR,Auto Scaling, …– Custom metrics from your application (use Put API call)• Insight, Alarms, Notifications• Start easily, auto-scale with your application• Sophisticated Automation– Use CloudWatch metrics with Auto Scaling todynamically scale EC2 instances
  • 21. Use CloudWatch to monitor & manage resource usage• Monitor resource utilization– Are you using the right instance type?– Have you left instances idle?– Is your instance usage level or bursty?• Manage resource utilization– Move bursty workloads to otherinstances– Rebalance your worker nodes– Scale nodes automatically with AutoScaling
  • 22. Use CloudWatch to create Billing Alerts• Alert when estimated charges reach threshold• Track an individual developer, or your whole business• Set up your billing alarm and actions
  • 23. Trusted AdvisorEnterprise Strength Monitoring/Optimization• Monitors and recommendsoptimizations for:– Cost– Security– Fault Tolerance– Performance• Available to customerswith Business andEnterprise-level supporthttp://aws.amazon.com/premiumsupport/trustedadvisor/
  • 24. Trusted Advisor: Cost Optimization Tips
  • 25. Trusted Advisor: Performance Tips
  • 26. Third-party services to optimize your AWS usage
  • 27. Scale OpportunisticallyOpportunity favors the preparedapplication
  • 28. Time-to-Result Case 1: Value of result quickly diminishesExample:EngineeringsimulationDelay  Loss ofproductivity,project slips
  • 29. Time-to-Result Case 2: Result is valuable…until it’s notExample:Weekendregression testsDelay  Minimalimpact until8:00AM Monday
  • 30. Consider Spot Instances for greater savings and scale• Spot instances in a nutshell– Run when Your Bid ≥ Spot Price– Spot instances = Spare EC2 instances– Might be interrupted at any time• Benefits– Savings: Up to 90% off On-Demand– Scale: Access up to 1,000s of EC2 instances• To use Spot– Decide on a bid price– Launch via Console, API, Auto Scaling– Monitor Bid Statuses via Console/API
  • 31. What applications work on Spot?• Good Spot applications are:– Delayable: to balance SLA/cost– Scalable: “embarrassingly parallel”– Fault-tolerant: can be terminated without losing all work– Portable across regions, AZs, instance types• Examples:– MapReduce (Hadoop, Amazon EMR)– Scientific Computing (Monte Carlo simulations)– Batch Processing (video transcoding)– Financial Computing (high-frequency trading algorithm backtesting)– and many others…Lucky Oyster crawled 3.4B Web Pages,building a 400M entry index in around14 hours for $100 (>85% savings)!
  • 32. Use Auto Scaling to dynamically scale your app• Auto Scaling auto-sizes your fleet– based on preset triggers and schedules• Integrates with CloudWatch metrics• Use Auto Scaling to– Improve customer experience, application performance– Maximize CPU/IO/Memory utilization– Optimize other metricsScale with Real-Time Demand
  • 33. Auto-Scaling Example: Netflix
  • 34. Follow the Money vs. Follow the Customer• Optimize utilization– Auto Scale on utilization metrics:CPU, memory, requests, connections, …• Optimize price paid– Scale with Spot instances when Spot prices are low– e.g., Run batch processes off-peak (nights, weekends) whenSpot prices are lower
  • 35. Follow the Money vs. Follow the Customer• Optimize customer experience with Auto Scaling• Example 1: Scale resources to meet customer demand– Video service Auto Scales instances to respond to customer web servicerequests• Example 2: Scale resources to ensure fresh results– A scientific paper search engine Auto Scales on queue depth (# of newdocs to crawl)– 10 instances steady state and up to 5,000+ to ensure minimumthroughput time• Example 3: Scale resources preemptively before large demand– A TV show marketing site scales up before the show and back downafter
  • 36. Utility HPC / BigData Orchestration Software• Scale clusters from50–50,000 cores• Error-handling,reliability• Data scheduling:internal  cloud• Automated Security,Encryption Framework• Audit, compliance• Reporting, chargeback Automate spot bidding =>
  • 37. Shared FSScale from 50–50,000+ coresCycleCloud Deploys Secured, Auto-scaled HPC ClustersUserCheck job loadCalculate ideal HPC clusterLegacyInternalHPCLoad-based Spot biddingProperly price the bidsManage Spot Instance lossSpot Instance Execute Nodes(auto-started & auto-stopped)FS /S3HPC ClusterOn-Demand Execute NodesData Scheduling to place dataHPC Orchestration toHandle Spot Instance Bid & Loss
  • 38. Conclusion (Part I):Fit the cloud to your product and business model• Use Only What You Need (and pay only for what you use!)• Measure and Manage• Scale Opportunistically
  • 39. An example putting it all together: Saving on BatchProcessinghttp://aws.amazon.com/architecture/3. ScaleOpportunistically:Auto Scale workernodes based onsize of input queue1. Pay Onlyfor What YouUse: Right-size yourcloudresources2. Monitor andManage your systemwith CloudWatch,Billing Alerts,Trusted Advisor
  • 40. Conclusion (Part II):Use the cloud to create new products & business modelsOn-Premises• Failure isexpensive• Experimentinfrequently• Less InnovationOptimized Cloud• Failure isinexpensive• Experiment earlyand often• More Innovation
  • 41. THANK YOUSteffen Krause@AWS_Aktuellskrause@amazon.de