3. eBay marketplace = economy of scale 9 23 2 2Bpage views/day 23Mlines of code 9Petabytesof data storage 94 200 $62 94M active users 200Mlive listings $62B2010 gross merchandise volume 6,000 75 6,000 application servers 75B database calls/day Commercial data warehouse 100x larger than Library of Congress
4. Cloud bursting: ↓ costs by ↑ efficiency 2000 1800 1600 1400 1200 1000 800 600 400 200 0 Fixed Cost Datacenter efficiency Cloud Burst Energy-proportional computing Excess Capacity Ideal Remove the peaks Request for Data Current Traffic text text text text text text text text Timeline
5. Datacenter efficiency TIER II 2(N+1) TIER IV N+1 Datacenter tier = level of redundancy Blend of Tier IV & Tier II Minimum 30% reduction in OpEx cost per MW enables cost effective operations/expansion. 50% reduction in CapEx for new lower-tier datacenters (Tier IV vs. Tier II) Burst Capacity (DC and Cloud)
6. Energy-proportional computing Traffic KWh cost / incentives / forecasts Meta Scheduler BMS metrics SLA Models Utilization (headroom) Availability Cloud Bursting SLC PHX (Tier 4) PHX (Tier2) Direct traffic to the most economical location / optimize placement within DC
7. Financial model structure Hourly demand(in average TPS) Hourly demand(in average TPS) Historical (from ODW) or predicted Historical (from ODW) or predicted Define a standard CU Define a standard CU Hourly computing units (CU) required Max TPS per CU Hourly computing units (CU) required Max TPS per CU Based on server inventoryand growth forecast Based on server inventoryand growth forecast Hourly CUs neededin external cloud Number of CU in internal cloud Hourly CUs neededin external cloud Number of CU in internal cloud Negotiated price with public cloud vendor Negotiated price with public cloud vendor Hourly cost of CUin external cloud Hourly total cost Hourly cost of CU in internal cloud Hourly cost of CUin external cloud Hourly total cost Hourly cost of CU in internal cloud HW/SW lease, networking, space, power, laborPredict future costs trends HW/SW lease, networking, space, power, laborPredict future costs trends Sum cost over period Sum cost over period Evaluate tradeoffs by varying size of internal cloud, and costs Evaluate tradeoffs by varying size of internal cloud, and costs
9. TSP ↑ = more attractive options for cloud bursts All self operated CUs TPS and FE Compute Units (CU) 24-Month Projects Load served by external cloud Historical TPS hourly data from ODW CU decreasing by Lease expiration Load served by self operated CU
10. Cost-benefit analysis DC CUs Impact on Project Annual Cost 9/2010 – 8/2011 DC CU Hr Cost = 1.07, 30 TPS per CU 60,000,000 Cloud cost ($/hour) Total Cost ($) 0.5 0.8 1 40,000,000 1.1 1.2 1.5 2 3 4 20,000,000 0 DCs $20M 0 DCs $30M 3K DCs $40M 3.5K DCs $45M 4K DCs $47M 4.8K DCs $52M 5K DCs $55M 0 2000 4000 6000 8000 DC CUs
eBay Inc. has two strong core businesses … that offer significant potential for growth and profitability. And we have a consistent track record of delivering on our commitments.We are a more customer-focused company.We are accelerating innovation across the board.And we have a unique combination of assets that position us well to capitalize aggressively on the inflection point we are seeing in the external environment.eBay/marketplaces around the world: North America: eBay.com and eBay.ca EU: eBay.co.uk eBay.de eBay.fr Etc. Asia: eBay.in eBay.au Korea: GMarketCore adjacency businesses: PayPalStubHub
Top 8 pools represent 1900 machines (Nov 2009). By defining a cutoff at 800 machines , it means that we would need to cloud burst 36800 VM/hours to cover the peak.And using $1/h on Azure(excluding bandwidth) = $36,800 / month1900 – 800 = 1100 machines x $1000 = $1.1M /monthImprove power efficiencyServer Consolidation (Sharing)Reduce DR capacity by implementing pre-emptionClicks 1, 2, 3: Align costs to utilizationData center efficiencyEnergy-proportional computingClick 4: “Fill out the valleys”Reuse spare capacity for experimentation, …Offer Infrastructure as a serviceClick 5: Chop off peaksClick 6: Move the peaks into the valleys (smooth out)Cloudburst (borrow capacity to address peaks)
Tiered approach more efficient, cost-effective, and reliable.Equate to flying airplane with 8 engines when 2 will do. Overcompensates, costs too much, reduces efficiency.Can allocate power/cost of other 6 engines to other applications at tiered levels, based on business need. Avoid spending the capital up front when you build lower-level tiers/Tier IIEvaluate systems based on level of business-critical applications/requirements.Place applications based on their resilience to failure. EXAMPLE: Search – a business-critical function – points immediately at another center if there is an issue, as opposed to a more time-consuming process to migrate a DB from one location to another. The search nodes can be in Tier II, the DB needs to stay in Tier IV (more sensitive to failures). We believe that 70%+ of our applications can reside in Tier II (half the capex, half the opex). Today, all of them live in Tier IV. Assign top tier operations – redundancy, cost, reliability – to mission-critical applications.Assign lower-level operational structure to less-critical applications.
Future state : based on utility cost, datacenter and infrastructure metrics, traffic will be directed to the most economical location while achieving SLA. consider utility companies incentives to shift traffic and power down or idle down infrastructure.
VISUALLY CONSISTENT – UPDATE LABELS – ADD AXESBalance: all in-house; all outsourceInflection points to choose some distributionLines rep various choice; dots rep various modelseBay’s method: Adopt an hybrid cloud approach (part on premise, part off premise) Use the on premise infrastructure for core business critical applications and infrastructure. Convert site to a cloud infrastructure to enable agility and increase average utilization Use the off premise cloud to offload capacity for peak demand, and new/experimental projects. Reduce on premise footprint.Anticipating double-digit transaction per second (TPS) increases YoY.Clarify TPS definition: Combination of internal and external factors, including processing, analytics, as well as site traffic and commerce transactions.As TPS increases, need options for addressing cloud bursting:Advantage: Adjust computing power to applications’ varying demandEnables higher resource utilization due to statistical multiplexing of loadEnables short time to market for new app to capture opportunities when they are identifiedGreen line: Servers operated by eBay Estimated ~ 5228 servers / 13130 CU’s Variety of HW types and virtualization status (Nov 2010)Cost based on 2 yr leaseComplete control; tried and known environmentParameters: Server cost per hourChange in the number of servers over timeBlue line: Internal cloud: Resources dedicated to a single operator who incurs constant cost per server regardless of actual loadTan line: External cloud: use and pay for server only when neededUsually pay by hour of reserved operationLight blue line: Hybrid cloud: A combination of internal and external clouds (part on-premise; part off-premise)Use the on premise infrastructure for core business critical applications and infrastructure.Convert site to a cloud infrastructure to enable agility and increase average utilizationUse the off premise cloud to offload capacity for peak demand, and new/experimental projects. Reduce on premise footprint.Internal cloud “stamps” 2 yr lease. Fixed cost independent of utilizationQuickly deploy and change appsShare load dynamically with self-operated serversParameters: number of stamps and installation/operation datesExternal cloud Paid for by the hour.Quickly turned on to carry traffic when exceeds capacity of DC-based serversParameters: hourly cost of server unit
As infrastructure is commoditized, eBay can decrease cloud costs as much as possible by working with external industry leaders.Allows eBay to focus on strengths of running world’s largest online marketplace: Business intelligence Commerce expertise Customer focus Reliability Security Global presence
