Paul Strong's presentation from Cybera Summit 08.

1. eBay –
Examples Of NGDC
Technologies At Work
Paul Strong
Distinguished Research Scientist,
eBay Research Labs
®
Cybera Summit 2008

2. Copyright Notice
© 2008 eBay Inc. All rights reserved.
• No part of these materials may be reproduced or
transmitted in any form, by any means (electronic,
photocopying, recording, or otherwise) without the
prior permission of eBay Inc.
• eBay and the eBay logo are registered
trademarks of eBay Inc.
• PayPal and the PayPal logo are registered
trademarks of PayPal, Inc.
• Other trademarks and brands are the property of
their respective owners.
• Please do not take our picture or record the
class/session without asking permission.
®
©2008, eBay Inc.

3. What Is A Next Generation Data Center?
• Is it the platform for SOA?
• Is it a Grid?
• Does it leverage Virtualization technologies?
• Is it full of blades?
• Is it Greener?
• Is it more efficient?
• Is it more automated?
®
©2008, eBay Inc.

4. What Is A Next Generation Data Center?
• Is it the platform for SOA?
• Is it a Grid?
• Does it leverage Virtualization technologies?
• Is it full of blades?
• Is it Greener?
• Is it more efficient?
• Is it more automated?
®
©2008, eBay Inc.

5. NGDC Is…
More Than A Collection Of Technologies
• Runs Business Processes driven by SLAs
• Is a Value Center, rather than a Cost Center
– A new dialog between IT and the Business
• Enables
– Internal utilities for core business functions/processes
– External utilities for business process elements
– Businesses based on business process mash-ups
– Opportunities for new platforms
®
©2008, eBay Inc.

6. Cloud Computing
®
©2008, eBay Inc.

7. Cloud Computing Is…
• A new way about thinking of business…
– Business = Codified core differentiating value
• Outsourcing everything that is not core
– Physical infrastructure →
Clouds, Platform as a Service (PaaS)
– Logical infrastructure (non-core business software) →
Software as a Service (SaaS)
• About being flexible
– Temporary outsourcing…
• Not a new set of technologies
– A new context for applying a set of evolving technologies that
are now more mature or capable
®
©2008, eBay Inc.

8. Cloud Computing Is About…
®
©2008, eBay Inc.

9. NGDC Is…
More Than A Collection Of Technologies
• Runs Business Processes driven by SLAs
• Is a Value Center, rather than a Cost Center
– A new dialog between IT and the Business
• Enables
– Internal utilities clouds for core business functions/processes
– External utilities clouds (PaaS, SaaS) for infrastructure and business
process elements
– Businesses based on business process mash-ups
– Opportunities for new platforms
®
©2008, eBay Inc.

10. NGDC, Friends & Relations
U
®
©2008, eBay Inc.

11. Managing NGDCs…
Is not about managing servers,
operating systems, disks etc.
®
©2008, eBay Inc.

12. ®
©2008, eBay Inc.

13. Why eBay Is A Useful Example
New Challenges
Extreme Engineering
The Bleeding Edge Everyday use
Technology trickle down/transfer
®
©2008, eBay Inc.

14. eBay – The 30 Second Introduction!
eBay users trade about $1,989 worth of goods on the site every second
On an average day on eBay…
• A car sells every 56 seconds
• A pair of shoes sell every 7 seconds
• A cell phones sells every 7 seconds
1.3m people make all or part of their living selling on
* ®
©2008, eBay Inc. *ACNielsen International Research, June 2006

15. eBay’s Drivers
• Extreme Scale
– 241m Registers Users, 112m+ Items, 7m+ New Items Per Day
• Extreme Growth
– Near exponential growth in listings for most of history – 13
years
• Extreme Agility
– Roll code to the site every 2 weeks
• Constant, predictable presence
– Must be 24x7x365
• Efficiency
Failure To Keep Up Is Not An Option!
®
©2008, eBay Inc.

16. eBay – The Simple Diagram
®
©2008, eBay Inc.

17. eBay Example #1
Making The Database Scale
• Second Database for failover
• CGI pools, Listings, Pages, and Search continued to scale horizontally
However …
By November 1999, the database servers approached their limits of physical growth.
S/W Load Balancer S/W Load Balancer S/W Load Balancer S/W Load Balancer
Web Server Web Server Web Server Apache
C++ COTS Search
OS OS OS UNIX
“CGIn” “Listings” “Pages” “Search”
RDBMS RDBMS
UNIX UNIX
1999 bull.ebay.com bear.ebay.com
®
©2008, eBay Inc.

18. eBay Example #1
Making The Database Scale
• Database quot;splitquot; technology.
• Logically partition database into separate instances.
• Horizontal scalability through 2000, but not beyond.
S/W Load Balancer S/W Load Balancer S/W Load Balancer S/W Load Balancer
Web Server Web Server Web Server Apache
C++ COTS Search
OS OS OS UNIX
“CGIn” “Listings” “Pages” “Search”
RDBMS RDBMS RDBMS RDBMS
UNIX UNIX UNIX UNIX
2000 bull.ebay.com bear.ebay.com chard.ebay.com cab/bongo.ebay.com
®
©2008, eBay Inc.

19. eBay Example #1
Virtualizing the Database
Application Servers
CATY
Attributes Catalogs Rules User Account Feedback Misc API Scratch
1…N
DB 1 DB 2 DB 3
• Separate Application notion of a database from physical implementation
• Databases may be combined and separated with no code changes
• Reduce cost of creating multiple environments (Dev, QA, …)
• Application can continue to function without non-critical data (markdown) ®
©2008, eBay Inc.

20. eBay Example #1
Virtualizing & Scaling the Database
November, 1999
November, 1999
®
©2008, eBay Inc.

21. eBay Example #1
Virtualizing & Scaling the Database
December, 2002
SAN
®
©2008, eBay Inc.

22. eBay Example #1
Virtualizing & Scaling the Database
• Scales Out
– 241 million registered users
– 112 million plus Items
– 7 million plus new items per day
– 50 billion SQL transactions per day
– 600+ production database instances (inc replicas)
– 100+ clusters
• Cheaper
– Smaller, potentially commodity, servers
• Highly Resilient
– 2-4 copies of everything
– Minimized impact of outage to [relatively] small sub-set of data
• Flexible/Agile
– Easy to change – database, schemas, partitioning etc.
– Minimal impact on architecture or code
®
©2008, eBay Inc.

23. eBay Example #2
Scaling The Application
®
©2008, eBay Inc.

24. eBay Example #2
Scaling The Application
• Partition code into functional areas
– Application is specific to a single area (Buying, Selling etc.)
– Domain contains common business logic across applications
• Restrict inter-dependencies
– Applications depend on Domains, not on other applications
– No dependencies among shared domains
User Application
User Application Selling Application
Selling Application Buying Application
Buying Application Billing Application
Billing Application Search Application
Search Application
Applications
User Domain
User Domain Selling Domain
Selling Domain Buying Domain
Buying Domain Billing Domain
Billing Domain Search Domain
Search Domain
Personalization Domain
Personalization Domain User Validation Domain
User Validation Domain Shared Billing Domain
Shared Billing Domain
Shared
Shared Buying Domain
Shared Buying Domain myEBay Domain
myEBay Domain Shared Search Domain
Shared Search Domain
Domains
Core Domain
Core Domain API Domain
API Domain
Lookup Domain
Lookup Domain ®
©2008, eBay Inc.

25. eBay Example #2
Scaling The Application
• Segment functions into separate application pools
– Minimizes/isolates DB dependencies
– Allows for parallel development, deployment and monitoring
ViewItem Pool SYI Pool
http://cgiX.ebay.com... http://cgiY.ebay.com...
Load
Load Load
Load
Balancer
Balancer Balancer
Balancer
Web
Web Web
Web Web
Web Web
Web Web Servers
Load
Load Load
Load
Balancer
Balancer Balancer
Balancer
AS
AS AS
AS AS
AS AS
AS AS
AS AS
AS App Servers
Load
Load
Balancer
Balancer
User Acct Caty1 Caty20+
®
©2008, eBay Inc.

26. eBay Example #2
Scaling The Application
• Everything behaves as loosely coupled services
• Minimize inter-dependencies
• Infrastructure is like a giant FPGA
– Potential to re-program by re-routing traffic
• Scales Up And Out (Choice)
– Scale out means scaled throughput and resilience
– 16000+ concurrent instances
– 8000+ servers (mainly blades)
• Efficiency
– Run traffic from different time zones on the same server but
different instances
®
©2008, eBay Inc.

27. Consequences
• Scale Out
– Pro – Scale, throughput, resilience, use commodity products
– Con – More to manage – complexity, relationships
• Virtualization
– Pro – Flexibility
– Con – more relationships to manage
• Commodity
– Interchangeable, choice, no lock in, lower unit cost
®
©2008, eBay Inc.

28. The Big Problem
Management complexity scales with this
# Relationships
# Relationships
# Components ®
©2008, eBay Inc.

29. Understanding Relationships
Service A is composed of
Persistence Sub-Service B
Business Logic Sub-Service C
A
Presentation Sub-Service D
B C D
®
©2008, eBay Inc.

30. Understanding Relationships
Business Logic Sub-Service C is composed of
A Load Balancing Service
Several Application Instances
A
B C D
App App LBS
®
©2008, eBay Inc.

31. Understanding Relationships
The Application Instances are hosted on
Operating System Instances
The Load Balancing Service is hosted on
A
A Load Balancer Operating System
B C D
App App LBS
OS OS LB
®
©2008, eBay Inc.

32. Understanding Relationships
The Operating System Instances are hosted on
Servers or Virtual Servers, which are in turn hosted on servers
The Load Balancer OS is hosted on
A
A Physical Load Balancer
B C D
App App LBS
OS OS LB
VS
Svr Svr LB
®
©2008, eBay Inc.

33. Interaction/Traffic Relationships
Starting To Look Complicated!
®
©2008, eBay Inc.

34. Relationships Are Everything!
• Everything is interconnected
• Changing one thing causes ripples
• How you connect things together determines
business functionality and business value
• Agility is the ability to change these relationships
dynamically (easier with loosely coupled services)
®
©2008, eBay Inc.

35. Relationships Are Everything!
• Virtualization is about standardizing relationships
and interposing/isolating one end from the other
• Understanding these relationships allows you to
– Tie business processes to the infrastructure they run on
– Map value to cost
– Understand and manage traffic flow
– Understand and manage provisioning etc.
• It’s all about managing relationships, not things!
®
©2008, eBay Inc.

36. Data Center Evolution
• Non-Deterministic
• Out of band changes
• Hosting other people’s code
• Emergent Behavior
• People and other software using things in unintended ways
• SO-Aaaaaaaghhhhh
• More relationships
®
©2008, eBay Inc.

37. Conclusions
• NGDC is not just about technology that enables
greater scaling, flexibility, resilience etc.
• NGDC has to be about changing the nature of the
data center and its relationship to the business
• The challenge is how to understand and manage
relationships, not just things!
– Relationships inside the datacenter
– Relationships to external services (SaaS et al)
– Relationships managed on external infrastructure (Clouds,
PaaS)
– Dynamically created and destroyed relationships as services
come and go!
®
©2008, eBay Inc.

38. Thank You
Paul Strong
pstrong@ebay.com
Distinguished Research Scientist
eBay Research Labs,
®
eBay Inc.