The document discusses how LinkedIn, a large professional social network, is built using Java technologies and agile practices. It focuses on LinkedIn's architecture, which has evolved from a monolithic design to a service-oriented architecture with over 50 services. It also describes how agile methods like short release cycles and engineering cards are used. Testing is a strong focus, with over 6,500 unit and integration tests as well as automated continuous integration.

1. LinkedIn: A Professional Social Network Built
with Java™ Technologies and Agile Practices
Nick Dellamaggiore, Principal Software Engineer
Eishay Smith, Senior Software Engineer

2. Learn how Java™ and agile practices are employed in
building large-scale consumer internet sites like
LinkedIn, the world’s largest professional network.
2008 JavaOneSM Conference | java.com.sun/javaone | 2

3. Agenda
LinkedIn(troduction)
Agile engineering process
Architecture
Building LinkedIn with Java™
Questions/Discussion
2008 JavaOneSM Conference | java.com.sun/javaone | 3

4. 2008 JavaOneSM Conference | java.com.sun/javaone | 4

5. LinkedIn
Your professional identity on the web
The site The numbers
• Launched in 2003 • 22 million members
• 3 degree network • 4+ million unique visitors/month
• Strong focus on privacy • 40 million page views/day
The company • 2 million searches/day
• Profitable since 2006 • 250K invitations sent/day
• 250 employees • 1 million answers posted
• Based in Mountain View, CA
• We’re hiring!
2008 JavaOneSM Conference | java.com.sun/javaone | 5

6. Engineering Process
Release often
• Short, 2-4 week development cycles
All dev tasks are broken down into small, manageable
engineering cards
Strong focus on testing
Minimize meetings, status (standup meetings as needed)
2008 JavaOneSM Conference | java.com.sun/javaone | 6

7. Engineering Cards
2008 JavaOneSM Conference | java.com.sun/javaone | 7

8. Testing
6500+ unit and integration tests
500 HtmlUnit tests
Large-scale Hudson deployment
• 20+ nodes
• Continuous integration build/smoke test on SVN commit
• Nightly test run (full test suite)
2008 JavaOneSM Conference | java.com.sun/javaone | 8

9. “Celebrities” Test Network
2008 JavaOneSM Conference | java.com.sun/javaone | 9

10. Hudson
2008 JavaOneSM Conference | java.com.sun/javaone | 10

11. Testing…
Lessons Learned
Test suite too focused on integration
• Long running time
• Long-running setup/teardown routine
• Exacerbated by move to SOA
• Tests become redundant due to overlapping integration tests
• Better: use mocks (EasyMock works great)
2008 JavaOneSM Conference | java.com.sun/javaone | 11

12. Architecture
2008 JavaOneSM Conference | java.com.sun/javaone | 12

13. LinkedIn Architecture: 2003-2005
2008 JavaOneSM Conference | java.com.sun/javaone | 13

14. LinkedIn Architecture: 2006
2008 JavaOneSM Conference | java.com.sun/javaone | 14

15. LinkedIn Architecture: Today
2008 JavaOneSM Conference | java.com.sun/javaone | 15

16. LinkedIn Architecture: Today
2008 JavaOneSM Conference | java.com.sun/javaone | 16

17. Service-oriented Architecture
Benefits Caveats
• Each service can scale • Diminished data integrity,
independently consistency
• Encourages decoupling, reduces • No cross-service transactions,
code+deployment dependencies referential integrity or joins
• Graceful degradation of • Backward compatibility
functionality • “8 Fallacies of Distributed
Computing”
Takeaways
• Not applicable to small or simple sites, startups
• Build services with interfaces, POJOs, dependency injection
• Be conscious of API design, granularity
2008 JavaOneSM Conference | java.com.sun/javaone | 17

18. Services
2008 JavaOneSM Conference | java.com.sun/javaone | 18

19. The Cloud
Backend server caching the entire LinkedIn Network
Graph cache
• Updated via Databus
• Persisted to disk on shutdown
Transient network cache
• Computationally intensive to build
• Bound to member session
• Sticky load balancing to Cloud
Graph operations:
• findRoute(m1, m2)
• visit(visitor, deg)
• visit(visitor, deg, since)
2008 JavaOneSM Conference | java.com.sun/javaone | 19

20. The Cloud
The numbers
22M nodes, 120 million edges
12 GB JVM heap
• In-memory caches implemented in C++, accessed via JNI
40 Cloud server instances deployed in production
2008 JavaOneSM Conference | java.com.sun/javaone | 20

21. LinkedIn News
Part of a new trend in LinkedIn
Crawling the web for news
• Distributed task management
• Parsing technologies: StAX, Rome
Indexing and searching
• Serving the right news to the right people
Duplications Duplications Duplications
• Real time problem, fast solution
• 500 articles batch = 125K compares
2008 JavaOneSM Conference | java.com.sun/javaone | 21

22. News Service Architecture
2008 JavaOneSM Conference | java.com.sun/javaone | 22

23. LinkedIn: 99% Pure Java™
Plumbing Services Tier Web Tier Tools
Spring Remoting LinkedIn Spring Louis (in-house fwk) Hudson (CI)
ActiveMQ Eh-cache Spring MVC Eclipse+Mylyn
Quartz Lucene Grails JIRA/Greenhopper
HttpClient Jetty DWR JUnit, HtmlUnit
2008 JavaOneSM Conference | java.com.sun/javaone | 23

24. Why we love Java™ at LinkedIn
Static typing is a lifesaver
• Huge codebase, 1M+ lines of code
• Refactoring with confidence
• Code navigation (via IDE)
• 20 active branches, compiler helps with merging
•
Scaling the engineering organization
• Currently 50+ engineers, 8 teams… and growing
• Excellent talent pool of Java™ engineers to hire from
Community
2008 JavaOneSM Conference | java.com.sun/javaone | 24

25. LinkedIn: A Professional Social Network Built
with Java™ Technologies and Agile Practices
Nick Dellamaggiore (http://www.linkedin.com/in/nick)
Eishay Smith (http://www.linkedin.com/in/eishay)
We’re hiring!