SlideShare ist ein Scribd-Unternehmen logo
1 von 41
Downloaden Sie, um offline zu lesen
LINKEDIN COMMUNICATION ARCHITECTURE


Ruslan Belkin, Sean Dawson

TS-5234
Learn how we at LinkedIn built and evolved scalable
communication platform for the world’s largest
professional network




                                        2008 JavaOneSM Conference | java.sun.com/javaone |   2
Agenda

 Why are we doing this talk
 LinkedIn Communication Platform at a glance
  • Evolution of LinkedIn Communication System
  • Evolution of the Network Updates System
 Scaling the system: from 0 to 22M members
 Q&A




                                        2008 JavaOneSM Conference | java.sun.com/javaone |   3
Why are we doing this talk?

 Share our experience in building the world-largest
 professional network in Java™
 Describe the evolution of the communication platform
 Share lessons we learned so you could benefit from our
 successes, mistakes and experience




                                       2008 JavaOneSM Conference | java.sun.com/javaone |   4
LinkedIn Communication Platform
Quick Tour




                            2008 JavaOneSM Conference | java.sun.com/javaone |   5
LinkedIn Communication Platform
Quick Tour




                            2008 JavaOneSM Conference | java.sun.com/javaone |   6
LinkedIn Communication Platform
Quick Tour




                            2008 JavaOneSM Conference | java.sun.com/javaone |   7
LinkedIn Communication Platform
The Numbers

  22M members
  130M connections
  2M email messages per day
  250K invitations per day




                              2008 JavaOneSM Conference | java.sun.com/javaone |   8
LinkedIn Communication Platform
The Setup

  Sun™ x86 platform and Sparc production hardware
  running Solaris™ Operating System
  100% Java programming language
  Tomcat and Jetty as application servers
  Oracle and MySQL as DBs
  ActiveMQ for JMS
  Lucene as a foundation for search
  Spring as a glue
  Mac for development



                                      2008 JavaOneSM Conference | java.sun.com/javaone |   9
LinkedIn Communication Platform

 The Communication Service         The network updates service
  •   Permanent message storage     • Short-lived notifications (events)
  •   InBox messages                • Distribution across various
  •   Emails                            affiliations and groups
  •   Batching, delayed delivery    •   Time decay
  •   Bounce, cancellation          •   Events grouping and
  •   Actionable content                prioritization
  •   Rich email content




                                                2008 JavaOneSM Conference | java.sun.com/javaone | 10
The Communication Service

 How is it different:
  • Workflow oriented
  • Messages reference other objects in the system
  • Incorporates email delivery
  • Batching of messages
  • Message cancellation
  • Delayed delivery, customer service review queues, abuse controls
  • Supports reminders and bounce notifications to users
 Has undergone continuous improvements throughout life
 of LinkedIn


                                              2008 JavaOneSM Conference | java.sun.com/javaone | 11
Message Creation




                   2008 JavaOneSM Conference | java.sun.com/javaone | 12
Message Delivery




                   2008 JavaOneSM Conference | java.sun.com/javaone | 13
The Communication Service

 Message Creation                        Message Delivery
  • Clients post messages via             • Message delivery is triggered by
    asynchronous Java Communications          clients or by scheduled processes
    API using JMS                         •   Delivery actions are asynchronous
  • Messages then are routed via          •   Messages can be batched for
    routing service to the appropriate        delivery into a single email message
    mailbox or directly for email         •   Message content is processed
    processing                                through the JavaServer Page™ (JSP™)
  • Multiple member or guest                  technology for pretty formatting
    databases are supported               •   The scheduler can take into account
                                              the time, delivery preferences,
                                              system load
                                          •   Bounced messages are processed
                                              and redelivered if needed
                                          •   Reminder system works the same
                                              way as message delivery system



                                                      2008 JavaOneSM Conference | java.sun.com/javaone | 14
The Communication Service

 SOA architecture
 Wireable components build around LinkedIn Spring
 extensions
 Spring HTTP-RPC
 Heavy use of JMS and asynchronous flows




                                      2008 JavaOneSM Conference | java.sun.com/javaone | 15
The Communication Service
Failure Recovery

  Possible failures:
   • Messages can bounce
   • Messages can get lost:
      •   Database problems
      •   Bugs in the code
      •   Bugs in the content processing of emails
      •   Various services may become unavailable
  Avoiding the downtime




                                                     2008 JavaOneSM Conference | java.sun.com/javaone | 16
The Communication Service
How do we scale it?

  Functional partitioning:
   • sent, received, archived, etc.
  Class partitioning:
   • Member mailboxes, guest mailboxes, corporate mailboxes
  Range partitioning:
   • Member ID range
   • Email lexicographical range
  Asynchronous flows




                                             2008 JavaOneSM Conference | java.sun.com/javaone | 17
Network Updates Service

 What is your network up to?
 The goal is to have a flexible service for distributing many
 types of short-lived updates
 Availability across a number of clients (web apps, RSS,
 API, LinkedIn Mobile, third-party…)




                                          2008 JavaOneSM Conference | java.sun.com/javaone | 18
Network Updates Service
Motivation


  Homepage circa 2007
  Poor UI
   • Cluttered
   • Where does new content go?
  Poor Backend Integration
   • Many different service calls
   • Takes a long time to gather all
     of the data




                                       2008 JavaOneSM Conference | java.sun.com/javaone | 19
Network Updates Service
Motivation


  Homepage circa 2008
  Clean UI
   • Eliminates contention for
     homepage real estate
  Clean Backend
   • Single call to fetch updates
   • Consistent update format




                                    2008 JavaOneSM Conference | java.sun.com/javaone | 20
Network Updates Service
Iteration 1

  Move existing homepage logic into a remote service,
  refactor homepage to use the new service
  Advantages
   • No user impact while API is being finalized
   • Improve performance by fetching updates in parallel
   • Reduce complexity of the web app
   • Updates become easily accessible to other clients




                                          2008 JavaOneSM Conference | java.sun.com/javaone | 21
Network Updates Service
Iteration 1 - API




                          2008 JavaOneSM Conference | java.sun.com/javaone | 22
Network Updates Service
Iteration 1 - API

  Pull-based architecture
  Collectors
   • Responsible for gathering data
   • Parallel collection to improve performance
  Resolvers
   • Fetch state, batch lookup queries, etc…
   • Use EHCache to cache global data (e.g., member info)
  Rendering
   • Transform each object into its XML representation



                                               2008 JavaOneSM Conference | java.sun.com/javaone | 23
Network Updates Service
Iteration 1 - Example

UpdateQueryCriteria query =
   UpdateQueryCriteria.createDefaultCriteria()
       .setMemberID(2)
       .setRequestedTypes(NetworkUpdateType.CONNECTION)
       .setCutoffDate(…)
       .setMaxNumberOfUpdates(10);

MyNetworkUpdatesService service =
   getService(MyNetworkUpdatesService.class);

DataTree update =
   service.getNetworkUpdatesSumary(query);


                                       2008 JavaOneSM Conference | java.sun.com/javaone | 24
Network Updates Service
Iteration 1 - Example


                        <updates>
                          <NCON>
                            <connection>
                              <id>2</id>

                          <firstName>Chris</firstNa
                          me>

                           <lastName>Yee</lastName>
                            </connection>
                          </NCON>
                        </updates>

                                2008 JavaOneSM Conference | java.sun.com/javaone | 25
Network Updates Service
Iteration 1

  Lessons learned:
   • Centralizing updates into a single service leaves a single point of
       failure
   •   Be prepared to spend time tuning the HttpConnectionManager
       (timeouts, max connections)
   •   While the system was stabilizing, it was affecting all users; should
       have rolled the new service out to a small subset!
   •   Don’t use “Least Frequently Used” (LFU) in a large EHCache—very
       bad performance!




                                                    2008 JavaOneSM Conference | java.sun.com/javaone | 26
Network Updates Service
Iteration 2

  Hollywood Principle: “Don’t call me, I’ll call you”
  Push update when an event occurs
  Reading is much quicker since we don’t have to search for
  the data!
  Tradeoffs
   • Distributed updates may never be read
   • More storage space needed




                                        2008 JavaOneSM Conference | java.sun.com/javaone | 27
Network Updates Service
Iteration 2 - Pushing Events




                               2008 JavaOneSM Conference | java.sun.com/javaone | 28
Network Updates Service
Iteration 2 - Reading Updates




                                2008 JavaOneSM Conference | java.sun.com/javaone | 29
Network Updates Service
Iteration 2

  Pushing Updates
   • Updates are delivered via JMS
   • Aggregate data stored in 1 CLOB column for each target user
   • Incoming updates are merged into the aggregate structure using
     optimistic locking to avoid lock contention
  Reading Updates
   • Add a new collector that reads from the Update Database
   • Use Digesters to perform arbitrary transformations on the stream
     of updates (e.g, collapse 10 updates from a user into 1)




                                                   2008 JavaOneSM Conference | java.sun.com/javaone | 30
Network Updates Service
Iteration 2

  Lessons learned:
   • Underestimated the volume of updates to be processed
   • CLOB block size was set to 8k, leading to a lot of wasted space
       (which isn’t reclaimed!)
   •   Real-time monitoring/configuration with Java Management
       Extension (JMX™) specification was extremely helpful




                                                  2008 JavaOneSM Conference | java.sun.com/javaone | 31
Network Updates Service
Iteration 3

  Updating a CLOB is expensive
  Goal: Minimize the number of CLOB updates
   • Use an overflow buffer
   • Reduce the size of the updates




                                      2008 JavaOneSM Conference | java.sun.com/javaone | 32
Network Updates Service
Iteration 3 - Overflow Buffer


  Add VARCHAR(4000) column
  that acts as a buffer
  When the buffer is full, dump it
  to the CLOB and reset
  Avoids over 90% of CLOB
  updates (depending on type),
  while still retaining the
  flexibility for more storage




                                     2008 JavaOneSM Conference | java.sun.com/javaone | 33
Scaling the system

 What you learn as you scale:           What to do:
  • A single database does not work     • Partition everything:
  • Referential integrity will not be        • by user groups
      possible                               • by domain
  •   Cost becomes a factor:                 • by function
      databases, hardware, licenses,    • Caching is good even when it’s
      storage, power                        only modestly effective
  •   Any data loss is a problem        •   Give up on 100% data integrity
  •   Data warehousing and analytics    •   Build for asynchronous flows
      becomes a problem                 •   Build with reporting in mind
  •   Your system becomes a target      •   Expect your system to fail at any
      for spamming exploits, data           point
      scraping, etc.
                                        •   Never underestimate growth
                                            trajectory


                                                   2008 JavaOneSM Conference | java.sun.com/javaone | 34
LinkedIn Communication Architecture

 Build with scalability in mind - never know when your
 business will take off
 Expect to do a lot of architecture and code refactoring as
 you learn and appreciate growth challenges




                                         2008 JavaOneSM Conference | java.sun.com/javaone | 35
LinkedIn Communication Architecture
Ruslan Belkin (http://www.linkedin.com/in/rbelkin)
Sean Dawson (http://www.linkedin.com/in/seandawson)

We are hiring!
The Communication Service
LinkedIn Spring Extensions
  Automatic context instantiation   Support for Builder Pattern
  from multiple spring files        Custom property editors:
  LinkedIn Spring Components         • Timespan (30s, 4h34m, etc.)
  Property expansion                 • Memory Size, etc.
  Automatic termination handling




                                              2008 JavaOneSM Conference | java.sun.com/javaone | 37
The Communication Service
LinkedIn Spring Extensions

Comm-server/
   cmpt/
       components/
              ccsServiceExporter.spring
              comm.spring
              jmx.spring
              comm-server.properties
                     corpMboxServiceExporter.spring
              main.spring
              comm-server.spring
              memberMboxServiceExporter.spring
              comm.properties
              guestMboxServiceExporter.spring
       build.xml
    impl/
       …
                                       2008 JavaOneSM Conference | java.sun.com/javaone | 38
The Communication Service
LinkedIn Spring Extensions

…
<bean id=quot;resolver”
class=quot;com.linkedin.comm.pub.impl.MessageAddressResolverquot;>
   <lin:config>
       <property name=quot;resolverDBquot; ref=quot;resolverDBquot;/>
       <property name=quot;eosquot; ref=quot;eosquot;/>
       <property name=quot;elsquot; ref=quot;eosquot;/>
       <property name=quot;memberAccessorquot;
ref=quot;coreMemberAccessorquot;/>
   </lin:config>
</bean>
…




                                      2008 JavaOneSM Conference | java.sun.com/javaone | 39
The Communication Service
LinkedIn Spring Extensions (Builder)

private final MessageAddressResolverDB _resolverDB;
…
MessageAddressResolver(Config config) { … }
…
public static class Config    {
   private MessageAddressResolverDB _resolverDB;

   public MessageAddressResolverDB getResolverDB() {
       return ConfigHelper.getRequired(_resolverDB);
   }
   public void setResolverDB(MessageAddressResolverDB
                              resolverDB) {
              _resolverDB = resolverDB;
   }
}/*Config*/

                                       2008 JavaOneSM Conference | java.sun.com/javaone | 40
The Communication Service
LinkedIn Spring Extensions (Components)

…
<lin:component
   id=quot;remoteContentCommunicationServicequot;
   location=quot;comm-server-client-cmptquot;>
   <lin:wire property-name=quot;activemq.producer.brokerURLquot;
                           property-
value=quot;${activemq.producer.brokerURL}quot;/>
   <lin:wire property-name=quot;comm.server.httpRpc.urlquot;
                           property-
value=quot;${leo.comm.server.httpRpc.url}quot;/>
</lin:component>
…




                                          2008 JavaOneSM Conference | java.sun.com/javaone | 41

Weitere ähnliche Inhalte

Was ist angesagt?

LinkedIn powerpoint
LinkedIn powerpointLinkedIn powerpoint
LinkedIn powerpointguest2137df
 
How To Really Use LinkedIn In 10 Slides
How To Really Use LinkedIn In 10 SlidesHow To Really Use LinkedIn In 10 Slides
How To Really Use LinkedIn In 10 SlidesBert Verdonck
 
(Master ppt) LinkedIn on Campus
(Master ppt) LinkedIn on Campus(Master ppt) LinkedIn on Campus
(Master ppt) LinkedIn on CampusLinkedIn
 
LinkedIn presentation
LinkedIn presentationLinkedIn presentation
LinkedIn presentationjkwong5
 
How to REALLY use LinkedIn - mini LinkedIn Presentation
How to REALLY use LinkedIn - mini LinkedIn PresentationHow to REALLY use LinkedIn - mini LinkedIn Presentation
How to REALLY use LinkedIn - mini LinkedIn PresentationBert Verdonck
 
LinkedIn training presentation
LinkedIn training presentationLinkedIn training presentation
LinkedIn training presentationNovell
 
Creating A Social Media Marketing Plan
Creating A Social Media Marketing PlanCreating A Social Media Marketing Plan
Creating A Social Media Marketing PlanCory Williamson
 
How to Use Social Media for Lead Generation
How to Use Social Media for Lead GenerationHow to Use Social Media for Lead Generation
How to Use Social Media for Lead GenerationHubSpot
 
2014 LinkedIn Company Presentation
2014 LinkedIn Company Presentation2014 LinkedIn Company Presentation
2014 LinkedIn Company PresentationLinkedIn
 

Was ist angesagt? (20)

LinkedIn powerpoint
LinkedIn powerpointLinkedIn powerpoint
LinkedIn powerpoint
 
Linkedin Training Basic
Linkedin Training   BasicLinkedin Training   Basic
Linkedin Training Basic
 
How To Really Use LinkedIn In 10 Slides
How To Really Use LinkedIn In 10 SlidesHow To Really Use LinkedIn In 10 Slides
How To Really Use LinkedIn In 10 Slides
 
(Master ppt) LinkedIn on Campus
(Master ppt) LinkedIn on Campus(Master ppt) LinkedIn on Campus
(Master ppt) LinkedIn on Campus
 
LinkedIn presentation
LinkedIn presentationLinkedIn presentation
LinkedIn presentation
 
LINKEDIN CONTENT MARKETING Source: LinkedIn
LINKEDIN CONTENT MARKETING Source: LinkedInLINKEDIN CONTENT MARKETING Source: LinkedIn
LINKEDIN CONTENT MARKETING Source: LinkedIn
 
Digital Marketing Real Estate Case Study
Digital Marketing Real Estate Case StudyDigital Marketing Real Estate Case Study
Digital Marketing Real Estate Case Study
 
LinkedIn ppt
LinkedIn pptLinkedIn ppt
LinkedIn ppt
 
LinkedIn Basics
LinkedIn BasicsLinkedIn Basics
LinkedIn Basics
 
How to REALLY use LinkedIn - mini LinkedIn Presentation
How to REALLY use LinkedIn - mini LinkedIn PresentationHow to REALLY use LinkedIn - mini LinkedIn Presentation
How to REALLY use LinkedIn - mini LinkedIn Presentation
 
LinkedIn training presentation
LinkedIn training presentationLinkedIn training presentation
LinkedIn training presentation
 
LinkedIn presentation
LinkedIn presentationLinkedIn presentation
LinkedIn presentation
 
LinkedIn Premium Playbook
LinkedIn Premium PlaybookLinkedIn Premium Playbook
LinkedIn Premium Playbook
 
Creating A Social Media Marketing Plan
Creating A Social Media Marketing PlanCreating A Social Media Marketing Plan
Creating A Social Media Marketing Plan
 
LinkedIn Marketing Strategies
LinkedIn Marketing StrategiesLinkedIn Marketing Strategies
LinkedIn Marketing Strategies
 
Linkedin 101 ppt
Linkedin 101 pptLinkedin 101 ppt
Linkedin 101 ppt
 
How to Use Social Media for Lead Generation
How to Use Social Media for Lead GenerationHow to Use Social Media for Lead Generation
How to Use Social Media for Lead Generation
 
LinkedIn for Students
LinkedIn for StudentsLinkedIn for Students
LinkedIn for Students
 
2014 LinkedIn Company Presentation
2014 LinkedIn Company Presentation2014 LinkedIn Company Presentation
2014 LinkedIn Company Presentation
 
LinkedIn Marketing PPT
LinkedIn Marketing PPTLinkedIn Marketing PPT
LinkedIn Marketing PPT
 

Ähnlich wie LinkedIn Communication Architecture

JavaOne Conference, 2008
JavaOne Conference, 2008JavaOne Conference, 2008
JavaOne Conference, 2008Sean Dawson
 
LinkedIn - A Professional Network built with Java Technologies and Agile Prac...
LinkedIn - A Professional Network built with Java Technologies and Agile Prac...LinkedIn - A Professional Network built with Java Technologies and Agile Prac...
LinkedIn - A Professional Network built with Java Technologies and Agile Prac...LinkedIn
 
LinkedIn - A Professional Network built with Java Technologies and Agile Prac...
LinkedIn - A Professional Network built with Java Technologies and Agile Prac...LinkedIn - A Professional Network built with Java Technologies and Agile Prac...
LinkedIn - A Professional Network built with Java Technologies and Agile Prac...Nick Dellamaggiore
 
Intro in JavaEE world (TU Olomouc)
Intro in JavaEE world (TU Olomouc)Intro in JavaEE world (TU Olomouc)
Intro in JavaEE world (TU Olomouc)blahap
 
ADDRESSING TOMORROW'S SECURITY REQUIREMENTS IN ENTERPRISE APPLICATIONS
ADDRESSING TOMORROW'S SECURITY REQUIREMENTS IN ENTERPRISE APPLICATIONSADDRESSING TOMORROW'S SECURITY REQUIREMENTS IN ENTERPRISE APPLICATIONS
ADDRESSING TOMORROW'S SECURITY REQUIREMENTS IN ENTERPRISE APPLICATIONSelliando dias
 
Server side push in Aldan 3
Server side push in Aldan 3Server side push in Aldan 3
Server side push in Aldan 3ALDAN3
 
Oracle WebLogic 12c New Multitenancy features
Oracle WebLogic 12c New Multitenancy featuresOracle WebLogic 12c New Multitenancy features
Oracle WebLogic 12c New Multitenancy featuresMichel Schildmeijer
 
Virtualizing a Virtual Machine
Virtualizing a Virtual MachineVirtualizing a Virtual Machine
Virtualizing a Virtual Machineelliando dias
 
Java EE8 - by Kito Mann
Java EE8 - by Kito Mann Java EE8 - by Kito Mann
Java EE8 - by Kito Mann Kile Niklawski
 
Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered
Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates UncoveredRuslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered
Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates UncoveredLinkedIn
 
Streaming to a New Jakarta EE
Streaming to a New Jakarta EEStreaming to a New Jakarta EE
Streaming to a New Jakarta EEJ On The Beach
 
Streaming to a new Jakarta EE
Streaming to a new Jakarta EEStreaming to a new Jakarta EE
Streaming to a new Jakarta EEMarkus Eisele
 
blueMarine Or Why You Should Really Ship Swing Applications
blueMarine  Or Why You Should Really Ship Swing  Applications blueMarine  Or Why You Should Really Ship Swing  Applications
blueMarine Or Why You Should Really Ship Swing Applications Fabrizio Giudici
 
Glassfish JEE Server Administration - JEE Introduction
Glassfish JEE Server Administration - JEE IntroductionGlassfish JEE Server Administration - JEE Introduction
Glassfish JEE Server Administration - JEE IntroductionDanairat Thanabodithammachari
 
Intorduction to struts
Intorduction to strutsIntorduction to struts
Intorduction to strutsAnup72
 
SPEC INDIA Java Case Study
SPEC INDIA Java Case StudySPEC INDIA Java Case Study
SPEC INDIA Java Case StudySPEC INDIA
 
Streaming to a new Jakarta EE / JOTB19
Streaming to a new Jakarta EE / JOTB19Streaming to a new Jakarta EE / JOTB19
Streaming to a new Jakarta EE / JOTB19Markus Eisele
 
Vikas chanalia_8.4+ EXP-Networker-RESUME
Vikas chanalia_8.4+ EXP-Networker-RESUMEVikas chanalia_8.4+ EXP-Networker-RESUME
Vikas chanalia_8.4+ EXP-Networker-RESUMEVikas Chanalia
 

Ähnlich wie LinkedIn Communication Architecture (20)

JavaOne Conference, 2008
JavaOne Conference, 2008JavaOne Conference, 2008
JavaOne Conference, 2008
 
LinkedIn - A Professional Network built with Java Technologies and Agile Prac...
LinkedIn - A Professional Network built with Java Technologies and Agile Prac...LinkedIn - A Professional Network built with Java Technologies and Agile Prac...
LinkedIn - A Professional Network built with Java Technologies and Agile Prac...
 
LinkedIn - A Professional Network built with Java Technologies and Agile Prac...
LinkedIn - A Professional Network built with Java Technologies and Agile Prac...LinkedIn - A Professional Network built with Java Technologies and Agile Prac...
LinkedIn - A Professional Network built with Java Technologies and Agile Prac...
 
Intro in JavaEE world (TU Olomouc)
Intro in JavaEE world (TU Olomouc)Intro in JavaEE world (TU Olomouc)
Intro in JavaEE world (TU Olomouc)
 
ADDRESSING TOMORROW'S SECURITY REQUIREMENTS IN ENTERPRISE APPLICATIONS
ADDRESSING TOMORROW'S SECURITY REQUIREMENTS IN ENTERPRISE APPLICATIONSADDRESSING TOMORROW'S SECURITY REQUIREMENTS IN ENTERPRISE APPLICATIONS
ADDRESSING TOMORROW'S SECURITY REQUIREMENTS IN ENTERPRISE APPLICATIONS
 
Server side push in Aldan 3
Server side push in Aldan 3Server side push in Aldan 3
Server side push in Aldan 3
 
Oracle WebLogic 12c New Multitenancy features
Oracle WebLogic 12c New Multitenancy featuresOracle WebLogic 12c New Multitenancy features
Oracle WebLogic 12c New Multitenancy features
 
Java EE 7 - Overview and Status
Java EE 7  - Overview and StatusJava EE 7  - Overview and Status
Java EE 7 - Overview and Status
 
Virtualizing a Virtual Machine
Virtualizing a Virtual MachineVirtualizing a Virtual Machine
Virtualizing a Virtual Machine
 
Java EE8 - by Kito Mann
Java EE8 - by Kito Mann Java EE8 - by Kito Mann
Java EE8 - by Kito Mann
 
Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered
Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates UncoveredRuslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered
Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered
 
Streaming to a New Jakarta EE
Streaming to a New Jakarta EEStreaming to a New Jakarta EE
Streaming to a New Jakarta EE
 
Streaming to a new Jakarta EE
Streaming to a new Jakarta EEStreaming to a new Jakarta EE
Streaming to a new Jakarta EE
 
blueMarine Or Why You Should Really Ship Swing Applications
blueMarine  Or Why You Should Really Ship Swing  Applications blueMarine  Or Why You Should Really Ship Swing  Applications
blueMarine Or Why You Should Really Ship Swing Applications
 
Glassfish JEE Server Administration - JEE Introduction
Glassfish JEE Server Administration - JEE IntroductionGlassfish JEE Server Administration - JEE Introduction
Glassfish JEE Server Administration - JEE Introduction
 
Intorduction to struts
Intorduction to strutsIntorduction to struts
Intorduction to struts
 
SPEC INDIA Java Case Study
SPEC INDIA Java Case StudySPEC INDIA Java Case Study
SPEC INDIA Java Case Study
 
Streaming to a new Jakarta EE / JOTB19
Streaming to a new Jakarta EE / JOTB19Streaming to a new Jakarta EE / JOTB19
Streaming to a new Jakarta EE / JOTB19
 
Nadeem_CV
Nadeem_CVNadeem_CV
Nadeem_CV
 
Vikas chanalia_8.4+ EXP-Networker-RESUME
Vikas chanalia_8.4+ EXP-Networker-RESUMEVikas chanalia_8.4+ EXP-Networker-RESUME
Vikas chanalia_8.4+ EXP-Networker-RESUME
 

Mehr von LinkedIn

How LinkedIn is Transforming Businesses
How LinkedIn is Transforming BusinessesHow LinkedIn is Transforming Businesses
How LinkedIn is Transforming BusinessesLinkedIn
 
Networking on LinkedIn 101
Networking on LinkedIn 101Networking on LinkedIn 101
Networking on LinkedIn 101LinkedIn
 
5 تحديثات على ملفك في 5 دقائق
5 تحديثات على ملفك في 5 دقائق5 تحديثات على ملفك في 5 دقائق
5 تحديثات على ملفك في 5 دقائقLinkedIn
 
5 LinkedIn Profile Updates in 5 Minutes
5 LinkedIn Profile Updates in 5 Minutes5 LinkedIn Profile Updates in 5 Minutes
5 LinkedIn Profile Updates in 5 MinutesLinkedIn
 
The Student's Guide to LinkedIn
The Student's Guide to LinkedInThe Student's Guide to LinkedIn
The Student's Guide to LinkedInLinkedIn
 
The Top Skills That Can Get You Hired in 2017
The Top Skills That Can Get You Hired in 2017The Top Skills That Can Get You Hired in 2017
The Top Skills That Can Get You Hired in 2017LinkedIn
 
Accelerating LinkedIn’s Vision Through Innovation
Accelerating LinkedIn’s Vision Through InnovationAccelerating LinkedIn’s Vision Through Innovation
Accelerating LinkedIn’s Vision Through InnovationLinkedIn
 
How To Tell Your #workstory
How To Tell Your #workstoryHow To Tell Your #workstory
How To Tell Your #workstoryLinkedIn
 
LinkedIn Q1 2016 Earnings Call
LinkedIn Q1 2016 Earnings CallLinkedIn Q1 2016 Earnings Call
LinkedIn Q1 2016 Earnings CallLinkedIn
 
The 2016 LinkedIn Job Search Guide
The 2016 LinkedIn Job Search GuideThe 2016 LinkedIn Job Search Guide
The 2016 LinkedIn Job Search GuideLinkedIn
 
LinkedIn Q4 2015 Earnings Call
LinkedIn Q4 2015 Earnings CallLinkedIn Q4 2015 Earnings Call
LinkedIn Q4 2015 Earnings CallLinkedIn
 
Banish The Buzzwords
Banish The BuzzwordsBanish The Buzzwords
Banish The BuzzwordsLinkedIn
 
LinkedIn Bring In Your Parents Day 2015 - Your Parents' Best Career Advice
LinkedIn Bring In Your Parents Day 2015 - Your Parents' Best Career AdviceLinkedIn Bring In Your Parents Day 2015 - Your Parents' Best Career Advice
LinkedIn Bring In Your Parents Day 2015 - Your Parents' Best Career AdviceLinkedIn
 
LinkedIn Q3 2015 Earnings Call
LinkedIn Q3 2015 Earnings CallLinkedIn Q3 2015 Earnings Call
LinkedIn Q3 2015 Earnings CallLinkedIn
 
LinkedIn Economic Graph Research: Toronto
LinkedIn Economic Graph Research: TorontoLinkedIn Economic Graph Research: Toronto
LinkedIn Economic Graph Research: TorontoLinkedIn
 
Freelancers Are LinkedIn Power Users [Infographic]
Freelancers Are LinkedIn Power Users [Infographic]Freelancers Are LinkedIn Power Users [Infographic]
Freelancers Are LinkedIn Power Users [Infographic]LinkedIn
 
Top Industries for Freelancers on LinkedIn [Infographic]
Top Industries for Freelancers on LinkedIn [Infographic]Top Industries for Freelancers on LinkedIn [Infographic]
Top Industries for Freelancers on LinkedIn [Infographic]LinkedIn
 
LinkedIn Quiz: Which Parent Are You When It Comes to Helping Guide Your Child...
LinkedIn Quiz: Which Parent Are You When It Comes to Helping Guide Your Child...LinkedIn Quiz: Which Parent Are You When It Comes to Helping Guide Your Child...
LinkedIn Quiz: Which Parent Are You When It Comes to Helping Guide Your Child...LinkedIn
 
LinkedIn Connect to Opportunity™ -- Stories of Discovery
LinkedIn Connect to Opportunity™ -- Stories of DiscoveryLinkedIn Connect to Opportunity™ -- Stories of Discovery
LinkedIn Connect to Opportunity™ -- Stories of DiscoveryLinkedIn
 
LinkedIn Q2 2015 Earnings Call
LinkedIn Q2 2015 Earnings CallLinkedIn Q2 2015 Earnings Call
LinkedIn Q2 2015 Earnings CallLinkedIn
 

Mehr von LinkedIn (20)

How LinkedIn is Transforming Businesses
How LinkedIn is Transforming BusinessesHow LinkedIn is Transforming Businesses
How LinkedIn is Transforming Businesses
 
Networking on LinkedIn 101
Networking on LinkedIn 101Networking on LinkedIn 101
Networking on LinkedIn 101
 
5 تحديثات على ملفك في 5 دقائق
5 تحديثات على ملفك في 5 دقائق5 تحديثات على ملفك في 5 دقائق
5 تحديثات على ملفك في 5 دقائق
 
5 LinkedIn Profile Updates in 5 Minutes
5 LinkedIn Profile Updates in 5 Minutes5 LinkedIn Profile Updates in 5 Minutes
5 LinkedIn Profile Updates in 5 Minutes
 
The Student's Guide to LinkedIn
The Student's Guide to LinkedInThe Student's Guide to LinkedIn
The Student's Guide to LinkedIn
 
The Top Skills That Can Get You Hired in 2017
The Top Skills That Can Get You Hired in 2017The Top Skills That Can Get You Hired in 2017
The Top Skills That Can Get You Hired in 2017
 
Accelerating LinkedIn’s Vision Through Innovation
Accelerating LinkedIn’s Vision Through InnovationAccelerating LinkedIn’s Vision Through Innovation
Accelerating LinkedIn’s Vision Through Innovation
 
How To Tell Your #workstory
How To Tell Your #workstoryHow To Tell Your #workstory
How To Tell Your #workstory
 
LinkedIn Q1 2016 Earnings Call
LinkedIn Q1 2016 Earnings CallLinkedIn Q1 2016 Earnings Call
LinkedIn Q1 2016 Earnings Call
 
The 2016 LinkedIn Job Search Guide
The 2016 LinkedIn Job Search GuideThe 2016 LinkedIn Job Search Guide
The 2016 LinkedIn Job Search Guide
 
LinkedIn Q4 2015 Earnings Call
LinkedIn Q4 2015 Earnings CallLinkedIn Q4 2015 Earnings Call
LinkedIn Q4 2015 Earnings Call
 
Banish The Buzzwords
Banish The BuzzwordsBanish The Buzzwords
Banish The Buzzwords
 
LinkedIn Bring In Your Parents Day 2015 - Your Parents' Best Career Advice
LinkedIn Bring In Your Parents Day 2015 - Your Parents' Best Career AdviceLinkedIn Bring In Your Parents Day 2015 - Your Parents' Best Career Advice
LinkedIn Bring In Your Parents Day 2015 - Your Parents' Best Career Advice
 
LinkedIn Q3 2015 Earnings Call
LinkedIn Q3 2015 Earnings CallLinkedIn Q3 2015 Earnings Call
LinkedIn Q3 2015 Earnings Call
 
LinkedIn Economic Graph Research: Toronto
LinkedIn Economic Graph Research: TorontoLinkedIn Economic Graph Research: Toronto
LinkedIn Economic Graph Research: Toronto
 
Freelancers Are LinkedIn Power Users [Infographic]
Freelancers Are LinkedIn Power Users [Infographic]Freelancers Are LinkedIn Power Users [Infographic]
Freelancers Are LinkedIn Power Users [Infographic]
 
Top Industries for Freelancers on LinkedIn [Infographic]
Top Industries for Freelancers on LinkedIn [Infographic]Top Industries for Freelancers on LinkedIn [Infographic]
Top Industries for Freelancers on LinkedIn [Infographic]
 
LinkedIn Quiz: Which Parent Are You When It Comes to Helping Guide Your Child...
LinkedIn Quiz: Which Parent Are You When It Comes to Helping Guide Your Child...LinkedIn Quiz: Which Parent Are You When It Comes to Helping Guide Your Child...
LinkedIn Quiz: Which Parent Are You When It Comes to Helping Guide Your Child...
 
LinkedIn Connect to Opportunity™ -- Stories of Discovery
LinkedIn Connect to Opportunity™ -- Stories of DiscoveryLinkedIn Connect to Opportunity™ -- Stories of Discovery
LinkedIn Connect to Opportunity™ -- Stories of Discovery
 
LinkedIn Q2 2015 Earnings Call
LinkedIn Q2 2015 Earnings CallLinkedIn Q2 2015 Earnings Call
LinkedIn Q2 2015 Earnings Call
 

Kürzlich hochgeladen

UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 

Kürzlich hochgeladen (20)

UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 

LinkedIn Communication Architecture

  • 1. LINKEDIN COMMUNICATION ARCHITECTURE Ruslan Belkin, Sean Dawson TS-5234
  • 2. Learn how we at LinkedIn built and evolved scalable communication platform for the world’s largest professional network 2008 JavaOneSM Conference | java.sun.com/javaone | 2
  • 3. Agenda Why are we doing this talk LinkedIn Communication Platform at a glance • Evolution of LinkedIn Communication System • Evolution of the Network Updates System Scaling the system: from 0 to 22M members Q&A 2008 JavaOneSM Conference | java.sun.com/javaone | 3
  • 4. Why are we doing this talk? Share our experience in building the world-largest professional network in Java™ Describe the evolution of the communication platform Share lessons we learned so you could benefit from our successes, mistakes and experience 2008 JavaOneSM Conference | java.sun.com/javaone | 4
  • 5. LinkedIn Communication Platform Quick Tour 2008 JavaOneSM Conference | java.sun.com/javaone | 5
  • 6. LinkedIn Communication Platform Quick Tour 2008 JavaOneSM Conference | java.sun.com/javaone | 6
  • 7. LinkedIn Communication Platform Quick Tour 2008 JavaOneSM Conference | java.sun.com/javaone | 7
  • 8. LinkedIn Communication Platform The Numbers 22M members 130M connections 2M email messages per day 250K invitations per day 2008 JavaOneSM Conference | java.sun.com/javaone | 8
  • 9. LinkedIn Communication Platform The Setup Sun™ x86 platform and Sparc production hardware running Solaris™ Operating System 100% Java programming language Tomcat and Jetty as application servers Oracle and MySQL as DBs ActiveMQ for JMS Lucene as a foundation for search Spring as a glue Mac for development 2008 JavaOneSM Conference | java.sun.com/javaone | 9
  • 10. LinkedIn Communication Platform The Communication Service The network updates service • Permanent message storage • Short-lived notifications (events) • InBox messages • Distribution across various • Emails affiliations and groups • Batching, delayed delivery • Time decay • Bounce, cancellation • Events grouping and • Actionable content prioritization • Rich email content 2008 JavaOneSM Conference | java.sun.com/javaone | 10
  • 11. The Communication Service How is it different: • Workflow oriented • Messages reference other objects in the system • Incorporates email delivery • Batching of messages • Message cancellation • Delayed delivery, customer service review queues, abuse controls • Supports reminders and bounce notifications to users Has undergone continuous improvements throughout life of LinkedIn 2008 JavaOneSM Conference | java.sun.com/javaone | 11
  • 12. Message Creation 2008 JavaOneSM Conference | java.sun.com/javaone | 12
  • 13. Message Delivery 2008 JavaOneSM Conference | java.sun.com/javaone | 13
  • 14. The Communication Service Message Creation Message Delivery • Clients post messages via • Message delivery is triggered by asynchronous Java Communications clients or by scheduled processes API using JMS • Delivery actions are asynchronous • Messages then are routed via • Messages can be batched for routing service to the appropriate delivery into a single email message mailbox or directly for email • Message content is processed processing through the JavaServer Page™ (JSP™) • Multiple member or guest technology for pretty formatting databases are supported • The scheduler can take into account the time, delivery preferences, system load • Bounced messages are processed and redelivered if needed • Reminder system works the same way as message delivery system 2008 JavaOneSM Conference | java.sun.com/javaone | 14
  • 15. The Communication Service SOA architecture Wireable components build around LinkedIn Spring extensions Spring HTTP-RPC Heavy use of JMS and asynchronous flows 2008 JavaOneSM Conference | java.sun.com/javaone | 15
  • 16. The Communication Service Failure Recovery Possible failures: • Messages can bounce • Messages can get lost: • Database problems • Bugs in the code • Bugs in the content processing of emails • Various services may become unavailable Avoiding the downtime 2008 JavaOneSM Conference | java.sun.com/javaone | 16
  • 17. The Communication Service How do we scale it? Functional partitioning: • sent, received, archived, etc. Class partitioning: • Member mailboxes, guest mailboxes, corporate mailboxes Range partitioning: • Member ID range • Email lexicographical range Asynchronous flows 2008 JavaOneSM Conference | java.sun.com/javaone | 17
  • 18. Network Updates Service What is your network up to? The goal is to have a flexible service for distributing many types of short-lived updates Availability across a number of clients (web apps, RSS, API, LinkedIn Mobile, third-party…) 2008 JavaOneSM Conference | java.sun.com/javaone | 18
  • 19. Network Updates Service Motivation Homepage circa 2007 Poor UI • Cluttered • Where does new content go? Poor Backend Integration • Many different service calls • Takes a long time to gather all of the data 2008 JavaOneSM Conference | java.sun.com/javaone | 19
  • 20. Network Updates Service Motivation Homepage circa 2008 Clean UI • Eliminates contention for homepage real estate Clean Backend • Single call to fetch updates • Consistent update format 2008 JavaOneSM Conference | java.sun.com/javaone | 20
  • 21. Network Updates Service Iteration 1 Move existing homepage logic into a remote service, refactor homepage to use the new service Advantages • No user impact while API is being finalized • Improve performance by fetching updates in parallel • Reduce complexity of the web app • Updates become easily accessible to other clients 2008 JavaOneSM Conference | java.sun.com/javaone | 21
  • 22. Network Updates Service Iteration 1 - API 2008 JavaOneSM Conference | java.sun.com/javaone | 22
  • 23. Network Updates Service Iteration 1 - API Pull-based architecture Collectors • Responsible for gathering data • Parallel collection to improve performance Resolvers • Fetch state, batch lookup queries, etc… • Use EHCache to cache global data (e.g., member info) Rendering • Transform each object into its XML representation 2008 JavaOneSM Conference | java.sun.com/javaone | 23
  • 24. Network Updates Service Iteration 1 - Example UpdateQueryCriteria query = UpdateQueryCriteria.createDefaultCriteria() .setMemberID(2) .setRequestedTypes(NetworkUpdateType.CONNECTION) .setCutoffDate(…) .setMaxNumberOfUpdates(10); MyNetworkUpdatesService service = getService(MyNetworkUpdatesService.class); DataTree update = service.getNetworkUpdatesSumary(query); 2008 JavaOneSM Conference | java.sun.com/javaone | 24
  • 25. Network Updates Service Iteration 1 - Example <updates> <NCON> <connection> <id>2</id> <firstName>Chris</firstNa me> <lastName>Yee</lastName> </connection> </NCON> </updates> 2008 JavaOneSM Conference | java.sun.com/javaone | 25
  • 26. Network Updates Service Iteration 1 Lessons learned: • Centralizing updates into a single service leaves a single point of failure • Be prepared to spend time tuning the HttpConnectionManager (timeouts, max connections) • While the system was stabilizing, it was affecting all users; should have rolled the new service out to a small subset! • Don’t use “Least Frequently Used” (LFU) in a large EHCache—very bad performance! 2008 JavaOneSM Conference | java.sun.com/javaone | 26
  • 27. Network Updates Service Iteration 2 Hollywood Principle: “Don’t call me, I’ll call you” Push update when an event occurs Reading is much quicker since we don’t have to search for the data! Tradeoffs • Distributed updates may never be read • More storage space needed 2008 JavaOneSM Conference | java.sun.com/javaone | 27
  • 28. Network Updates Service Iteration 2 - Pushing Events 2008 JavaOneSM Conference | java.sun.com/javaone | 28
  • 29. Network Updates Service Iteration 2 - Reading Updates 2008 JavaOneSM Conference | java.sun.com/javaone | 29
  • 30. Network Updates Service Iteration 2 Pushing Updates • Updates are delivered via JMS • Aggregate data stored in 1 CLOB column for each target user • Incoming updates are merged into the aggregate structure using optimistic locking to avoid lock contention Reading Updates • Add a new collector that reads from the Update Database • Use Digesters to perform arbitrary transformations on the stream of updates (e.g, collapse 10 updates from a user into 1) 2008 JavaOneSM Conference | java.sun.com/javaone | 30
  • 31. Network Updates Service Iteration 2 Lessons learned: • Underestimated the volume of updates to be processed • CLOB block size was set to 8k, leading to a lot of wasted space (which isn’t reclaimed!) • Real-time monitoring/configuration with Java Management Extension (JMX™) specification was extremely helpful 2008 JavaOneSM Conference | java.sun.com/javaone | 31
  • 32. Network Updates Service Iteration 3 Updating a CLOB is expensive Goal: Minimize the number of CLOB updates • Use an overflow buffer • Reduce the size of the updates 2008 JavaOneSM Conference | java.sun.com/javaone | 32
  • 33. Network Updates Service Iteration 3 - Overflow Buffer Add VARCHAR(4000) column that acts as a buffer When the buffer is full, dump it to the CLOB and reset Avoids over 90% of CLOB updates (depending on type), while still retaining the flexibility for more storage 2008 JavaOneSM Conference | java.sun.com/javaone | 33
  • 34. Scaling the system What you learn as you scale: What to do: • A single database does not work • Partition everything: • Referential integrity will not be • by user groups possible • by domain • Cost becomes a factor: • by function databases, hardware, licenses, • Caching is good even when it’s storage, power only modestly effective • Any data loss is a problem • Give up on 100% data integrity • Data warehousing and analytics • Build for asynchronous flows becomes a problem • Build with reporting in mind • Your system becomes a target • Expect your system to fail at any for spamming exploits, data point scraping, etc. • Never underestimate growth trajectory 2008 JavaOneSM Conference | java.sun.com/javaone | 34
  • 35. LinkedIn Communication Architecture Build with scalability in mind - never know when your business will take off Expect to do a lot of architecture and code refactoring as you learn and appreciate growth challenges 2008 JavaOneSM Conference | java.sun.com/javaone | 35
  • 36. LinkedIn Communication Architecture Ruslan Belkin (http://www.linkedin.com/in/rbelkin) Sean Dawson (http://www.linkedin.com/in/seandawson) We are hiring!
  • 37. The Communication Service LinkedIn Spring Extensions Automatic context instantiation Support for Builder Pattern from multiple spring files Custom property editors: LinkedIn Spring Components • Timespan (30s, 4h34m, etc.) Property expansion • Memory Size, etc. Automatic termination handling 2008 JavaOneSM Conference | java.sun.com/javaone | 37
  • 38. The Communication Service LinkedIn Spring Extensions Comm-server/ cmpt/ components/ ccsServiceExporter.spring comm.spring jmx.spring comm-server.properties corpMboxServiceExporter.spring main.spring comm-server.spring memberMboxServiceExporter.spring comm.properties guestMboxServiceExporter.spring build.xml impl/ … 2008 JavaOneSM Conference | java.sun.com/javaone | 38
  • 39. The Communication Service LinkedIn Spring Extensions … <bean id=quot;resolver” class=quot;com.linkedin.comm.pub.impl.MessageAddressResolverquot;> <lin:config> <property name=quot;resolverDBquot; ref=quot;resolverDBquot;/> <property name=quot;eosquot; ref=quot;eosquot;/> <property name=quot;elsquot; ref=quot;eosquot;/> <property name=quot;memberAccessorquot; ref=quot;coreMemberAccessorquot;/> </lin:config> </bean> … 2008 JavaOneSM Conference | java.sun.com/javaone | 39
  • 40. The Communication Service LinkedIn Spring Extensions (Builder) private final MessageAddressResolverDB _resolverDB; … MessageAddressResolver(Config config) { … } … public static class Config { private MessageAddressResolverDB _resolverDB; public MessageAddressResolverDB getResolverDB() { return ConfigHelper.getRequired(_resolverDB); } public void setResolverDB(MessageAddressResolverDB resolverDB) { _resolverDB = resolverDB; } }/*Config*/ 2008 JavaOneSM Conference | java.sun.com/javaone | 40
  • 41. The Communication Service LinkedIn Spring Extensions (Components) … <lin:component id=quot;remoteContentCommunicationServicequot; location=quot;comm-server-client-cmptquot;> <lin:wire property-name=quot;activemq.producer.brokerURLquot; property- value=quot;${activemq.producer.brokerURL}quot;/> <lin:wire property-name=quot;comm.server.httpRpc.urlquot; property- value=quot;${leo.comm.server.httpRpc.url}quot;/> </lin:component> … 2008 JavaOneSM Conference | java.sun.com/javaone | 41