Did you know that eHarmony is responsible for 5% of all new US marriages, and that more than 600,000 people already got married to partners they met on eHarmony? eHarmony was founded to give people a better chance at finding happy, passionate, and fulfilling relationships. During this talk I describe how we go about creating Compatible matches, and how we leverage Big Data technologies to accomplish that goal. Specifically, I discuss how we take Billion+ potential matches that we find through MongoDB, store them in a Voldemort NoSQL datastore, and then run multiple Hadoop jobs to come up with a filtered list based on Machine Learned models. Our Hadoop clusters are in-house, high density, low power Seamicro installations, and we use Spring Batch and Spring Data Hadoop to orchestrate the Hadoop jobs.

2. DAVID GEVORKYAN
@davidgev
davidgevorkyan

3. WHO ARE WE?

4. EHARMONY CREATES
THE HAPPIEST,
MOST PASSIONATE
AND MOST FULFILLING
RELATIONSHIPS*
*ACCORDING TO A RECENT STUDY

5. 438
MARRIAGES PER DAY

6. THE DIFFERENCE?

7. THE DIFFERENCE?
Compatibility Matching System®
COMPAT I B I L I T Y
MATCHING
AFFINITY
MATCHING
MATCH
DISTRIBUTION

8. THE DIFFERENCE?
Compatibility Matching System®
COMPAT I B I L I T Y
MATCHING
AFFINITY
MATCHING
MATCH
DISTRIBUTION

9. UNIDIRECTIONAL USER DEFINED CRITERIA
Nicolette

10. BIDIRECTIONAL
UNIDIRECTIONAL USER DEFINED CRITERIA
Leo
Ian
Steve
Nicolette

11. BIDIRECTIONAL
UNIDIRECTIONAL USER DEFINED CRITERIA
Leo
Ian
Steve
Nicolette

15. 150
ques)ons
Personality
Values
A5ributes
Beliefs

16. Intellect
Energy
Sociability
Ambition
Humor
Spirituality
Kindness
Curiosity

17. COMPATIBILITY MATCHING
USER DEFINED
CRITERIA
COMPAT I B I L I T Y
MODELS
MONGO DB
VOLDEMORT

18. MONGO DB
DATA STORE NEEDS
POWERFUL
INDEXING
MODELS
FAST MULTI -
ATTRIBUTE
SEARCHES
EASY TO
MAINTAIN
60M+
QUERIES
per day

19. MONGO DB
WINS
AUTO
SCALING
BUILT- I N
SHARDING
AUTO
BALANCING
MMS

20. VOLDEMORT ?
THAT NAME
SOUNDS FAMILIAR

21. VOLDEMORT
DATA STORE NEEDS
CRUD
OPERATIONS
VARIED
TRANSACTION
SIZES
BILLION+
POTENTIAL
MATCHES
per day

22. VOLDEMORT
WINS
AUTO
REPLICATION
AUTO
PARTITIONING
PLUGGABLE
SERIALIZATION

23. AFFINITY MATCHING
Compatibility Matching System®
COMPAT I B I L I T Y
MATCHING
AFFINITY
MATCHING
MATCH
DISTRIBUTION

24. 65 30
3000 miles

25. Comm probability Distance in Miles
PROB
0 1 3 7 15 63 255 1023 4095

27. 4
-­‐
8
in
Comm probability Height difference in cm
PROB
-29 -25 -21 -17 -13 -9 -6 -3 0 3 6 9 12 16 20 24 28 32 36 40 44 48 52 56

28. WORDS TO USE

29. WORDS TO USE

30. SOME INSIGHT

31. DATA NEEDS FOR AFFINITY
50M+ REGISTERED USERS
103 ATTRIBUTES
250M+
PHOTOS
107 DAILY MATCHES
4B+ QUESTIONNAIRES
ANSWERED

32. COMMUNICATION AGGREGATES
EVENT LISTENER
SERVICE
USER ACTIVITY
SERVICE
10K EVENTS
PER SECOND
HOURLY, DAILY
~5MS
RESPONSE
TIMES
USER
SERVICE
TOTAL

33. OFFLINE BATCH JOBS
USER
SERVICE
1+GB
Compressed
Protocol
MAP-SIDE JOINS
Buffers
( T B ) SCORING
PAIRINGS
SERVICE
!
750M
Compressed
Protocol
Buffers
BILLION+
POTENTIAL
MATCHES

34. AMAZON
EMR
AWS DIRECT
CONNECT
IN-HOUSE
SEAMICRO
256 NODES
50TB STORAGE
LOW OPERATIONAL COST
LOW POWER CONSUMPTION
PREDICTABLE COMPLETION TIMES
DATA RETRIEVAL LATENCY

35. MODEL RETRAINING
distcp
Protocol
Buffers
from
Offline
Jobs

36. MATCH DISTRIBUTION
Compatibility Matching System®
COMPAT I B I L I T Y
MATCHING
AFFINITY
MATCHING
MATCH
DISTRIBUTION

37. Delivering the right matches
at the right time to as many
people as possible across
the entire network

40. PROB
2 2

41. PROB
2 2

42. PROB

43. MONITORING

44. metrics.codahale.com

45. We Are Hiring!
jobs.eharmony.com

46. THANK YOU
QUESTIONS?
@davidgev

47. CREDITS:
Visual Elements From
The Noun Project
http://thenounproject.com