The online advertising industry is based on identifying users with cookies, and showing relevant ads to interested users. But there are many data providers, many places to target ads and many people browsing online. How can we identify users across data providers? The first step in solving this is by cookie mapping: a chain of server calls that pass identifiers across providers. Sadly, chains break, servers break, providers can be flaky or use caching and you may never see the whole of the chain. The solution to this problem is constructing an identity graph with the data we see: in our case, cookie ids are nodes, edges are relations and connected components of the graph are users. In this talk I will explain how Hybrid Theory leverages Spark and GraphFrames to construct and maintain a 2000 million node identity graph with minimal computational cost.

3. WHOAMI
> Ruben Berenguel (@berenguel)
> PhD in Mathematics
> Lead Data Engineer at Hybrid Theory
> Preferred stack is Python, Go and Scala

4. Part 1 Set up
Part 2 The identity graph
Part 3 Speed up and improvements

5. Part 1: Set up
Adtech
What are cookies, really?
What is cookie mapping?
The identity problem

6. PROGRAMMATIC ADTECH
FIND USERS SATISFYING SOME
CRITERIA

7. PROGRAMMATIC ADTECH
FIND USERS SATISFYING SOME
CRITERIA
> Visited pages of category ABC

8. PROGRAMMATIC ADTECH
FIND USERS SATISFYING SOME
CRITERIA
> Visited pages of category ABC
> Are interested in concept XYZ

9. PROGRAMMATIC ADTECH
FIND USERS SATISFYING SOME
CRITERIA
> Visited pages of category ABC
> Are interested in concept XYZ
> Are likely to want to buy from our
client RST

10. TO FIND THEM WE NEED
THEIR BROWSE AND/OR BEHAVIOUR
DATA
! "

11. TO FIND THEM WE NEED
THEIR BROWSE AND/OR BEHAVIOUR DATA
! "
TO DELIVER FOR OUR
CLIENTS WE NEED
A WAY TO SHOW THEM ADS
!

12. COOKIES
ARE USED TO HELP
WEBSITES
TRACK EVENTS
AND STATE
AS USERS BROWSE

13. THERE ARE TWO KIND OF
COOKIES
FIRST PARTY (SESSION, STATE…)
THIRD PARTY (EVENT TRACKING…)

24. WE GET BROWSE DATA FROM USERS ON
THE WEB FROM DATA PROVIDERSA
A
Event logs with cookies provided in batch by data providers

25. WE GET BROWSE DATA FROM USERS IN THE
WEB FROM DATA PROVIDERS
WE GET BROWSE DATA FROM USERS
BROWSING OUR CLIENT WEBSITEB
B
Event logs with cookies generated from our servers, via our pixels

26. HOW DO WE CONNECT
BOTH DATA SOURCES?
THE IDENTIFIERS WE GET
FROM BOTH SIDES ARE
UNRELATED! !

27. MAPPING SERVERS
AND
THE MAPPING CHAIN

41. THE IDENTITY
PROBLEM

43. BASIC SOLUTION

44. BASIC SOLUTION
> Coalesce (merge on nulls) chains based on one id

45. BASIC SOLUTION
> Coalesce (merge on nulls) chains based on one id
> Is not as complete as the graph approach because…

46. BASIC SOLUTION
> Coalesce (merge on nulls) chains based on one id
> Is not as complete as the graph approach because…
> Requires one stable identifier

48. Part 2: The identity graph
Rethink the problem as a graph
Connected components in big data

59. ENTER GRAPHFRAMES

60. BASIC SPARK GRAPH
FRAMEWORK: GRAPHX
IT IS MESSAGE-PROPAGATIONC
,
GRAPH-PARALLEL, LOW LEVEL
C
Like the Pregel API

61. BASIC SPARK GRAPH
FRAMEWORK: GRAPHX
IT IS MESSAGE-PROPAGATION (PREGEL
API) GRAPH-PARALLEL, LOW LEVEL
GRAPHFRAMES ARE TO DATAFRAMES
AS GRAPHX IS TO RDDS

62. ALTERNATIVES CONSIDERED…
Apache Giraph harder maintenance
Neo4J harder scalability
AWS Neptune too new

63. INPUT SHOULD BE FORMATTED AS A DATAFRAME OF EDGES
src dst (…)
partner_1_! partner_2_⍺ 1617963647…
partner_1_2 partner_3_⭘ 1617963647…
partner_2_𝛄 partner_3_ 1617963654…
⁞ ⁞ ⁞

64. CONNECTED
COMPONENTS IN BIG
DATA
THE LARGE STAR - SMALL
STAR ALGORITHM

79. OUTPUT LAYOUT
Component Id Partner / Cookie Id Timestamp
10234 partner_1_! 1617963647
10234 partner_2_⍺ 1617963647
5534 partner_1_2 1617963654
⁞ ⁞ ⁞

80. To map from Partner A to Partner B

81. To map from Partner A to Partner B
> Given an id Partner_A_X,

82. To map from Partner A to Partner B
> Given an id Partner_A_X,
> we find the connected component id for the node
Partner_A_X,

83. To map from Partner A to Partner B
> Given an id Partner_A_X,
> we find the connected component id for the node
Partner_A_X,
> we find all the nodes of the form Partner_B_* for the
component above

84. IMPACT OF MOVING FROM AN ADHOC
PROCESS TO A GRAPH PROCESS

85. IMPACT OF MOVING FROM AN ADHOC
PROCESS TO A GRAPH PROCESS
> Partner integration: from 2 months to 1 week

86. IMPACT OF MOVING FROM AN ADHOC
PROCESS TO A GRAPH PROCESS
> Partner integration: from 2 months to 1 week
> Users mapped uplift: around 20%

87. IMPACT OF MOVING FROM AN ADHOC
PROCESS TO A GRAPH PROCESS
> Partner integration: from 2 months to 1 week
> Users mapped uplift: around 20%
> Mapping "quality": competitive (within 5%) with industry
leaders

88. Part 3: Speed up and improvements
Data cleanup
Cheap refresh
Machine tuning
Potential improvement

89. DATA CLEANUP

90. INVALID IDENTIFIERS

91. INVALID IDENTIFIERS
LIKE NA OR 0 OR XYZ
(OR FRAUDULENT CALLS TO A MAPPING SERVER)

92. NODE PRUNING

93. NODE PRUNING
TO PREVENT HUGE COMPONENTS
IN THE COOKIE CASE, BY EXPIRING COOKIES NOT SEEN IN N DAYS

94. COMPONENT DESTRUCTION

95. COMPONENT DESTRUCTION
TO LIMIT COMPONENT SIZE
ARTIFICIALLY
IF THE DATA IS FULLY CLEAN WE CAN ASSUME NO USER HAS MORE THAN M IDENTIFIERS

96. WHAT IS THE FASTEST WAY TO BUILD A
2 BILLION NODES GRAPH DAILY?
!

97. WHAT IS THE FASTEST WAY TO BUILD A
2 BILLION NODES GRAPH DAILY?
NOT DOING IT

98. THE EASY WAY

107. MACHINE
TUNING
FOR LARGE
GRAPHS

108. GO LARGE AND TUNE UP

109. GO LARGE AND TUNE UP
> the process is memory hungry

110. GO LARGE AND TUNE UP
> the process is memory hungry
> the process is shuffle hungry

111. GO LARGE AND TUNE UP
> the process is memory hungry
> the process is shuffle hungry
BETTER TO HAVE FEW, LARGE,
MACHINES

112. GO LARGE AND TUNE UP
> the process is memory hungry
> the process is shuffle hungry
BETTER TO HAVE FEW, LARGE,
MACHINES
AND GIVE EXECUTORS MORE
MEMORY THAN YOU'D THINK

113. IMPACT OF ADAPTIVE
QUERY EXECUTION
(AQE)
AQE USES RUNTIME STATISTICS TO
HELP THE COST BASED OPTIMIZER
(CBO) AND SPEED UP SPARK

114. IMPACT OF ADAPTIVE
QUERY EXECUTION
(AQE)
AQE USES RUNTIME STATISTICS TO HELP
THE COST BASED OPTIMIZER (CBO) AND
SPEED UP SPARK
USING SPARK 3.X WITH AQE ACTIVE
HAS A 30-40% SPEED UP

115. FURTHER IMPROVEMENTS

116. FURTHER IMPROVEMENTS
> Easy: Move storage to Delta Lake

117. FURTHER IMPROVEMENTS
> Easy: Move storage to Delta Lake
> Hard: implement union-find-shuffle instead of large star -
small star

118. THANKS!

119. Get the slides from my github:
github.com/rberenguel/
The repository is
identity-graphs

121. References
Connected Components in MapReduce and Beyond (ACM)
Connected Components in MapReduce and Beyond (slides)
Partition Aware Connected Component Computation in Distributed Systems
Building Graphs at a Large Scale: Union Find Shuffle
Adaptive Query Execution: Speeding up SparkSQL at runtime
Pregel: A System for Large-Scale Graph Processing
GraphX
GraphFrames
Apache Giraph
Neo4J
AWS Neptune
Databricks' Delta Lake: high on ACID

122. Related talks
Massive-Scale Entity Resolution Using the Power of Apache Spark and Graph
Maps and Meaning: Graph-based Entity Resolution in Apache Spark & GraphX
Building Identity Graph at Scale for Programmatic Media Buying Using Apache Spark and
Delta Lake
Building Identity Graphs over Heterogeneous Data
Optimize the Large Scale Graph Applications by using Apache Spark with 4-5x
Performance Improvements
GraphFrames: Graph Queries In Spark SQL
Using GraphX/Pregel on Browsing History to Discover Purchase Intent

123. Reference Image attribution
Graphs Ruben Berenguel ! (Generative art with p5js)
Bulb Alessandro Bianchi (Unsplash)
Bubbles Marko Blažević (Unsplash)
Chair Volodymyr Tokar (Unsplash)
Cookie Dex Ezekiel (Unsplash)
Loupe Agence Olloweb (Unsplash)
Map Timo Wielink (Unsplash)
Mask Adnan Khan (Unsplash)
Newspaper Rishabh Sharma (Unsplash)
Party Adi Goldstein (Unsplash)
Socket Kelly Sikkema (Unsplash)
Spray JESHOOTS.COM (Unsplash)
Tuning gustavo Campos (Unsplash)
Web Shannon Potter (Unsplash)

124. Resources
Unicode table

125. EOF