It is common for consumer Internet companies to start off with popular third-party tools for analytics needs. Then, when the user base and the company grows, they end up building their own analytics data pipeline and query engine to cope with their data scale, satisfy custom data enrichment and reporting needs and achieve high quality of their data. That’s exactly the path that was taken at Grammarly, the popular online proofreading service. In this session, Grammarly will share how they improved business and marketing analytics, previously done with Mixpanel, by building their own in-house analytics engine and application on top of Apache Spark. Chernetsov wil touch upon several Spark tweaks and gotchas that they experienced along the way: – Outputting data to several storages in a single Spark job – Dealing with Spark memory model, building a custom spillable data-structure for your data traversal – Implementing a custom query language with parser combinators on top of Spark sql parser – Custom query optimizer and analyzer when you want not exactly sql – Flexible-schema storage and query against multi-schema data with schema conflicts – Custom aggregation functions in Spark SQL

1. Building a Versatile Analytics Pipeline
On Top Of Apache Spark
Misha Chernetsov, Grammarly
Spark Summit 2017
June 6, 2017

3. Data Team Lead @ Grammarly
Building Analytics Pipelines (5 years)
Coding on JVM (12 years), Scala + Spark (3 years)
About Me: Misha Chernetsov
@chernetsov

4. Tool that helps us better understand:
● Who are our users?
● How do they interact with the product?
● How do they get in, engage, pay, and how long do they stay?
Analytics @ Consumer Product Company

5. We want our decisions to be
data-driven
Everyone: product managers, marketing, engineers, support...
Analytics @ Consumer Product Company

6. data
analytics
report
Analytics @ Consumer Product Company

7. Calendar Day
Number of
unique active
users by day
Example Report 1 – Daily Active Users
dummy data!dummy data!

8. Example Report 2 – Comparison of Cohort Retention Over Time
dummy data!dummy data!

9. Ads
Email
Social
Number of
users who
bought a
subscription.
Split by traffic
source type
(where user
came from)
Calendar Day
Example Report 3 – Payer Conversions By Traffic Source
dummy data!dummy data!

10. ● Landing page visit
○ URL with UTM tags
○ Referrer
● Subscription purchased
○ Is first in subscription
Example: Data

11. Everything is an Event
Example: Data
{
"eventName": "page-visit",
"url": "...?utm_medium=ad",
…
}
{
"eventName": "subscribe",
"period": "12 months",
…
}

12. Enrich and/or Join
Example: Data
{
"eventName": "page-visit",
"url": "...?utm_medium=ad",
…
}
{
"eventName": "subscribe",
"period": "12 months",
…
}
Slice by Plot

13. capture enrich index query
Analytics @ Consumer Product Company

14. capture enrich index query
Use 3rd Party?
1. Integrated Event Analytics
2. UI over your DB
Reports are not tailored for your
needs, limited capability.
Pre-aggregation / enriching
is still on you.
Hard to achieve accuracy and trust.

15. capture enrich index query
Build Step 1: Capture

16. ● Always up, resilient
● Spikes / back pressure
● Buffer for delayed processing
Kafka
Capture
REST
{
"eventName": "page-visit",
"url": "...?utm_medium=paid",
…
}

17. Long-term
Storage
StreamKafka
Save To Long-Term Storage

18. Cassandra
micro-batch
Kafka
Save To Long-Term Storage
val rdd = KafkaUtils.createRDD[K, V](...)
rdd.saveToCassandra("raw")

19. capture enrich index query
Build Step 2: Enrich

20. Enrichment 1: User Attribution

21. Enrichment 1: User Attribution
{
"eventName": "page-visit",
"url": "...?utm_medium=ad",
"fingerprint": "abc",
…
}
{
"eventName": "subscribe",
"userId": 123,
"fingerprint": "abc",
…
}

22. Enrichment 1: User Attribution
{
"eventName": "page-visit",
"url": "...?utm_medium=ad",
"fingerprint": "abc",
…
}
{
"eventName": "subscribe",
"userId": 123,
"fingerprint": "abc",
…
}
"attributedUserId": 123,

23. t
Non-authenticated:
userId = null
Authenticated:
userId = 123
fingerprint = abc
(All events from a
given browser)
Enrichment 1: User Attribution

24. t
Non-authenticated:
userId = null
attributedUserId = 123
Authenticated:
userId = 123
fingerprint = abc
(All events from a
given browser)
Enrichment 1: User Attribution

25. Authenticated:
userId = 756
Enrichment 1: User Attribution
tfingerprint = abc
(All events from a
given browser)
Authenticated:
userId = 123
Heuristics to
attribute those

26. Authenticated:
userId = 756
Enrichment 1: User Attribution
tfingerprint = abc
(All events from a
given browser)
Authenticated:
userId = 123
Heuristics to
attribute those

27. rdd.mapPartitions { iterator =>
val buffer = new ArrayBuffer()
iterator
.takeWhile(_.userId.isEmpty)
.foreach(buffer.append)
val userId = iterator.head.userId
buffer.map(_.setAttributedUserId(userId)) ++ iterator
}
Enrichment 1: User Attribution

28. rdd.mapPartitions { iterator =>
val buffer = new ArrayBuffer()
iterator
.takeWhile(_.userId.isEmpty)
.foreach(buffer.append)
val userId = iterator.head.userId
buffer.map(_.setAttributedUserId(userId)) ++ iterator
}
Enrichment 1: User Attribution

29. rdd.mapPartitions { iterator =>
val buffer = new ArrayBuffer()
iterator
.takeWhile(_.userId.isEmpty)
.foreach(buffer.append)
val userId = iterator.head.userId
buffer.map(_.setAttributedUserId(userId)) ++ iterator
}
Enrichment 1: User Attribution

30. rdd.mapPartitions { iterator =>
val buffer = new ArrayBuffer()
iterator
.takeWhile(_.userId.isEmpty)
.foreach(buffer.append)
val userId = iterator.head.userId
buffer.map(_.setAttributedUserId(userId)) ++ iterator
}
Enrichment 1: User Attribution

31. rdd.mapPartitions { iterator =>
val buffer = new ArrayBuffer()
iterator
.takeWhile(_.userId.isEmpty)
.foreach(buffer.append)
val userId = iterator.head.userId
buffer.map(_.setAttributedUserId(userId)) ++ iterator
}
Enrichment 1: User Attribution
Can grow big and
OOM your worker for
outliers who use
Grammarly without
ever registering

32. By default we
should operate
in User Memory
(small fraction).
Spark & Memory
User Memory
100% - spark.memory.fraction = 25%
Spark Memory
spark.memory.fraction = 75%
Let’s get into
Spark Memory
and use its
safety features.

33. rdd.mapPartitions { iterator =>
val buffer = new SpillableBuffer()
iterator
.takeWhile(_.userId.isEmpty)
.foreach(buffer.append)
val userId = iterator.head.userId
buffer.map(_.setAttributedUserId(userId)) ++ iterator
}
Enrichment 1: User Attribution
Can safely grow in
mem while enough
free Spark Mem. Spills
to disk otherwise.

34. Spark Memory Manager & Spillable Collection
Memory Disk

35. Spark Memory Manager & Spillable Collection
Memory Disk

36. Spark Memory Manager & Spillable Collection
Memory Disk

37. Spark Memory Manager & Spillable Collection
Memory
×2
Disk

38. Spark Memory Manager & Spillable Collection
Memory Disk

39. Spark Memory Manager & Spillable Collection
Memory Disk

40. Spark Memory Manager & Spillable Collection
Memory
Spill to Disk
Disk

41. Spark Memory Manager & Spillable Collection
Memory Disk
Spill to Disk

42. Spark Memory Manager & Spillable Collection
Memory Disk

43. trait SizeTracker {
def afterUpdate(): Unit = { … }
def estimateSize(): Long = { … }
}
Call on every append.
Periodically estimates size
and saves samples.
Extrapolates
SizeTracker

44. trait Spillable {
abstract def spill(inMemCollection: C): Unit
def maybeSpill(currentMemory: Long, inMemCollection: C) {
try x2 if needed
}
}
Spillable
call on every append to collection

45. public long acquireExecutionMemory(long required, …)
public void releaseExecutionMemory(long size, …)
TaskMemoryManager

46. ● Be safe with outliers
● Get outside User Memory (25%), use Spark Memory (75%)
● Spark APIs: Could be a bit friendlier and high level
Custom Spillable Collection

47. Enrichment 2: Calculable Props

48. Enrichment Phase 2: Calculable Props
{
"eventName": "page-visit",
"url": "...?utm_medium=ad",
"fingerprint": "abc",
"attributedUserId": 123,
…
}
{
"eventName": "subscribe",
"userId": 123,
"fingerprint": "abc",
…
}

49. Enrichment Phase 2: Calculable Props
{
"eventName": "page-visit",
"url": "...?utm_medium=ad",
"fingerprint": "abc",
"attributedUserId": 123,
…
}
{
"eventName": "subscribe",
"userId": 123,
"fingerprint": "abc",
"firstUtmMedium": "ad",
…
}

50. val firstUtmMedium: CalcProp[String] =
(E "url").as[Url]
.map(_.param("utm_source"))
.forEvent("page-visit")
.first
Enrichment Phase 2: Calculable Props Engine & DSL

51. ● Type-safe, functional, composable
● Familiar: similar to Scala collections API
● Batch & Stream (incremental)
Enrichment Phase 2: Calculable Props Engine & DSL

52. Enrichment Pipeline with Spark

53. Raw
Kafka
Spark Pipeline
Stream:
Save Raw Kafka
Stream:
User Attr.
User-attributed
Kafka
Stream:
Calc Props
Enriched and
Queryable
Batch:
User Attr.
Batch:
Calc Props

54. batch
micro-batchmicro-batchmicro-batch
Cassandra
Kafka
Spark Pipeline
Kafka
Cassandra
Kafka
Parquet on
AWS S3
batch

55. ● Connectors for everything
● Great for batch
○ Shuffle with spilling
○ Failure recovery
● Great for streaming
○ Fast
○ Low overhead
Spark Pipeline

56. batch
micro-batchmicro-batchmicro-batch
Cassandra
Kafka
Spark Pipeline
Kafka
Cassandra
Kafka
Parquet on
AWS S3
batch

57. job
Multiple Output Destinations
Kafka Kafka
CassandraCassandra

58. val rdd: RDD[T]
rdd.sendToKafka(“topic_x”)
rdd.saveToCassandra(“table_foo”)
rdd.saveToCassandra(“table_bar”)
Multiple Output Destinations: Try 1

59. rdd.saveToCassandra(...)
rdd.forEachPartition(...)
sc.runJob(...)
Multiple Output Destinations: Try 1

60. job
Multiple Output Destinations: Try 1
Kafka Kafka
CassandraCassandra

61. job
Multiple Output Destinations: Try 1 = 3 Jobs
Kafka Kafka
job
Kafka
Table 1
job
Kafka
Table 2

62. val rdd: RDD[T]
rdd.cache()
rdd.sendToKafka(“topic_x”)
rdd.saveToCassandra(“table_foo”)
rdd.saveToCassandra(“table_bar”)
Multiple Output Destinations: Try 2

63. job
Multiple Output Destinations: Try 2 = Read Once, 3 Jobs
Kafka Kafka
job
Cache
Table 1
job
Cache
Table 2

64. rdd.forEachPartition { iterator =>
val writer = new BufferedWriter(
new OutputStreamWriter(new FileOutStream())
)
iterator.forEach { el =>
writer.writeln(el)
}
writer.close() // makes sure this writes
}
Writer

65. rdd.forEachPartition { iterator =>
val writer = new BufferedWriter(
new OutputStreamWriter(new FileOutStream())
)
iterator.forEach { el =>
writer.writeln(el)
}
writer.close() // makes sure this writes
}
Writer

66. rdd.forEachPartition { iterator =>
val writer = new BufferedWriter(
new OutputStreamWriter(new FileOutStream())
)
iterator.forEach { el =>
writer.writeln(el)
}
writer.close() // makes sure this writes
}
Writer

67. rdd.forEachPartition { iterator =>
val writer = new BufferedWriter(
new OutputStreamWriter(new FileOutStream(...))
)
iterator.forEach { el =>
writer.writeln(el)
}
writer.close() // makes sure this writes
}
● Buffer
● Non-blocking
● Idempotent / Dedupe
Writer

68. andWriteToX = rdd.mapPartitions { iterator =>
val writer = new XWriter()
val writingIterator = iterator.map { el =>
writer.write(el)
}.closing(() => writer.close)
}
AndWriter

69. andWriteToX = rdd.mapPartitions { iterator =>
val writer = new XWriter()
val writingIterator = iterator.map { el =>
writer.write(el)
}.closing(() => writer.close)
}
AndWriter

70. andWriteToX = rdd.mapPartitions { iterator =>
val writer = new XWriter()
val writingIterator = iterator.map { el =>
writer.write(el)
}.closing(() => writer.close)
}
AndWriter

71. val rdd: RDD[T]
rdd.andSaveToCassandra(“table_foo”)
.andSaveToCassandra(“table_bar”)
.sendToKafka(“topic_x”)
Multiple Output Destinations: Try 3

72. job
Multiple Output Destinations: Try 3
Kafka Kafka
CassandraCassandra

73. ● Kafka
● Cassandra
● HDFS
Important! Each andWriter will consume resources
● Memory (buffers)
● IO
And Writer

74. capture enrich index query
Build Step 3: Index

75. Index
● Parquet on AWS S3
● Custom partitioning: By eventName and time interval
● Append changes, compact + merge on the fly when querying
● Randomized names to maximize S3 parallelism
● Use s3a for max performance and and tweak for S3 read-after-write
consistency
● Support flexible schema, even with conflicts!

76. Some Stats
● Thousands of
events
per second
● Terabytes of
compressed
data

77. capture enrich index query
Build Step 4: Query

78. ● DataFrames
● Spark SQL Scala dsl / Pure SQL
● Zeppelin
Hardcore Query

79. ● Plot by day
● Unique visitors
● Filter by country
● Split by traffic source (top 20)
● Time from 2 weeks ago to today
Casual Query

80. Option 1: SQL
Quickly gets complex

81. Too expensive
to build, extend
and support
Option 2: UI

82. SEGMENT “eventName”
WHERE foo = “bar” AND x.y IN (“a”, “b”, “c”)
UNIQUE
BY m IS NOT NULL
TIME from 2 months ago to today
STEP 1 month SPAN 1 week
Option 3: Custom Query Language

83. SEGMENT “eventName”
WHERE foo = “bar” AND x.y IN (“a”, “b”, “c”)
UNIQUE
BY m IS NOT NULL
TIME from 2 months ago to today
STEP 1 month SPAN 1 week
Option 3: Custom Query Language
Expressions

84. ● Segment, Funnel, Retention
● UI & as DataFrame in Zeppelin
● Spark <= 1.6 – Scala Parser Combinators
● Reuse most complex part of expression parser
● Relatively extensible
Option 3: Custom Query Language

85. Option 3: Custom Query Language

86. ● Custom versatile analytics is doable and enjoyable
● Spark is a great platform to build analytics on top of
○ Enrichment Pipeline: Batch / Streaming, Query, ML
● Would be cool to see even deep internals slightly more extensible
Conclusion

87. We are hiring!
olivia@grammarly.com
https://www.grammarly.com/jobs

88. Thank you!
Questions?