With the right combination of open source projects, you can have a high concurrency and low latency spark jobs for doing data analysis. We'll show both REST and JDBC access to access data from a persistent spark context and then show how the combination of Spark Job Server, Spark Thrift Server and Apache Kudu can create a scalable backend for low latency analytics.

1. High concurrency,
Low latency analytics
using Spark/Kudu
Chris George

2. Who is this guy?

3. Tech we will talk about:
Kudu
Spark
Spark Job Server
Spark Thrift Server

4. What was the
problem?

5. Apache Kudu

6. History of Kudu

7. Columnar vs other
types of storage

8. What if you could update
parquet/ORC easily?

9. HDFS vs Kudu vs
HBase/Cassandra/xyz

10. Kudu is purely a storage
engine accessible
through api

11. To add sql queries/more
advanced sql like
operations

12. Impala vs Spark

13. Kudu Slack Channel

14. Master and Tablets in
Kudu

15. Range and Hash
Partitioning

16. Number of cores =
number of partitions

17. Partitioning can be on
1+ columns

18. Composite primary keys important to filter
on the key in order
A, B, C
i.e. don’t scan for just B if possible
it will be expensive

19. Scans on a tablet is single
threaded but you can do 200+
scans on a tablet concurrently

20. To find your scale... load up a single
tablet
Insertion, Update, Deletes.. concurrently
Until it doesn't meet your performance

21. Partitioning is
extremely important

22. Kudu client is java
Python connectors coming
C++ client

23. Java Client loops through
tablets, but not
concurrently

24. But you can code the
multithread or contribute

25. Predicates on any
column

26. Summary of why
Kudu?

27. Predicates/Projections on
any column very quickly
at scale

28. Spark

29. Spark Datasource api:

30. Reads CSV
val df = sqlContext.read
.format("com.databricks.spark.csv")
.option("header", "true") // Use first line of all files as
header
.option("inferSchema", "true") // Automatically infer
data types
.load("cars.csv")
Writes CSV
val selectedData = df.select("year", "model")
selectedData.write
.format("com.databricks.spark.csv")
.option("header", "true")
.save("newcars.csv")

31. But these are often simlified as:
val parquetDataframe =
sqlContext.read.parquet(“people.parquet")
parquetDataframe.write.parquet("people.parquet")

32. I wrote the current version
of the Kudu
Datasource/Spark
Integration

33. There are limitations
with the datasource api

34. Save Modes for datasource api:
append, overwrite, ignore, error

35. append = insert

36. overwrite = truncate + insert

37. ignore = create if not exists

38. error = throw exception
if the data exists

39. What if I want to
update? Nope

40. What about deletes?
Not individually

41. So how do you support
updates/deletes?

42. By not using the
datasource api.. but I'll talk
more about that in a minute

43. Immutability of
dataframes

44. So why use
datasource api?

45. Because it's smarter
than it appears for reads

46. Pushdown predicates
and projections

47. Pushdown predicates:
val df =
sqlContext.read.options(Map("kudu.master" ->
"kudu.master:7051","kudu.table" ->
“kudu_table")).kudu
df.filter("id" >= 5).show()

48. Datasource has knowledge of what
can be pushed down to underlying
store
and what can not.

49. Why am I telling you
this?

50. Cause if you want things to
be fast you need to know
what is not pushed down!

51. EqualTo
GreaterThan
GreaterThanOrEqual
LessThan
LessThanOrEqual
And
https://github.com/cloudera/kudu/blob/master/j
ava/kudu-
spark/src/main/scala/org/apache/kudu/spark/k
udu/DefaultSource.scala#L159

52. Did you notice whats missing?
EqualTo
GreaterThan
GreaterThanOrEqual
LessThan
LessThanOrEqual
And

53. "OR"
So spark will use it's optimizer to run two
separate kudu scans for the OR
"IN" is coming very soon, nuanced
performance details

54. btw if you register the dataframe as
a temp table in spark
"select * from someDF where
id>=5" will also do pushdowns

55. "select * from someDF
where id>=5" will also do
pushdowns

56. things like select * from someDF
where lower(name)="joe"
will pull the entire table into memory
probably a bad thing

57. Projections will also be pushed down to
kudu so your not retrieving the entire row
df.select("id", "name")
select id, name from someDf

58. Looked at lots of existing
datasource’s to design
kudu’s

59. How does kudu do
updates/deletes in spark?

60. // Use KuduContext to
create, delete, or write to
Kudu tables
val kuduContext = new
KuduContext("kudu.master:7051")

61. // Insert data
kuduContext.insertRows(df, "test_table")
// Delete data
kuduContext.deleteRows(filteredDF, "test_table")
// Upsert data
kuduContext.upsertRows(df, "test_table")
// Update data
val alteredDF = df.select("id", $"count" + 1)
kuduContext.updateRows(alteredDF, “test_table")
http://kudu.apache.org/docs/developing.html

62. Upserts are handled server side for performance
Upserts can also be handled through datasource api:
df.write.options(Map("kudu.master"->
"kudu.master:7051", "kudu.table"-
>"test_table")).mode("append").kudu

63. You can also create,
check existence and
delete tables through api

64. Additional notes:
Kudu datasource currently works with spark 1.x
Next release it will support both 1.x and 2.x
It's being improved on regular basis

65. Number of partitions on the dataframe is
related to how many tablets/partitions are
related to the filter.
Partition scans are parallel and have locality
awareness in spark

66. Be sure to set spark locality wait to
something other small for low latency
(3 seconds is the spark default)

67. Spark Job Server
(SJS)

68. Created for low
latency jobs on spark

69. Persistent contexts
Reduces runtime of hello world
type of job from 1 second to 10 ms

70. Rest based api to:
Run Jobs
Create contexts
Check status of job both async/sync

71. Creating a context calls spark
submit (in separate jvm mode)
Uses akka to communicate
between rest and spark driver

72. To create a persistent context you need:
cpu cores + memory footprint
name to reference it by
factory to use for the context.
ie HiveContextFactory vs SqlContextFactory

73. Our average job time is
30ms when coming through
api for simpler retrievals

74. Jobs need to implement an interface
context will be passed in
DON’T CREATE YOUR OWN
SQLCONTEXT!!

75. Currently only supports
spark 1.x
2.x is coming soonish

76. Keeps track of job
runtimes in nice ui along
with additional metrics

77. You can cache data and it
will be available to later
jobs

78. You can also load objects and
they are available to later jobs
via NamedObject interface

79. Persistent context can be
run in seperate JVM or
within SJS

80. It does have some
sharp edges though...

81. Due to jvm classloader
contexts need to be restarted
on deploy to pick up new code

82. Some settings:
spark.files.overwrite = true
context-per-jvm = true
spray-can: parsing.max-content-length = 256m
spray-can: idle-timeout = 600 s
spray-can: request-timeout = 540 s
spark.serializer =
"org.apache.spark.serializer.KryoSerializer"
filedao vs sqldao backend
have to build from source/no binary for SJS

83. hive-site.xml
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:derby:memory:myDB;create=true</value>
<description>JDBC connect string for a JDBC
metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>org.apache.derby.jdbc.EmbeddedDriver</value>
<description>Driver class name for a JDBC
metastore</description>
</property>

84. SparkThrift Server
Extended/Reused hive
thrift server

85. I run the following on a persistent context:
sc.getConf.set("spark.sql.hive.thriftServer.singleSes
sion", "true")
sqlContext.setConf("hive.server2.thrift.port", port) //
port to run thrift server on
HiveThriftServer2.startWithContext(sqlContext)

86. Now I can connect using
hive-jdbc
odbc (microsoft or simba)

87. Run a job with joins/ or even just a
basic dataframe through
datasource api and
registerTempTable

88. val df =
sqlContext.read.options(Map("kudu.master"
-> "kudu.master:7051","kudu.table" ->
“kudu_table")).kudu
df.registerTempTable

89. You could also potentially
cache/persist via spark and
register that way assuming joins
are expensive

90. Now you can run queries
as if it was a traditional
database

91. Hey thats great, but how fast?
500 ms average response time
200 concurrent complex queries
1+ Billion rows with 200+ columns
sql queries with 5 predicates, min,max,count some
values and group by on 5 columns
No spark caching

92. We take this a step farther and do
complex dataframes and it is made
available as a registered temp
table

93. Questions… If we run out
of time send me
questions on slack