Detecting Events on the Web in Real Time with Java, Kafka and ZooKeeper - James Stanier

Dr. James Stanier | Head of Analytics | Brandwatch.com | jamess@brandwatch.com

Coming Up/
• Me, Brandwatch and new problems
• Apache Kafka
• Processing data in Java
• Distributing work with Zookeeper
• Managing state
© 2014 Brandwatch | www.brandwatch.com 2

Who?

Dr. James Stanier
Head of Analytics, Brandwatch
@jstanier | jamess@brandwatch.com

Brandwatch

Where we are/
• Brighton
• New York
• San Francisco
• Berlin
• Stuttgart

What Brandwatch does/
Crawl Store and
Index
Analyse
3
Present
4
• Crawl 70M+
sites including
key social
networks
• 27 languages
• Powerful search
operators
• 20Bn + indexed
URLs
• Years of historical
data
• Automated topic &
sentiment analysis
in all 27
languages
• Automate
common tasks
including alerts
• Advanced
analytics modules
• Automatic
categorisation
with rules
• Custom
dashboards
• Reporting &
alerts

Brandwatch
Analytics

Data/ Aggregation

Data/ Classification

Data/ Not just top level metrics

Development/ What do we use?

Data/ The numbers
• 50+ Java Web Crawlers
• 10+ Historical crawlers for new queries
• Twitter via GNIP (now Twitter)
• 70M+ query matches per day

The speed of
social

A new challenge

21
The challenge/ The signal from the noise

The challenge/ at scale
• 100K+ user queries
• 70M+ mentions per day
• Polling the database for mentions would take
8hrs for one pass

The Problem/ How we handled it…
Crawler 1
Crawler 2
Crawler
n-1
Crawler
N
Kafka
Cluster
Signals
Signals
Signals
Processing
cluster
Signals
handler
JVM
DB
Mentions
Mentions

Kafka

Step 1/ Kafka
Crawler 1
Crawler 2
Crawler
n-1
Crawler
N
Kafka
Cluster Mentions

Kafka/ What is it?
• Apache Kafka is a publish-subscribe messaging
system rethought as a distributed commit log
• Apache top level project November 2013
• Started at LinkedIn

Kafka/ is…
• Fast: hundreds of MBs read/write per second
from thousands of clients
• Scalable: clustered, partitioned over many
machines, expanded without downtime
• Durable: messages persisted to disk and
replicated in cluster

Kafka/ Terminology
• Kafka maintains feeds of messages
called topics
• Programs that publish messages are
called producers
• Programs that subscribe to messages
are called consumers
• Kafka is a cluster of servers called brokers

Kafka/ How it’s used...
Producer Producer Producer
Kafka
Cluster
Consumer Consumer Consumer

Kafka/ Written to disk?
http://q.acm.org/detail.cfm?id=1563874

Kafka/ Bending, not breaking
http://engineering.gnip.com/tag/kafka/

Kafka/ The anatomy of a topic
0 1 2 3 4 5 6
0 1 2 3 4
0 1 2 3 4 5
Partition 0
Partition 1
Partition 2
Old New
Writes

Kafka/ Warning: ordering
Kafka guarantees a total ordering per partition,
not per whole topic

Kafka/ Try it out!
> tar -xzf kafka_2.9.2-0.8.1.tgz!
> cd kafka_2.9.2-0.8.1!
> bin/zookeeper-server-start.sh config/
zookeeper.properties!
> bin/kafka-server-start.sh config/
server.properties!
!

Kafka/ Try it out!
> bin/kafka-topics.sh --create --zookeeper
localhost:2181 --replication-factor 1 --partitions
1 --topic test!
> bin/kafka-console-producer.sh --broker-list
localhost:9092 --topic test!
Hello JAX!!
> bin/kafka-console-consumer.sh --zookeeper
localhost:2181 --topic test --from-beginning!
Hello JAX!!

Kafka/ With Java
<dependency>!
!<groupId>org.apache.kafka</groupId>!
!<artifactId>kafka_2.10</artifactId>!
!<version>0.8.1</version>!
</dependency>!

Kafka/ With Java
Properties props = new Properties();!
props.put("metadata.broker.list",
"broker1:9092,broker2:9092");!
props.put("serializer.class",
"kafka.serializer.StringEncoder");!
props.put("partitioner.class",
"example.producer.SimplePartitioner");!
props.put("request.required.acks", "1");!
ProducerConfig config = new ProducerConfig(props);!
Producer<String, String> producer = new
Producer<String, String>(config);!

Kafka/ Partitioning
public class SimplePartitioner implements Partitioner<String> {!
public SimplePartitioner (VerifiableProperties props) {!
!
public int partition(String key, int numberOfPartitions) {!
return md5hash(key) % numberOfPartitions;!
}!
}!

Kafka/ Sending from the crawlers
String message = toJson(...);!
KeyedMessage<String, String> message = new!
!KeyedMessage<String, String>("query.mentions",
!queryId, message);!
producer.send(message);!

Step 1/ Done
Crawler 1
Crawler 2
Crawler
n-1
Crawler
N
Kafka
Cluster Mentions

Processing

42
Processing/ What’s happening now?

Step 2.1/ One processing JVM
Crawler 1
Crawler 2
Crawler
n-1
Crawler
N
Kafka
Cluster Mentions
Signals
processor

Processing/ A wild tweet appears!
Mention
date: 14/10/2014 6:10PM
pageType: twitter
author: @javadude
hashtags: [#jaxlondon, #amazingtalk, #greatshoes]
mentionedTweeters: [@jstanier]
text: “@jstanier is at #jaxlondon tonight
#amazingtalk #greatshoes”

Processing/ Storing hashtags
Map<Date, Multiset<String>>!
!
Initialise with the last 24 hours

!
Mention
date: 14/10/2014 6:10PM

!
Mention
date: 14/10/2014 6:10PM
add(“#jaxlondon”)!
add(“#amazingtalk”)!
add(“#greatshoes”)!

Processing/ Cycling the buckets
@Scheduled(cron = "0 0 * * * *")!
public void cycleBuckets() {!
Date oldest = buckets.lastKey();!
removeBucket(oldest);!
DateTime newest = new!
! ! !DateTime(buckets.firstKey());!
addBucket(newest.plusHours(1).toDate());!
}!

Processing/ Detecting spikes
• At regular intervals
• For each hashtag
• Convert to a timeseries [5, …. 1002, 5499]
• Use our super secret detection algorithm
• Give a score to it
• If score > threshold, it’s interesting
• Send it on a new Kafka topic

Processing/ What we just did
#hashtag
data
model

Processing/ But we also track…
author
data
model
sentiment
data
model
page type
data
model
link share
data
model
#hashtag
data
model
country
data
model
volume
data
model

Processing/ …for one query
“JAX London” query
author
data
model
sentiment
data
model
page type
data
model
link share
data
model
#hashtag
data
model
country
data
model
volume
data
model

Processing/ 100K+ queries and rising

Processing/ We need more JVMs
But how do we share the workload?

Distribution
of work

Step 2.2/ A cluster of processing JVMs
Crawler 1
Crawler 2
Crawler
n-1
Crawler
N
Mentions Signals
© 2014 Brandwatch | www.brandwatch.com
Kafka
Cluster
Signals
Processing
cluster
Mentions

Distribution/ An atomic unit of work
© 2014 Brandwatch | www.brandwatch.com
Signals
Processing
cluster
?

Distribution/ Leader election
A way of deciding who is the leader for a task in
a group of distributed nodes

Distribution/ Zookeeper
A way of coordinating and managing distributed
applications

Zookeeper/ It’s like a file system
/brandwatch
/feature_1 /feature_2

Zookeeper/ At the command line

Distribution/ In Java
http://curator.apache.org/curator-framework

Distribution/ Recipes

Distribution/ Instantiating Curator
!
CuratorFrameworkFactory!
.builder()!
.connectString(zkQuorum)!
.namespace(namespace)!
.build()!
! ! ! !.start();!

Distribution/ Offering jobs
/brandwatch
/signals
/queries
/15846 /1268589
Manager
JVM
DB

Distribution/ Adding nodes
!
public void createZNode(String queryId) {!
try {!
client.create().forPath(ZK_NODE_PREFIX + queryId);!
} catch (NodeExistsException e) {!
log.debug("Node {} was already created.”, queryId);!
}!
}!

Distribution/ Deleting nodes
!
public void removeZNode(String queryId) {!
try {!
client.delete().forPath(ZK_NODE_PREFIX + queryId);!
} catch (NoNodeException e) {!
log.debug("Node {} was already deleted.”, queryId);!
}!
}!

Distribution/ Leader election 101
/brandwatch
/signals
/queries
/15846 /1268589
Processing
JVM 1
Processing
JVM 2
Processing
JVM 3

Distribution/ Leader election 101
/brandwatch
/signals
/queries
/15846 /1268589
1 2 3
Processing
JVM 1
Processing
JVM 2
Processing
JVM 3

Distribution/ The leader dies
/brandwatch
/signals
/queries
/15846 /1268589
2 3
Processing
JVM
Processing
JVM

Distribution/ The dead rises again
/brandwatch
/signals
/queries
/15846 /1268589
2 3 4
Processing
JVM
Processing
JVM
Processing
JVM

Distribution/ Curator: LeaderLatch
!
public LeaderLatch(CuratorFramework client, String latchPath)!
Parameters:!
!
client - the client!
latchPath - the path for this leadership group!

Distribution/ LeaderLatch recipe
public class WorkerManager implements PathChildrenCacheListener {!
!
private Map<Integer, LeaderLatch> leaderLatches = newHashMap();!
!
@Override!
public void childEvent(CuratorFramework client, !
PathChildrenCacheEvent event) {!
// Handle adds and removes here!!
}!
}!

Distribution/ Curator: Starting up
@PostConstruct!
public void initialise() throws Exception {!
List<ChildData> currentData = newArrayList(initialisePathChildrenCache());!
log.info("Pre creating workers for {} existing queries", currentData.size());!
for (ChildData childData : currentData) {!
int queryId = parseQueryIdFromPath(childData.getPath());!
startLeaderElection(queryId);!
}!
}!

Distribution/ Curator: PathChildrenCache
private List<ChildData> initialisePathChildrenCache() throws Exception {!
pathChildrenCache.start(StartMode.BUILD_INITIAL_CACHE);!
pathChildrenCache.getListenable().addListener(this);!
List<ChildData> currentData = pathChildrenCache.getCurrentData();!
return currentData;!
}!

Distribution/ Curator: Adding a node
@Override!
public void childEvent(CuratorFramework client, PathChildrenCacheEvent event) { !
ChildData childData = event.getData();!
switch (event.getType()) {!
case CHILD_ADDED:!
queryId = parseQueryIdFromPath(childData.getPath());!
if (!haveLeaderLatchForQuery(queryId)) {!
startLeaderElection(queryId);!
}!
break;!

Distribution/ Curator: Deleting a node
// Continued...!
case CHILD_REMOVED:!
queryId = parseQueryIdFromPath(childData.getPath());!
removeLeaderLatchForQuery(queryId);!
break;!
default:!
break;!
}!
}!

Distribution/ Almost there?
We are processing long running jobs
What about workers getting overloaded?

Distribution/ After leader election
1. Take leadership
2. Hit max queries?
a. No – go to 3
b. Yes – give up leadership, try again
3. Start working

Distribution/ Now we’re almost there?
Actually, no…

Distribution/ Infinite election
At capacity!
Processing JVM
Elected
for 1328
Elected
for 1328
At capacity! At capacity!
Elected
for 1328
Processing JVM Processing JVM

Distribution/ Solution
Processing JVM
Elected
for 1328
At capacity!
Elected
for 1328
Refused 1328

Distribution/ Solution
At capacity!
Processing JVM
Elected
for 1328
Elected
for 1328
Refused 1328
At capacity! At capacity!
Elected
for 1328
Refused 1328
Refused 1328

State

State/ CAP theorem
Availability
CA AP
CP

State/ Snapshotting of worker data
If one worker dies, we want the other to pick up
where it left off
Regular snapshotting to HBase

State/ Serialisation and compression
Serialise and compress using Kryo
~ 0.5MB per query, but a lot are very small

Step 2.2/ Done!
Crawler 1
Crawler 2
Crawler
n-1
Crawler
N
Kafka
Cluster
Signals
Signals
Signals
Processing
cluster
Signals
handler
JVM
DB
Mentions
Mentions

Monitoring

Monitoring/ Statsd and Graphite

Closing remarks

Summary/ Using this architecture
Now
• Smarter alerts (email, push, in-browser)
• Monitoring crises/events as they happen
Underway
• Automatic clustering of spikes into events
• Historical analysis of trends

Say hello/
jamess@brandwatch.com
UK: +44 (0)1273 358 635
@brandwatch | @jstanier
www.brandwatch.com

Q&A

Detecting Events on the Web in Real Time with Java, Kafka and ZooKeeper - James Stanier

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (10)

Ähnlich wie Detecting Events on the Web in Real Time with Java, Kafka and ZooKeeper - James Stanier

Ähnlich wie Detecting Events on the Web in Real Time with Java, Kafka and ZooKeeper - James Stanier (20)

Mehr von JAXLondon2014

Mehr von JAXLondon2014 (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Detecting Events on the Web in Real Time with Java, Kafka and ZooKeeper - James Stanier