Akka in Production: Our Story

Akka in Production
Our Story
Evan Chan
PNWScala 2013

Saturday, October 19, 13

Who is this guy?
• Staff Engineer, Compute and Data Services, Ooyala
• Building multiple web-scale real-time systems on top of
C*, Kafka, Storm, etc.
• github.com/velvia
• Author of ScalaStorm, Scala DSL for Storm
• @evanfchan

2

WANT REACTIVE?
event-driven, scalable, resilient and responsive

3

SCALA AND AKKA
AT OOYALA

4

COMPANY OVERVIEW
Founded in 2007
Commercially launch in 2009
230+ employees in Silicon Valley, LA, NYC,
London, Paris, Tokyo, Sydney & Guadalajara
Global footprint, 200M unique users,
110+ countries, and more than 6,000 websites
Over 1 billion videos played per month
and 2 billion analytic events per day
25% of U.S. online viewers watch video
powered by Ooyala

CONFIDENTIAL—DO NOT DISTRIBUTE

5

How we started using Scala
• Ooyala was a mostly Ruby company - even MR jobs
• Lesson - don’t use Ruby for big data
• Started exploring Scala for real-time analytics and MR
• Realized a 1-2 orders of magnitude performance boost
from Scala
• Today use Scala, Akka with Storm, Spark, MR,
Cassandra, all new big data pipelines


Ingesting 2 Billion Events / Day
Consumer watches
video

Storm

Nginx

Raw Log
Feeder

Kafka

New Stuff


Livelogsd - Akka/Kafka file tailer
Current
File

Coordinator

Rotated
File

File
Reader
Actor

File
Reader
Actor
Kafka

Kafka Feeder


Rotated
File 2

Storm - with or without Akka?
Kafka
Spout

• Actors talking to each other within a
bolt for locality
• Don’t really need Actors in Storm

Bolt

• In production, found Storm too
complex to troubleshoot

Actor
Actor


• It’s 2am - what should I restart?
Supervisor? Nimbus? ZK?

Akka Cluster-based Pipeline
Kafka
Consumer

Kafka
Consumer

Kafka
Consumer

Kafka
Consumer

Kafka
Consumer

Spray
endpoint

Spray
endpoint

Spray
endpoint

Spray
endpoint

Spray
endpoint

Cluster
Router

Cluster
Router

Cluster
Router

Cluster
Router

Cluster
Router

Processing
Actors

Processing
Actors

Processing
Actors

Processing
Actors

Processing
Actors


Lessons Learned
• Still too complex -- would we want to get paged for this
system?
• Akka cluster in 2.1 was not ready for production (newer
2.2.x version is stable)
• Mixture of actors and futures for HTTP requests
became hard to grok
• Actors were much easier for most developers to
understand


Simplified Ingestion Pipeline
Kafka
Partition
1

Kafka
SimpleConsumer

Kafka
Partition
2

Kafka
SimpleConsumer

• Kafka used to partition
messages
• Single process - super
simple!
• No distribution of data

Converter Actor

Converter Actor

Cassandra Writer
Actor

Cassandra Writer
Actor


• Linear actor pipeline very easy to understand

STACKABLE ACTOR TRAITS

13

Why Stackable Traits?
• Keep adding monitoring, logging, metrics, tracing code
gets pretty ugly and repetitive
• We want some standard behavior around actors -- but
we need to wrap the actor Receive block:
class someActor extends Actor {
def wrappedReceive: Receive = {
case x => blah
}
def receive = {
case x =>
println(“Do something before...”)
wrappedReceive(x)
println(“Do something after...”)
}
}


Start with a base trait...

trait
/**
*
*/
def

ActorStack extends Actor {
Actor classes should implement this partialFunction for standard
actor message handling
wrappedReceive: Receive

/** Stackable traits should override and call super.receive(x) for
* stacking functionality
*/
def receive: Receive = {
case x => if (wrappedReceive.isDefinedAt(x)) wrappedReceive(x) else unhandled(x)
}
}


Instrumenting Traits...
trait Instrument1 extends ActorStack {
override def receive: Receive = {
case x =>
println("Do something before...")
super.receive(x)
println("Do something after...")
}
}

trait Instrument2 extends ActorStack {
case x =>
println("Antes...")
super.receive(x)
println("Despues...")
}
}


Now just mix the Traits in....
• Traits add instrumentation; Actors stay clean!
• Order of mixing in traits matter
class DummyActor extends Actor with Instrument1 with Instrument2 {
def wrappedReceive = {
case "something" => println("Got something")
case x => println("Got something else: " + x)
}
}

Antes...
Do something before...
Got something
Do something after...
Despues...


PRODUCTIONIZING AKKA

18

Our Akka Stack
• Spray - high performance HTTP
• SLF4J / Logback
• Yammer Metrics
• spray-json
• Akka 2.x
• Scala 2.9 / 2.10


On distributed systems:
“The only thing that matters is
Visibility”

20

Using Logback with Akka
• Pretty easy setup
• Include the Logback jar
• In your application.conf:
event-handlers = ["akka.event.slf4j.Slf4jEventHandler"]

• Use a custom logging trait, not ActorLogging
• ActorLogging does not allow adjustable logging levels
• Want the Actor path in your messages?
•


org.slf4j.MDC.put(“actorPath”, self.path.toString)

Using Logback with Akka

trait Slf4jLogging extends Actor with ActorStack {
val logger = LoggerFactory.getLogger(getClass)
private[this] val myPath = self.path.toString
logger.info("Starting actor " + getClass.getName)
case x =>
org.slf4j.MDC.put("akkaSource", myPath)
super.receive(x)
}
}


Akka Performance Metrics
• We deﬁne a trait that adds two metrics for every actor:
• frequency of messages handled (1min, 5min, 15min
moving averages)
• time spent in receive block
• All metrics exposed via a Spray route /metricz
• Daemon polls /metricz and sends to metrics service
• Would like: mailbox size, but this is hard


Akka Performance Metrics

trait ActorMetrics extends ActorStack {
// Timer includes a histogram of wrappedReceive() duration as well as moving avg of rate
of invocation
val metricReceiveTimer = Metrics.newTimer(getClass, "message-handler",
TimeUnit.MILLISECONDS, TimeUnit.SECONDS)
case x =>
val context = metricReceiveTimer.time()
try {
super.receive(x)
} finally {
context.stop()
}
}
}


Performance Metrics (cont’d)


Flow control
• By default, actor mailboxes are unbounded
• Using bounded mailboxes
• When mailbox is full, messages go to DeadLetters
• mailbox-push-timeout-time: how long to wait
when mailbox is full
• Doesn’t work for distributed Akka systems!
• Real ﬂow control: pull, push with acks, etc.
• Works anywhere, but more work


Flow control (Cont’d)
• A working ﬂow control system causes the rate of all
actor components to be in sync.
• Witness this message ﬂow rate graph of the start of
event processing:


VisualVM and Akka
• Bounded mailboxes = time spent enqueueing msgs


VisualVM and Akka

• My dream: a VisualVM plugin to visualize Actor
utilization across threads


Tracing Akka Message Flows
• Stack trace is very useful for traditional apps, but for
Akka apps, you get this:
at akka.dispatch.Future$$anon$3.liftedTree1$1(Future.scala:195) ~[akka-actor-2.0.5.jar:2.0.5]
at akka.dispatch.Future$$anon$3.run(Future.scala:194) ~[akka-actor-2.0.5.jar:2.0.5]
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:94) [akka-actor-2.0.5.jar:2.0.5]
at akka.jsr166y.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1381) [akka-actor-2.0.5.jar:2.0.5]
at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259) [akka-actor-2.0.5.jar:2.0.5]
at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975) [akka-actor-2.0.5.jar:2.0.5]
at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479) [akka-actor-2.0.5.jar:2.0.5]
at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) [akka-actor-2.0.5.jar:2.0.5]

• What if you could get an Akka message trace?
--> trAKKAr message trace <-akka://Ingest/user/Super --> akka://Ingest/user/K1: Initialize
akka://Ingest/user/K1 --> akka://Ingest/user/Converter: Data


trait TrakkarExtractor extends TrakkarBase with ActorStack {
import TrakkarUtils._
val messageIdExtractor: MessageIdExtractor = randomExtractor
case x =>
lastMsgId = (messageIdExtractor orElse randomExtractor)(x)
Collector.sendEdge(sender, self, lastMsgId, x)
super.receive(x)
}
}

• Trait sends an Edge(source, dest, messageInfo) to a
local Collector actor
• Aggregate edges across nodes, graph and proﬁt!


Good Akka development practices
• Don't put things that can fail into Actor constructor
• Default supervision strategy stops an Actor which
cannot initialize itself
• Instead use an Initialize message
• Put your messages in the Actor’s companion object
• Namespacing is nice


PUTTING IT ALL TOGETHER

35

Akka Visibility, Minimal Footprint

trait InstrumentedActor extends Slf4jLogging with ActorMetrics with TrakkarExtractor
object MyWorkerActor {
case object Initialize
case class DoSomeWork(desc: String)
}
class MyWorkerActor extends InstrumentedActor {
def wrappedReceive = {
case Initialize =>
case DoSomeWork(desc) =>
}
}


Next Steps

• Name?
• Open source?
• Talk to me if you’re interested in contributing


THANK YOU
And YES, We’re HIRING!!
ooyala.com/careers


Akka in Production: Our Story

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Mehr von Evan Chan

Mehr von Evan Chan (15)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Akka in Production: Our Story