Underscore, Tel Aviv, May 2018

1. Writing Asynchronous
Programs with
&
Yardena Meymann, 15/5/2018
Underscore Tel Aviv

2. About Me
I’m a senior software developer, architect and trainer with
over 20 years experience in the industry, over decade in
Java and in recent 5 years almost exclusively Scala.
I worked on large enterprise projects at Cisco, HP and
VMware, then joined Feature.fm to build its backend
infrastructure from scratch.
I’m also a programming language enthusiast and one of the
organizers of Sayeret Lambda meet-up.

3. About
• Smart marketing & advertising for the music
industry
• Marketing suite: Smart Links, Pre-Save & Pre-
Order, Action Pages
• Cross-Channel Self-Serve Ad Platform: Streaming
services sponsored songs, music blog content,
audio ads, social ads
• Analytics & Insights
• Over 50,000 artists and labels, including all 3
major labels use Feature.fm

4. Feature.fm Backend
• ≈20 services (Scala, node.js, go), Docker on DC/OS
• API server and several UI servers
• MongoDB, ElasticSearch, Kafka, Druid
• Spark
• ≈1000 ad requests and events per second (peak time)
• Integrations with over a dozen external services

5. Aiming at
100% async
We don’t want to wait
for anything
By User Minesweeper on en.wikipedia - Minesweeper, CC BY-SA 3.0,
https://commons.wikimedia.org/w/index.php?curid=1302402

6. Async in Scala
• Future / Promise
• Akka Actors
• Akka Streams (impl. of Reactive Streams)
• Async/Await (successor of CPS)
• Monix
• ScalaZ Task
• …

7. Futures

8. Basic usage is simple

9. Error Handling
If you handle Failure(_) or implement recover() are you really
covered? What if your code throws an exception?
• callbacks are “covered”
• but first call may not be
• if a function returns a future is all the code there asynchronous?
• compiler magic will often happen on your thread
• example, reactive-mongo:
• collection.find(query).one[Info]
• def one[T](implicit reader: BSONReader[T],…): Future[Option[T]]
• trait BSONReader[B <: BSONValue, T] { def read(bson: B): T }

10. Error Handling
• If you write a method that returns a Future, convert all
failures to Future Failure
• If you take by-name parameters or functions, wrap
invocation in a Future fromTry Try(…)
• Better yet, do not assume that parameter functions are trivial,
allow them to be asynchronous
• Start with a trivial completed Future (Future.unit in 2.12)
• If you call a method you don’t 100% trust, wrap the call in
Try or try/catch

11. Some Problems
• What if a Future does not complete?
Someone will eventually time out, but they
may not know the cause

12. Some Problems
• How do you retry a future? multiple times?
Declining retries?

13. Some Problems
• Multiple futures
Future sequence Seq(f1, f2, f3, …)
• How many futures can/should you pass?

14. Actors

15. Actors
• Message passing
• Supervision
• Safe state management
• Can be distributed
• Mailbox can be persisted
… many other advantages

16. Akka Vocabulary
• ActorSystem - the world in which actors live, usually a singleton.
You can’t use akka without it.
• All our service classes have it as an implicit parameter, it allows us to
create actors and use akka utilities whenever we need
• ExecutionContext.global -> system.dispatcher
• ActorRef - stable actor reference
• Props - actor factory function, system.actorOf(props) -> ref
• ActorContext - like system, but inside an actor
• Scheduler - system.scheduler very useful addition

17. Limit Future Execution Time
with the Scheduler
import akka.pattern.after
import scala.concurrent.duration._
val system: ActorSystem = ???
implicit val ec = system.dispatcher
val breakOnTimeout: Future[Nothing] =
after(2.seconds, system.scheduler)
(Future failed new TimeoutException(…))
Future firstCompletedOf List(
/* your async call */,
breakOnTimeout
)

18. Retries
• You can use recoverWith
def retry[T](f: () => Future[T], delays: Seq[FiniteDuration])
(implicit ec: ExecutionContext, s: Scheduler): Future[T] = {
f() recoverWith {
case _ if delays.nonEmpty =>
after(delays.head, s)(retry(f, delays.tail)
}
}
• https://github.com/softwaremill/retry
does not depend on Akka

19. Actor <-> Future
• Actor message send to Future: Ask pattern (? operator)
• import akka.pattern.ask
• You will also need an akka.util.Timeout (explicit or implicit)
• Future to message send: Pipe pattern (pipeTo operator)
• import akka.pattern.pipe
• import context.dispatcher or system.dispatcher
• How to communicate failure?
• Handle akka.actor.Status.Failure
• Log whenever supervision gets involved

20. Testing Actors
• akka.testkit.TestKit
• ScalaTest: XSpecLike, DefaultTimeout, ImplicitSender,
BeforeAndAfterAll
• Mockito
• expectMsg(…)
• probe = TestProbe() can be used with dynamic actor hierarchies
to test interactions between actors
• With static hierarchies, allow to override props of child actor
to support DI (“Externalize child making from the parent”)
• https://github.com/miguno/akka-mock-scheduler

21. Measuring Actors
• Integrate with DropWizard (aka CodaHale) metrics
• https://github.com/erikvanoosten/metrics-scala
• Monitor error rates
• Monitor dead letters

22. Actor vs Future
• Future is one operation, Actors are usually more long-
lived, handling series of operations
• Use actors to model complex behaviors, to guard state
• Use actors to gain fine grained control over execution
• Beware of loosing type safety with Actors (or use Akka
Typed)
• Beware of “lost errors” and undetected actor flickering

23. Streams

24. Akka Streams
• Implementation of ReactiveStreams specification
• http://www.reactive-streams.org/
• Publisher -> Source
• Processor -> Flow
• Subscriber -> Sink
• We use streams for:
• Http Server (akka-http)
• Http Service Client (akka-http w/ connection pool)
• Kafka reading and writing (reactive-kafka)

25. Stream vs Future vs Actor
• Problem: Future sequence Seq(f1, f2, f3, …)
• Actor mailbox overflow
• Solution: Dynamic Push-Pull
• Push when consumer is faster
• Pull when producer is faster
• Switch automatically between the above
• Batching demand allows batching data
Publisher Subscriber
data
demand

26. Working with Streams
• Flow, Source, Sink
• Stages, built-in/custom
• Graph in a general case, often
a sequence
• Processed items go through
stages, finite/infinite flows
• Build a blueprint, run to trigger
materialization
• Implemented with Akka Actors

27. Many Built-in Stages
• Map, filter, reduce, groupBy
• Asynchrony level control
• Batching, buffering
• Rate-limiting, retries
• …

28. How Do You Start
• There is a steep learning curve, we are still learning as we go
• To start, get your implicits ready:
• implicit val system: ActorSystem = ???
• implicit val materializer = ActorMaterializer()
• implicit val context = system.dispatcher
• With Akka-Http, you also need (for example)
• implicit val serialization = jackson.Serialization
• implicit val formats = org.json4s.DefaultFormats ++
org.json4s.ext.JodaTimeSerializers.all + org.json4s.ext.UUIDSerializer + …

29. Handling Stream Errors
• Streams are materialized to Actors, so Supervision
.withAttributes(supervisionStrategy(Supervision.resumingDecider))
• Put it last!
• Recover tells you the error, but not the element that
caused it
• Wrap the processing in a Try and pass original element or
a key aside it through the stream (Input,Try[Output])

30. Keeping Streams Alive
• We have many infinite streams in our system
• While it’s easy to test inner parts of the stream, source or
sink may encapsulate external service connection and are
the components with high risk of failure
• We often hold stream reference under a “watchdog”
Actor, if the stream fails we kill the watchdog that restarts
and recreates the Stream afresh

31. Streams vs Actors
• Use streams:
• when you need back-pressure
• if you model asynchronous computation that can be composed
from existing building blocks (map, reduce, group-by, etc.)
• to reuse components - compose flows
• to interact with other reactive libs
• Alpakka
• Use Actors for distribution or persistence, advanced features
• StreamRefs experimental
• Use Actors for custom behavior

32. HTTP Server
• Streaming is mostly transparent for the implementer
• Http().bindAndHandle(routes, host, port)
• def bindAndHandle(handler: Flow[HttpRequest, HttpResponse,
Any], interface: String, port: Int …)
HttpRequest HttpResponse
Source SinkFlow
def routes: Route =
path("status") {
get {
onComplete((statusActor ? IsHealthy).mapTo[Boolean]) {
case Success(true) => complete(OK, Message(“Server is up"))
case Success(false) => complete(ServiceUnavailable, Message("Server is unavailable"))
case Failure(e) => complete(InternalServerError, Message("Server error"))
}
}
} ~ …

33. HTTP Server
• Request and Response bodies are Streams, marshaling
and un-marshaling asynchronous
• Request body
val rawEntity = extract(_.request.entity.dataBytes.runWith(Sink.reduce[ByteString](_ ++ _)))
• Response body
implicit val jsonStreamingSupport: JsonEntityStreamingSupport = EntityStreamingSupport.json()
pathPrefix("journal") {
path(Segment) { persistenceId =>
get {
complete(journal.eventsById(persistenceId))
} } }
def eventsById(id: String): Source[JObject, _] =
readJournal.eventsByPersistenceId(id, 0L, Long.MaxValue).map(e =>
("pId", e.persistenceId) ~ ("seqNr", e.sequenceNr) ~ …
).completionTimeout(1.minute)

34. HTTP Client
• Http().cachedHostConnectionPool[T](host, port):
Flow[(HttpRequest, T), (Try[HttpResponse], T), HostConnectionPool]
• T is the context object
• Measure time
• Associate request with response and return individual Futures
by using T = Promise[HttpResponse]

35. Kafka
• Producer
• Producer.plainSink(producerSettings)
• Producer.commitableSink(producerSettings)
• Producer.flow(producerSettings)
• Consumer
• Consumer.committableSource(consumerSettings, topics)
• Consumer.atMostOnceSource(consumerSettings, topics)
• CommittableOffsetBatch
• Consumer.plainSource(consumerSettings, subscription)
• Consumer.plainPartitionedManualOffsetSource(…)

36. Alpakka
New generation of EIP: stream-aware, reactive, integration
pipelines
• AMQP Connector
• Apache Geode connector
• Apache Solr Connector
• AWS DynamoDB Connector
• AWS Kinesis Connector
• AWS Lambda Connector
• AWS S3 Connector
• AWS SNS Connector
• AWS SQS Connector
• Azure Storage Queue Connector
• Cassandra Connector
• Elasticsearch Connector
• File Connectors
• FTP Connector
• HBase connector
• IronMq Connector
• JMS Connector
• MongoDB Connector
• MQTT Connector
• OrientDB Connector
• Server-sent Events (SSE) Connector Slick (JDBC) Connector
• Spring Web
• Unix Domain Socket Connector

37. Keeping Promises - HTTP
private val poolClientFlow = Http().cachedHostConnectionPool[Promise[HttpResponse]]("akka.io")
private val q =
Source.queue[(HttpRequest, Promise[HttpResponse])](QueueSize, OverflowStrategy.dropNew)
.via(poolClientFlow)
.toMat(Sink.foreach({
case ((Success(resp), p)) => p.success(resp)
case ((Failure(e), p)) => p.failure(e)
}))(Keep.left)
.run()
def handle(request: HttpRequest): Future[HttpResponse] = {
val responsePromise = Promise[HttpResponse]()
q.offer(request -> responsePromise).flatMap {
case QueueOfferResult.Enqueued => responsePromise.future
case QueueOfferResult.Dropped => Future.failed(new RuntimeException(“…”))
case QueueOfferResult.Failure(ex) => Future.failed(ex)
case QueueOfferResult.QueueClosed => Future.failed(new RuntimeException(“…”))
}
}

38. Keeping Promises - Kafka
private val q = Source
.queue[ProducerMessage.Message[String, Array[Byte], Promise[Long]]]
(Props(classOf[KafkaPublisherActor[Promise[Long]]]))
.via(Producer.flow(producerSettings))
.map(x => x.message.passThrough.complete(Try(x.offset)))
.to(Sink.ignore)
def send(topic: String, value: Array[Byte], key: String): Future[Long] = {
val responsePromise = Promise[Long]()
queue.offer(request -> responsePromise).flatMap {
case QueueOfferResult.Enqueued => responsePromise.future
case QueueOfferResult.Dropped => Future.failed(new RuntimeException(“…”))
case QueueOfferResult.Failure(ex) => Future.failed(ex)
case QueueOfferResult.QueueClosed => Future.failed(new RuntimeException(“…”))
}
}

39. Resources and Links
• https://akka.io/docs/
• https://blog.softwaremill.com/synchronous-or-
asynchronous-and-why-wrestle-with-
wrappers-2c5667eb7acf
• http://viktorklang.com/blog/
• https://gist.github.com/viktorklang/9414163
• https://www.slideshare.net/lutzh/20150702-
reactive-streams-tel-aviv
• http://arild.github.io/akka-workshop/
• http://kazuhiro.github.io/scala/akka/akka-http/
akka-streams/2016/01/31/connection-pooling-
with-akka-http-and-source-queue.html
• https://github.com/hseeberger/akka-
http-json
• https://blog.softwaremill.com/
measuring-response-time-in-akka-
http-7b6312ec70cf
• https://github.com/akka/alpakka
• http://blog.colinbreck.com/
• https://github.com/scala/scala-async,
https://github.com/monix/monix,
https://github.com/scalaz/scalaz
• https://github.com/ListnPlay/RiverSong

40. Register to the First Israeli
Functional Programming
Conference
http://www.flip-il.org/
July 23, 2018
Academic College
Tel-Aviv Yafo

41. Thank You!
WE ARE HIRING!