Our software needs to become reactive, this realization is widely understood: we need to consider responsiveness, maintainability, elasticity and scalability from the outset. Not all systems need to implement all these to the same degree, specific project requirements will determine where effort is most wisely spent, but in the vast majority of cases the need to go reactive will demand that we design our applications differently.
In this presentation we explore several architecture elements that are commonly found in reactive systems (like the circuit breaker, various replication techniques, or flow control protocols). These patterns are language agnostic and also independent of the abundant choice of reactive programming frameworks and libraries, they are well-specified starting points for exploring the design space of a concrete problem: thinking is strictly required!
8. Result: Maintainability & Fexibility
• decoupled responsibility—decoupled teams
• develop pieces at their own pace
• continuous delivery
• Microservices: Single Responsibility Principle
8
9. Implementation: Message-Driven
• focus on communication between
components
• model message flows and protocols
• common transports: async HTTP, *MQ, Actors
9
18. Circuit Breaker Pattern
• well-known, inspired by electrical engineering
• first published by M. Nygard in «Release It!»
• protects both ways:
• allows client to avoid long failure timeouts
• gives service some breathing room to recover
18
19. Circuit Breaker Example
19
private object StorageFailed extends RuntimeException
private def sendToStorage(job: Job): Future[StorageStatus] = {
// make an asynchronous request to the storage subsystem
val f: Future[StorageStatus] = ???
// map storage failures to Future failures to alert the breaker
f.map {
case StorageStatus.Failed => throw StorageFailed
case other => other
}
}
private val breaker = CircuitBreaker(
system.scheduler, // used for scheduling timeouts
5, // number of failures in a row when it trips
300.millis, // timeout for each service call
30.seconds) // time before trying to close after tripping
def persist(job: Job): Future[StorageStatus] =
breaker
.withCircuitBreaker(sendToStorage(job))
.recover {
case StorageFailed => StorageStatus.Failed
case _: TimeoutException => StorageStatus.Unknown
case _: CircuitBreakerOpenException => StorageStatus.Failed
}
21. Multiple-Master Replication Patterns
• this is a tough problem with no perfect solution
• requires a trade-off to be made between
consistency and availability
• consensus-based focuses on consistency
• conflict-free focuses on availability
• conflict resolution gives up a bit of both
• each requires a different programming model
and can express different transactional behavior
21
22. Consensus-Based Replication
• strong coupling between replicas to ensure
that all are “on the same page”
• unavailable during network outages or certain
machine failures
• programming model “just like a single thread”
• Postgres, Zookeeper, etc.
22
23. Replication with Conflict Resolution
• requires conflict detection
• resolution without user intervention will have to
discard some updates
• detection/resolution unavailable during partitions
• programming model “like single thread” with
caveat
• popular RDBMS in default configuration offer this
23
24. Conflict-Free Replication
• express updates such that they can be merged
• cannot express “non-local” constraints
• all expressible updates can be performed under
any conditions without losses or inconsistencies
• replicas may temporarily be out of sync
• different programming model, explicitly distributed
• Riak 2.0, Akka Distributed Data
24
27. Saga Pattern: Background
• Microservice Architecture means distribution
of knowledge, no more central database
instance
• Pat Helland:
• “Life Beyond Distributed Transactions”, CIDR 2007
• “Memories, Guesses, and Apologies”, MSDN blog 2007
• What about transactions that affect multiple
microservices?
27
28. Saga Pattern
• Garcia-Molina & Salem: “SAGAS”, ACM, 1987
• Bank transfer avoiding lock of both accounts:
• T₁: transfer money from X to local working account
• T₂: transfer money from local working account to Y
• C₁: compensate failure by transferring money back to X
• Compensating transactions are executed during
Saga rollback
• concurrent Sagas can see intermediate state
28
29. Saga Pattern
• backward recovery:
T₁ T₂ T₃ C₃ C₂ C₁
• forward recovery with save-points:
T₁ (sp) T₂ (sp) T₃ (sp) T₄
• in practice Sagas need to be persistent to
recover after hardware failures, meaning
backward recovery will also use save-points
29
30. Example: Bank Transfer
30
trait Account {
def withdraw(amount: BigDecimal, id: Long): Future[Unit]
def deposit(amount: BigDecimal, id: Long): Future[Unit]
}
case class Transfer(amount: BigDecimal, x: Account, y: Account)
sealed trait Event
case class TransferStarted(amount: BigDecimal, x: Account, y: Account) extends Event
case object MoneyWithdrawn extends Event
case object MoneyDeposited extends Event
case object RolledBack extends Event
31. Example: Bank Transfer
31
class TransferSaga(id: Long) extends PersistentActor {
import context.dispatcher
override val persistenceId: String = s"transaction-$id"
override def receiveCommand: PartialFunction[Any, Unit] = {
case Transfer(amount, x, y) =>
persist(TransferStarted(amount, x, y))(withdrawMoney)
}
def withdrawMoney(t: TransferStarted): Unit = {
t.x.withdraw(t.amount, id).map(_ => MoneyWithdrawn).pipeTo(self)
context.become(awaitMoneyWithdrawn(t.amount, t.x, t.y))
}
def awaitMoneyWithdrawn(amount: BigDecimal, x: Account, y: Account): Receive = {
case m @ MoneyWithdrawn => persist(m)(_ => depositMoney(amount, x, y))
}
...
}
33. Example: Bank Transfer
33
override def receiveRecover: PartialFunction[Any, Unit] = {
var start: TransferStarted = null
var last: Event = null
{
case t: TransferStarted => { start = t; last = t }
case e: Event => last = e
case RecoveryCompleted =>
last match {
case null => // wait for initialization
case t: TransferStarted => withdrawMoney(t)
case MoneyWithdrawn => depositMoney(start.amount, start.x, start.y)
case MoneyDeposited => context.stop(self)
case RolledBack => context.stop(self)
}
}
}
34. Saga Pattern: Reactive Full Circle
• Garcia-Molina & Salem note:
• “search for natural divisions of the work being
performed”
• “it is the database itself that is naturally partitioned into
relatively independent components”
• “the database and the saga should be designed so that
data passed from one sub-transaction to the next via
local storage is minimized”
• fully aligned with Simple Components and isolation
34
36. Conclusion
• reactive systems are distributed
• this requires new (old) architecture patterns
• … helped by new (old) code patterns &
abstractions
• none of this is dead easy: thinking is
required!
36