How microservices fail, and what to do about it

How Microservices Fail…
and what to do about it.
Richard Rodger
@rjrodger

github.com/rjrodger/nodezoo
https://aws.amazon.com/message/5467D2/

Pattern Matching
Service discovery is an anti-pattern.
Instead, make messages first-class citizens.
Use message data to define patterns,
and these patterns define a language!

Transport Independence
Services should not know about each other,
or how to send messages.
Services are fully defined by:
message patterns that they recognise, and
message patterns that they emit.

// a search message
{
"role": "search", // a namespace
"cmd": "search", // this is a command
"query": "ldap", // some data
}
!
// the pattern to match
role:search,cmd:search

// some nodezoo message patterns
!
role:search,cmd:search // do a search
role:search,cmd:insert // insert into index
role:info,cmd:get // get module info
role:npm,cmd:get // get npm data
role:npm,info:change // module changed!
role:info,req:part // need module info
role:info,res:part // here's module info
!
!
!

role:search,cmd:search
role:info,cmd:get
synchronous request/response

role:npm,info:change
asynchronous "winner-take-all"
(actor)

role:info,req:part
asynchronous "ﬁre-and-forget"
(publish/subscribe)
role:info,res:part

(publish/subscribe)
(actor)
synchronous "sidewinder"
(side effects!)
synchronous/
asynchronous
consumed/
observed

senecajs.org

code
(branch: msdub201509)

kintsugi ⾦金継ぎ
(golden joinery)

(publish/subscribe)
(actor)
(side effects!)
How do these break?

(publish/subscribe)
(actor)
(side effects!)

failure mode
"Slow downstream"
B responses are getting slower,
consuming As resources.
mitigation
Drop B.
A should consider B dead.
Or: the transport should handle this.

failure mode
"Upstream overload"
A is sending messages to B
at a higher rate than B can handle.
mitigation
Back-pressure from B.
A should accept back pressure
notiﬁcations and scale back.
Assumes B is doing more work than A.

failure mode
"Lost Actions"
A is sending messages to B,
and C is listening. But perhaps the
latest version of C is broken?
mitigation
Measure message ﬂow rates.
Do the ﬂow ratios match
the business rules?

failure mode
"Broken Contracts"
A and B are using a newer
message schema,
but you forgot about C.
mitigation
Don't use contracts!
Message schemas are a
net negative and hinder
multi-version deployment.

failure mode
"Poison Message"
A is sending messages that crash B.
B keeps restarting and trying to handle
the poison message. Now nothing works!
mitigation
B should drop out of date messages
on the ﬂoor. B should maintain a list of
recently seen messages and ignore duplicates.
B sends bad messages to the "Dead Letter" log.

failure mode
"Guaranteed Delivery ... ain't"
B expects at-most-once, exactly-one,
or at-least-once delivery of
unique messages. This is not possible.
mitigation
Idempotency.
Where possible, duplicate messages
should have no bad effects.

failure mode
"Emergent Behaviour"
Strange loops and unexplained
message paths make it hard to understand
what the system is doing.
mitigation
Correlate messages.
Attach correlation identiﬁers to messages
so that you can trace the ﬂow of causality.

failure mode
"Catastrophic Collapse"
You've introduced feedback
that grows exponentially.
And you've no idea how to ﬁx it.
mitigation
Have a kill switch.
Microservices aren't a
silver bullet. Sometimes you
need to selectively reboot.

P( success ) = 1
P( failure ) < ε
vs.

// apoptosis
setTimeout(function(){
process.exit(0)
}, 60*60*1000*Math.random())
!
!
!
!
!
!
!
!
!

Thanks!
Richard Rodger
@rjrodger

How microservices fail, and what to do about it

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Andere mochten auch

Andere mochten auch (7)

Ähnlich wie How microservices fail, and what to do about it

Ähnlich wie How microservices fail, and what to do about it (20)

Mehr von Richard Rodger

Mehr von Richard Rodger (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

How microservices fail, and what to do about it