Microservices on Mesos & Netflix OSS @ Cloud Solution Day 2016

1. Microservices on Mesos & Netflix
OSS
By hieu@hoiio.com

2. Before we start
(or ground rules)
● Forgive my shaking voice
● Feel free to tell me I’m stupid
● Feel free to interrupt me for
question

3. Agenda
● About Hoiio Stack
● Phoenix Deployment
○ Mesos/Marathon
● Netflix OSS @ Hoiio
○ Service Discovery
■ Consul
■ Eureka
○ API Gateway
● (Monitoring)

4. About Hoiio

5. ● VoIP/SIP
● SMS/Email
● Connected Apps
● HR suite
Services

6. ● Distributed multi-service
Micro-services
● RabbitMQ as message broker
Peer-to-peer via HTTP
● Kafka as event-store
● JAVA/Python/... modules
● Entirely on AWS
Platform
zuul
auth
billingsip
HTTP

7. Phoenix Deployment
Mesos/Marathon

8. ● Abstract cluster of machines to
a single “black-box” machine
● Master nodes, Slave/Agent
nodes
● Tasks are submitted to master
● Master schedules job to one of
the slaves
Mesos

9. http://www.slideshare.net/spodila/prezo-tovmware-june2015

10. ● Framework running on top of
Mesos
● Manage tasks config, number of
instance,...
● Healthcheck
● REST interface
● Mesos as OS, Marathon as Task
Manager
Marathon

11. ● Framework running on top of
Mesos
● Manage tasks config, number of
instance,...
● Healthcheck
● REST interface
● Mesos as OS, Marathon as Task
Manager
Marathon
Mesos
Slave
Mesos
Master
Marathon
CPU/M
emory
Kernel
Schedu
ler
Task
Manager

13. ● Docker as container
○ Supported by Mesos
○ Use AWS ECR as private repo/
Private repo running on Marathon
● Marathon performs healthcheck
and replaces unhealthy
instances
● Replacement takes seconds!
Phoenix?

14. {
"id": "ms-uat-xxx",
"mem": 384,
“cpu”: 0.5,
...

16. Service Discovery
Netflix Eureka

17. ● Eureka Server & Client
● Server route are replicated
● Each Client hold a copy of route
table
● Route table are updated in
background
https://github.com/Netflix/eureka/wiki/
Eureka-at-a-glance

18. ● Eureka
○ Eureka server tracks which service
is running where (which ip and
port?)
○ All records are replicated to all
eureka-clients
● Ribbon
○ Pick a server from records replica
on local eureka-client
○ Make request to picked server
○ Retry if configured
10.0.12.16:1234 10.0.140.21:4321
10.0.140.26:6789
Eureka
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
10.0.12.16:1234
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
R
10.0.140.21:4321
10.0.140.21:4321

19. Auth1
Routes
● Single-point-of-failure? Not
really
○ Route table are replicated
○ Each client has a copy
○ Routes are queried from local copy
● When Eureka is down
○ New servers are not updated
○ Might call to a dead server ->
retry on local server list with
Ribbon
SIP
Auth2
HTTP
Routes
Eureka
Server
Routes
Routes

20. String moduleVipAddress = "call.hoiio.info"
Observable<HttpResponse> response = HoiioRibbonRequest.getInstance().makeRequest(
moduleVipAddress,
UUID.randomUUID().toString(),
httpRequest);

21. ● Timeout and Retry
○ Defined in HoiioRibbonRequest
○ Default:
■ Timeout: 10s
■ Retry:
● Same server: 0
● Next server: 3
○ Can be re-configured
10.0.12.16:1234
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
R
10.0.140.21:4321
10.0.140.21:4321

22. HttpClient httpClient;
RetryPolicy retryPolicy;
String moduleVipAddress = "call.hoiio.info"
Integer sameServerRetry = 1
Integer nextServerRetry = 1
retryPolicy = new RetryPolicy(
new DefaultLoadBalancerRetryHandler(
sameServerRetry,
nextServerRetry,
true
)
)
}
Integer timeout = 60;
httpClient = new HttpClient(500, 50, timeout*1000);
Configuration config = new Configuration(moduleVipAddress, httpClient, retryPolicy);
Observable<HttpResponse> httpResponse = HoiioRibbonRequest.getInstance().makeRequest(
config,
correlationId,
httpRequest);

23. Service Discovery
Consul

24. ● Clustering with agent on each
instance
● Service info is shared in cluster
● Agent has REST interface to
register/deregister/checks/quer
y/…
● Zuul-pronted as primary
reversed proxy
Implementation
service.json
service.json
Zuul

26. HoiioConsulLoadBalancer lb = new HoiioConsulLoadBalancer(appName, ConsulService.Info.environment(), tag);
HttpResponse httpResponse;
try {
httpResponse = lb.execute(new HttpCmd(httpRequest))
} catch (NoServerException ignored) {
ZuulLogger.logger.error("No server for " + appName)
httpResponse = responseFactory.get().newHttpResponse(
new BasicStatusLine(HttpVersion.HTTP_1_1, 503, "Service not available"),
null);
}

27. API Gateway
with Netflix Zuul and Archaius

28. ● Single gateway for API
● API mapping for easy
understanding
● Optimize number of request
called
● Reject malformed request
Problems
sms
auth
billingsip
HTTP

29. ● Why Zuul?
○ Apps does not have Eureka Client
○ Cron jobs
○ Exposing API
● What Zuul does
○ Represent API caller (Apps,
Cronjob, Partner,...) to talk to
modules (act as a proxy)
■ Relay request
■ Retry
○ Authenticate request
10.0.12.16:1234
10.0.140.21:4321
10.0.140.26:6789
Eureka
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
Z
Z
/a/b/c
10.0.140.26:6789
10.0.12.16:1234
/a/b/c -> /a/c
/a/c
Microservice

30. ● Pre, Route, Post Filter
○ Groovy filter
○ Has priority
● Integrate with Archaius for
Dynamic configuration
● Integrate with Eureka/Consul
for service discovery
Netflix Zuul
Reject
malformed
Authenticate
Route using
Eureka
Ribbon/
Eureka
Add header
pre
route
post
Archaius
Route mapping
/sms/send /sms/send ->
{“module”:”sms”,
“uri”:”sendOneSms”}
/sendOneSms

31. ● Timeout and Retry
○ Zuul represents API callers to talk
to modules -> must tell Zuul
timeout and retry for each API
○ Default values
■ Timeout: 10s
■ Retry:
● Same server: 0
● Next server: 3
{
"vipAddress": "auth.hoiio.info",
"module": "auth",
“apis”: [
{
"from":"/v1/otp",
"to": "/private/v1/otp",
"type": "private",
"timeout": 60,
"retry": {
"same": 1,
"next": 2
}
}
]
}

34. Monitoring

35. ● Remember Consul?
● Consul watch
○ Trigger action when a service
status changes
Service status
service.json
{
"service": {
"name": "MS-Apps-1-46",
"tags": ["prod"],
"address": "10.0.14.10",
"port": 8080,
"checks": [
{
"script":
"/opt/consul/bin/MS-Apps-1-46-healthcheck.sh",
"interval": "60s"
}
]
}
}

36. ● Metric sources:
○ CollectD/cAdvisor
○ Cloudwatch
● Metric storage:
○ InfluxDB
● Visualization:
○ Grafana
Instance stats
Kapacitor
Cloudwatch
CAS
slack
/sms

38. Thank you!

39. We are hiring!
● Fresh web engineer
● Senior web engineer
● Internship