Mesos has continued to add support for a new, easy to use framework API: the HTTP API. In this talk we first provide an overview of the different parts of the HTTP API including scheduler, executor, and operator API. Next we provide insight in how the new API can be used by framework writers with best practices and common pitfalls experienced while implementing HTTP API support for ArangoDB which is among the first production frameworks to utilize this new feature. From this experience we present guidelines on how to change a framework over from the old API to the new HTTP API and compare both implementations side to side to show the differences (or different usage patterns) of both APIs. This talk is equally useful for framework authors who directly want to start from scratch with the new HTTP API. Video recording: https://www.youtube.com/watch?v=o6RQlT35mmo&index=3&list=PLbzoR-pLrL6pLSHrXSg7IYgzSlkOh132K

1. HTTP API in Action
Max Neunhöffer, ArangoDB
Jörg Schad, Mesosphere

2. © 2016 Mesosphere, Inc. All Rights Reserved. 2
Jörg Schad
Distributed Systems Engineer,
Mesosphere
@joerg_schad
Max Neunhöffer
Senior Software Architect
ArangoDB
@neunhoef

3. © 2016 Mesosphere, Inc. All Rights Reserved. 3
HTTP v1 APIs
● Mesos 1.0 Milestone
● Actually multiple APIs
○ Framework
■ Scheduler
■ Executor
○ Operator
● RPC-based HTTP API
● Language independent
schedulers!

4. © 2016 Mesosphere, Inc. All Rights Reserved. 4
In the beginning
there was libmesos

5. © 2016 Mesosphere, Inc. All Rights Reserved. 5
The old
days…
● Not portable
● Hard to
debug
● Upgrade
Dependency
● No
Versioning
Scheduler
Java
Library
Native
Library
mesos.jar libmesos.so
Mesos Master

6. © 2016 Mesosphere, Inc. All Rights Reserved. 6
Networking
● Container?
● Firewall
Scheduler
Java
Library
Native
Library
mesos.jar libmesos.so
Mesos Master

7. © 2016 Mesosphere, Inc. All Rights Reserved. 7
Let us break down
walls….

8. © 2015 Mesosphere, Inc. All Rights Reserved.
Mesos APIs
8
Scheduler Master Agent Executor
Scheduler API Executor API
Framework API
Master API Agent API
Operator
Operator API
Internal API

9. © 2016 Mesosphere, Inc. All Rights Reserved. 9
But then business
demanded
FAST DATALet us build something new!

10. © 2016 Mesosphere, Inc. All Rights Reserved. 10
API V1 Goals
● Work inside firewalls / containers
● Allow pure language client libraries
● API versioning
● Backwards compatible API evolution
● Well documented

11. FRAMEWORK PROTOCOL
• Every call is a HTTP POST request
• Content-Type: application/json or application/x-protobuf
• Successful SUBSCRIBE call results in a “200 OK” streaming response
• Record-IO formatted events
• Persistent connection
• All successful non-SUBSCRIBE calls result in “202 Accepted”
•

12. WORKFLOW
Scheduler Master
SUBSCRIBE

13. POST /api/v1/scheduler HTTP/1.1
{
"type" : "SUBSCRIBE",
"subscribe" : {
"framework_info" : {
"user" : "foo",
"name" : "Example HTTP Framework"
}
}
}
WORKFLOW
SUBSCRIBE
Scheduler Master

14. WORKFLOW
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Mesos-Stream-Id: 130ae4e3-6b13-4ef4-baa9-9f2e85c3e9af
<event length>{
"type": "SUBSCRIBED",
"subscribed": {
"framework_id": {
"value": "12220-3440-12532-2345"
},
"heartbeat_interval_seconds": 15
}
}<more events>
<more events>
SUBSCRIBE
Streaming Response
MasterScheduler

15. WORKFLOW
Scheduler Master
Offers Event
<event-length>
{
"type" : "OFFERS",
"offers" : [
{
"offer_id":{"value": "12214-23523-O235235"},
"framework_id":{"value": "12124-235325-32425"},
"agent_id":{"value": "12325-23523-S23523"},
"hostname":"agent.host",
"resources":[...],
"attributes":[...],
"executor_ids":[]
}
]
}
Agent

16. WORKFLOW
Scheduler Master
ACCEPT
POST /api/v1/scheduler HTTP/1.1
Mesos-Stream-Id: 130ae4e3-6b13-4ef4-baa9-9f2e85c3e9af
{
"framework_id" : {"value" : "12220-3440-12532-2345"},
"type" : "ACCEPT",
"accept" : {
"offer_ids" : [
{"value" : "12220-3440-12532-O12"},
{"value" : "12220-3440-12532-O12"}
],
"operations" : [ {"type" : "LAUNCH", "launch" : {...}} ],
"filters" : {...}
}
}
Agent

17. Scheduler Master
WORKFLOW
ACCEPT
Response
HTTP/1.1 202 Accepted
Agent

18. WORKFLOW
Scheduler Master
ACCEPT
Launch Task
Agent

19. WORKFLOW
Scheduler Master
Launch Task
Executor
SUBSCRIBE
Agent
ACCEPT

20. SCHEDULER DISCONNECTIONS & PARTITIONS
• Master tracks the persistent subscription connection
• Reconnect within framework failover timeout
• Periodic HEARTBEAT events sent by master

21. Operator API
• Master and Agent API
• Human Operator
•
• Event Stream
• Avoid Polling /state_summary
• RecordIO format
•
21
POST /api/v1/executor HTTP/1.1
Host: agenthost:5051
Content-Type: application/json
Accept: application/json
{
"type": "SUBSCRIBE",
}

22. Operator API
• Master and Agent API
• Human Operator
•
• Event Stream
• RecordIO format
•
22
<event-length>
{
"type": "TASK_UPDATED",
"task_updated": {
"task_id": {
"value": "42154f1b-adcd-4421-bf13-8bd11adfafaf"
},
"framework_id": {
"value": "49154f1b-8cf6-4421-bf13-8bd11dccd1f1"
},
"agent_id": {
"value": "2915adf-8aff-4421-bf13-afdafaf1f1"
},
"executor_id": {
"value": "adfaf-adff-2421-bf13-adf23tafa21"
},
"state" : "TASK_RUNNING"
}
}

23. Versioning
• Corresponding major versions
• API compatibility
• Protobuf like
http://localhost:5050/api/v1/scheduler
http://localhost:5051/api/v2/executor

24. API STATUS
• Scheduler API & Executor API : Stable
• Operator API : Experimental
• Client library implementations

25. © 2015 Mesosphere, Inc. All Rights Reserved. 25
● multimodel
Database
● extensible

26. © 2016 Mesosphere, Inc. All Rights Reserved. 26
Writing a framework scheduler for a stateful service is hard:
● A part of the scheduling happens in the framework (accepting offers).
● Getting the scheduling logic right is difficult
● A framework needs to be resilient (use zookeeper)
● The ArangoDB framework consists of 5140 SLOC in C++
● It handles
● deployment,
● dealing with persistent volumes,
● a part of the failover, and
● a part of the up and down scaling.
ArangoDB Mesos Framework

27. © 2016 Mesosphere, Inc. All Rights Reserved. 27
libmesos makes it challenging to develop frameworks:
● The framework executable is usually deployed as Docker image.
● There are two connections needed between framework and Mesos master in
different directions (network isolation problems).
● One is limited to C++, Java or Python.
● The libmesos source is huge: 190K SLOC
● A Docker image to build a framework with libmesos needs >3GB
● Documentation is mostly hidden in the source.
● Debugging is a challenge (Docker, Mesos, binary messages, network issues).
Motivation ArangoDB

28. © 2016 Mesosphere, Inc. All Rights Reserved. 28
Solution: Use a SDK that works via the HTTP protocol.
The HTTP protocol provides the following advantages:
● any language works that can to HTTP/JSON
● JSON is clear-text and easier to debug
● using an SDK removes the amount of boilerplate code
● do not need libmesos
● much smaller footprint
Prototype: Mesos framework SDK for node.js
Still: scheduling and failure management
HTTP based ArangoDB
Writing Stateful Frameworks - Ken Sipe & Jörg Schad
Saturday, 10:25

29. © 2016 Mesosphere, Inc. All Rights Reserved. 29
Challenges
● State Abstraction missing (MESOS-2916)
● Mesos-framework
● Persistent Volumes
● Failover
● Scaling
● Operations (e.g., Updates)

30. © 2016 Mesosphere, Inc. All Rights Reserved. 30
● Shim
● Switch between old/new API
● Independent code
● Mesos-framework
● JS based SDK
● Not production!
● DC/OS SDK
How to get started?
var scheduler = new Scheduler({
"masterUrl": "172.17.10.103", // If Mesos DNS is used this would be
"leader.mesos", otherwise use the actual IP address of the leading master
"port": 5050,
"frameworkName": "My first Command framework",
"logging": {
"level": "debug" // Set log Level to debug (default is info)
},
"restartStates": ["TASK_FAILED", "TASK_KILLED", "TASK_LOST",
"TASK_ERROR", "TASK_FINISHED"], // Overwrite the restartStates (by default,
TASK_FINISHED tasks are NOT restarted!)
"tasks": {
"sleepProcesses": {
"priority": 1,
"instances": 3,
"commandInfo": new Mesos.CommandInfo(
"sleep 10;"
),
"resources": {
"cpus": 0.2,
"mem": 128,
"ports": 1,
"disk": 0
}}}
Writing Stateful Frameworks - Ken Sipe & Jörg Schad
Saturday, 10:25

31. © 2016 Mesosphere, Inc. All Rights Reserved. 31
● Use libraries
● Shim for easy switching
● Persistent Connection
● Connection: keep-alive
● Transfer-Encoding: chunked
● HTTP Pipelining
● React to heart beats
● Implement authentication schema
Best Practices

32. © 2016 Mesosphere, Inc. All Rights Reserved. 32
Thank You!
Learn more by visiting dcos.io and arangodb.com.

33. © 2016 Mesosphere, Inc. All Rights Reserved.
THANK YOU!
33