Доклад Мэтта Хиза на HighLoad++ 2014.

1. Scaling Micro-
Services in Go
Matt Heath, Hailo

5. PHP
API
Load Balancer
PHP
API
PHP
API
PHP
API
Java
App
Load Balancer
PHP
API
PHP
API
MySQL
Redis MySQL Slave
eu-west-1

6. ALL
HAIL
THE
MONOLITH

8. PHP
API
Load Balancer
PHP
API
PHP
API
PHP
API
Java
App
Load Balancer
PHP
API
PHP
API
MySQL
Redis MySQL Slave
eu-west-1

9. PHP
API
MySQL
PHP
API
PHP
Service
Java
Service
ELB
ELB ELB
PHP
API
PHP
API
per city multi-region
ELB
Cassandra
PHP
Service
MySQL MySQL
Java
App
Java
App
Java
App

10. PHP
API
MySQL
PHP
Service
Java
Service
ELB
ELB ELB
PHP
API
PHP
API
per city multi-region
ELB
Cassandra
PHP
Service
MySQL MySQL
Java
App
Java
App
Java
App
PHP
API

11. PHP
API
MySQL
PHP
Service
Java
Service
ELB
ELB ELB
PHP
API
PHP
API
per city multi-region
ELB
Cassandra
PHP
Service
MySQL MySQL
Java
App
Java
App
Java
App
PHP
API

12. Communication paths
with two programmers
Communication paths
with three programmers
Communication paths
with four programmers
Communication paths
with five programmers
Communication paths
with ten programmers

13. DEVELOPING NEW FEATURES
UNCLEAR RESPONSIBILITIES
SLOW FAILOVER
LACK OF AUTOMATION

15. “construct a highly agile and
highly available service from
ephemeral and assumed broken
components” - Adrian Cockcroſt

16. Cassandra
Go
Service
Go
Service
Java
Service
ELB
Go “Thin” API
Go
Service
us-east-1 eu-west-1
Go
Service
Java
Service
ELB
Go “Thin” API
RabbitMQ Message Bus
(federated clusters per AZ)
RabbitMQ Message Bus
(federated clusters per AZ)
Cassandra

18. Logic
Service
Handler
Storage

19. Handler
Library for building services that talk Protobuf via RMQ
Logic
Storage
Service
go-platform-layer

20. package handler
import (
"..."
)
func Foo(req *server.Request) (proto.Message, errors.Error) {
}

21. service.Register(
&service.Endpoint{
Name: "foo",
Mean: 100,
Upper95: 200,
Handler: handler.Foo,
}
)

22. Handler
Library for building services that talk Protobuf via RMQ
Logic
Storage go-service-layer
Self-configuring external service adapters
Service
go-platform-layer

23. package handler
import (
"github.com/hailocab/gomemcache/memcache"
)
func Foo(req *server.Request) (proto.Message, errors.Error) {
item, err := memcache.Get("key")
if err != nil {
// handle error
}
// ...
}

24. Handler go-platform-layer
Library for building services that talk Protobuf via RMQ
Logic
Storage go-service-layer
Self-configuring external service adapters
Services get for free:
• Provisioning
• Service discovery
• Configuration
• Monitoring
• Authentication/authorisation
• AB testing
• Self configuring connectivity
to third-party services
Service

28. CI Pipeline (Janky/Jenkins)
Amazon S3
Provisioning Service Provisioning Service Provisioning Service
Provisioning Manager / Scheduler

29. CI Pipeline (Janky/Jenkins)
Amazon S3
Provisioning Service Provisioning Service Provisioning Service
Provisioning Manager / Scheduler

30. AUTOMATIC SERVICE DISCOVERY
Provisioning Service
Rabbit MQ
Provisioning Service
Discovery
Service
New
Service

31. AUTOMATIC SERVICE DISCOVERY
Provisioning Service
Rabbit MQ
Discovery
Service
Binding
Service
Provisioning Service
New
Service

32. SMALL INDEPENDENT SERVICES
SINGLE RESPONSIBILITY
EASE PAIN, SCALE RAPIDLY

33. CLOUD NATIVE / ANTIFRAGILE
EXPECT FAILURE
AUTOMATE EVERYTHING

34. TESTING

35. LOAD
FAILURE
DEGRADATION

36. 15,000 JOBS/HOUR
50,000 DRIVERS
30,000+ REQ/S

37. CONTINUOUS
PRODUCTION
TESTING

38. MONITORING

39. AUTOMATIC HEALTHCHECK REGISTRATION
Provisioning Service
Rabbit MQ
Monitoring
Service
Provisioning Service
Service
Publish
Healthchecks

43. DISTRIBUTED
TRACING

44. hailo~2~api api.v1.customer service.customer
hailo~2~api api.v1.customer service.customer

45. hailo~2~api api.v1.customer service.customer
REQ
REP
REQ
REP
IN
OUT
IN
OUT
hailo~2~api api.v1.customer service.customer

46. {
"timestamp": 1410262798427145176,
"traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765",
"type": "OUT",
"messageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120",
"parentMessageId": "",
"from": "com.hailocab.hshell",
"to": "com.hailocab.service.nearest-driver.search",
"hostname": "ip-10-13-2-251",
"az": "eu-west-1a",
"handlerInstanceId": “server-com.hailocab.service.nearest-driver-18bd089e-8ef1-4ca1-75cb-8...c”,
"duration": 11222094
}
{
"timestamp": 1410262798416053450,
"traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765",
"type": "REQ",
"messageId": "6404dd1e-c995-48a9-73dc-9edb1380f0bf",
"parentMessageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120",
"from": "com.hailocab.service.nearest-driver",
"to": "com.hailocab.service.zoning.search",
"hostname": "ip-10-13-2-251",
"az": "eu-west-1a"
}

47. Phosphor
Host Instances
Publish
Service A
Trace Library
chan
goroutine
UDP
Service B
Trace Library
chan
goroutine
UDP
Trace
Service
Dashboards
Monitoring
In-memory
Aggregates
Optional
persistant
storage

48. var traceChan chan []byte
func init() {
// Use a buffered channel
traceChan = make(chan []byte, 200)
// Fire off a background worker for this channel
defaultClient = NewClient(traceChan)
go defaultClient.publisher()
}
// Send, drops trace if the backend is at capacity
func Send(msg []byte) {
select {
case traceChan <- msg:
// Success
default:
// Default case fired if channel is full
// Ensures this is non blocking
}
}

49. Tracing: 33eda743-f124-435c-71fc-3c872bbc98e6
2014-09-07 02:20:19.867 [/] [START] → -
2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-3-51] [REQ] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers -
2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-2-203] [IN] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers -
2014-09-07 02:20:19.868 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features -
2014-09-07 02:20:19.869 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features -
2014-09-07 02:20:19.876 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.feature-flags → com.hailocab.service.hob.list -
2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-168] [IN] com.hailocab.service.hob → com.hailocab.service.config.compile -
2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.service.feature-flags → com.hailocab.service.hob.list -
2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.hob → com.hailocab.service.config.compile -
2014-09-07 02:20:19.883 [eu-west-1a/ip-10-11-3-168] [OUT] com.hailocab.service.hob → com.hailocab.service.config.compile - 5.59 ms
2014-09-07 02:20:19.886 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.hob → com.hailocab.service.config.compile - 8.40 ms
2014-09-07 02:20:19.887 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 9.72 ms
2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 13.23 ms
2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 20.58 ms
2014-09-07 02:20:19.890 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 22.59 ms
2014-09-07 02:20:19.902 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.36 ms
2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 1.97 ms
2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search -
2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.10 ms
ERR - com.hailocab.service.fare.basefare: Missing config at xxx
2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.06 ms
ERR - com.hailocab.service.fare.basefare: Missing config at xxx
2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search -
2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search -
2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search -
2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 0.20 ms
2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 2.25 ms
2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch -
2014-09-07 02:20:19.912 [eu-west-1a/ip-10-11-3-227] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch -
2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 9.46 ms
2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime -
2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-227] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 7.58 ms
2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime -
2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 0.06 ms
2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 1.77 ms
2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 14.02 ms
2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 15.48 ms
2014-09-07 02:20:19.941 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated -
2014-09-07 02:20:19.945 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated -
2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 1.82 ms
2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 6.01 ms
2014-09-07 02:20:19.948 [eu-west-1a/ip-10-11-2-203] [OUT] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 80.46 ms
2014-09-07 02:20:19.950 [eu-west-1a/ip-10-11-3-51] [REP] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 82.71 ms

55. SO, GO?

56. SMALL
SIMPLE
EASY TO READ & LEARN

57. CONCURRENCY PRIMITIVES
INTERFACE SUPPORT

58. EASY DEPLOYMENT &
MANAGEMENT IN PRODUCTION
AND AT SCALE

59. СПАСИБО!

60. IMAGES
HMS President: Roger Marks
Orbital Ion Cannon: www.rom.ac
Go Gophers: Renee French
Utopia/Dystopia: Dylan Glynn
Control Room: NASA