Paris API meetup #30 @Dailymotion HQ - "Using Go to build a REST API: yes, it’s a good match!" by Vincent BEHAR (https://twitter.com/vbehar) and Mina GENDI (https://github.com/m-gendi), Senior backend developers at Dailymotion. [Recorded Video]: http://www.dailymotion.com/video/x6890ra Fast, easy to write/read and deploy, we know all that. But Go is also DSL friendly, so we’ll see how we can write a custom DSL to define our endpoints, and test them with BDD-style scenarios. We will also see how we can handle versioning, and the gotchas to avoid. And we’ll finish with some monitoring love, of course. Paris API meetup #30 link: https://www.meetup.com/ParisAPI/events/243855587/?_locale=fr-FR

1. © 2017
Using Go to build a REST API

2. © 2017
Who we are
Vincent BEHAR
twitter.com/vbehar
Mina GENDI
github.com/m-gendi

3. © 2017
Agenda
• Our project
• First challenges
• REST Endpoints
• API Documentation
• Testing
• API versioning
• Monitoring

4. © 2017
• Programmatic Advertising
• Complex & rich domain
• Full control over programmatic chain
• From scratch
• Link publishers with advertisers
OUR PROJECT
AdTech Platform
• REST API
• Used to generate all the rules for the ad-servers
• 239 endpoints
• 54 main entity types
• Could have been any language
• So what were the benefits of using Go ?
API

5. © 2017
FIRST CHALLENGES
Start easy
func getPublishers(w http.ResponseWriter, r *http.Request) {
publishers, _ := store.LoadPublishers(...)
json.NewEncoder(w).Encode(publishers)
}
• Write simple http.HandlerFunc
• Lots of copy/paste when it gets more complex:
• Error handling
• Pagination
• Filtering
• Sorting
• Logging
• Monitoring
• …

6. © 2017
FIRST CHALLENGES
Keep It DRY
• Identify common use-cases:
• GET Collection (54)
• GET Resource (49)
• POST Resource (30)
• PUT Resource (28)
• DELETE Resource (27)
• …
• Write a helper func per use-case
func getPublishers(w http.ResponseWriter, r *http.Request) {
return getCollection(w, r, "publisher", &models.Publisher{})
}
func getPublisher(w http.ResponseWriter, r *http.Request) {
return getResource(w, r, "publisher", &models.Publisher{})
}

7. © 2017
FIRST CHALLENGES
• Same use-case, but different behaviour
• Parameters hell
func getPublishers(w http.ResponseWriter, r *http.Request) {
return getCollection(w, r, "publisher",
&models.Publisher{}, true, "", false, nil, "", "", nil, true)
}
But…

8. © 2017
SOLUTIONS
Options Struct
func getPublishers(w http.ResponseWriter, r *http.Request) {
return getCollection(w, r, GetCollectionOptions{
Type: "publisher",
Model: &models.Publisher{},
...
}
}
• Extensible
• Readable

9. © 2017
SOLUTIONS
Functional Options
func getPublishers(w http.ResponseWriter, r *http.Request) {
return getCollection(w, r,
withType("publisher"),
withModel(&models.Publisher{}),
…)
}
• https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis
• Options are functions that operates on some internal objects used by getCollection
• Extensible

10. © 2017
OUR SOLUTION
Composite Literal
// GetPublishers is a REST endpoint that retrieves all publishers
var GetPublishers = Endpoint{
For: http.MethodGet,
At: "/v1/publisher",
HandledBy: GetCollection{
Of: "publisher",
Using: sliceOf(models.Publisher{}),
With: publishersLoader,
},
}
• All in 1 place: path, method, handler
• Declarative
• Easy to read
• Extensible
• It’s just Go

11. © 2017
REST ENDPOINTS
Implementation
type Endpoint struct {
For string // method
At string // path
HandledBy http.Handler
}
func (ep Endpoint) ServeHTTP(w http.ResponseWriter, r *http.Request) {
ep.HandledBy.ServeHTTP(w, r)
}
• Endpoint is an http.Handler
• Just a wrapper around another http.Handler

12. © 2017
REST ENDPOINTS
Handlers
type GetCollection struct {
Of string
ParsedBy CollectionRequestParser
Using ModelFactory
With CollectionLoader
}
func (h GetCollection) ServeHTTP(w http.ResponseWriter, r *http.Request) {
model := h.Using()
if h.ParsedBy == nil {
h.ParsedBy = paginationParser.And(sortingParser).And(filterParser)
}
if h.With == nil {
h.With = defaultCollectionLoader
}
[...]
}

13. © 2017
REST ENDPOINTS
Handlers
type PostResource struct {
Of string
Using ModelFactory
ValidatedBy Validator
With ResourceStorer
}
func (h PostResource) ServeHTTP(w http.ResponseWriter, r *http.Request) {
model := h.Using()
err := json.NewDecoder(r.Body).Decode(model)
validationErr, err := h.ValidatedBy.validate(ctx, model)
h.With(ctx, StoreItemOptions{ TableName: h.Of }, model)
[...]
}

14. © 2017
REST ENDPOINTS
Example
var GetSitesByPublisher = Endpoint{
For: http.MethodGet,
At: "/v1/publisher/{pubid:[0-9]+}/site",
HandledBy: GetNestedCollection{
Of: "site",
Using: sliceOf(models.Site{}),
With: nestedSitesLoader,
From: Parent{
Type: "publisher",
MappedBy: "pubid",
Using: instanceOf(models.Publisher{}),
With: nonSoftDeletedResourceLoader,
},
},
}

15. © 2017
API DOCUMENTATION
• Manually written
• Generated using apidocjs.com
• Lots of duplicate information
• API Documentation is often outdated
• Everything is already written and accessible from our Endpoints
• Couldn’t it be re-used ?
Current status

16. © 2017
API DOCUMENTATION
Kubernetes API Documentation
https://github.com/kubernetes/api/blob/master/core/v1/types.go
https://kubernetes.io/docs/api-reference/v1.8/#pod-v1-core

17. © 2017
API DOCUMENTATION
• API Documentation Generation
• Use Go stdlib to parse the source code
• go/ast, go/doc, go/parser, go/token
• Use reflection to retrieve the « models »
• Use godoc for both internal doc and API doc
• Use request/response fields from internal models (Go Struct)
• Generate swagger, markdown, …
Future

18. © 2017
INTEGRATION TESTING
• Lots of copy/paste
• Un-friendly error messages:
• We can do better
Who cares about tests ?
--- FAIL: TestGetAdCategories (0.03s)
ad_category_test.go:44: Expected 8 Got 7

19. © 2017
INTEGRATION TESTING
BDD-style scenarios
Scenario{
Name: "Load all Ad Categories",
Given: IHave{
RESTCollection{
Of: "ad_category",
WithAPIPrefix: "/v1",
},
},
When: IDo{
GetCollection{
At: httpServer.URL,
},
},
Then: ItShould{
BeAnHTTPResponse{
WithStatusCode: http.StatusOK,
WithBody: MatchingJSON{
jsonValueAt("result.#").EqualToValue(26),
jsonValueAt("result.0.childId").EqualToValue("IAB1"),
},
},
},
}.test(t)

20. © 2017
INTEGRATION TESTING
BDD-style scenarios
Error:
Not equal: 27 (expected)
!= 26 (actual)
Messages: [TestGetAdCategories]
Given I have a REST collection of ad_category
When I do a GET collection request at http://127.0.0.1:
62526/v1/ad_category?sort=id
Then it should be an HTTP response
with body matching JSON value at result.#
• For integration tests
• Use the net/http/httptest pkg
• Custom scenario: nice output message:

21. © 2017
INTEGRATION TESTING
Implementation
type Scenario struct {
Name string
Given Given
When When
Then Then
}
type When interface {
run(context.Context, *testing.T) context.Context
}
// GetCollection is a When step
type GetCollection struct {
At string // At is the base URL of the HTTP server
}
func (c GetCollection) run(ctx context.Context, t *testing.T)
context.Context {
resp, _ := http.Get(...)
return context.WithValue(ctx, contextKeys.HTTPResponse, resp)
}

22. © 2017
INTEGRATION TESTING
Result
• Easier to read
• Re-usable blocks
• Clear separation of concerns

23. © 2017
API VERSIONING
• Product requirements are constantly changing
• Breaking changes
• API users need to adapt
Product change vs API users

24. © 2017
API VERSIONING
• Don’t break old clients !
• Use an HTTP header to pass the « API level » implemented by the client
• Adapt the request to the current API level implemented by the server
• Handle the request as usual
• Adapt the response to the API level implemented by the client
• Use HTTP middleware
• Just JSON transformation
API Levels

25. © 2017
API VERSIONING
Declaration
func init() {
Adapters{
{
ForLevel: 2,
AppliesTo: rest.Endpoints{
rest.GetTimezones,
},
AdaptResponseWith: jsonResponseBodyAdapter(adaptTimezones),
},
{
ForLevel: 2,
AppliesTo: rest.Endpoints{
rest.GetTimezone,
},
AdaptResponseWith: jsonResponseBodyAdapter(adaptTimezone),
},
}.mustRegister()
}

26. © 2017
API VERSIONING
Types
type Adapter struct {
ForLevel int
AppliesTo rest.Endpoints
AdaptRequestWith RequestAdapter
AdaptResponseWith ResponseAdapter
}
type RequestAdapter interface {
Adapt(*http.Request) (*rest.APIError, error)
}
type ResponseAdapter interface {
Adapt(r *http.Request, statusCode int, headers http.Header,
body *bytes.Buffer) error
}

27. © 2017
API VERSIONING
Implementation
func Adapters(inner http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
level := ParseRequestedAPILevel(r.Header)
adapters := restadapters.ForEndpointAndLevel(r.Method, path, level)
adapters.SortByLevelAsc()
for _, adapter := range adapters {
adapter.AdaptRequestWith.Adapt(r)
}
}
inner.ServeHTTP(rw, r)
adapters.SortByLevelDesc()
for _, adapter := range adapters {
adapter.AdaptResponseWith.Adapt(r, statusCode, w.Header(), buf)
}
}
})
}

28. © 2017
API VERSIONING
• Request
• You can only read it once!
• Use a bytes.Buffer
• Response
• Can’t re-read from it
• http.ResponseWriter is not easy to wrap
• Use github.com/felixge/httpsnoop
• Capture status code, headers, and body
Gotchas!

29. © 2017
MONITORING
• Endpoints performances
• SQL queries
• More insights in the requests
• Different behaviour based on the request body
What we need

30. © 2017
MONITORING
DataDog, APM and Context
• Using github.com/DataDog/dd-trace-go
• Tracing from HTTP handlers to database calls
• Use the context pkg
• Tracing information is stored in the context
• Each func has a ctx as its first parameter
• Easily create custom « span » (validation, store, …)

31. © 2017
MONITORING
API Endpoints
SQL Requests

32. © 2017
MONITORING
Down to the SQL level

33. © 2017
MONITORING
More insights on the requests
if span, ok := tracer.SpanFromContext(r.Context()); ok {
payload, _ := ioutil.ReadAll(r.Body)
span.SetMeta("http.payload", string(payload))
span.SetMeta("http.remote_ip", remoteip.FromRequest(r))
span.SetMeta("http.user_agent", r.UserAgent())
span.SetMeta("api.level", strconv.Itoa(apiLevel))
}
• Middleware
• Add request metadata
• Request body
• Some headers
• …

34. © 2017
MONITORING
Single request details

35. © 2017
CONCLUSION
• Easy to read/write
• Performance
• Easy to fit to our needs, using just Go and the stdlib
• Fun to work with!
Go for a REST API ?

36. © 2017
Thank you