There are plenty of patterns and literature about how to organize systems in traditional languages like Java or C#. The same isn’t exactly true for functional programming languages, specially Clojure. At Nubank we had to figure it out over 3 years of writing tens of microservices in Clojure, using some of our knowledge from other languages and some creativity.
Several patterns emerged, several mistakes were made and eventually we came up with a very sustainable and scalable way to write new code even for developers which are new to Clojure. This talk will explore those learnings.
Presented at Clojure Remote 2017
10. Datomic
Immutability
Datomic stores the changes/transactions, not just the
data
-append only
-db as a value
-everything is data (transaction, schema, entities)
12. AWS + Docker
Immutability
Ability to spin machines with a given image/
configuration
-Each build generates a docker image
-Each deploy spins a new machine with the new
version
-As soon as the new version is healthy, old version is
killed. (blue-green deployment)
15. System map
Components
{:database #SomeDatabase{...}
:http-client #HttpClient{...}
:kafka #Kafka{...}
:auth #AuthCredentials{...}
...}
-Created at startup
-Entrypoints (e.g http server or kafka consumers) have access to all
components the business flows need
-dependencies of a given flow are threaded from the entry point until
the end, one by one if possible
-Thus no static access to system map! (e.g via a global atom)
-Any resemblance to objects and classes is just coincidence ;)
18. Simplicity
Pure functions
-easier to reason about, fewer moving pieces
-easier to test, less need for mocking values
-parallelizable by default, no need for locks or STMs
19. Datomic
Pure functions
-Datomic’s db as a value allows us to consider a function
that queries the database as a pure function
-db is a snapshot of the database at a certain point in time.
-So, querying the same db instance will always produce the
same result
20. Impure functions
Pure functions
-functions that produce side effects should be marked as
such. We use `!` at the end.
-split code which handles and transforms data from code
that handles side effects
-should be moved to the borders of the flow, if possible
-Consider returning a future/promise like value, so side
effect results can be composed (e.g with manifold or
finagle)
https://github.com/ztellman/manifold
https://github.com/twitter/finagle
22. Schema
Legacy
Majority of our code base was written before clojure.spec existed,
so I’ll be talking about the Schema library instead. Most principles
apply to clojure.spec as well.
23. Schema/Spec
Documentation
-Clojure doesn’t force you to write types
-parameter names are not enough
-declaring types helps a lot when glancing at the function
-values can be verified against a schema
24. Function declaration
Schema/spec
-All pure functions declare schemas for parameters and
return value
-All impure functions declare for parameters and don’t
declare output type if it’s not relevant.
-Validated at runtime in dev/test environments, on every
function call
-Validation is off on production.
25. Wire formats
Schema/Spec
-Internal schemas are your domain models
-Wire schemas are how you expose data to other services/
clients
-If they are different, you can evolve internal schemas
without breaking clients
-Need an adapter layer
-wire schemas are always validated on entry/exit points,
specially in production
-single repository for all wire schemas (for all 60+ services)
-caveat: this repository has a really high churn. Beware
28. Ports and Adapters
Definition
Core logic is independent to how we can call it (yellow)
A port is an entry-point of the application (blue)
An adapter is the bridge between a port and the core logic (red)
http://www.dossier-andreas.net/software_architecture/ports_and_adapters.html
http://alistair.cockburn.us/Hexagonal+architecture
29. Ports and Adapters (Nubank version)
Extended Definition
Pure business logic (green)
Controller logic wires the flow between the ports (yellow)
A port is an entry-point of the application (blue)
An adapter is the bridge between a port and the core logic (red)
30. Ports (Components)
Ports and Adapters
-Ports are initialised at startup
-Each port has a corresponding
component
-Serializes data to a transport
format (e.g JSON, Transit)
-Usually library code shared by
all services
-Tested via integration tests
HTTP
Kafka
Datomic
File Storage
Metrics
E-mail
31. Adapters (Diplomat)
Ports and Adapters
-Adapters are the interface to
ports
-Contain HTTP and Kafka
consumer handlers
-Adapt wire schema to
internal schema
-Calls and is called by
controller functions
-Tested with fake versions of
the port components, or
mocks
HTTP
Kafka
Datomic
File Storage
Metrics
E-mail
32. Controllers
Ports and Adapters
-Controllers wires the flow
between entry-point and the
side effects
-Only deals with internal
schemas
-Delegates business logic to
pure functions
-Composes side effect results
-Tested mostly with mocks
HTTP
Kafka
Datomic
File Storage
Metrics
E-mail
33. Business Logic
Ports and Adapters
-Handles and transforms
immutable data
-Pure functions
-Best place to enforce
invariants and type checks
(e.g using clojure.spec)
-Can be tested using
generative testing
-Should be the largest part of
the application
HTTP
Kafka
Datomic
File Storage
Metrics
E-mail
34. Microservices
Ports and Adapters
-Each service follows about
the same design
-Services communicate with
each other using one of the
ports (e.g HTTP or Kafka)
-Services DON’T share
databases
-HTTP responses contain
hypermedia, so we can
replace a service without
having to change clients
-Tested with end to end tests,
with all services deployed
35. Clojure is simple
Keep your design simple
Keep your architecture simple
SÃO PAULO, BRASIL