Modern distributed architectures are increasingly composed of large numbers of decoupled, asynchronous components. In AWS, these components are plumbed together via services like SQS, SNS, Kinesis and S3 often integrated via small and frequent numbers of microservices or lambdas. But how do you test these architectures if they are cloud native? It’s 2018, and we can do better than deploying the entire stack and running a battery of E2E tests against them. In his talk, Matt will demonstrate how you can scale development of large-scale systems across teams, technology and process, and unlock the agility of your cloud-native architecture. WARNING: there will be code (https://github.com/mefellows/serverless-testing-example)

1. Deploying large-scale, serverless
and asynchronous systems
Matt Fellows
@matthewfellows
...without integrated tests!

2. Architecture

3. 3
Evolution
Architecture

4. 4
Release
Architecture > Evolution
MoreEvolved
Time
Monolith
SOA
Serverless
Microservices
Testing
Culture
Unit tests,
TDD, BDD
Contract
tests
Waterfall
Outsourcing
API
Testing
Scenario
Tests
Fn
Not all primitives
are invisible!
Co-evolved
practices:
XP
AgileLean
Multiple E2E
Test Env.
Chaos
Engineering
DevOps
CD
CI

5. 5
If you can’t build a good monolith you
shouldn’t be doing microservices
Architecture > Microservices

6. Entry criteria to
make µServices
● Ability to write a modular monolith
● Strong grasp on your domain model
● Ability to design loosely coupled services
○ = state + transactions
● Ability to scale and decouple teams
● Ability to deploy continuously
● A strong “DevOps” culture
● Be comfortable with decentralised,
complex systems + uncertainty
Architecture > Microservices

7. ...the µServices
ride
● Ability to write a modular monolith
● Strong grasp on your domain model
● Ability to design loosely coupled services
○ = state + transactions
● Ability to scale and decouple teams
● Ability to deploy continuously
● A strong “DevOps” culture
● Be comfortable with decentralised,
complex systems + uncertainty
Architecture > Microservices

8. ...the µServices
ride
● Ability to write a modular monolith
● Strong grasp on your domain model
● Ability to design loosely coupled services
○ = state + transactions
● Ability to scale and decouple teams
● Ability to deploy continuously
● A strong “DevOps” culture
● Be comfortable with decentralised,
complex systems + uncertainty
Architecture > Microservices

9. ...the µServices
ride
● Ability to write a modular monolith
● Strong grasp on your domain model
● Ability to design loosely coupled services
○ = state + transactions
● Ability to scale and decouple teams
● Ability to deploy continuously
● A strong “DevOps” culture
● Be comfortable with decentralised,
complex systems + uncertainty
Architecture > Microservices

10. WHAT WE ARE KNOWN FOR
10
Microservices
Evolution Drivers for levelling up
● Increase time-to-value
● Increase utilisation / cost
reduction
● Raise abstraction; focus on
composition
● Scale product teams
● Massively distributed,
real-time apps
● API contracts (e.g. OAS)
● Ubiquity of cloud computing
● Rise of containers
● Unreliable networking /
components
● Rise of the Service Mesh
● Improved observability
Architecture > Microservices

11. Functions + Serverless

12. 12
Architecture > Functions + Serverless (Cloud Native)
https://aws.amazon.com/retail/data-analytics/

13. 13 https://www.slideshare.net/AmazonWebServices/new-launch-data-driven-apps-with-graphql-aws-appsync-deep-dive-mbl402-reinvent-2017
Architecture > Functions + Serverless (Cloud Native)

14. 14 https://aws.amazon.com/blogs/big-data/analyze-a-time-series-in-real-time-with-aws-lambda-amazon-kinesis-and-amazon-dynamodb-streams/
Architecture > Functions + Serverless (Cloud Native)

15. 15 https://aws.amazon.com/blogs/big-data/automating-analytic-workflows-on-aws/
Architecture > Functions + Serverless (Cloud Native)

16. 16 https://www.slideshare.net/nateware/aws-gdc2015-gameanalyticsfinal
Architecture > Functions + Serverless (Cloud Native)

17. 17
Architecture > Functions + Serverless (Cloud Native)
https://aws.amazon.com/retail/data-analytics/

18. 18
Much Compose! So speed!
Architecture > Functions + Serverless (Cloud Native)Architecture > Functions + Serverless (Cloud Native)

19. 19
BUT.
Architecture > Functions + Serverless (Cloud Native)

20. 20
If you can’t build good microservices
you shouldn’t be doing serverless
Architecture > Functions + Serverless (Cloud Native)

21. 21
If you can’t build a good Fn you
shouldn’t be doing FaaS
Architecture > Functions + Serverless (Cloud Native)

22. 22
So, what practices do we need?
Architecture > Functions + Serverless (Cloud Native)

23. AWS Meetup Lounge

24. 24
Demo: AWS Meetup Lounge
CloudFront
API Gateway
Authentication
You!
awslounge.onegeek.com.au
Twitter Fetch
Twitter feed
Twitter Push
IoT Message
Broker
Sentiment
Analysis
Checkpoint
Amazon ML
#awsmelb

25. Challenges w Serverless
Blasphemy!

26. 26
Vendor lock-in
Challenges > Vendor lock-in

27. 27 https://twitter.com/alambert/status/958050456101642241
Challenges > Vendor lock-in

28. WHAT WE ARE KNOWN FOR
28
Options
● Multi-cloud = lowest
common denominator
● There is always a vendor
(even if it’s you!)
● You’ll never be completely
vendor agnostic
● Multi-cloud abstraction
● Avoid altogether
● Structure your code better
Challenges
Challenges > Vendor lock-in

29. WHAT WE ARE KNOWN FOR
29
Options
● Multi-cloud = lowest
common denominator
● There is always a vendor
(even if it’s you!)
● You’ll never be completely
vendor agnostic
● Multi-cloud abstraction
● Avoid altogether
● Structure your code better
Challenges
Challenges > Vendor lock-in

30. Lambda
Handler
Challenges > Vendor lock-in > Lambda Anti-pattern
{
"Records": [
{
"EventVersion": "1.0",
"EventSubscriptionArn": "somearn",
"EventSource": "aws:sns",
"Sns": {
...
}
}
]
}
{
"Success": true
}

31. MQ Handler
WebSocket
handler
MQ Adapter WS Adapter
Azure Function
Adapter
Google Cloud Fn
Adapter
Azure Function
Handler
Google Cloud Fn
Handler
Lambda
Handler
Lambda
Adapter
Service
Repository
Adapter
(Anti-corruption layer)
Ports
Business Logic
Collaborators ...
Challenges > Vendor lock-in > Ports and Adaptors

32. 32
Challenges > Vendor lock-in > Ports and Adaptors > Sentiment Analysis Example
Twitter feed
Sentiment
Analysis
IoT Message
Broker

33. Challenges > Vendor lock-in > Ports and Adaptors > Lambda Port

34. Challenges > Vendor lock-in > Ports and Adaptors > SNS Adapter

35. Challenges > Vendor lock-in > Ports and Adaptors > Lambda Sentiment Service

36. Summary
Architecture > Microservices
● Separate protocol handling from
business logic
● Business logic shouldn’t change with
introduction of a new protocol
● Port, Adapter and Business Logic are
independently testable

37. 37
How do you test locally?
Challenges > Testing Locally

38. WHAT WE ARE KNOWN FOR
38
Options
Challenges > Testing Locally
● Stub services (e.g. Localstack)
● Stub API (e.g. Moto)
● Multiple handlers
● Unit tests

39. 39
Challenges > Testing Locally

40. WHAT WE ARE KNOWN FOR
40
Options
● Feature arms race
● Trustworthy?
● Integration style = slower
● All cloud providers?
● Stub services (e.g. Localstack)
● Stub API (e.g. Moto)
● Multiple handlers
● Unit tests
Challenges > Testing Locally
Challenges

41. WHAT WE ARE KNOWN FOR
41
Writing good unit tests
● Follow test pyramid
● Stub out AWS APIs
● Test your business logic
separately from any
serverless features (e.g.
lambda handler)
● Callback pyramid of death
● Tests are hard to write
● Your business logic knows
about AWS
● You have to stub multiple
services per test
● Large methods
● Confusing control flow
Challenges > Testing Locally > Unit Test
Code Smells

42. 42
Challenges > Testing Locally > Unit Test > Twitter Fetch
Checkpoint
Twitter Fetch

43. Challenges > Testing Locally > Unit Test > Lambda Service
Look ma, no AWS
Code

44. Challenges > Testing Locally > Unit Test > Lambda Service
Look ma, no AWS
Code

45. 45
Challenges > Testing Locally > Unit Test > Sentiment Analysis Example
Twitter feed
Sentiment
Analysis

46. Integration-style test
of handler
Challenges > Testing Locally > Unit Test > Lambda Handler

47. Summary
Architecture > Microservices
● Business logic should be tested
separately from port + adapter
● Majority of tests, should be fast unit
tests
● Stub AWS Services where required
● (Lambda) Handlers can be locally
integration tested with pre-canned JSON
events

48. 48
Async - who is my consumer?
Challenges > Async

49. 49
Async - delay in feedback
Challenges > Async

50. WHAT WE ARE KNOWN FOR
50
Options
● Local integration testing
● Cloud env. integration testing
● Contract testing
Challenges > Async

51. WHAT WE ARE KNOWN FOR
51
Options
● Can’t run complete local
environment
● Many moving parts
● Problems with integration
testing...
● Local integration testing
● Cloud env. integration testing
● Contract testing
Challenges
Challenges > Async

52. WHAT WE ARE KNOWN FOR
52
Options
● Can’t run complete local
environment
● Many moving parts
● Problems with integration
testing...
● Local integration testing
● Cloud env. integration testing
● Contract testing
Challenges
Challenges > Async

53. 53
Contract tests
Challenges > Async

54. 54
IoT Message
Broker
Sentiment
Analysis
Contract
{
'Positive': 0.234,
'Negative': 10.123,
'Neutral': 7.334,
'Mixed': 0
}
Consumer
Provider
Challenges > Async > Contract Tests

55. 55
CloudFront
API Gateway
Authentication
You!
awslounge.onegeek.com.au
Twitter Fetch
Twitter feed
Twitter Push
IoT Message
Broker
Sentiment
Analysis
Checkpoint
Amazon ML
#awsmelb
Challenges > Async > Contract Tests

56. 56
Authentication Twitter Fetch
Twitter feed
Twitter Push
IoT Message
Broker
Sentiment
Analysis
Checkpoint
#awsmelb
Challenges > Async > Contract Tests
Contract
Contract
Contract
Contract
Contract
Contract
Contract
Contract

57. WHAT WE ARE KNOWN FOR
57
Problems with automated e2e tests
● Slow
● Easy to break
● Hard to fix
● Scales badly across teams
● Lots of set up maintenance
● $$ potentially costly
Challenges > Async > Contract Tests

58. 58
C P
mock
Challenges > Async > Contract Tests
Understanding Test Symmetry

59. WHAT WE ARE KNOWN FOR
59
Test symmetry
Solved problems New problems
● Hard to keep both sides in
sync
● Fast feedback
● Few dependencies
● No dedicated environment
● Reliable
● Easy to debug
Challenges > Async > Contract Tests

60. 60
C P
mock
Challenges > Async > Contract Tests

61. 61
C P
mock
Challenges > Async > Contract Tests

62. PPCC
mock
contract
Challenges > Async > Contract Tests

63. Bug turnaround -
minutes
Contracts tests
Challenges > Async > Contract Tests

64. Know before you
commit
Contracts tests
Challenges > Async > Contract Tests

65. Make changes with
speed and confidence
Contracts tests
Challenges > Async > Contract Tests

66. Deploy independently
Contracts tests
Challenges > Async > Contract Tests

67. Better API design
(Consumer)
Contracts tests
Challenges > Async > Contract Tests

68. 68
● Open source
● Multiple languages
○ Javascript
○ Go
○ Python
○ JVM
○ .NET
○ + more
pact.io
● HTTP contracts
● Message contracts

69. How to be taller
...and ride that Fn

70. Summary
70
Summary
Problems Solutions
● Microservices+
● Create cloud and protocol
anti-corruption layer
● Mock out AWS SDK in unit tests
● Use contract tests
● Reduce reliance on end-to-end
environments
● Ad-hoc exploratory testing
● Serverless complexity >
uServices
● Testing cloud native services
● Avoiding vendor lock-in
● Handling asynchronous Fn’s

71. Deploying large-scale, serverless
and asynchronous systems
Matt Fellows
@matthewfellows
...without integrated tests!