Writing serverless functions is quite easy. And also combining some of them to build up a more complex system is not too complicated at all. But what happens if these tiny functions are not working as expected? How to trace and monitor each function by itself and the system as a whole? Is there any change to debug? And what about testing? Do we always have to deploy to the cloud or are there any local solutions to emulate the productive environment? And by the way, how can we automate all of this? Lots of questions are waiting to be answered. Required audience experience basic knoweldge of serverless computing Objective of the talk The audience will learn how to handle and “survive” serverless scenarios in real life. During the session I will demonstrate with the help of a demo application (aws lambda based) how to test, trace, monitor and debug servlerless functions.

1. #WISSENTEILEN
Lars Röwekamp | CIO New Technologies | open knowledge GmbH
@_openKnowledge | @mobileLarson
Surviving Serverless
in the Real-Life

2. ABOUT OPEN KNOWLEDGE
Software development and IT consulting
#WISSENTEILEN

3. ABOUT ME
Who am i?
• CIO New Technologies
• Enterprise & Mobile
• Author, Speaker, Coach & Mentor
• Snowboard & MTB Enthusiast
• Traveller between the worlds
Lars Röwekamp (a.k.a. @mobileLarson)
#WISSENTEILEN
LR

4. #WISSENTEILEN
„Run Code,
not Servers!“

5. #WISSENTEILEN
“Run your business code
highly-available
in the cloud in response
to events and scale
without any servers to
manage.“*
*(AWS Lambda product description)

6. #WISSENTEILEN
“Run your business code
highly distributed
and event driven in a non
transparent environment
with no single
point of control.“*
*(my personal interpretation)

7. „Run Code, not Servers“, they said!
„It will be fun“, they said

8. #WISSENTEILEN
The Road to the Cloud ...
The Serverless Showcase

9. #WISSENTEILEN
AWS Cloud
Use-Case: Upload Image
upload image
with additional
information

10. #WISSENTEILEN
AWS Cloud
Store raw Image
1
Use-Case: Upload Image
upload image
with additional
information

11. #WISSENTEILEN
AWS Cloud
Store raw Image
Store Image Information
1
2
Use-Case: Upload Image
upload image
with additional
information

12. #WISSENTEILEN
AWS Cloud
AWS Step Functions workflow: Store Image
Store raw Image
Store Image Information
1
2
Use-Case: Upload Image
upload image
with additional
information

13. #WISSENTEILEN
AWS Cloud
AWS Step Functions workflow: Store Image
Create ThumbnailStore raw Image
Store Image Information
1
2
Use-Case: Upload Image
upload image
with additional
information

14. #WISSENTEILEN
AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2
Use-Case: Upload Image
upload image
with additional
information

15. #WISSENTEILEN
AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2
Use-Case: Upload Image
upload image
with additional
information

16. #WISSENTEILEN
AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2
Use-Case: Upload Image
upload image
with additional
information

17. #WISSENTEILEN
AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2
Use-Case: Upload Image
upload image
with additional
information

18. #WISSENTEILEN
AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2
Use-Case: Upload Image
upload image
with additional
information

19. #WISSENTEILEN
„What can go
wrong?“

20. #WISSENTEILEN
AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

21. #WISSENTEILEN
AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

22. #WISSENTEILEN
AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

23. #WISSENTEILEN
AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

24. #WISSENTEILEN
AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

25. #WISSENTEILEN
AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

26. #WISSENTEILEN
AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

27. #WISSENTEILEN
AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

28. #WISSENTEILEN
AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

29. #WISSENTEILEN
AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

30. #WISSENTEILEN
• internal errors
• abnormalities (e.g. unexpected or invalid calls)
• lost or duplicated events
• high latency / timeouts
• security attacks (e.g. DoS/DDoS)
• personalized SLAs / usage plans / unexpected peaks of workload
• customer location changes
• runtime cost explosion / usage pattern evolves over time
What could possibly go wrong?

31. #WISSENTEILEN
Real-Life Agenda:
How to develop?

32. #WISSENTEILEN
Real-Life Agenda:
How to monitor?
How to test?
How to fix?

33. #WISSENTEILEN
How to monitor?
your serverless application

34. #WISSENTEILEN
With well-planned monitoring, we should be able to
• anticipate disruptions
• quickly identfiy the source of problems
• trigger automated recovery processes
• trigger alarms
Real-Life Monitoring

35. #WISSENTEILEN
Real-Life Monitoring
Business
KPI
UX
SLA
“products per
order”
“average order
value”
“abandonment
rate”
“first contentful
paint”
“first
meaningful
paint”
“first
interaction”
“availability”
“latency”
“durability”
“consistency”

36. #WISSENTEILEN
Well-planned monitoring should take various aspects into account
• reliability: components and communication
• usage: functional and non functional
• performance: duration, latentcy and timeouts
• security: access rights, attacts
• costs: current and trends
Real-Life Monitoring

37. #WISSENTEILEN
The 4 Pillars of Monitoring
3
2 4
1
Tracing Metrics
Alerting
Logging

38. 3
2
Tracing Metrics
4
4
Alerting
#WISSENTEILEN
The 4 Pillars of Monitoring
Represents state of an
application.
When things go wrong, we
need the logs to establish
what change in state
caused the error.
1
Logging
Logging

39. 3
Metrics
4
1
Alerting
Logging
#WISSENTEILEN
The 4 Pillars of Monitoring
Tracing
2
Represents a single user‘s
journey through an entire
stack of an application.
It is often used for
optimisation purpose.
Tracing

40. 2
Tracing
4
1
Alerting
Logging
#WISSENTEILEN
The 4 Pillars of Monitoring
3
Metrics
Represents a measuring
point aggregated over a
span of time.
Helps to determine the
systems current health
and trends.
Metrics

41. 3
2
Tracing Metrics
1
Logging
#WISSENTEILEN
The 4 Pillars of Monitoring
4
Alerting
The component of a
monitoring system that
performs actions based on
changes in metric values.
Mostly used to auto-heal
system or inform person in
charge.
Alerting3
2
Tracing Metrics
1
Logging
#WISSENTEILEN
The 4 Pillars of Monitoring
4
Alerting
The component of a
monitoring system that
performs actions based on
changes in metric values.
Mostly used to auto-heal
system or inform person in
charge.
Alerting3
2
Tracing Metrics
1
Logging
#WISSENTEILEN
The 4 Pillars of Monitoring
4
Alerting
The component of a
monitoring system that
performs actions based on
changes in metric values.
Mostly used to auto-heal
system or inform person in
charge.
Alerting

42. #WISSENTEILEN
Monitoring Strategy #1: DIY
Logging
Tracing
Metrics
Alerting
AWS Cloud

43. #WISSENTEILEN
Monitoring Strategy #1: DIY
Logging
Tracing
Metrics
Alerting
AWS Cloud
universal Log
Logging

44. #WISSENTEILEN
Monitoring Strategy #1: DIY
Logging
Tracing
Metrics
Alerting
AWS Cloud
universal Log
Logging
DevOps

45. #WISSENTEILEN
Monitoring Strategy #1: DIY
Logging
Tracing
Metrics
Alerting
AWS Cloud
universal Log
Archive
Logging
Tracing
DevOps

46. #WISSENTEILEN
Monitoring Strategy #1: DIY
Logging
Tracing
Metrics
Alerting
AWS Cloud
universal Log
Archive
Logging Metrics
Tracing
DevOps

47. #WISSENTEILEN
Monitoring Strategy #1: DIY
Logging
Tracing
Metrics
Alerting
AWS Cloud
universal Log
Alarm
Archive
Logging Metrics
Tracing
Alerting
DevOps

48. #WISSENTEILEN
Monitoring Strategy #2: Platform Services
AWS Cloud
Logging
Logging
Tracing
Metrics
Alerting

49. #WISSENTEILEN
Monitoring Strategy #2: Platform Services
AWS Cloud
Logging
Alerting
Metrics
“BASIC ALERTING FOR FREE”
“BASIC METRICS FOR FREE”
Logging
Tracing
Metrics
Alerting

50. #WISSENTEILEN
Monitoring Strategy #2: Platform Services
Logging
Metrics
Alerting
AWS Cloud
Alarm
Logging
Alerting
Metrics
Tracing

51. #WISSENTEILEN
Monitoring Strategy #2: Platform Services
Logging
Tracing
Metrics
Alerting
AWS Cloud
Logging
Alerting
Metrics
Tracing (still DIY)
Alarm
Metrics

52. #WISSENTEILEN
Monitoring Strategy #2: Platform Services
Logging
Tracing
Metrics
Alerting
AWS Cloud
Alarm
Logging
Tracing
Alerting
Tracing (DIY)
Metrics

53. #WISSENTEILEN
Monitoring Strategy #2: Platform Services
Logging
Tracing
Metrics
Alerting

54. #WISSENTEILEN
Monitoring Strategy #2: Platform Services
Logging
Tracing
Metrics
Alerting

55. #WISSENTEILEN
Monitoring Strategy #2: Platform Services
Logging
Tracing
Metrics
Alerting

56. #WISSENTEILEN
„But what, if I live in a
heterogenous world?“

57. #WISSENTEILEN
On-PremiseAWS Cloud Microsoft Cloud
My World to monitor
Logging
Tracing
Metrics
Alerting
Monitoring Strategy #3: External Services
?

58. #WISSENTEILEN
Monitoring Strategy #3: External Services
ELK-Tracing:
CloudTrail &
S3 Logstash
Input Plugin

59. #WISSENTEILEN
Monitoring Strategy #3: External Services
Logz.io-Tracing:
built-in support

60. #WISSENTEILEN
Monitoring Strategy #3: External Services
Dashbird:
built-in support

61. #WISSENTEILEN
Monitoring Strategy #3: External Services
On-PremiseAWS Cloud Microsoft Cloud
Logging
Tracing
Metrics
Alerting
My World to monitor
Plug-in Plug-in Plug-in

62. #WISSENTEILEN
„Logging data is expensive,
so better log wisely!“

63. #WISSENTEILEN
What kind of „benchmarks“ do we want for our monitoring?
• collect rich set of system and application metrics
• metrics and logs should not add user-facing latency
• metrics and logs should appear in real-time
• metrics should be granular and correlated
Monitoring Best Practices

64. #WISSENTEILEN
#1 avoid user-facing latency
Monitoring Best Practices
AWS Cloud
My Lambda logs
log
stream
log
data
async
sync
Log Aggregator
log
data
1
very fast and cheap
2
3
time consuming and “expensive”
parse
log stream

65. #WISSENTEILEN
#2 collect rich set of system and application metrics
Monitoring Best Practices
AWS Cloud
My Lambda logs
log
stream
log
data
async
sync
Log Aggregator
metrics
custom
metrics
custom
metrics
log
data
2
3
1
very fast and cheap
parse
log stream

66. #WISSENTEILEN
#3 avoid unnecessary costs
Monitoring Best Practices
AWS Cloud
My Lambda logs
log
stream
log
data
async
sync
Log Aggregator
metrics
log & custom
metrics
custom
metrics
custom
metrics
log
data
archive
logs
1
2

67. #WISSENTEILEN
#4 correlate logs and metrics
Monitoring Best Practices
AWS Cloud
My Lambda logs
log
stream
log
data
async
sync
Log Aggregator
metrics
log & custom
metrics
custom
metrics
custom
metrics
log
data
archive
Logs
1
correlation
ID

68. #WISSENTEILEN
#5 use ENV vars to enable/disable debug logging at Edge server
Monitoring Best Practices
AWS Cloud
My Lambda logs
log
stream
log
data
async
sync
Log Aggregator
metrics
log & custom
metrics
custom
metrics
custom
metrics
log
data
archive
Logs
DEBUG
on/off
ENV var
1
2

69. #WISSENTEILEN
How to test?
your serverless application

70. #WISSENTEILEN
„But even though the app
was well-tested, something
happened overnight!“

71. #WISSENTEILEN
Testing, the classical way

72. #WISSENTEILEN
Testing, the serverless way

73. #WISSENTEILEN
Testing, the serverless way

74. #WISSENTEILEN
Testing, the serverless way
Unit-Tests for Serverless Functions
• seperation of logic and infrastructure
• mock dependencies
• jUnit
• testNG

75. #WISSENTEILEN
Testing, the serverless way
Integration Tests for Cloud Components
• dev/test components in the cloud*
• mock/fake cloud components
• S3rver
• dynamodb-local
• kinesalite
• node-lambda
• fake-sns
• elasticmq
(* $$$, a.k.a. kind of expensive)

76. #WISSENTEILEN
Testing, the serverless way
End-to-End test for User-Stories
• cloud test stages*
• local cloud emulation
• SAM local (Amazon)
• LocalStack (Atlassian)
• docker-lambda (LambCI.org)
(* $$$, a.k.a. kind of expensive)

77. #WISSENTEILEN
SAM* local (public beta)
(* SAM = Serverless Application Model)
How to test via SAM local?

78. #WISSENTEILEN
„Based on AWS SAM, SAM Local is an AWS CLI
tool that provides an environment for you to develop,
test, and analyze your serverless applications locally
before uploading them to the Lambda runtime.“
• start local gateway from a SAM template
• validate SAM template
• generate sample payload for various event sources
How to test via SAM local?

79. #WISSENTEILEN
LocalStack (v. 0.8.5*)
(* or maybe higher? Changes frequently)
How to test with LocalStack?

80. #WISSENTEILEN
localStack
How to test with LocalStack?

81. #WISSENTEILEN
localStack
How to test with LocalStack?

82. #WISSENTEILEN
localStack
• easy to use mocking framework*
• builds up local cloud environment**
• Base Edition (free) & Pro Edition ($$, coming soon)
• fast growing community
• 100k test runs in 08.2017 für 0$
*based an established mocking frameworks
** at this moment AWS „only“, others shall follow
How to test with LocalStack?

83. #WISSENTEILEN
„Don‘t let your users
test your code!“

84. #WISSENTEILEN
What kind of „benchmarks“ do we want for our testing?
• test functional changes fast and cost efficient
• test basic integrational changes fast and cost efficient
• test integrational changes as „real“ as possible
• test use cases and user stories in total
Testing Best Practices

85. #WISSENTEILEN
#1 seperate business logic from infrastructure inside your functions
Testing Best Practices
AWS CloudOn-Premise
handler
logic
unit test candidate
e
i
u
u
integration test candidate
end-to-end test candidate
u

86. #WISSENTEILEN
#2 mock cloud infrastructure components of your functions
Testing Best Practices
AWS CloudOn-Premise
handler
logic
unit test candidate
e
i
u
u
integration test candidate
end-to-end test candidate
u
um

87. #WISSENTEILEN
#3 use local environment for functional testing (e.g. SAM local)
Testing Best Practices
AWS CloudOn-Premise
handler
logic
unit test candidate
e
i
u
u
integration test candidate
end-to-end test candidate
uvia SAM local
via SAM local
SAM
yaml
TEST
u
u

88. #WISSENTEILEN
#4 use local environment for trigger integration testing
Testing Best Practices
AWS CloudOn-Premise
handler
logic
unit test candidate
e
i
u
u
integration test candidate
end-to-end test candidate
uvia SAM local
via SAM local
SAM
yaml
TEST
u
u
i
i

89. #WISSENTEILEN
#5 use local „cloud“ components for basic integration testing
Testing Best Practices
AWS CloudOn-Premise
handler
logic
unit test candidate
e
i
u
u
integration test candidate
end-to-end test candidate
u
via DynamoDB local
via FakeS3 via SAM local
via SAM local
SAM
yaml
TEST
u
u
i
i
i
i
warning: local cloud
components can only proof
functional correctness but
not infrastructural
correctness like DLQs,
timeouts, throttling, SLAs, …

90. #WISSENTEILEN
#6 use temporary integration cloud for partial integration testing
Testing Best Practices
AWS CloudOn-Premise
handler
logic
unit test candidate
e
i
u
u
integration test candidate
end-to-end test candidate
uvia SAM local
via SAM local
SAM
yaml
TEST
u
u
via DynamoDB local
via FakeS3
i
i
Temorary Intregration #Dev1
ii
INT
i
i

91. #WISSENTEILEN
#7 use permanent integration cloud for end-to-end testing
Testing Best Practices
AWS CloudOn-Premise
handler
logic
unit test candidate
e
i
u
u
integration test candidate
end-to-end test candidate
uvia SAM local
via SAM local
SAM
yaml
TEST
u
u
via DynamoDB local
via FakeS3
i
i
Permament IntregrationINT
e
e
e
e
i
i

92. #WISSENTEILEN
Real-Life
Summary

93. #WISSENTEILEN
Writing hundreds of serverless functions is easy, integrating them in
contrast is kind of hard. Therefore we must be able to …
• avoid problems in advance -> test on all levels
• anticipate disruptions -> collect and analyze metrics
• quickly identfiy the source of problems -> via logging and tracing
• trigger automated recovery processes -> set up CI/CD pipeline
• trigger alarms -> establish alerting system
Surviving Serverless in the Real-Life

94. #WISSENTEILEN
„It is all about
automation!“

95. No servers, no stress!

96. ? ? ?
FRAGEN
#WISSENTEILEN

97. Kontakt
LARS RÖWEKAMP
CIO NEW TECHNOLOGIES
lars.roewekamp@openknowledge.de
+49 (0)441 4082 – 101
@mobileLarson
@_openknowledge
OFFENKUNDIGGUT
#WISSENTEILEN

98. Bildnachweise
#1: © FavrikaSimf – shutterstock.com
#8: © vasakna – fotolia.com
All other pictures inside this presentation orginate from
pixabay.com or were drawn by mysef.
#WISSENTEILEN