Modern software systems now increasingly span cloud, on-premise, and remote embedded devices & sensors. These distributed systems bring challenges with data, connectivity, performance, and systems management, so for business success we need to design and build with operability as a first class property. In this talk, we explore five practical, tried-and-tested, real world techniques for improving operability with many kinds of software systems, including cloud, Serverless, on-premise, and IoT: - Logging as a live diagnostics vector with sparse Event IDs - Operational checklists and 'Run Book dialogue sheets' as a discovery mechanism for teams - Endpoint healthchecks as a way to assess runtime dependencies and complexity - Correlation IDs beyond simple HTTP calls - Lightweight 'User Personas' as drivers for operational dashboards These techniques work very differently with different technologies. For instance, an IoT device has limited storage, processing, and I/O, so generation and shipping of logs and metrics looks very different from the cloud or Serverless case. However, the principles - logging as a live diagnostics vector, Event IDs for discovery, etc. - work remarkably well across very different technologies.

1. Practical Operability
Techniques for Teams
Matthew Skelton
Skelton Thatcher Consulting
skeltonthatcher.com / @SkeltonThatcher
IPEXPO Europe, ExCeL, London – 05 October 2017

2. Today
What is operability?
Modern logging
Run Book dialogue sheets
Endpoint healthchecks
Correlation IDs
User Personas for dashboards

3. Training
1-day tutorial with exercises
Book via Unicom: http://www.unicom.co.uk/workshops.html

4. You
Software Developer
Tester / QA
DevOps Engineer
Team Leader
Head of Department

5. Operability:
use modern logging, Run Book
dialogue sheets, endpoint
healthchecks, correlation IDs,
and user personas as
team collaboration techniques

7. About us
Co-founders at Skelton Thatcher Consulting
Matthew Skelton Rob Thatcher

8. Team-first digital transformation
30+ organisations
UK, US, EU, India, China

9. We build modern capabilities
by mentoring your teams

10. Team Guide to
Software Operability
Matthew Skelton & Rob Thatcher
skeltonthatcher.com/publications
Download a free sample chapter

11. Practical Operability
Techniques for Teams

12. What is operability?

13. Operability
making software work well
in Production

15. Logging with Event IDs

17. logging with Event IDs
reduce time to detect problems
increase team engagement
increase configurability
enhance collaboration
#operability

18. search by
event
Event ID
{Delivered,
InTransit,
Arrived}

19. transaction
trace
Correlation ID
612999958…

20. How many distinct event
types (state transitions)
in your application?

22. represent distinct states

23. enum
Human-readable sets:
unique values, sparse,
immutable
C#, Java, Python, node
(Ruby, PHP, …)

24. Technical
Domain
public enum EventID
{
// Badly-initialised logging data
NotSet = 0,
// An unrecognised event has occurred
UnexpectedError = 10000,
ApplicationStarted = 20000,
ApplicationShutdownNoticeReceived = 20001,
MessageQueued = 40000,
MessagePeeked = 40001,
BasketItemAdded = 60001,
BasketItemRemoved = 60002,
CreditCardDetailsSubmitted = 70001,
// ...
}

25. BasketItemAdded = 60001
BasketItemRemoved = 60002

26. log using Event IDs
with a modern
‘structured logging’ library

27. Example: video processing
On-demand processing of TV
advertisements
Ad-agency  TV broadcaster
High throughput
Glitch-free video & audio

28. Storage I/O
Worker Job
Queue
Upload

31. Example: video processing
Discover processing bottlenecks
Trigger alerts via LogEntries /
HostedGraphite
Report on KPIs
Target areas for improvement

32. Run Book dialogue sheets

34. Run Book dialogue sheets
Checklists for typical operational
considerations
Team-friendly exploration

35. runbooktemplate.infoRun Book dialogue sheets

36. System characteristics
Hours of operation
During what hours does the service or system actually need to operate? Can portions or features of the
system be unavailable at times if needed?
Hours of operation - core features
(e.g. 03:00-01:00 GMT+0)
Hours of operation - secondary features
(e.g. 07:00-23:00 GMT+0)
Data and processing flows
How and where does data flow through the system? What controls or triggers data flows?
(e.g. mobile requests / scheduled batch jobs / inbound IoT sensor data )
…

37. http://runbooktemplate.info/

38. runbooktemplate.infoRun Book dialogue sheets

39. Endpoint healthchecks

41. endpoint healthchecks
Every runnable app/service/daemon
exposes /status/health
An HTTP GET to the endpoint returns:
200 – "I am healthy"
500 – "I am sick"

42. endpoint healthchecks
Use JSON as a response type –
parsable by both machines and
humans!

43. endpoint healthchecks
For databases and other non-HTTP
components, run a lightweight HTTP
service in front of the component
200 / 500 responses

44. https://github.com/Lugribossk/simple-dashboard

45. Correlation IDs

47. ‘Unique-ish’ identifier for each request
Passed through downstream layers

48. Unique-ish ID

49. Synchronous HTTP:
X-HEADER e.g. X-trace-id
X-trace-id: 348e1cf8
If header is present, pass it on
(Yes, RFC6648, but this is internal only)

50. Asynchonous (queues, etc.):
Message Attributes, name:value pair
e.g. "trace-id":"348e1cf8"
AWS SQS: SendMessage() / ReceiveMessage()
Log the Correlation ID if present

51. Example: electronic trading
High speed, low latency
Trading options & derivatives
Connected to stock exchanges
Sub-millisecond timings
> £1 million per day traded

55. Correlations IDs for trading
Evidence for timely operation
Help identify bottlenecks
Target areas for perf tuning
Identify race conditions
Increase operability

56. Lightweight user personas

58. Lightweight user personas:
Ops Engineer
Test Engineer
Build & Deployment Engineer
Service Owner

59. Lightweight user personas:
Consider the User Experience (UX) of
engineers and team members using
and working with the software

60. http://www.keepitusable.com/blog/?tag=alan-cooper

61. https://www.geckoboard.com/blog/visualisation-upgrades-
progressing-towards-a-more-useful-and-beautiful-dashboard/

62. Lightweight user personas:
What data does the User Persona
need visible on a dashboard in order
to make decisions rapidly & safely?

63. Summary

64. Operability
making software work well
in Production

65. logging with Event IDs
use enum-based Event IDs to
explore runtime behaviour
and fault conditions

66. Run Book dialogue sheets
explore and establish
operational requirements as
a team, around a physical
table, together

67. endpoint healthchecks
HTTP 200 / 500 responses to
/status/health call with
JSON details – good for
tools and humans

68. Correlation IDs
trace execution using
correlation IDs:
synchronous (HTTP X-trace-id)
async (SQS MessageAttribute)

69. lightweight user personas
explore the UX and needs of
different roles for rapid
decisions via dashboards

70. Operability
use modern logging, Run Book
dialogue sheets, endpoint
healthchecks, correlation IDs,
and user personas as
team collaboration techniques

71. Team Guide to
Software Operability
Matthew Skelton & Rob Thatcher
skeltonthatcher.com/publications
Download a free sample chapter

72. Training
1-day tutorial with exercises
Book via Unicom: http://www.unicom.co.uk/workshops.html

73. Resources
• Training: Practical Operability for Developers and Testers – led
by Matthew Skelton and Rob Thatcher – 1-day workshop –
http://www.unicom.co.uk/practical-operability-for-developers-
and-testers.html
• Team Guide to Software Operability by Matthew Skelton and Rob
Thatcher (Skelton Thatcher Publications, 2016)
http://operabilitybook.com/
• Run Book template & Run Book dialogue sheets
http://runbooktemplate.info/

74. Questions?
Twitter: @SkeltonThatcher | #operability
email: questions@skeltonthatcher.com

75. thank you
@SkeltonThatcher
skeltonthatcher.com