Weitere ähnliche Inhalte Ähnlich wie Declarative observability management for Microservice architectures (20) Mehr von Sven Bernhardt (12) Kürzlich hochgeladen (20) Declarative observability management for Microservice architectures1. © OPITZ CONSULTING 2023 / Interner Gebrauch
Declarative o11y management for your µArchitectures 1
Gummersbach, June 3, 2023
Sven Bernhardt
DECLARATIVE O11Y MANAGEMENT
FOR MICROSERVICE ARCHITECTURES
2. © OPITZ CONSULTING 2023 / Interner Gebrauch
THAT‘S ME
Declarative o11y management for your µArchitectures 2
Sven Bernhardt
Cloud-Native enthusiast, API & integration geek. Always curious how new
technologies and concepts can help to make things more valuable and efficient.
@sbernhardt
https://svenbernhardt.wordpress.com/
Chief Architect / Integration Evangelist
OPITZ CONSULTING Deutschland
GmbH
3. © OPITZ CONSULTING 2023 / Interner Gebrauch
Declarative o11y management for your µArchitectures 3
CHALLENGES
01
SOLUTION STRATEGY
02
CONCLUSION
04
USE CASES
03
4. © OPITZ CONSULTING 2023 / Interner Gebrauch
Declarative o11y management for your µArchitectures 4
CHALLENGES
01
5. © OPITZ CONSULTING 2023 / Interner Gebrauch
PREPARE FOR THE UNKNOWN
Declarative o11y management for your µArchitectures 5
Things we are aware of
and understand
Things we understand,
but are not aware of
Things we are aware of
but we don’t understand
Things we are neither
aware of nor understand
6. © OPITZ CONSULTING 2023 / Interner Gebrauch
MONITORING
Declarative o11y management for your µArchitectures 7
¢ Historically: Up/Down-Status of a system
¢ Now: Depict the overall health status of a
system
¢ The monitoring system should answer
¢ What is broken?
¢ Why is it broken?
¢ Usually includes levels
¢ Network
¢ Machine
¢ Application
Quote from the SRE Book by Google
“Your monitoring system should address two
questions: what’s broken, and why? The “what’s
broken” indicates the symptom; the “why”
indicates a (possibly intermediate) cause. “What”
versus “why” is one of the most important
distinctions in writing good monitoring with
maximum signal and minimum noise.”
7. © OPITZ CONSULTING 2023 / Interner Gebrauch
EXAMPLE: SYMPTOMS & CAUSE
Declarative o11y management for your µArchitectures 8
Google SRE Book about the correlation between symptoms and causes:
“Note that in a multi-layered system, one person’s symptom is another person’s cause!”
Symptom (“What?”) Cause (“Why?”)
Webserver responds with 500 or 404 Database doesn’t accept new connections, as it is in
maintenance mode
Connection reset while initializing database connection
pool
Wrongly configured firewall rules (IP ranges)
Slow responses for web frontend Backend system is busy with wrongly configured batch job
8. © OPITZ CONSULTING 2023 / Interner Gebrauch
OBSERVABILITY – JUST A FANCY
NEW TERM?
Declarative o11y management for your µArchitectures 9
¢ Oftentimes defined as a superset of
monitoring
¢ Monitoring
¢ Alerting/visualization
¢ Distributed systems tracing infrastructure
¢ Log aggregation/analytics
¢ Goals:
¢ Enhance debuggability of a system
¢ Learn as much as you can about your system
environment
SRE book what isn’t the goal of monitoring:
”It can be tempting to combine monitoring
with other aspects of inspecting complex
systems, such as detailed system profiling,
single-process debugging, tracking details
about exceptions or crashes, load testing, log
collection and analysis, or traffic inspection.
While most of these subjects share
commonalities with basic monitoring, blending
together too many results in overly complex
and fragile systems.”
9. © OPITZ CONSULTING 2023 / Interner Gebrauch
CHARACTERISTICS OF MODERN MONITORING SOLUTIONS
Declarative o11y management for your µArchitectures 10
¢ Must be
¢ Understood as platform
¢ Simple to use
¢ Simple to onboard new applications to
¢ Should be built upon open standards (e.g.
Open Telemetry)
¢ Should only collect data from which one can
make sense of
This is where many traditional
monitoring solutions are strong
10. © OPITZ CONSULTING 2023 / Interner Gebrauch
APPS / SYSTEMS MUST BE
MONITORABLE!
Declarative o11y management for your µArchitectures 11
¢ Modern software today should be “able”
¢ “Able”-Criteria:
Verifi
Test
Oper
Read
Evolv
Maintain
Observ
Scal
Recover
Prov
Afford
Deploy
Audit
Impenetr
Debugg
Credits to Lucas Jellema: https://www.slideshare.net/lucasjellema/code-with-honor-on-
the-importance-and-beauty-of-software-programming-rmoug-2021-training-days
Able
11. © OPITZ CONSULTING 2023 / Interner Gebrauch
FROM MONOLITHIC TO CLOUD-NATIVE ARCHITECTURES
Declarative o11y management for your µArchitectures 13
Traditional approach
Monolithic architecture
Cloud-native approach
Microservice architecture
12. © OPITZ CONSULTING 2023 / Interner Gebrauch
FROM CENTRALIZED TO DECENTRALIZED ARCHITECTURES
Declarative o11y management for your µArchitectures 14
Centralized
STATIC
ON-PREM
MONOLITH
VIRTUAL MACHINES
MANUAL CHANGE PROCESS
Decentralized
DYNAMIC
CLOUD / MULTI-CLOUD
MICROSERVICES / SERVERLESS
CONTAINERS, KUBERNETES
AUTOMATED CI/CD TOOL CHAIN
# Services & APIs
CONTROL AND VISIBILITY
13. © OPITZ CONSULTING 2023 / Interner Gebrauch
CONSISTENCE AND EFFICIENCY CHALLENGE
Declarative o11y management for your µArchitectures
¢ Multi-layered Observability
challenge
¢ Edge
¢ App
¢ In-App
15
14. © OPITZ CONSULTING 2023 / Interner Gebrauch
Declarative o11y management for your µArchitectures 16
SOLUTION STRATEGY
02
15. © OPITZ CONSULTING 2023 / Interner Gebrauch
CONCEPT: API GATEWAY
Declarative o11y management for your µArchitectures 17
¢ Single entry point for clients to access
Services
¢ No matter the implementation technology
¢ No matter the deployment model (Monolithic
or µService)
¢ Provides a consistent governance model
¢ Decouples Client and Service implementation
¢ Is deployed separately in its own instance
¢ Deployment models:
¢ Bundled data and control plane
¢ Independent data and control plane
Source: https://tinyurl.com/yxbds3cd
16. © OPITZ CONSULTING 2023 / Interner Gebrauch
CONCEPT: SERVICE MESH
Declarative o11y management for your µArchitectures 18
¢ Efficient implementation of cross-cutting
concerns with respect to service
integration challenges
¢ Everything is a service!
¢ Cloud-native apps deployed to Kubernetes
¢ Non Cloud-native workloads
¢ Should be independent of
¢ Architecture (e.g. Monolithic or µService)
¢ Platform (e.g. VMs, Containers, Kubernetes)
Dedicated infrastructure layer that
makes service-to-service communication
more reliable, secure and observable
17. © OPITZ CONSULTING 2023 / Interner Gebrauch
API GATEWAY & SERVICE MESH TO EMPOWER E2E CONNECTIVITY
Declarative o11y management for your µArchitectures 19
¢ API Gateway is integrated in the Mesh
¢ Is just another service
¢ Gateway proxy just handles outbound traffic
¢ Benefits:
¢ Increased Developer experience
¢ Seamless observability
¢ Consistent security
¢ Resilience & Reliable connectivity
¢ Flexibility
GW DP
CLIENT
PUBLIC TRAFFIC
GW DP
MESH CP
MESH 1 MESH 2
18. © OPITZ CONSULTING 2023 / Interner Gebrauch
Business
Logic (B)
Policy (P)
SHARED POLICY LAYER
Declarative o11y management for your µArchitectures 20
OTHER
MONOLITH
MICROSERVICES 3RD-PARTY
Routing
Load Balancing
Authentication
Authorization
Logs & Metrics
Rate Limiting
Caching
Validation
Transformation
MULTI-CLOUD & KUBERNETES
ON-PREMISE
HOSTING
B
Shared Policy Layer (Kong Gateway)
B B B B B B
B B B
Serverless
(e.g. AWS
Lambda)
More applications and services
More languages and protocols
REST}
{
More deployment types
19. © OPITZ CONSULTING 2023 / Interner Gebrauch
COLLECTING DATA AT THE EDGE LEVEL
Declarative o11y management for your µArchitectures 21
¢ Kong Plugins to emit respective data
¢ HTTP / TCP Log
¢ Prometheus
¢ Zipkin
¢ Open Telemetry
¢ Kong EE provides more information OOTB
(Vitals)
¢ # API calls (per API resource)
¢ # Errors / Successful requests
¢ Gateway might be deployed as
¢ Kubernetes Ingress Controller
¢ Standalone Gateway (on VM or Bare Metal)
20. © OPITZ CONSULTING 2023 / Interner Gebrauch
COLLECTING DATA AT THE APP-LEVEL
Declarative o11y management for your µArchitectures 22
¢ Kuma Observability policies are used to emit
needed data
¢ TrafficLog
¢ TrafficMetrics
¢ TrafficTrace
¢ Metrics data can be collected for Data and
Control plane
¢ Insights into Mesh Topology with Service Map
¢ Options for Mesh Gateway
¢ Kong
¢ Kubernetes Gateway API (if operated on K8s)
21. © OPITZ CONSULTING 2023 / Interner Gebrauch
ANALYZING AND MONITORING THE DATA
Declarative o11y management for your µArchitectures
¢ Using Grafana Stack to create a 360-degree view
¢ Component usage:
¢ Visualization: Grafana
¢ Logging: Loki (Log Shipping: FluentD / FluentBit / Promtail)
¢ Metrics: Prometheus (for long-term storage Cortex / Thanos)
¢ Tracing: Tempo
¢ Alerting: Prometheus Alert Manager
¢ Operating models
¢ Self-managed on-prem
¢ Grafana SaaS offering
23
22. © OPITZ CONSULTING 2023 / Interner Gebrauch
CONCEPTUAL O11Y ARCHITECTURE
Declarative o11y management for your µArchitectures 24
¢ Flexible, cloud-agnostic approach
¢ Independent of architecture and platform
¢ VM / Bare Metal
¢ Containers / K8s
¢ Cloud / On-prem
¢ Easily extensible
¢ Completely based on Open Source
¢ Declarative approach (no code changes)
23. © OPITZ CONSULTING 2023 / Interner Gebrauch
Declarative o11y management for your µArchitectures 25
USE CASES
03
24. © OPITZ CONSULTING 2023 / Interner Gebrauch
#1 DATA PLATFORM – CHALLENGE / GOALS
Declarative o11y management for your µArchitectures 26
¢ Distributed on-prem architecture based on VMs and Containers
¢ Ensure SLAs and operability of the data platform
¢ Insights to data access and processing in a Data Lake scenario
¢ Using o11y data to being able to analyze and optimize data access and processing
¢ Ingestion
¢ Processing
¢ Analysis
25. © OPITZ CONSULTING 2023 / Interner Gebrauch
#1 DATA PLATFORM – SOLUTION ARCHITECTURE
Declarative o11y management for your µArchitectures 27
¢ On-prem only scenario
¢ Challenge for platform team
¢ Patching / Upgrading
¢ Backup / Recovery
¢ Extending platform
¢ Monitoring platform
¢ Host-level: Checkmk
¢ App-level: Grafana Stack (Self-hosted)
26. © OPITZ CONSULTING 2023 / Interner Gebrauch
#1 DATA PLATFORM – SOLUTION ARCHITECTURE
Declarative o11y management for your µArchitectures 28
¢ O11y data is delivered by agents:
¢ Host Metrics: Checkmk Agents
¢ Mesh / App Metrics: Envoy, Prometheus
Exporters, Spring Boot Actuator
¢ Logs: Envoy, Promtail, FluentD
¢ Traces: Envoy, Spring Boot / Micrometer
¢ Agent distribution:
¢ Automation of agent distribution (Ansible)
¢ Envoy proxies for apps are automatically
initiated (if applicable)
27. © OPITZ CONSULTING 2023 / Interner Gebrauch
#2: INTEGRATION PLATFORM – CHALLENGES / GOALS
Declarative o11y management for your µArchitectures 29
¢ Distributed hybrid architecture based on VMs and Containers
¢ Ensure SLAs and operability of the integration platform (critical system)
¢ Insights to cloud-native integration flows
¢ Support API lifecycle management
¢ Support platform team to proatively operate the platform
¢ Increase debugability of the distributed platform in case of application errors
¢ Transparency about data usage
¢ Monitor overall platform state (not only infra)
¢ Insight into data flows with respect to state & performance
28. © OPITZ CONSULTING 2023 / Interner Gebrauch
#2: INTEGRATION PLATFORM – SOLUTION ARCHITECTURE
Declarative o11y management for your µArchitectures 30
¢ Distributed, hybrid architecture
¢ Monitoring Platform:
¢ Grafana Stack (Grafana Cloud)
¢ O11y data is delivered by agents:
¢ Mesh / App Metrics: Envoy, Prometheus
Exporters, Spring Boot Actuator
¢ Logs: Envoy
¢ Traces: Envoy, Java agent
29. © OPITZ CONSULTING 2023 / Interner Gebrauch
#2: INTEGRATION PLATFROM – TRACING EXAMPLE VIEW
Declarative o11y management for your µArchitectures 31
30. © OPITZ CONSULTING 2023 / Interner Gebrauch
#2: INTEGRATION PLATFROM – EXAMPLE ERROR DASHBOARD
Declarative o11y management for your µArchitectures 32
31. © OPITZ CONSULTING 2023 / Interner Gebrauch
Declarative o11y management for your µArchitectures 33
CONCLUSION
04
32. © OPITZ CONSULTING 2023 / Interner Gebrauch
HOPE IS NOT A STRATEGY!
Declarative o11y management for your µArchitectures 34
¢ We need to build monitorable apps /
systems
¢ Having a consistent Monitoring strategy is
key to build sustainable solutions
¢ NOTE: Having good monitoring in place
doesn’t mean that we compeltly get rid of
failures
¢ Just collect important information from
which you make sense, e.g. in dashboards or
alerts
Source: Unsplash
33. © OPITZ CONSULTING 2023 / Interner Gebrauch
THINK IN PLATFORMS!
Declarative o11y management for your µArchitectures 35
¢ Declarative o11y with API Gateway & Service
Mesh
¢ Provide a shared policy layer to configure
respective aspects
¢ Think in platforms!
¢ Make it easy to use, operate and extend
¢ For product teams (Developer
experience)
¢ For platform teams
34. © OPITZ CONSULTING 2023 / Interner Gebrauch
WHAT SHOULD BE MONITORED?
Declarative o11y management for your µArchitectures 36
¢ The amount of monitoring data is almost
infinite
¢ Limit the amount of data collected to
relevant ones
¢ Focus on the four golden signals mentioned
in Google SRE Book:
¢ Latency
¢ Traffic
¢ Errors
¢ Saturation
35. © OPITZ CONSULTING 2023 / Interner Gebrauch
Declarative o11y management for your µArchitectures 38
Q & A
36. © OPITZ CONSULTING 2023 / Interner Gebrauch
IN CASE OF ANY QUESTIONS…
Declarative o11y management for your µArchitectures 39
Sven Bernhardt
Chief Architect / Integration Evangelist |
Oracle ACE Director
OPITZ CONSULTING Deutschland GmbH
Kirchstrasse 6, 51647 Gummersbach, Germany
Phone: +49 172 2193529
Mail: sven.bernhardt@opitz-consulting.com
Twitter: @sbernhardt
Blog: https://svenbernhardt.wordpress.com