Operationalizing Multi Cluster Istio and Lessons Learned and Developing Ambient Mesh

1. Operationalizing Multi Cluster
Istio: Lessons Learned and
Developing Ambient Mesh
Kevin Dorosh
Engineer @ Solo.io
Istio Contributor

2. 01
02
03
04
05
06
2
Istio Intro
Securing traffic
Resilience
Ambient Mesh
Ambient Mesh Demo
Brief Multicluster Considerations
What this talk will cover

3. 3
Connectivity & communication challenges

4. 4
Solo.io solves API connectivity & communication challenges

5. 5
Communication without Istio
frontend backend

6. 6
Istio
• Service discovery / Load balancing
• Secure service-to-service communication
• Traffic control / shaping / shifting
• Policy / Intention based access control
• Traffic metric collection
• Service resilience
• API / programmable interface
frontend backend
Istio

7. 7
Istio Architecture
https://istio.io/latest/docs/concepts/security/

8. 8
Secure outbound traffic
Application
Cloud Services
API Endpoints
Malicious website

9. 9
Find external services using the Kiali service graph

10. 10
Find external services using Prometheus
https://istio.io/latest/blog/2019/monitoring-external-service-traffic/

11. 11
Find external services using Envoy access logging
[2021-03-17T13:52:48.695Z] "GET /headers HTTP/1.1 -" 200 - "-" "-" 0 2128 63 62 "-" "curl/7.52.1"
"66e67f6d-4fa9-41e1-a103-1078b62221a5" "httpbin.org" "54.91.118.50:80" PassthroughCluster 10.44.0.7:34694
54.91.118.50:80 10.44.0.7:34692 - allow_any
log for HTTP:
curl http://httpbin.org/headers
[2021-03-17T13:56:04.611Z] "- - -" 0 - - - "-" 930 5818 153 - "-" "-" "-" "-" "54.91.118.50:443" PassthroughCluster
10.44.0.7:47868 54.91.118.50:443 10.44.0.7:47866 - -
log for HTTPS/TCP:
spec:
meshConfig:
accessLogFile: /dev/stdout
frontend

12. 12
Create ServiceEntries for external services
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: httpbin-ext
spec:
hosts:
- httpbin.org
ports:
- number: 80
name: http
protocol: HTTP
resolution: DNS
location: MESH_EXTERNAL
$ istioctl proxy-config endpoints ratings-v1-fd78f799f-gj8td | grep httpbin
34.199.75.4:80 HEALTHY OK outbound|80||httpbin.org
34.231.30.52:80 HEALTHY OK outbound|80||httpbin.org
54.166.163.67:80 HEALTHY OK outbound|80||httpbin.org
54.91.118.50:80 HEALTHY OK outbound|80||httpbin.org
https://istio.io/latest/docs/reference/config/networking/service-entry/

13. 13
Block undefined external endpoints
myapp
envoy
Change outboundTrafficPolicy to REGISTRY_ONLY
NOTE: Traffic can be configured to bypass envoy using annotations:
traffic.sidecar.istio.io/excludeOutboundIPRanges
traffic.sidecar.istio.io/excludeOutboundPorts
spec:
meshConfig:
outboundTrafficPolicy:
mode: REGISTRY_ONLY
https://istio.io/latest/docs/tasks/traffic-management/egress/egress-control/

14. 14
Direct external traffic thru egress gateways
myapp
envoy
curl http://httpbin.org/headers
Egress GW
httpbin.org
• Dedicated traffic exit points
• Only Egress gateways can run on internet enabled nodes
• Meet compliance requirements
• Use Network Policies on workloads to prevent bypassing the egress gateway
https://istio.io/latest/blog/2019/egress-traffic-control-in-istio-part-1/
https://istio.io/latest/docs/tasks/traffic-management/egress/egress-gateway/

15. 15
Encrypting traffic
frontend backend
frontend backend
fake-backend

16. 16
Istio Identity
$ istioctl dashboard envoy productpage-v1-5d9b4c9849-l8jjq.default
http://localhost:15000

17. 17
Istio mutual TLS mode
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: istio-system # can be granular
spec:
mtls:
mode: PERMISSIVE # or STRICT
• Change PERMISSIVE to STRICT
after onboarding all your
workloads to the mesh
• Can be disabled at the port level
frontend backend
Another App

18. 18
Verify secure traffic using Kiali

19. 19
Verify secure traffic using Envoy access logging
format: "[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%
%PROTOCOL% %DOWNSTREAM_TLS_VERSION%" %RESPONSE_CODE% …
$ kubectl logs reviews-v1-6549ddccc5-f28vj -c istio-proxy
[2021-03-17T00:52:08.689Z] "GET /reviews/0 HTTP/1.1 TLSv1.2" 200 - "-" "-" 0 295 8 7 "-" "Mozilla/5.0 (Macintosh;
Intel Mac OS X 11_2_3)
Add %DOWNSTREAM_TLS_VERSION% to your access log formatting:
Check the proxy logs:

20. 20
Check for envoy listeners
$ istioctl pc listener reviews-v1-6549ddccc5-f28vj | grep inbound
0.0.0.0 15006 Trans: tls; Addr: *:9080 Cluster: inbound|9080||
PERMISSIVE:
$ istioctl pc listener reviews-v1-6549ddccc5-f28vj | grep inbound
0.0.0.0 15006 Trans: tls; App: Istio HTTP Plain; Addr: *:9080 Cluster: inbound|9080||
0.0.0.0 15006 Trans: raw_buffer; Addr: *:9080 Cluster: inbound|9080||
STRICT:

21. 21
Controlling workload access
catalog
orders
GET GET
PUT
POST
PUT
ui database

22. 22
AuthorizationPolicy
apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
name: "database-viewer"
namespace: default
spec:
selector:
matchLabels:
app: database
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/default/sa/myapp-orders"]
to:
- operation:
methods: ["GET"]
paths: ["/data"]
orders
PUT
database
Can also target Ingress Gateway
Optional
Can also support IP blocks
Or DENY
https://istio.io/latest/docs/reference/config/security/authorization-policy/

23. 23
Use access logs to create your rules
$ kubectl logs reviews-v2-76c4865449-6wblx -c istio-proxy
…
[2021-03-17T02:27:18.143Z] “POST /reviews/0 HTTP/1.1 TLSv1.2" 200 - "-" "-" 0 379 29 28 "-" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"
"2dc83abd-71e1-40f5-9a6d-5f02dfceaa78" "reviews:9080" "127.0.0.1:9080" inbound|9080|| 127.0.0.1:59238 10.44.0.8:9080
10.44.0.9:37276 outbound_.9080_._.reviews.default.svc.cluster.local default
…
[2021-03-17T02:29:31.493Z] "GET /ratings/0 HTTP/1.1 -" 200 - "-" "-" 0 48 3 2 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X
11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"
"a51c2b7b-a8e3-417e-8566-4258b734c287" "ratings:9080" "10.44.0.7:9080" outbound|9080||ratings.default.svc.cluster.local
10.44.0.8:51176 10.56.48.22:9080 10.44.0.8:56738 - default
https://istio.io/latest/docs/tasks/observability/logs/access-log/
IP of pod sending the request
(kubectl get pods -o wide)
method
Outbound traffic going to ratings
Inbound traffic

24. 24
Debugging Authorization Policies
$ istioctl x authz check mydatabasepod
ACTION AuthorizationPolicy RULES
ALLOW database-viewer.default 1
ALLOW _anonymous_match_nothing_ 1
$ curl —-request POST database:9080
RBAC: access denied
$ kubectl logs mydatabasepod -c istio-proxy
[2021-03-17T19:31:34.595Z] "GET / HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-"
"curl/7.52.1" "8510e2fd-53f7-4a5a-95da-8796d90c51a2" “database:9080" "-" - - 10.44.1.8:9080 10.44.0.7:51260
outbound_.9080_._.database.default.svc.cluster.local -
Response to caller when denied:
Check what policies are applied on a pod:
Access log entry on the server sidecar:

25. 25
Request Authentication
JWT
catalog
orders
GET
POST
ui

26. Request Authentication
apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
name: "jwt-example"
spec:
selector:
matchLabels:
app: orders
jwtRules:
- issuer: "example-provider"
jwksUri: "https://example-provider.com/jwks.json"
ui
JWT
orders
https://istio.io/latest/docs/tasks/security/authorization/authz-jwt/

27. 27
Improve application resilience
frontend backend

28. Timeouts, circuit breaking and rate limiting
apiVersion:
networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: reviews
spec:
hosts:
- reviews
http:
- route:
- destination:
host: reviews
subset: v2
timeout: 0.5s
apiVersion:
networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: httpbin
spec:
host: httpbin
trafficPolicy:
connectionPool:
tcp:
maxConnections: 1
http:
http1MaxPendingRequests: 1
maxRequestsPerConnection: 1
outlierDetection:
consecutive5xxErrors: 1
interval: 1s
baseEjectionTime: 3m
maxEjectionPercent: 100
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: filter-local-ratelimit-svc
namespace: istio-system
spec:
workloadSelector:
labels:
app: productpage
configPatches:
- applyTo: HTTP_FILTER
listener:
filterChain:
filter:
name: "envoy.http_connection_manager"
patch:
…
stat_prefix: http_local_rate_limiter
token_bucket:
max_tokens: 10
tokens_per_fill: 10
fill_interval: 60s
…
https://istio.io/latest/docs/tasks/traffic-management/request-timeouts/
https://istio.io/latest/docs/tasks/traffic-management/circuit-breaking/
https://istio.io/latest/docs/tasks/policy-enforcement/rate-limit/

29. Istio Ambient
Mesh

30. Istio - The Industry’s Leading Service Mesh
2017
Istio Launched
2022
Ambient Mesh
Launched
Data Plane
Enhancements
2019-20
7 New Community Releases
1000s Production Customers
~ 1000 Community Contributors
2022
CNCF
2019-2022

31. Introducing Istio Ambient Mesh
31 | Copyright © 2022
A new, open source contribution to the Istio project, that
defines a new sidecar-less data plane.
Solo.io and Google are the lead contributors to Istio
Ambient Mesh.
Cost
Reduction
Simplify
Operations
Improve
Performance

32. What is Istio Ambient Mesh?
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
Proxy
Istio Sidecar Data Plane
1 Pod/Container = 1 Proxy
Ambient Mesh Data Plane
1 Node = 1 Proxy
Move from Sidecar Proxy per-pod architecture
to a Proxy per-node architecture.
“Making the Mesh
Transparent to Applications”
● Reduced Compute Cost
● Improve Business Continuity
● Increase Business Flexibility
● Simplified Operations
● Reduced Maintenance
● Simplified Upgrades
● Easier to Add Applications
● Less Day-2-Day Complexity
● Adapt to Application Needs
● Offer SLAs for Applications
● Many Apps = 1 Platform
Application
Team
● Mesh is transparent to Apps
● Applications won’t break
● Flexible Performance Available
● Manage Security vs Performance
Business
Owner
Platform
Team

33. Istio enables Zero-Trust Security
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
L4 Proxy
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
Istio Security with Sidecar Proxy Istio Security with Ambient Mesh
L4 Proxy
L7 Proxy
● All traffic goes through Proxy
● Proxy manages mTLS, Identity
● Proxy manages L7 Application Filters | Policies
● All traffic goes through Proxy
● L4 Proxy manages mTLS, Identity
● L7 Proxy manages L7 Application Filters | Policies

34. Istio enables Service Mesh flexibility
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
Proxy
Istio Sidecar Data Plane
1 Pod/Container = 1 Proxy
Ambient Mesh Data Plane
1 Node = 1 Proxy
Istio Control Plane
Cost Flexibility
Operational Flexibility
Performance Flexibility

35. Architecture: Visualizing ztunnel

36. Architecture: Visualizing ztunnel

37. HBONE - The protocol used to connect nodes
HTTP Based Overlay Network Encapsulation protocol
source: https://www.solo.io/blog/understanding-istio-ambient-ztunnel-and-secure-overlay/

38. Latest Ambient Developments
Rust Ztunnel
source: https://www.solo.io/blog/rust-ztunnel-istio-ambient-mesh/
https://www.semanticscholar.org/paper/A-work-stealing-scheduling-framework-supporting-Wang-Ji/2b956f8fcd744c4aaf8b0a
ed417f4706cc046545

39. Ambient Mesh
Demo

40. Multicluster
Considerations

41. 41 | Copyright © 2022

42. 42 | Copyright © 2022

43. 43 | Copyright © 2022

44. 44 | Copyright © 2022

45. Thank You!