Meetup by CloudZone @Google offices, Tel-Aviv, Israel - 15.10.2018

1. Managing Microservices with Istio 1.0 on GKE
SERVICE MESH DEEP DIVE

2. • We Participate in CNCF K8s & Istio’s Meetings

3. Used to be in System
Used to be a developer
A devops Engineer
An amateur pianist
> WHOAMI

4. AGENDA – we’re not going to cover it all
Introduction
Service Mesh Concept
Istio & it’s Architecture
Installation
Request Routing
Resiliency
Monitoring Mesh services
OpenTracing & App changes
Visualizing
Kiali
Integ. With Stackdriver
Traffic director
Mutual TLS Authentication
End-User Authentication
Authorization
Observability Multicluster/ Mesh Exp.
Traffic management Security
Integ. With Stackdriver
Traffic director
GCP Specifics

5. What’s a Service Mesh
…network of microservices and the interactions between them…
…As it grows in size and complexity, it becomes harder to understand and manage. Its requirements
can include discovery, load balancing, failure recovery, metrics, and monitoring…
…more complex operational requirements, like A/B testing, canary releases, rate limiting, access
control, and end-to-end authentication.
A service mesh solution is a configurable layer that
attends these requirements, in secure, usable way
…usually implemented by providing a proxy sidecar…

6. Istio
• Backed by Google, IBM & Lyft
• The best-known service solution mesh today (Poor LinkerD/Conduit).
• Supported natively on Kubernetes
• First version, 0.1 – may 2017 (Checked heavily, by many)
• Current version 1.0.2 (1.0 is the 1st GA, Production Ready version)
• Leverages the sidecar model
• Uses Envoy for its ambassador proxies

7. • Means A messenger, Ambassador
• High-performance L4/L7 proxy written in C++, designed for observability
• Support HTTP/2 & gRPC
• Mediates all inbound/outbound traffic for services inside the Istio mesh
• Istio (pilot in particular) leverages many of Envoy’s features
• Injected as a sidecar container in a the service’s kubernetes pod.
• Part of the CNCF
• Current v1.8 (1.9-dev)
Envoy – The Dataplane

8. Why use Istio?
• Enjoy features without application code changes
• Supports HTTP, HTTP2, gRPC, Webscokets, MongoDB
• Zone & Region aware load-balancing
• Service-2-service mTLS authentication & authorization
• Smart traffic routing & many resiliency features e.g. circuit breaking
• Automatic metrics/app monitoring, logging & tracing
• Mesh expansions and multiuser features.

9. Control-plane Architecture
• Pilot (Service Discovery for envoys, Traffic management, Resiliency)
• Converts high level rules to envoy specifics
• Decouples service discovery platform specifics
• Mixer (Enforcement, Telemetry)
• Enforces access control & policies
• Collects telemetry from proxies and other services
• Citadel (Security)
• Provides mTLS with service/user-2-service authentication
• Manages credentials and identity at scale
• Manages certificates lifecycle for workloads

10. • Use helm for production installation – Helm install / Helm template
• Helm can be used for easy low level customization of Istio’s components and feature
• Setups scalable control plane by default
• Helm 2.10+ will make life easier (manually create CRDs and remove)
• Provides ansible playbooks for installing on a VMs or Openshift
• Sidecar Injection can be done manually with istioctl (kube-inject)
• Or automatically with the side-car-injector (controlled then with policy and annotations)
Istio Setup & Sidecar Injection

11. Bookinfo app

12. • Pilot manages the Envoys & configures rules for routing traffic between the Envoys
• Envoys are propagated with rules in an eventual consistency way.
• Istio uses concept of service versions – a sub division of service instances
• Routing to versions according to headers, weights, tags of source/dest. and more
• Common scenarios: A/B Testing, Canary deployments, %-splits, Community features
• Mirroring production traffic
Traffic Management – Request Routing

13. • Istio’s supports zone-aware load balancing modes: (envoy supports more)
• Round-robin, random, weighted least request (Zone-aware)
• Envoys use Pilot’s service disc. Interface to update load balancing pools.
• Envoy distributes traffic to instances in its pool.
• Envoy periodically checks health of upstream service instances.
• Envoy ejects/return instances to the pool with the circuit breaker pattern
• Kubernetes built-in load balancing is bypassed.
Traffic Management – Load Balancing

14. • Istio & Envoy provide recovery & resiliency features:
• Timeouts
• Retries
• Concurrent connections and request rate limits
• Active periodic health checks on load balancing members
• Passive health checks with outlier detection (from real resonses)
• Active & passive health checks minimize the chances for unhealthy pod to receive traffic
• Combination of all reduce request failure and impact on latency to minimum.
• Override with x-envoy-upstream-rq-timeout-ms & x-envoy-max-retries
• Fault Injection – Simulate failures & loaded downstream services
Traffic Management – Handling failures

15. • Ingress traffic to the mesh should also flow through an envoy
• Make traffic mgmt. enabled for front microservices.
• Gateway+Ingress Gateway is Istio’s replacement for k8s Ingress
• Unlike K8s Ingress, Istio Ingress support L4-L7
• Support UDP, HTTP/2, gRPC, Webscoekts
• Ingress Gateway is an Envoy proxy, in an Edge mode
• Can use Ingress Gateway as an API manager.
• Egress: By default, external services are unavailable for mesh services
• Set ServiceEntry for allowing access to external hosts from within the Mesh
• By default, mesh external hosts are not registered as services in the mesh – you get 404
Traffic Management – Ingress/Egress

16. • 4 API resources: VirtualService, DestinationRule, ServiceEntry, Gateway
• VirtualService – Rules as for how request is routed to a service
• DestinationRule – Which Policies to set once VirtualService routing occurred
• ServiceEntry – Enable requests to services outside the Mesh
• Gateway – HTTP/2/TCP edge LoadBalancer for ingress traffic for an application
Traffic Management – Configuration

17. Traffic Management – Subsets
kind: VirtualService
metadata:
name: reviews
spec:
hosts:
- reviews
- reviews.default.svc...
- reviews.other-ns.svc...
http:
- route:
- destination:
host: reviews
subset: v1
weight: 75
- destination:
host: reviews
subset: v2
weight: 25
timeout: 10s
retries:
attempts: 3
perTryTimeout: 2s
kind: DestinationRule
metadata:
name: reviews
spec:
host: reviews
trafficPolicy:
loadBalancer:
simple: RANDOM
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
Review Pods
Label:
Version=v1
Review Pods
Label:
Version=v2
75% 25%
random Round_robin
100 max conn.
All microservices
In k8s, percentage is controlled with physical replicas

18. Traffic Management – Conditional Rules
kind: VirtualService
metadata:
name: reviews
spec:
hosts:
- reviews
http:
- match:
- sourceLabels:
- version: v1
- app: productpage
- (uri/method/headers/
port, headers…)
route:
- destination:
host: reviews
subset: v2
weight: 25
- route
- destination:
host: reviews
subset: v1
weight: 75
kind: DestinationRule
metadata:
name: reviews
spec:
host: reviews
trafficPolicy:
loadBalancer:
simple: RANDOM
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
Review Pods
Label:
Version=v1
Review Pods
Label:
Version=v2
75% 25%
random Round_robin
Product page
Pods
Label:
Version=v1
Demo
Rest of the
microservices

19. Traffic Management – Timeouts, Retries, CircuitBreaking
kind: DestinationRule
metadata:
name: httpbin
spec:
host: httpbin
trafficPolicy:(portLevelSettings)
connectionPool:
tcp:
maxConnections: 1
http:
http1MaxPendingRequests: 1
maxRequestsPerConnection: 1
http2MaxRequest: 1
outlierDetection:
consecutiveErrors: 5
interval: 1s
baseEjectionTime: 3m
maxEjectionPercent: 100
kind: VirtualService
metadata:
name: reviews
spec:
hosts:
- reviews
http:
- route:
- destination:
host: reviews
subset: v1
weight: 100
timeout: 1s
retries:
attempts: 2
perTryTimeout: 1s

20. Traffic Management – Mesh external services
kind: VirtualService
metadata:
name: google
spec:
hosts:
- www.google.com
tls:
- match:
- port: 443
sni_hosts:
- www.google.com
route:
- destination:
host: www.google.com
port:
number: 443
weight: 100
timeout: 1s
kind: ServiceEntry
metadata:
name: google
spec:
hosts:
- www.google.com
ports:
- number: 80
name: https
protocol: HTTPS
resolution: DNS
location: MESH_EXTERNAL
kind: DestinationRule
metadata:
name: google
spec:
host:www.google.com
trafficPolicy:
connectionPool:
http:
http1MaxPendingRequests: 10
maxRequestsPerConnection: 10

21. Traffic Management – Gateways & Ingress
kind: Gateway
metadata:
name: example-apps
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- ”*.example.com"
kind: VirtualService
metadata:
name: httpbin
spec:
hosts:
- ”great-app.example.com"
gateways:
- example-apps
http:
- match:
- uri:
prefix: /status
- port: 80
route:
- destination:
port:
number: 8000
host: ”great.default"
Show Gateway
kind: VirtualService
metadata:
name: httpbin
spec:
hosts:
- ”amazing-app.example.com"
gateways:
- example-apps
http:
- match:
- uri:
prefix: /catalog
- port: 80
rewrite
uri: /newcatalog
route:
- destination:
port:
number: 8000
host: ”great.default"

22. Traffic Management – Mirroring
• Send a copy of live traffic to a mirrored service
• Test production with minimum risk with real live traffic
• Occurs out of band of the production service - not impact on latency
kind: VirtualService
metadata:
name: httpbin
spec:
hosts:
- httpbin
http:
- route:
- destination:
host: httpbin
subset: v1
weight: 100
mirror:
host: httpbin
subset: v2

23. Security – Microservices
• Microservices require:
• Defense against man in the middle
• A way to provide identify to microservices, so it can provide access control
• Auditing – who access what and when
• Istio provides that at scale while:
• No change needs to be made to app code
• Integrate with existing security systems
• Build security on top of untrusted networks

24. Security – Identity in Istio
• Identity is fundamental in any security infrastructure
• Citadel manages identities with certificates in SPIFFE format
• SPIFFE – open-standard for identities in cloud native domain
• https://www.slideshare.net/prabathsiriwardena/cloud-native-identity-with-spiffe
• Istio uses SPIRE – an implementation of SPIFFE
• In k8s, the SPIFFE identity is in the form of spiffe://cluster.local/ns/<namespace>/sa/<serviceAcctName>
• e.g spiffe://cluster.local/ns/default/sa/a-microservice
• Citadel automates key & certificate rotation at scale

25. Security – Identity Lifecycle on K8s
K8s API Server
watch for SAs
1. Citadel watches k8s api for ServiceAccounts, then generates cert. & key-pairs
2. Citadel mounts the relevant cert. & key pair with a k8s secret to the Pod
3. Citadel watches the certificates and rotates them by re-writing the k8s secret
4. Pilot generates map of which identity is authorized to run what services
Products Microservice Pod
SA
products
Gen/Rotate cert/KP for
products microservices
certs the Istio-proxy
Envoy container App container
SA products
secret
Istio-certs
secret
Secure naming
products <-> SPIFFE://clus..

26. Security – Microservices authentication
• Service-to-service authentication with citadel generated certificates – upgrades channel to be encrypted
• Pilot watches Policies & DestinationRules, and updates the proxies with how to perform authentication
• Mutual TLS – In addition, called microservice validates the caller client certificates
• App can be http, while mTLS auth. occurs transparently - no handling of certificates in code
• End-user authentication enable proxies to validate user’s JWT tokens before reaching the app

27. Security – AuthenticationPolicy & DestinationRule
kind: "DestinationRule"
metadata:
name: ”a-microservice"
spec:
host: a-microservice.bar.svc.cluster.local
trafficPolicy:
tls:
mode: DISABLE
portLevelSettings:
- port:
number: 8080
tls:
mode: ISTIO_MUTUAL
kind: "Policy"
metadata:
name: ”a-microservice"
spec:
targets:
- name: a-microservice.bar…
ports:
- number: 8080
peers:
- mtls: {}
DEMO
Currently, Istio supports only mTLS is, thus
DestinationRules must be used if AuthPolicy
requires mTLS

28. Security – End User Authentication
kind: "Policy"
metadata:
name: "jwt-example"
spec:
targets:
- name: httpbin
origins:
- jwt:
issuer: "testing@secure.istio.io"
jwksUri: "https://...” <– jwt web key set to verify the token
principalBinding: USE_ORIGIN
JWKS is a set of keys containing the public keys that should be used to verify any JWT

29. Security – Microservices Authorization
• Toggle with RbacConfig singleton API resource instasnce
• Configure with ServiceRole & ServiceRoleBinding
• ServiceRole defines the permission e.g. Service names, HTTP methods
• ServiceRoleBinding can refer a user, group or service
• In mTLS, caller microservice validates server identity is authorized to run it
• Istio Authorization is like WAF, while NetworkPolicy is on L3-L4
• End-user authorization – Istio refer JWT claims in Istio RBAC role bindings
kind: ServiceRole
metadata:
name: books-review-reader
namespace: default
spec:
rules:
- services: [”a-microservice.ns"]
paths: ["*/reviews"]
methods: ["GET”,”HEAD”]
constratins:
- key: destaintion.labels[“version”]
value: [“v1”,”v2”]
(- key: ip/port/namespace/user/headers)
kind: ServiceRoleBinding
metadata:
name: allow-reviews-to-product-ms
namespace: default
spec:
subjects:
- user: "service-account-a"
roleRef:
kind: ServiceRole
name: books-review-reader

30. Security – Microservices Authorization
kind: ServiceRole
metadata:
name: reviews-reader
namespace: default
spec:
rules:
- services: [”reviews.ns"]
paths: ["*/reviews"]
methods: ["GET”,”HEAD”]
constratins:
- key: destaintion.labels[“version”]
value: [“v1”,”v2”]
(- key: ip/port/namespace/user/headers)
kind: ServiceRoleBinding
metadata:
name: allow-reviews-to-product
namespace: default
spec:
subjects:
- user: ”product-service-account"
roleRef:
kind: ServiceRole
name: reviews-reader

31. Observability – Polcies & Telemetry
• Mixer provides telemetry collection and policy enforcement
• Policies such as Rate limits & black/white listing
• Envoy checks Mixer for policies for each requests (with cache)
• Envoy buffers telemetry on requests, and deliver to Mixer
• Like Pilot, Mixer abstracts away policy & telemetry infra backends
• Mixer holds a shared cache in case external backends are unavailable
• Telemetry collected from Envoys is attribute based.

32. Observability - Monitoring
Microservice Pod
Envoy container
kind: prometheus
metadata:
name: handler
namespace: istio-system
spec:
metrics:
- name: request_total
instance_name: requestcount.metric.istio-system
kind: COUNTER
label_names:
- destination_service
- response_code
Handler def.
Who scrape what
from mixer
kind: metric
metadata:
name: requestcount
namespace: istio-system
spec:
dimensions:
destination_service: destination.service.host
.
.
response_code: response.code | 200
value: "1"
What to report Mixer
Bind handler to metric
According to Rule
kind: rule
metadata:
name: promhttp
namespace: istio-system
spec:
actions:
- handler: handler.prometheus
instances:
- requestcount.metric
match: context.protocol == "http"
Envoy request attributes
Handlers e.g. Fluentd, denier…

33. Observability - Tracing
• Tracing is usually based on aggregating spans - when and where the request was in the microservices path
• Envoy supports Zipkin (or Zipkin compatible e.g. Jaeger) and Lightstep – Both OpenTracing API based
• Istio comes with Jaeger (ingest & visualize) as default
• Envoy sidecars generate and propagate B3 Headers and send them to Jaeger
• It is the Application responsibility to propagate these headers (OpenTracing instrumentation)
• In SpringBoot2 you can import opentracing-spring-jaeger-cloud-starter
• NodeJs npm install jaeger-client | npm install zipkin / zipkin-instrumentation-express
• Some do all the work for you / in other you need to build a tracer that propagates/sends B3 Headers

34. Observability – Monitoring & Tracing
App
container
Sidecar
µService A Pod (HTTP/s Service)
Mixer
Scape
Visualize
Report
metrics
App
container
Sidecar
µService A Pod (HTTP/s Service)
App
container
Sidecar
µService A Pod (TCP Service)
Telemetry
Sidecar span
App. component span
Aggregated
view
Report
opentracing
spans

35. Observability – A Word about Kiali
Service Mesh Observability

36. Observability – Debugging the mesh
• istioctl proxy-status - Pilot & Envoys sync status
• Istioctl proxy-status <pod-name> - Envoy | Pilot configuration diff
• istioctl authn tls-check – debugging mTLS authentication
• Istioctl proxy-config cluster / endpoint – show services and their endpoints in envoys config
• Istioctl proxy-config route – route configuration of envoy

37. Multicluster MESH
K
Hosting K8s Cluster
Multicluster – Separate meetup
Istio-system
Istio Cont. plane
exposed with I/LBs
Root CA
K
Hosting K8s Cluster
Istio-system
Istio-remote
K
Hosting K8s Cluster
Istio-system
Istio-remote
VM /
Bare metal
Envoy
Sign certificates
Mesh expansion
Register Sidecar
Mesh expansion / Multicluster requires Pods ips available outside the cluster. E.g. aws_vpc_cni, gke native clusters

38. Google Cloud & Istio
• Integration with Stackdriver for metrics and logging
• Traffic Director – Multicloud/Hybrid managed Istio Service Mesh
• Integration with Google Cloud Endpoints

39. Thank you