AWS Community Day 2022 David Kirk Hybrid Local Development Environments with EKS

1. Hybrid Development
Environments
David Kirk

2. What’re we talkin about?
- What’s the problem we’re trying to solve?
- What is a Hybrid Development Environment?
- How we’ve implemented it
- Practical considerations

3. About Me
- Started programming in Second Life
- 7 years fulltime - all DevOps
- d@vidkirk.com

4. What’s the Problem?
- Microservice Development
- Dozens of services
- Computational Overhead
- Software complexity grows faster than computational capability
- Cognitive Overhead
- “How do I run all these things?”
- Organizational Overhead
- Consistent Dev Environments
- Sharing work

5. Common Solutions
- Bare Metal
- Inconsistent across org
- Deploy complexity, or run a subset
- Compute constrained
- docker-compose
- Containerized, common deploy definition
- Complexity integrating into local development
- Not productionlike

6. What do we really want here?
- Simple to deploy
- Simple to maintain
- Consistent across devs
- Productionlike

7. - Access Cluster Traffic
- Outbound and inbound
- “Local is in the cluster”
Hybrid Dev Environment!
- k8s Cloud + Local
- Namespaced per dev
- Develop & Debug locally
- Productionlike
- Less toolchain/config work
- Devs exposed to Ops in day-to-day work
Dev’s Namespace
Dev’s Laptop
Main API
Backstage
Backstage
Intercepted
Traffic

8. Here’s a Diagram
Dev 2’s
Namespace
Dev 3’s
Namespace
Dev 1’s
Namespace
Pod A
Pod B
Pod C
Dev 1’s Laptop
Service D
Networking
Magic
Dev EKS
Cluster

9. Have any of y’all had
experience with
environments like this?

10. Implementation Challenges
- Still lots of Complexity
- How do you build everything?
- How do you deploy everything?
- How do you connect?
- Simplify Tooling
- Create common tools wherever possible
- Only introduce complexity where necessary
- Simplify Concepts
- Minimize the number of workflows
- Minimize cognitive overhead

11. Simplify Tooling
- Building & Deploying
- Complex systems are cool…
- Let’s just focus on the basics
- Single Entry Point - Skaffold
- Define all builds in here
- Define all deploys in here
- `skaffold build` and `skaffold deploy`
- Same Tool for Staging & Production
- Different config per environment

12. Single Build & Deploy Path
- Your workflows should not be pets
- Minimize cognitive overhead
- Reduce tooling development workload
- Single Build Path
- Tag by git hash
- No need to rebuild same hash
- Cache hit everything you didn’t change
- Single Deploy Path
- Environment-specific config
- Helm bakes the yaml
- k8s only updates what changes
Build &
Deploy

13. Implementation Details - Build & Deploy
- platform-k8s
- Build & Deploy Source of Truth
- Common Framework
- All engineers can contribute - 21 of 29 have!
- Shared Docker Daemon
- DinD in the cluster
- DOCKER_HOST=tcp://dind.dind:2374
- Don’t slow down dev machines
Docker-in-Docker
in the cluster
Dev
Laptop
Dev
Laptop
Dev
Laptop

14. Implementation Details - Cluster Interaction
- Telepresence
- Route traffic to cluster
- Intercept from cluster
- k9s
- Invaluable k8s introspection tool
- Logs, deploy status, etc

15. Implementation Details - Managed Services
- Run them in pods
- RDS -> MySQL, MSK -> Kafka, etc
- Disposable
- Easy to experiment with
- Data generation scripts
- Developers need to make these
- Critical for making stacks useful
- 3rd Party Services
- Case-By-Case
- Multitenant if possible, mock if not

16. What’s the Workflow Like?
1. repo-sync
2. make build-dev
3. make deploy-dev
4. telepresence connect
5. dev work!
If it breaks, turn it off and on again!

17. Pitfalls
- Pin your telepresence version
- Different versions step on each other
- This is unorthodox
- Most devs haven’t worked like this
- Be prepared to answer the same question 10,000 times
- It pays off - Devs love it & have Ops experience
- Everything needs to conform
- No pets
- Have to be strict about this

18. But it’s worth it!
- Devs are big fans
- Learning curve, but not too steep
- Learn to fix prod
- Learn to debug k8s
- One toolchain
- High Achievers Contribute to Tooling
- Yohei spent a week building data seeding
- Swaroop implemented KEDA - now in production
- Nick implemented mobile testing w/ EEs running API
- Jorde implemented web UI testing - started in Dev, now in Staging
- Productionlike

19. That’s it!