Slide from my talk on "Instant developer onboarding with self-contained repositories". https://sched.co/l9yG Code examples on: https://github.com/Yshayy/self-contained-repositories Conference Recordings will be added once it will be public

1. Instant Self-contained Development
Environments for Everyone
Yshay Yaacobi
CTO @ Livecycle
yshay@livecycle.io
yshayy

2. TWEEKATHON

3. @yshayy
About me
● CTO & co-founder of livecycle
● Full-stack developer for quite a time
● Passionate about cloud development,
backend architecture, UX/DX, functional
programming and Docker
● Creator & maintainer of Tweek - an open-source, “cloud-
native” feature management solution
● Things I care about in code: simplicity, consistency and elegance

4. @yshayy
About Livecycle
● Next generation collaboration tools for development teams
● Continuous playground environments
● Bridging the gap between coders & non-coders
● Soon in public beta

5. How does it feel like to start working on a new
complex codebase?

6. @yshayy
Try to build & run
● Wrong OS
● Missing or conflicting SDKs and/or PL runtimes
● Package managers throw “random” errors

7. @yshayy
Read README.md and try to make it work
● Run some magic scripts (and watch them fails)
● Change hosts file
● Install required tools & dependencies on the environment
● Install Root CA?

8. @yshayy
Try to develop
● Debugging doesn’t work
● IDE have problem with autocomplete or dependencies
● Code watch & build doesn’t work and we need watchman
● HMR doesn’t work because of websocket issues
● Problems with external dependencies
● CORS

9. Integration tests?

10. Do it all over again after few months of working
on something else....

11. @yshayy
Why is it so difficult?
● Different OSes (or versions)
● Lots of versions and fragmentation of sdk & runtimes
● Work on my machine syndrome
● Vast amount of different toolchains, IDEs, extensions
● Complex development flows that are difficult to setup and easily break:
○ Debugging, watching+building, hot-reloading, docker mounts
● Developers’ machines are polluted and overloaded with tools
● Environments and tools tend to change rapidly in active repositories
● ----> Waste of time and frustration!

12. The dream
Development environments that are...

13. Consistent
Provide the same predictable experience...

14. Reproducible
It’s possible to destroy & rebuild them

15. Isolated
Don’t or get affected by other environments

16. Self-Contained
All dependencies and tools needed for development
are defined & packaged inside

17. UNBREAKABLE
We won’t struggle countless hours to get things
working again

18. @yshayy
Running our environments in containers

19. @yshayy
Before we begin
● All examples & slides are available on Github
● All tools used in this presentation are OSS or free-to-use projects
● Most examples here are far from bullet-proof and some of them use tools that
can be considered experimental

20. @yshayy
Example #1 - qeesung/image2ascii
CLI tool for creating ASCII art from images
● Project written in Go
● Before Go modules

21. @yshayy
The Development Container
● Integration with SCM
● Remote code editing
● Remote terminal

22. @yshayy
Configuring our environment
● Setting Runtime/SDKS/CLIs
● Setting environment variables and path
● Configure our shell
● Defining extensions

23. @yshayy
Example #2 - yshayy/email-sender
Simple Flask app to send email based on SendGrid example
New challenges
● Running & interacting with a server
● Managing secrets
● Debugging

24. @yshayy
Secrets Encryption
● Secrets sit inside the repository
● Using Mozilla Sops for encrypting secrets
● GPG keys are nice for start, but it can also integrate with cloud encryptions-as-
service solution such as KMS, KeyVault
● MetaData is saved unencrypted which make it easy to do diffing and check
history
● Practice usually used in GitOps context
● Other solutions - git-secret, git-crypt, Kamus, SealedSecrets, etc...

25. @yshayy
IDE settings
● Launch settings - launch.json
● Port forwarding - forwarding port on the localhost

26. @yshayy
#3 HabitRPG/Habitica
OSS Web-based RPG for organizing your life
New Challenges
● Huge project
● Front end
● Backend
● DB

27. @yshayy
Full-stack application
● Docker-compose
● DB Image
● Additional tools

28. @yshayy
Data seeding
● Basic scripts
● Alternatively, we can clone data from staging/production

29. @yshayy
Use reverse proxy
● Route services to several subdomains instead of ports
● Wildcard “localhost” dns-es (localtest.me, xip)
● Traefik - a very simple and developer friendly reverse proxy

30. The next one is personal...

31. @yshayy
#4 - Soluto/Tweek
Cloud-native open-source feature flags and configuration management
New challenges
● Several microservices
● Several DBs/Messaging systems
● Cross service communications
● Polyglot environment

32. @yshayy
Complex architecture

33. @yshayy
“Nested” containers
● Docker-in-docker vs docker-from-docker
● Development in nested containers with Tilt + Docker-Compose
○ Watching & rebuilding on every code commit
○ Remote debugging
○ “Hot” code reloading if possible
● Things can get slower...

34. @yshayy
“Mock” cloud dependencies
● Docker images of databases (redis, mongo, postgres, etc…)
● Wire-compatible solutions (Minio, OIDC mock server)
● Manual mocks
● Full frameworks (localstack)
● Encrypted credentials with dedicated tenants. (or dynamic provisioning)

35. @yshayy
#5 - kubecost/cost-model
Tool for managing kubernetes costs
New challenges
● We need kubernetes
● Metrics server
● Prometheus

36. @yshayy
What can we do with Kubernetes?
● Kubernetes local development is already difficult
○ Fragmentation - Mini-kube, Docker for Desktop k8s, micro-k8s, kind, k3s, etc…
○ Versioning
○ Upgrading
● Using a single kubernetes distro+version can make life easy

37. @yshayy
Kubernetes in dev-container
● K3S - minimal kubernetes distribution
● K3D - Make it easy to run k3s and a dedicated registry inside Docker
● Stable and cluster can be re-created
● Helm controller for installing helm charts declaratively
● Tilt facilitate building/pushing/running

38. @yshayy
Docker Host
Dev Container
IDE
Tilt
Docker-in-Docker
Registry
K3S Node
ContainerD
App
To put it “simply”....

39. @yshayy

40. No more demos…

41. @yshayy
Containerized development environments
● Development environment configuration is also source-controlled and
correspond to the application code.
● Developer machine stays clean
● Can scale well to multiple environments without conflicts
● Can run locally or remotely

42. @yshayy
Our setup
● ~10 microservices in Golang & typescript/js
● Frontend with HMR
● Our own Kubernetes CR and controllers
● External dependencies (GH api, sendgrid, auth0, etc…)
● DB + Graphql engine
● A full blown CI system
● Container Registry
● Dynamic dns subdomains
● SSL Certs for local development
● Multiple CLI/SDKs for kubernetes and code-generation

43. @yshayy
Results
● Time to teardown and download/rebuild all cluster and dependencies <15m
● Time to build/run/test code changes < 10s
● Time to onboard new developer < 3h
(including remote provisioning of a dedicated host on AWS)
● Time to introduce new tool and update dev-environments if needed <5m)
● No “works on my machine” occurrences
● No strain on developer machines
● Developers need to deal less with secrets
● Our team work on both M1 and Intel Macs

44. @yshayy
Future optimizations
● Shared build cache
● Snapshots (for reducing time or sharing state)
● Using a cloud provider optimized for dev-machines (cost, location, hibernation,
cpu/ram, etc...)

45. @yshayy
Drawbacks
● Creating the initial setup can take some time
● Many tools, some are bleeding-edge
● Additional code to manage
● Dev-Environments are not standardized yet
● Coupling to VS-Code, Docker, Git and Linux
● Some performance issues
● Security challenges between development and production context

46. @yshayy
Alternatives to VS Code?
● It’s possible to use terminal based code-editors
● GitPod.io & Theia have similar features with gitpod.yaml
● Jetbrains Projector
● Run local IDE with Docker mounts

47. @yshayy
What about serverless?
● Should work although some pieces might be missing
● FaaS frameworks and sdks usually can run locally
● For the other PaaS features - cloud mocking frameworks, emulators or wire
compatible solutions. (localstack, minio, etc…)
● If necessary, throw IaC tools to the mix (pulumi/terraform) for dynamic
provisioning

48. @yshayy
What about native mobile?
● The problem still exists, developers can engage in epic battles with mobile IDEs,
workspace, build and debug.
● It might be possible to stream applications with VNC/WebRTC during
development, but experience is not optimal
● Mobile-emulators are heavy and can require nested virtualization to perform
● Container ecosystem is optimized for Linux
● IDEs for mobile are very tailored for mobile development
● Might be easier/possible with more cross-platform frameworks such as react-
native/flutter

49. @yshayy
Part of a larger trend to put more stuff in the repository
● Linting, style guidelines
● Declarative application dependencies
● Documentation
● OpenAPI Specification
● CI Pipelines definition
● Workflows (PR)
● Design systems
● Infrastructure-As-Code
● Secrets (encrypted)
● Dashboards/alerts/SLA configuration
● Notebooks
● ...

50. @yshayy
Self-Contained Repositories
● All code, tools, knowledge, definitions and processes related to project resides
in the repository.
● Git as the single source of truth
● Code is more accessible, lowering the barrier of entry
● Applications are portable
● Fine-grained developer experience
● Emerging tools ecosystem

51. @yshayy
Patterns & Cheatsheet
Challenge Solution Exampletools
Basic SDK/Runtime
dependencies
Development in
container
VSCode+Docker, GitPod
Cloud dependencies Compatible dockerized
implementations
Minio, redis
Initial application data Data seeding Scripts, replicating from cloud
More hardware Remote environment Docker-machine, codespaces
Multi-Container Apps Nested Containers Dind, compose, tilt
Kubernetes Apps Nested Kubernetes Kind, k3d, tilt, skaffold
Exposing network
dependencies
Reverse proxy, wild card
development dns
Traefik, nginx, *.xip, *.localtest.me
Serverless Mock cloud frameworks Localstack

52. Thank you
@yshayy
https://github.com/yshayy/self-contained-repositories

53. Questions