Continuous Deployment is all over software companies. We will look at how to transfer some of the methods to hardware startups. Exploring embedded devices as an iterative process rather than from a traditional engineering approach.
- why is hardware so hard
- hardware test types
- over the air updates
- A case for continuous deployment
- vendors hate you
- libvirt test host hardware abstraction
- A quick look at a pratical jenkins setup
6. bankruptcysoftwaremanufacture
Hardware is Hard
- Hardware is done at massive scale
- Hence it used to be a waterfall business
- Fixing anything costs millions and takes years
- Hence test suits are massive and take forever
designspecify
8. Embedded Software is Hard
- work on very few, very
expensive prototypes
- Test requires specialised jigs and
probes
- Manual reset required when
hardware gets stuck
- All the tools are sudo and bad
9. Embedded Software is Hard
Typical test script you get from vendors:
- sudo apt-get -y install some-stuff-that-doesnt-exist
- sudo echo > /dev/something #never worked
- sudo someflashtool-binary --device=/dev/ttyUSB0
- sudo ./usbthing --find-device-automagically
- sudo scp 192.168.1.2:/test.xml /home/android/meh.
cvs
- echo probably worked. like i care. lol
10. The 2000 Solution
- TTY to LAN
- Allows sharing hardware within teams
- Conflict resolution: “shout across the
office”
- I can’t believe people still do this
- Usb controllable PSU
- To “reboot” during test cycles
14. … or something...
- Anything using kernel namespaces
- Or in this case libvirt
15. Hub Agent at Nokia
1 usb hub
=
1 virtual machine
16. Hub Agent
udev
Hub agent
New device on hubNew usb hub
Spawn agent VM
Register agent
Attach device to agent
Set agent label
17. Hub Agent at Superscale
Docker now supports usb.
Yey!
18. Over The Air Updates
CD Pipeline would be incomplete without delivery
Don’t build devices without internet in 2016.
Continuously deliver updates
19. Over The Air Updates
1,000,000 SCADA systems
are exposed on the internet
- Used in Factories, Nuclear plants,
Hospitals,...
- 700+ vulnerabilities
- No OTA = no Fix
- => Have OTA, plz
20. Over The Air Updates
Typical memory layout
system
user data / config state
bootloader
calibration
Vendor code from hell
Done in factory. Don’t touch.
Stuff to delete in CI.
Always test from clean state
OTA goes here
21. Over The Air Updates
Why OTA is full-image
➢ This is hardware, it doesn’t change.
➢ We must test a known state
➢ OTA scales to billions
22. Over The Air Updates
Classic staged rollouts before CD
The thing
git push git pull
Build & test
Roll to 1000 customersRelease for real
27. Vendors
Hardware vendors are “agile” too.
memory chip origin: “found box in garage”
➢ Yield test != works
➢ Have every single variant in your test rack
➢ Write down chip versions on incoming
batches
➢ Beware of changes even within batches
➢ Thanks to continuous deployment, we actually
CAN test all variants on git push in parallel
28. Vendors
Destruction tests you want on every variant
➢ Flash memory until death (must survive
specified number of cycle)
➢ Max that cpu, forever, until it desolders
➢ Battery charging cycles.
➢ Extreme environment conditions
➢ Find creative ways of destroying stuff
➢ (yeah, you, don’t trust that vendor)
29. Exit
Internet hardware is hard, implement CD early
➢ Flash memory until death (must survive
specified number of cycle)
➢ Max that cpu, forever, until it desolders
➢ Battery charging cycles.
➢ Extreme environment conditions
➢ Find creative ways of destroying stuff