This presentation, from the Embedded Linux Conference Europe in October 2016, discusses how resinOS was built, highlights some of its key features, and shares a roadmap for future development and contribution.
resinOS is the latest open-source tool built by resin.io to enable the future of hardware with the tools of modern software. resinOS is a simple yet powerful operating system that brings standard Docker containers to embedded devices and works on a wide variety of device types and architectures. resinOS was born from the team’s experience deploying embedded containers across device types and has been battle-tested in production environments.
You can download resinOS at https://resinos.io
4. Be the embedded OS of choice for containers in IoT
Create a community around containers for IoT
Modern security features
Minimal footprint
Production ready
Mission
5. Started 4 years ago
Modern devops practices to the embedded world
Naturally leaned towards containers
Ported Docker to ARMv6
Ported Docker to ARMv5
Fixes upstreamed
History - resin.io
6. Needed an OS for our platform
Tried a modified Arch
Tried a modified TinyCore
Both had important shortcomings
History - resinOS
7. Started in January 2014 as internal project
Used Yocto as a base
Open sourced in July 2015
Currently under very active development
It’s been running in production for 2.5 years
History - resinOS
11. meta-resin
meta-resin
meta-resin-common
Jethro overlayer Fido overlayer Daisy overlayer
Main resinOS layer
Automatic aufs patching
BSP independent kernel configuration
Can prepopulate docker images
Kernel headers for out-of-tree module development
https://github.com/resin-os/meta-resin
12. Environment defined in a Dockerfile
Predictable host configuration
Docker image artifacts
You can use the OS as a container
resin/resinos:<version>-<board>
Build system
https://github.com/resin-os/resin-yocto-scripts
13. Separate rootfs and root state
We know exactly which services write to disk
Dual root partition
data partition auto-expands on first boot
Partition layout
rootA databoot rootB state
14. Forced us to investigate all writes
Configuration stored in state partition
Network configuration
Random seed
Clock at shutdown
Some state is stored in tmpfs
DHCP leases
Limited logs
Read-only root
16. Compartmentalisation of failures
Device can survive data partition corruption
Most I/O activity happens in there
Root partition is never written to while in use
We strive to do atomic operations everywhere
Reliability
19. Leverage a lot of systemd features
Adjusting OOM score for critical services
Running services in separate mount namespaces
Very easy dependency management
NTP
Socket activation for SSH
Saves RAM since ssh is running only when needed
Systemd
20. DNS is hard
dnsmasq
Integration of Docker with host’s dnsmasq
NetworkManager
Excellent D-Bus API
ModemManager
Excellent D-Bus API
Lots of documentation
Networking
21. AUFS driver
Allows support for NAND based devices
Currently on docker 1.10.3
Backported stability patches
Journald logging driver
Avoids SD card wear
Seccomp enabled
Docker
22. All logs end up in journald
In RAM 8MB buffer by default
Configurable log persistence
Journald allows for structured logs
Container logs are annotated with metadata
Easy to send logs to a central location to store and process
Log management
24. Some boards have internal storage
Image for these boards is a flasher
Automatic copying to internal storage
Feedback through LEDs
Two stage flashing
25. So many options
It’s one of our biggest focus areas
resinhup is our current approach
Takes advantage of dual root partition
Validates everything before changing the state
It’s still experimental
Host OS updates
https://github.com/resin-os/resinhup/
26. ● Used by
CoreOS, ChromiumOS, Ubuntu Snappy
Brillo, Mender.io
But wastes a lot of space
We’re experimenting with more advanced approaches
ostree
docker
Dual root partition method
27. Integration with docker
It uses docker to pull the OS image
It then unpacks and applies it
Leveraging important docker features
Signed images
Programmatic API for fetching
Open question: can unify containers and host?
ResinHUP
https://github.com/resin-os/resinhup/
28. Automatic emulated testing
● We support virtual QEMU boards
● Automated basic testing on every PR
○ Booting
○ Networking
● Integrated with our Jenkins
https://github.com/resin-io/autohat
29. Automatic hardware testing
● Manual testing doesn’t scale
○ Currently 22 boards
● We built a board that instruments boards
○ GPIO
○ Provisioning
○ SD muxing
○ Wifi testing
https://github.com/resin-io/autohat-rig
30. ARM64
● Coming soon
ARMv6
● RPI Zero
● RPI model 1 A+
ARMv5
● TS7700
Device support
ARMv7
● Raspberry Pi 2
● Raspberry Pi 3
● Samsung Artik 5
● SamsungArtik 10
● Beaglebone Black
● Beaglebone Green
● Beaglebone Green Wireless
● Odroid C1/C1+
● Odroid XU4
● SolidRun Hummingboard i2
● Boundary Devices Nitrogen6x
● Parallella Board
● VIA 820 board
● Zynq zc702
● TS4900 single and Quad
X86_32
● Intel Edison
X86_64
● Intel NUC
31. Device support
● Easy to add new boards
● Meta-resin handles
○ Userspace
○ Image generation
○ Kernel configuration
33. ● How do you..
○ Configure network credentials?
○ Provision a device?
○ Develop on the board?
○ Get logs?
Development tools
34. ● Development images have
○ Open SSH server
○ Docker socket exposed over TCP
○ mDNS exposed metadata
● Device is at <hostname>.local
Development mode
35. ● Image configuration
● Wifi credentials
● Hostname
● Persistent logging
Resin Device Toolbox
$ rdt configure ~/Downloads/resinos-dev.img
? Network SSID super_wifi
? Network Key super_secure_password
? Do you want to set advanced settings? Yes
? Device Hostname resin
? Do you want to enable persistent logging? no
Done!
36. ● Automatically detects removable storage
● Won’t wipe your drive!
● Validates after writing
Resin Device Toolbox
$ sudo rdt flash ~/Downloads/resinos-dev.img
? Select drive /dev/disk3 (7.9 GB) - STORAGE DEVICE
? This will erase the selected drive. Are you sure? Yes
Flashing [========================] 100% eta 0s
Validating [========================] 100% eta 0s
37. Docker development
Finds device in local network
Continously syncs code into the container
Rebuilds when necessary
Resin Device Toolbox
$ rdt push --source .
* Building..
- Stopping and Removing any previous 'myapp' container
- Removing any existing container images for 'myapp'
- Building new 'myapp' image
38. ● More than 500 images for each supported device type
● Debian, Fedora, Alpine
● Nodejs, python, golang, Java
● Follow docker conventions
Base Images
https://github.com/resin-io-library/base-images
Ποιοι είμαστε
Γιατί κάνουμε το resin
Πως δουλεύει το resin
We will upload the slides
The links next to the github logo link to the repo for what is mentioned in the slide
We believe containers give some unique capabilities when applied to IoT
Developers use very familiar tools
They don’t have to reboot for every iteration
If their container completely crashes the OS is still there to recover
But also have some unique challenges
A container in the cloud interacts with its environment by requiring computation, memory, networking, storage
All of these are very well abstracted and have no different if they are virtual or not
You can’t have a virtual GPIO
Containers in IoT have to interface with hardware sometimes
Devices can be connected with to very poor networks or via 3G, bandwidth matters
We’re also aiming at making the OS developer friendly and providing the tools that allow you to build great applications
We’ve made a lot of progress during the past years and now we’re making an official release as a standalone project
We were managing a fleet of 200 devices across London
Most of the tooling felt primitive. It was a hard problem
We wanted to bring more modern tools to the embedded world
Bridge yocto world with docker/cloud world
Started with `git push` workflow, only supported nodejs projects
Most distributions are not optimised for remote management and embedded devices
We had a very early prototype which sometimes it would start being slow
Turned out filesystem indexing was kicking off in the background
We had to manually build tooling around creating images, installing software
Most distributions support a limited set of architectures
resinOS is the result of our efforts to have a reliable system in production
Has been running on thousands of devices
We moved the development to the open a bit more than a year ago
We’re now making it a proper open source project
Yocto had drawbacks, namely steep learning curve, resource-hungry builds
Benefits outweight them by far
We can disable compile-time features that we don’t use
Very big community in the embedded space
Provides the right tools for distribution maintainer
For each board we support there is a GH repo with a well-defined structure
We manage yocto dependencies by using git submodules
We can move each board independently through versions without blocking the development of meta-resin
Normally each such repo has no code, but sometimes some glue code is put there in a non-submoduled layer
This is where the bulk of the work is happening
We need AUFS for docker - we’ll talk about it later on why we chose this backend
Meta-resin auto-detects if the kernel provided by the bsp has aufs support (like beaglebone) and automatically applies patches if not
Meta-resin automatically extends the kernel config provided by the BSP to enable the right features for container to run
Prepopulating docker images means easier mass-production/provisioning
We use a docker container for the build env yocto runs in
Well defined versions of host dependencies
One of the artifacts of the build process is a docker image containing the userspace and boot files
We are actually pushing this image in dockerhub for every release
This can be used as a base for a container or for testing
It will also become relevant in the talk when we talk about OS updates
Meta-resin creates this layout by default
Data partition is where all containers and container state is kept
State partition is where partition state is kept
We generally try to keep the boot partition as a FAT partition
This allows our development tools to read and write to it cross-platform
We chose to have a read-only root partition to make failures much less likely
During board boot we bind-mount a lot of relevant directories from the state partition to the appropriate locations
Another approach, used by OpenWRT, would be having a read-only root (squashfs in their case) and using overlayfs on top.
We didn’t want to be dependent on a union filesystem for the rootfs
Some POSIX semantics are lost
You can still get random mutations of the rootfs that will bite you later when you update
The goal of the OS is to be fully updatable but also flexible. If all the application code was living in the host then a complex host OS update would be required for every change.
This is where containers come in.
Our userspace is designed to do the absolute necessary things to bring up the board and connect to the internet
Using this distinction we can have very fast reboot-less updates for containers
But we can still update the host with traditional methods but much more infrequently
The initial versions of our OS were based on sysvinit
But we quickly found ourselves having a hard time managing dependencies
Sometimes restarting a service wouldn’t kill all its processes
Each program was managing its own logs
We plan to leverage it a lot more in the future
Networking outside the data center is a wild place
Glibc only supports max 3 nameservers
In some networks DHCP dns is broken
In some networks google DNS is broken
In some networks DNS server take forever to reply
In some networks DNS servers reply with the wrong data
D-Bus can be easily exposed to containers
Have also tried connman
We had to patch it to convince it to not do NTP
Documentation buried into repo and examples in test suite
We didn’t have a good way of persisting a config file describing 3G connections
ModemManager gets us that
In general much more developer friendly
Overlay needs 3.18+ and uses a lot of inodes
Overlay2 needs 4.0+
BTRFS is not stable enough, especially for low storage devices
Devicemapper doesn’t work on NAND
Zfs… yeah
Docker’s default logging driver will keep appending your stdout into a very big JSON file on disk
We configure docker to send logs to journald
Meta-resin also enables seccomp in the kernel and docker uses a default profile for all containers
We don’t allow anything to write its own logs
One can easily send the stream of journald logs to a remote server
If configured, ResinOS can store logs in the
Yocto by default produces the host OS image but sometimes this needs to be written to internal storage
Dependening on the board this might involve a few steps
ResinOS can create a “flasher” type image that copies itself into internal storage and shuts down the board
There are so many options for OS updates. I think there are 4 talks on this subject in this ELCE
Resinhup is how are addressing our current production needs
It’s still experimental and tested in a very carefully selected set of customers
Host OS updates is a very hot topic and is definitely
One important feature we added recently is integration with docker
Pulling down an fs image using a tarball felt wrong when we had docker
Currently we’re using docker only for pulling an image and then copy it to the rootfs
Docker has a lot of nice features that can be leveraged. Signed images for example
Automate all possible ways of booting a board ( SD card / DFU / Network booting? ).* Emulate the peripherals that could be connected to the SBC ( GPIO/SPI/I2C/Bluetooth).* All external hardware used to run the tests should be on USB to enable for portability of the entire rig.* Simulate and test different network conditions under which a SBC operates.* Accompanying software components should have an easy DSL/way of writing and extending tests, so that peripheral abstractions can be re-used.* Accompanying software component should enable Continuous Integration with Jenkins* All custom hardware designs should be in KiCad to allow for easy contribution as the project is open source.
Automate all possible ways of booting a board ( SD card / DFU / Network booting? ).* Emulate the peripherals that could be connected to the SBC ( GPIO/SPI/I2C/Bluetooth).* All external hardware used to run the tests should be on USB to enable for portability of the entire rig.* Simulate and test different network conditions under which a SBC operates.* Accompanying software components should have an easy DSL/way of writing and extending tests, so that peripheral abstractions can be re-used.* Accompanying software component should enable Continuous Integration with Jenkins* All custom hardware designs should be in KiCad to allow for easy contribution as the project is open source.
20 boards
Normally you have to connect with ethernet/serial cable
Configuration is tricky
Flashing
All of our code is open source and our goal is to build a community around this project
We’ll be more than happy to chat with you on our gitter channel