This document discusses a project that aims to facilitate container virtualization and migration across network edge nodes using OpenStack cloud infrastructure. The key objectives of the project are to research container image format support in OpenStack Zun, implement alternatives to pulling images from DockerHub, investigate passing parameters from Zun CLI to RunC, design checkpoint/restore functionality, and design a migration workflow. The project finds that OCI is the compatible image format. It implements pulling images from other URLs and storing them in Glance. It designs checkpoint/restore using CRIU to save runtime state.

1. Virtualization and Migration in Edge
computing – Cloud computing models using
OpenStack
Seva Sai praveen ISE2017013

2. Introduction
● This project is a part of SmartMe initiative by the Mobile and Distributed Systems Lab, University
of Messina, Italy.
● It focuses on facilitating container virtualization and migration across network Edge nodes on
OpenStack Cloud infrastructure.
● The primary use-case of the SmartME project is to leverage the OpenStack platform and
facilitate it to work for IoT devices on the Edge layer of the network.
● In continuation to that we also performed a thorough investigation to understand the thin line
between Cloud computing and Edge computing prior to working on mechanisms to leverage the
OpenStack.

3. SmartME
● SmartME is an ongoing project, led by the University of Messina and the Messina municipality
aiming at designing, deploying, and implementing a smart city architecture and infrastructure in
Messina.
● Low-cost microcontroller boards equipped with sensors and actuators are installed on buses,
lamp posts, and buildings of local institutions, all over the urban area to collect data and
information.
● Why OpenStack?

4. Introduction to terminology
Edge Computing
● Facilitates the operation of compute, storage, and networking services between end devices
and Cloud computing datacentres
● Distribution of the communication, computation, and storage resources and services on or close
to devices and systems in the control of end-users
● In this case, the IoT devices themselves act as nodes/Edges of the OpenStack Infrastructure
Often serves as a complement to Cloud computing, rather than substitute

5. Container
A container is a standard unit of software that packages up code and all its dependencies so the
application runs quickly and reliably from one computing environment to another. A Docker container
image is a lightweight, standalone, executable package of software that includes everything needed to
run an application: code, runtime, system tools, system libraries and settings.
Container Image
Container images are the snapshots of a running or a suspended Container. Container images
become containers at runtime and in the case of Docker containers - images become containers when
they run on Docker Engine.

6. Fig: Containers vs Virtual machines

7. OpenStack
● OpenStack is an open-source software platform for Cloud computing, mostly deployed as
infrastructure-as-a-service , whereby virtual servers and other resources are made available to
customers.
● The software platform consists of interrelated components that control diverse, multi-vendor
hardware pools of processing, storage, and networking resources throughout a data center.
● It assists in managing the fleet of IoT devices deployed across the smart city.
● Our project is built entirely on OpenStack infrastructure.

8. Zun
Zun is an OpenStack Container service. It aims to provide an API service for running application
containers without the need to manage servers or clusters. This is the core of implementation work
RunC
RunC, a lightweight universal container runtime, is a command-line tool for spawning and
running containers according to the Open Container Initiative (OCI) specification.

9. Migration
● It refers to the process of moving a virtual machine or an application between different physical
machines.
● It can be stateful or stateless

10. Project Objectives
● Research container image format support for OpenStack Zun
○ Zun currently uses runC to spawn containers from images pulled from DockerHub. We
need to make sure there is compatibility of dockerHub container images with runC and
Zun.
○ In order to ensure compatibility of dockerHub images and checkpoint-commited images
which are to be migrated across Edge devices, the underlying format of dockerHub
images and runC container runtime needs to be thoroughly investigated.

11. Project Objectives
● Research and implement alternatives to DockerHub for pulling container-images
○ The system in its current implementation only has support for images pulled via
DockerHub.
○ However, in order to make the process of migration simple and seamless it is intended to
allow the system to pull container images stored at any end URL, and successfully spawn
them as containers on the zun managed nodes.

12. Project Objectives
● Investigate the passing of parameters to RunC container runtime from Zun CLI
○ Since Zun uses runC as container runtime and any commands and parameters passed to
Zun are to be finally run on the container, a thorough inspection of how exactly the
commands and parameters that are being passed to Zun are resulting in the actions of
container needs to done.

13. Project Objectives
● Design checkpoint/restore functionality in addition to container commit
○ It is the most critical functionality that must be targeted before migration can be achieved.
○ In terms of a single process system implies that the runtime and execution state of the
process is saved on the persistent memory as a snapshot along with allowing us to
restore the process from its previous snapshot whenever desired.
○ In addition to allowing checkpoint/restore inside the container runtime, the container
image may also be modified with the process snapshot so that future containers may be
spawned from the modified image itself.

14. Project Objectives
● Design migration workflow of container image from one node to another
○ Checkpoint functionality of container saves the runtime of process in persistent storage.
○ Now, in order to ensure the smooth migration of the container, running container needs to
be paused.
○ This paused container should then be replicated and saved in desired container image
format. The container image is then required to be migrated to target node and
respawned.

15. Methodology
● OpenStack Zun uses runC to spawn containers. runC, can only spawn containers which abide
by OpenContainer Initiative Image Specification (Open Container Image - OCI format).
● OpenStack Zun, by default, uses dockerHub as disk/container-image repository to pull images.
These images on dockerHub are in Docker V2 Image format, which itself is based on OCI image
format
● Consequently, the images which are pulled from dockerHub are compatible with runC, and
thereby Zun, and can be spawned to form Containers. [9][10]
● In order to use custom images, it must be ensured that the images that are being used are
compliant with the OCI Image specification.
1. Research container image format support for OpenStack Zun
Docker Registry API to be standardised in OCI - https://blog.docker.com/2018/04/docker-registry-api-standardized-oci/
Docker leads OCI release of V1.0 runtime and image format specifications - https://blog.docker.com/2017/07/oci-release-of-v1-0-runtime-and-image-format-specifications/

16. Fig: Container Image compatibility

17. 2. Research and implement alternatives to DockerHub for pulling
container-images
● Glance
○ The OpenStack Glance is an image service which provides discovering, registering,
retrieving for disk and server images.The OpenStack Glance is a central repository for
virtual images.
○ Glance has RESTful API that allows retrieval of the actual images.
○ Glance supports wide range of image formats.
○ Container images can be pulled from and committed to Glance image repository of the
Cloud infrastructure via Glance command line interface.
Image formats support of Glance - https://docs.OpenStack.org/python-
glanceclient/latest/cli/details.html
Glance command line API - https://docs.OpenStack.org/python-glanceclient/latest/cli/details.html

18. ● Zun API expansion to use images stored at any endpoint
○ The Zun API parser is modified to create a new pullimage command that can take any
URL endpoint along with an image name, and download the OCI specification compatible
image from that URL. In addition, the single command is able to add that image to the
OpenStack image repository - Glance.
○ pullimage functionality uses python library functions to fetch and download file(container
image) over HTTP/HTTPS from any URL end point.
○ The downloaded container image is now added the local OpenStack Glance repository
using Glance command line API and is now available to use by Zun from the Glance
image repository.
○ These images must be OCI image format compliant and only then can they be
successfully spawned into working containers using the Zun API.

19. ● The architecture of Zun involves basically 3 layers:
○ Layer 1: Zun API for container management
○ Layer 2: Docker-py: Python library for Docker container management
○ Layer 3: RunC container runtime
● In order to passing parameters to the RunC container, we must leverage the Docker layer as
per the architecture of Zun in order to avoid future conflicts in design and maintaining
consistency.
3. Investigate the passing of parameters to RunC container runtime from
Zun CLI

20. 4. Design checkpoint/restore functionality in addition to container commit
● Zun commit allows containers that are spawned along with any files on board to be persisted as a
new container-image on the host system.
● However, does not retain the state of any running process inside the container.
● The use cases enlisted as part of the SmartMe project involve the checkpointing of
processes(preserve the runtime) and their migration and subsequent deployment on a different
node along the network Edge.
● This could not be achieved using current Zun API and thus we implemented a new technique to
achieve this.
● The flowchart on the next slide depicts the use of CRIU (Checkpoint/Restore In User-space) tools
to achieve the aforementioned functionality.

21. Fig: Migration workflow

22. 5. Design migration workflow of container image from one node to another
The workflow for achieving migration is as follows:
● Pause and checkpoint the container’s runtime using CRIU checkpoint functionality
● Commit the checkpoint-ed container to glance repo using Zun commit functionality
● Transfer the container image from one node to another using remote sync protocol or existing
migration techniques
● Save container image in target node’s glance repo
● Spawn the container in target node to the start the container and restore the persistent state
● Restore the run time of the process using CRIU restore

23. Results: Container format
● The spawning of containers as well as their migration is directly linked with the container image
format through the choice of tools and protocols that would eventually be used for migration.
● The default container runtime being runC for Zun, the migration techniques researched in the
future must be compliant with the image format that is being used in Zun.
● The findings of the research show that OCI (Open Container Initiative) is the format specification
that Zun is currently compliant with and any and all tools and technologies being used must
support this image format specification in order to integrate with Zun’s container management.

24. Results: Alternatives to DockerHub
● A new API specification was created in Zun that brings together functionality from Zun and
Glance (OpenStack image hosting service) to allow the user to download and spawn containers
from OCI compliant images that are hosted at any URL endpoint eg GDrive, AWS S3 etc.
● The use of pullimage functionality and subsequent spawning makes it easy to intermediately
host images on private servers before being pulled by other Edge nodes and spawned as
containers.
● The use of generic file hosting services allows us to host and store container images that are
not Docker V2 compliant but are OCI compliant and thus can be used successfully with Zun
container management service.

25. pullimage command
● zun pullimage “<Endpoint URL>” “<name of the image>”
● pullimage command pulls the image from the source and stores it in the OpenStack Image
repository Glance.
● Once the pulling and storing is done we then spawn the container using the image.
● zun run --image-driver glance “image name” ls

26. Fig: Description of the pulled image

27. Fig: Description of the spawned container

28. Fig: Container running status

29. Results: Passing of parameters
● RunC is completely wrapped around a DockerPy sublayer for management of containers. Thus,
Zun CLI is limited to invoking the relevant docker-py library functions only.
● Zun does not have any direct interaction with RunC although the containers that are being
spawned by Zun via the use of Docker-py library are essentially using a RunC runtime. So, all the
parameters passed to Zun via CLI will be invoking respective dockerPy library functions.
● As mentioned before, the implementation of various flags and parameters in the Zun API allows us
to have full control over the RunC parameters as long as it is supported in dockerPy.

30. Results: Checkpoint/Restore Design
● The mechanism for Checkpoint/Restore is the core of the migration process. We were able to
successfully create a working flow of commands and calls. By following the mentioned steps we
would be able to :
○ Stop a running process on the container and save it’s runtime and state as a set of files
on the container.
○ Commit the files to a new container image that can be migrated and deployed to a new
node or allow the current node to be stopped and killed with no loss to the progress of the
system.
○ Spawn the committed container image on new/same node and restore the saved process
to resume from the same state where it was paused.
○ Critical to the process of migration of running processes

31. Fig: Spawning a container for CRIU

32. Fig: commit operation

33. Fig: Glance image list

34. Conclusion and Future Scope
● Since everything after the completion of the earlier step exists as files in the controlling system,
it can be easily moved to any other node through remote sync protocols or existing container
migration techniques in order to continue execution of the runtime in the new node.
● The work done as part of the project creates a foundation to continue the work container
migration on network Edge by the team at University of Messina.
● The work involved as part of this project is open-source and is available for everyone to view
and leverage.

35. References
[1] Carlo Puliafito, Enzo Mingozzi, Carlo Vallati, Francesco Longo, Giovanni Merlino - Virtualization
and Migration at the Network Edge: an Overview
[2] Roberto Morabito - Virtualization on Internet of Things Edge Devices with Container Technologies:
a Performance Evaluation - May 2017 - IEEE Access PP(99)
[3] Antonio Puliafito et al - Building a Smart City Service Platform in Messina with the #SmartME
Project - 2018 32nd International Conference on Advanced Information Networking and Applications
Workshops
[4] Antonio Puliafito et al - Stack4Things: An OpenStack-Based Framework for IoT - 2015 3rd
International Conference on Future Internet of Things and Cloud
[5] Giovanni Merlino, Dario Bruneo et al - Stack4Things: integrating IoT with OpenStack in a Smart
City context
[6] Dario Bruneo - Head in a Cloud: An approach for Arduino YUN virtualization - 2017 Global Internet
of Things Summit

36. References
[7] Flávio Ramalho, Augusto Neto - Virtualization at the network Edge: A performance comparison -
2017 IEEE WoWMoM
[8] Stack4Things - https://github.com/MDSLab/stack4thing
[9] Docker Registry API to be standardised in OCI - https://blog.docker.com/2018/04/docker-registry-
api-standardized-oci/
[10] Docker leads OCI release of V1.0 runtime and image format specifications -
https://blog.docker.com/2017/07/oci-release-of-v1-0-runtime-and-image-format-specifications/
[11] Image formats support of Glance - https://docs.openstack.org/python-
glanceclient/latest/cli/details.html
[12] Glance command line API - https://docs.openstack.org/python-glanceclient/latest/cli/details.html
[13] Containers replacing Virtual machines - https://blog.docker.com/2018/08/containers-replacing-
virtual-machines/
[14] OpenStack DevStack installation - https://opendev.org/openstack/devstack.
[15] OpenStack DevStack documentation - https://docs.openstack.org/devstack/latest/
[16] OpenStack Horizon Dashboard Demo -https://www.youtube.com/watch?v=z6ftW7fUdp4

37. Thank You