Freedom of choice is one of the key concepts in the SDN and NFV revolution we are seeing today. OPNFV is at the heart of this revolution yet very limited freedom of choice has existed on the hardware architecture side. However, with the work done in the Armband project, ARM servers are now an alternative hardware architecture for Brahmaputra deployments. The Armband team has ported the OPNFV Fuel Project to support deployments on ARM servers. The necessary code changes have been upstreamed through the OPNFV armband project. End users are now able to download or build their own Brahmaputra OPNFV ISO ready for ARM and install it using available OPNFV documentation. In addition to this and to further the OPNFV VNF ecosystem, a full specification OPNFV Pharos lab based on ARM servers was built by Enea for running continuous integration (CI) and continuous deployment (CD). In this presentation, we will walk you through the experiences gained in this process, the challenges and how they were overcome and what is coming next.

1. June 20–23, 2016 | Berlin, Germany

2. OPNFV on ARM
Hardware Freedom of Choice Has Arrived
Florin Dumitrașcu, ENEA

3. Armband Project Scope
 Add support for ARM Architecture-based servers in OPNFV
 Drive population of ARM-based servers in OPNFV Labs
 Work with upstream community to release OPNFV on
multi-architectures, on any relevant silicon required by the industry
What we do …
3OPNFV on ARM, OPNFV Summit 2016

4. The Team for Brahmaputra on ARM
4OPNFV on ARM, OPNFV Summit 2016

5. Armband Project Numbers
Commits per repository Authors per repository
Main committers to date: ENEA, Cavium, ARM
Source: http://projects.bitergia.com/opnfv/browser/scm-repos.html?page=1
5OPNFV on ARM, OPNFV Summit 2016

6. Armband Project Facts
o Development
• Git: https://git.opnfv.org/cgit/armband/
• Gerrit: https://gerrit.opnfv.org/gerrit/#/admin/projects/armband
• Jira: https://jira.opnfv.org/projects/ARMBAND/issues/
o Testing
• Dashboard: https://testresults.opnfv.org/dashboard/# (Fuel, arm-pod1)
o Continuous Integration / Continuous Deployment
• Jenkins: https://build.opnfv.org/ci/view/armband
o Artifacts for Brahmaputra
• ISO, documentation: http://artifacts.opnfv.org/armband.html
o Communication
• Wiki: https://wiki.opnfv.org/display/armband
• IRC Channel: #opnfv-armband @ Freenode
6OPNFV on ARM, OPNFV Summit 2016

7. ENEA Pharos Lab (Kista, Sweden)
• Applied Micro X-Gene 2, ARMv8-64 8 cores @ 2.4GHz
• 32GB RAM , 1x128GB SSD, 2x1TB HDD
• 1x10Gbps SFP+ NICs, 2x1Gbps NICs (RJ45)
• AMD Opteron A1100, ARMv8-64 Cortex A57, 8 cores
• 2 x 16GB RAM (32 GB total), 1 TB HDD
• 2 x 10Gbps NIC
https://wiki.opnfv.org/display/pharos/Enea-pharos-lab
• Cavium Networks CN8890-CRB 1S ThunderX
• ARMv8.1-64 48 cores @ 1.8GHz
• 8 x 16GB RAM (128GB total), 1 x 500GB HDD
• 1 x 40Gbps QSFP+ NIC, 2 x 10Gbps SFP+ NICs, 1 x 1Gpbs NIC
(RJ45, IPMI interface)• Cavium Networks CN8890-CRB 1S ThunderX
• ARMv8.1-64 48 cores @ 1.8GHz
• 8 x 16GB RAM (128GB total), 1 x 500GB HDD
• 1 x 40Gbps QSFP+ NIC, 2 x 10Gbps SFP+ NICs, 1 x 1Gpbs NIC
(RJ45, IPMI interface)
7OPNFV on ARM, OPNFV Summit 2016

8. Armband Workflow
clone repository
make submodules-init
make patches-import
modify and commit
make build
make patches-export
make submodules-clean
(create submodule)
1
2
3
4
5
6
7
8
$ git clone ssh://<Linux Foundation User>@gerrit.opnfv.org:29418/armband
$ git submodule -b stable/8.0 add --name fuel-web
https://github.com/openstack/fuel-web.git upstream/fuel-web
Apply patches from patches/<sub-project>/* to respective submodules in
/upstream/<sub-project>. Branch “armband-workbench” created in submodules.
All the sub-projects are registered as git submodules.
Call make submodules-init to initialize them.
Modify sub-projects for whatever you need. Commit your changes when you
want them taken into account in the build.
Build to validate changes.
Each commit on “armband-workbench” branch of each sub-project will be exported
to the patches /<sub-project>/ via “git format-patch”. Only commit patches !
Clean workbench branches.
8OPNFV on ARM, OPNFV Summit 2016

9. Armband Fuel Installer
o Patching upstream repositories for ARM
• OpenStack Fuel
• Fuel@OPNFV
• Fuel plugins
o Using Git Submodules
• submodules defined in .gitmodules file
• url – upstream repository
• branch – lock on upstream branch
• upstream – path for local workspace (clone)
o Pros
• Not an explicit upstream fork
• Organize patches
• Patch files before building
• Gerrit automatically pulls upstream changes into
submodules (requires patch rebasing)
o Cons
• Overhead for simple patches
• Cannot pin Fuel to a tag like Fuel@OPNFV does
Source: https://git.opnfv.org/cgit/armband/tree/.gitmodules
9OPNFV on ARM, OPNFV Summit 2016

10. /armband
Armband Directory Structure
/armband/patches
/armband/upstream Source: https://git.opnfv.org/cgit/armband/tree
/armband/patches/fuel-library
10OPNFV on ARM, OPNFV Summit 2016

11. aptly create mirror
Armband Package Repositories (1)
Local mirror of
fuel-infra repository
Upstream fuel-infra
repository
http://mirror.fuel-infra.org/mos-repos/ubuntu/
aptly update mirror
Local fuel-infra
package repository
aptly repo importaptly repo create
Snapshot of local
fuel-infra pkg repository
aptly snapshot create
Published local fuel-infra
package repository
(internal use)
http://<local-ip>/repo/mirantis/mos-repos/ubuntu/
aptly publish snapshot
Published snapshots
archive
(internal use)
http://<local-ip>/repo/archive
Contains both binary-amd64 and binary-
arm64 packages. However, in this
repository, only non-native packages
(“Architecture: all”) are contained for
binary-arm64 (e.g. “nova-api”)
11OPNFV on ARM, OPNFV Summit 2016

12. Native fuel-infra pkg built on
arm64, e.g. “ceph”
aptly repo add
gbp-buildpackage
(build/patch/version bump)
Native arm64 local fuel-
infra packages repository
Snapshot of arm64 local
fuel-infra pkg repository
aptly snapshot create
Armband arm64 ports packages
E.g. “kernel, qemu, grub-efi-arm64”
aptly repo add
gbp-buildpackage
(arm64 specific ports)
Armband arm64 local
packages repository
Snapshot of Armband
arm64 local pkg repository
aptly snapshot create
Published local fuel-
infra package repository
(internal use)binary-arm64 / Architecture: arm64
(e.g. “ceph, mongodb, galera”)
Armband Package Repositories (2)
binary-amd64 / Architecture: all, amd64
binary-arm64 / Architecture: all
(e.g. “nova-api”)
binary-arm64 / Architecture: arm64
(e.g. “kernel, qemu, grub-efi-arm64”)
Published Armband
package repository
(public)
aptly snapshot merge
12OPNFV on ARM, OPNFV Summit 2016

13. Armband Ubuntu Trusty Ports
Component ARM64 Notes
ARM64 Kernel 4.2.0 re-packaged from Yarmouth repositories.
One kernel that currently supports ALL our arm64 targets: Cavium, APM, AMD.
GRUB-EFI-ARM64 Bootstrapping nodes from Fuel's Cobbler requires an arm64 netloader, compatible with
syslinux like config (e.g. using MAC address as config name).
DMIDECODE dmidecode in Trusty does not support all arm64 targets properly, repackage with fixes from
upstream repository.
OHAI Add arm64 CPU details detection. Add ethtool support for reading L1 info.
QEMU Bump QEMU to 2.5 for multiple arm64 fixes and newly added support.
UDEV Provide a predictable eth naming scheme for arm64 (aarch64) boards with
on-SoC ethernet devices, like APM Mustang, AMD Softiron.
LIBGUESTFS Update to libguestfs 1.32.2 and repackage for Trusty. Fixes lots of dependecies for arm64.
LSHW Fix Ubuntu Trusty bug on arm64: "AArch64: slow cpuinfo due to redundant loop“.
OPNFV-QUAGGA Build opnfv-quagga required by ODL plugin on Ubuntu Trusty arm64 build.
13OPNFV on ARM, OPNFV Summit 2016

14. Fuel-infra ports for arm64 architecture
Component ARM64 Notes
MYSQL Version bump to MySQL 5.6.28 and add wsrep patch 25.13 for Galera support.
GALERA Version bump to Galera 3-25.3.14.
LIBVIRT Version bump to libvirt 1.3.2.
Rebase libvirt required by Qemu 2.5 on Ubuntu Trusty arm64 build GIC support.
Force GIC version detection by default on ARMv7/8.
CEPH Add arm64 architecture to debian/control (for ceph-fuse[-dbg]).
CIRROS Upstream package available for arm64. Add it to Armband fuel-infra repository.
MONGODB Enable arm64 Java Script (V8) support on arm64.
14OPNFV on ARM, OPNFV Summit 2016

15. Armband Functest
o Deployment scenarios
• Currently running os-odl_l2-nofeature-ha scenario only.
• Work in progress to activate os-nosdn-nofeature-ha, os-odl_l3-nofeature-ha.
o Tempest
• Using Cirros TestVM for arm64.
• Live instance snapshot and VM live migration require kernel and qemu
patching for GICv3 on ThunderX targets (affects 15 test cases).
o Rally
• Fix Rally to allow customization of RAM size for instance flavors.
These were hardcoded to 64M, 128M causing out of memory on arm64 in some
scenarios (https://review.openstack.org/#/c/315171/).
o vIMS
• Live test working with vIMS instantiated manually.
• Work in progress to port Cloudify on arm64 CentOS for vIMS orchestration.
o Yardstick
• Work in progress to port test images and tools on arm64.
15OPNFV on ARM, OPNFV Summit 2016

16. On the way to Colorado Release
o Migrate Armband Fuel installer from Brahmaputra (Liberty) to Colorado (Mitaka)
• Rebase Fuel components in Armband git repository.
• Rebase OPNFV components in Armband git repository.
• Update Armband package repository.
o Validate and extend where needed OpenStack Mitaka functionality on arm64
• Fix bugs and workarounds reported in Armband release notes aligned to Brahmaputra 3.0.
• Cherry-pick upstream fixes (if made available) for problems reported in same notes.
http://artifacts.opnfv.org/armband/brahmaputra/docs/release-notes.html
o Yardstick support
• Modify the script to build composite x86 + arm64 image with all packages needed by Yardstick.
• Validate ARM compatibility – tools, etc.
o Full orchestration of Clearwater vIMS with Cloudify for Functest
• Build CentOS7 arm64 image with everything required by Cloudify.
• Integrate in Functest.
16OPNFV on ARM, OPNFV Summit 2016

17. Upstream challenges
o Unify x86 and arm64 packages in a single mirror
o Fuel plugin builder currently using lots of hardcoding
o Common Fuel build (Fuel master) for x86 and arm64
environment deploys
17OPNFV on ARM, OPNFV Summit 2016

18. Questions
OPNFV on ARM, OPNFV Summit 2016 18

19. Thank you !