"Session ID: BUD17-402 Session Name: Virtual Perspectives on Cross-compilation - BUD17-402 Speaker: Alex Bennée Track: Virtualization ★ Session Summary ★ Cross compiling (building software for one machine type on another) is something that should be familiar to most ARM developers. While kernel cross-compilation is well supported the story for user space is more complex. There are dedicated build systems for creating an entire rootfs from scratch but the overhead of learning them is quite for a single program. The usual fall-back of a linux-user chroot can be quite fiddly to set up and is often unique to each developer's set-up. Virtualisation developer Alex Bennée offers some solutions that utilise Docker's container based approach, optionally with QEMU's linux-user emulation to do builds directly from the command line. In contrast to custom chroots Docker allows the creation of well specified, re-buildable containers that can be simply deployed by new developers. He will talk about the ongoing work to make building test binaries for foreign architectures in QEMU as simple as a make invocation. --------------------------------------------------- ★ Resources ★ Event Page: http://connect.linaro.org/resource/bud17/bud17-402/ Presentation: https://www.slideshare.net/linaroorg/bud17402-virtual-perspectives-on-crosscompilation Video: https://youtu.be/8q9gWreQYZU --------------------------------------------------- ★ Event Details ★ Linaro Connect Budapest 2017 (BUD17) 6-10 March 2017 Corinthia Hotel, Budapest, Erzsébet krt. 43-49, 1073 Hungary --------------------------------------------------- Keyword: Virtualization, cross-compilation http://www.linaro.org http://connect.linaro.org --------------------------------------------------- Follow us on Social Media https://www.facebook.com/LinaroOrg https://twitter.com/linaroorg https://www.youtube.com/user/linaroorg?sub_confirmation=1 https://www.linkedin.com/company/1026961"

1. Virtualperspectivesoncross-
compiling
AlexBennée
LinaroConnectBUD17-402
1

2. Budapest 2017
WhoAmI
Alex Bennée
Senior Virtualisation
Engineer
@ Linaro since 2013
Mostly work on QEMU's TCG
2 . 1

3. Budapest 2017
VirtualisationSolvesEverything!
2 . 2

4. Budapest 2017
TalkStructure
2 . 3

5. Budapest 2017
UseCase-KernelCompile
3 . 1

6. Budapest 2017
TheKernelCompile
Running:
Gives:
export PATH=${HOME}/${COMPILER}/bin:$PATH
export ARCH=arm
export CONFIG_CROSS_COMPILE=arm-linux-gnueabihf-
make -j9 | tail -n 3
LD arch/arm/boot/compressed/vmlinux
OBJCOPY arch/arm/boot/zImage
Kernel: arch/arm/boot/zImage is ready
3 . 2

7. Budapest 2017
KernelCompilesAreEasy!
Self Contained
No external libraries
so
no headers
no shared objects
no static archives
no pkg-con g
3 . 3

8. Budapest 2017
UseCase-SimpleTestCase
4 . 1

9. Budapest 2017
TheCode
#include <stdio.h>
int main(int argc, char **argv)
{
printf("Hello Worldn");
}
4 . 2

10. Budapest 2017
TheCompile
Running:
Gives:
aarch64-linux-gnu-gcc --static simple-hello.c -o simple-hello
file simple-hello
simple-hello: ELF 64-bit LSB executable, ARM aarch64, version 1
(SYSV), statically linked, for GNU/Linux 3.7.0,
BuildID[sha1]=4712144fd1986c295641da66f7f5d40d35a2b1fb, not
stripped
4 . 3

11. Budapest 2017
UseCase-Usingotherlibraries
5 . 1

12. Budapest 2017
TheCode
#include <stdio.h>
#include <glib.h>
int main(int argc, char **argv)
{
g_message("Hello Worldn");
}
5 . 2

13. Budapest 2017
ANativeCompile
Getting include paths:
Compiling:
Gives:
pkg-config glib-2.0 --cflags --libs
-I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-
2.0/include -lglib-2.0
gcc glib-hello.c ${FLAGS} -o glib-hello-native
file glib-hello-native
glib-hello-native: ELF 64-bit LSB executable, x86-64, version 1
(SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-
64.so.2, for GNU/Linux 2.6.32,
BuildID[sha1]=1a4db3259cd31f564d2ec60654ef478d372b20d2, not
stripped
5 . 3

14. Budapest 2017
ACrossCompile
Running:
Gives:
FLAGS="$(pkg-config glib-2.0 --cflags --libs)"
aarch64-linux-gnu-gcc glib-hello.c ${FLAGS} -o glib-hello-arm64
/usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-
gnu/bin/ld: cannot find -lglib-2.0
collect2: error: ld returned 1 exit status
5 . 4

15. Budapest 2017
UseCase-Multi-archuser-spacebuild
6 . 1

16. Budapest 2017
EnablingARM64onDebian
dpkg --add-architecture arm64
apt-get update -qq
apt-get install -yy libglib2.0-dev:arm64
apt-get build-dep -a arm64 qemu
6 . 2

17. Budapest 2017
CompileFlags
Call:
For:
aarch64-linux-gnu-pkg-config glib-2.0 --cflags --libs
-I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-
2.0/include -L/usr/lib/aarch64-linux-gnu -lglib-2.0
6 . 3

18. Budapest 2017
Compile
Running this:
Gives:
aarch64-linux-gnu-gcc glib-hello.c ${FLAGS} -o glib-hello-arm64-
pkgconfig
file glib-hello-arm64-pkgconfig
glib-hello-arm64-pkgconfig: ELF 64-bit LSB executable, ARM
aarch64, version 1 (SYSV), dynamically linked, interpreter
/lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0,
BuildID[sha1]=cf29dbefb15b5c9936760e765f313e0ba4c3399a, not
stripped
6 . 4

19. Budapest 2017
Multi-ArchPros/Cons
7 . 1

20. Budapest 2017
SingDebian'sPraises
10 supported
architectures
Many additional "ports"
Excellent maintainers ;-)
7 . 2

21. Budapest 2017
Caveats
Secondary architecture must match
versions
even minor di erence will fail
Not all packages are multi-arch "clean"
ARM is pretty good, but not complete
Very Debian
Even Ubuntu needs ports
Not portable to other
7 . 3

22. Budapest 2017
OtherOptions
Chroot
Cross-build
system
Buildroot
OpenEmedded
8 . 1

23. Budapest 2017
ChrootIssues
Often project speci c
Unknown
provenience
Can be ddly to set-
up
8 . 2

24. Budapest 2017
BuildSystems
Builds full-stack
Toolchain
Libraries
Project Manifest
Designed as Distro
builders
Massive overkill
8 . 3

25. Budapest 2017
OurWishlist
Concisely de ned build-
environment
Widely available across distros
Fully buzzword compliant
8 . 4

26. Budapest 2017
Docker
9 . 1

27. Budapest 2017
De nition
FROM debian:stable-slim
# Setup Emdebian
RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >>
/etc/apt/sources.list
RUN curl http://emdebian.org/tools/debian/emdebian-toolchain-
archive.key | apt-key add -
# Duplicate deb line as deb-src
RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >>
/etc/apt/sources.list
# Add arm64 multiarch
RUN dpkg --add-architecture arm64
# Install what we need
RUN apt update
RUN apt install -yy crossbuild-essential-arm64
RUN apt-get build-dep -yy -a arm64 qemu
9 . 2

28. Budapest 2017
Dockerguestnotes
Defaults to
root
Self-contained le-
system
RUN id alex 2>/dev/null || useradd -u 1000 -U alex
9 . 3

29. Budapest 2017
Invocation
Running:
Gives:
docker run --rm
--user=$(id -un) -v $(pwd):$(pwd) -w $(pwd)
qemu:debian-arm64-cross
aarch64-linux-gnu-gcc glib-hello.c
-I/usr/include/glib-2.0
-I/usr/lib/aarch64-linux-gnu/glib-2.0/include
-L/usr/lib/aarch64-linux-gnu -lglib-2.0
-o glib-hello-arm64-docker
glib-hello-arm64-docker: ELF 64-bit LSB executable, ARM aarch64,
version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-
aarch64.so.1, for GNU/Linux 3.7.0,
BuildID[sha1]=cf29dbefb15b5c9936760e765f313e0ba4c3399a, not
stripped
9 . 4

30. Budapest 2017
UseCase-WhenMulti-Archdoesn't
work
Multi-Arch might be broken
Development OS may not be
Debian
10 . 1

31. Budapest 2017
Ifonly
10 . 2

32. Budapest 2017
linux-userinDocker
Guest thinks it is an ARM system, with complete
ARM rootfs
docker run --rm -i
--user=$(id -un) -v $(pwd):$(pwd) -w $(pwd)
qemu:debian-armhf-user
uname -a
Linux 52fb9e5738b4 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 armv7l GNU/Linux
10 . 3

33. Budapest 2017
Caveats
binfmt_misc not
containerised
static QEMU safer
slower than native cross
10 . 4

34. Budapest 2017
UseCase-QEMUBuildSystem
11 . 1

35. Budapest 2017
Why?
Portable build recipes
used by
Building guest test
cases
Patchew
11 . 2

36. Budapest 2017
Help
make docker
Build QEMU and run tests inside Docker containers
Available targets:
docker: Print this help.
docker-test: Run all image/test combinations.
docker-clean: Kill and remove residual docker testing
containers.
docker-TEST@IMAGE: Run "TEST" in container "IMAGE".
Note: "TEST" is one of the listed test
name,
or a script name under
$QEMU_SRC/tests/docker/;
"IMAGE" is one of the listed container
name."
docker-image: Build all images.
docker-image-IMAGE: Build image "IMAGE".
docker-run: For manually running a "TEST" with
"IMAGE"
Available container images:
gentoo travis fedora min-glib debian-s390x-cross centos6
ubuntu debian-bootstrap debian-arm64-cross debian debian-armhf-

37. cross
Available tests:
test-clang test-mingw test-build test-quick test-full
Available tools:
travis
Special variables:
TARGET_LIST=a,b,c Override target list in builds.
EXTRA_CONFIGURE_OPTS="..."
Extra configure options.
IMAGES="a b c ..": Filters which images to build or run.
TESTS="x y z .." Filters which tests to run (for docker-
test).
J=[0..9]* Overrides the -jN parameter for make
commands
(default is 1)
DEBUG=1 Stop and drop to shell in the created
container
before running the command.
NOUSER Define to disable adding current user to
containers passwd.
NOCACHE=1 Ignore cache when build images.
EXECUTABLE=<path> Include executable in image.
11 . 3

38. Budapest 2017
Examples
make docker-test-build@centos6 J=9
make docker-test-quick@travis J=9
make docker-test-build@debian-arm64-cross TARGET_LIST="arm-
softmmu,arm-linux-user" J=9
make docker-test-build@debian-ppc-user TARGET_LIST="ppc-softmmu"
J=9
11 . 4

39. Budapest 2017
Summary
Kernel compiles are the simple case
User-space adds complexity
Multi-Arch helps with cross-compiling
Docker allows us to paper over the cracks
Combine Docker with QEMU for double the
fun
12 . 1

40. Budapest 2017
Questions?
13 . 1

41. ThankYou
#BUD17
For further information: www.linaro.org
BUD17 keynotes and videos on: connect.linaro.org
14 . 1

42. Budapest 2017
ExtraSlides
15 . 1

43. Budapest 2017
DebianBootstrap
#!/bin/sh
# Simple wrapper for debootstrap, run in the docker build context
FAKEROOT=`which fakeroot 2> /dev/null`
# debootstrap < 1.0.67 generates empty sources.list, see
Debian#732255
MIN_DEBOOTSTRAP_VERSION=1.0.67
if [ -z $FAKEROOT ]; then
...
if [ -z "${DEB_ARCH}" ]; then
...
if [ -z "${DEB_TYPE}" ]; then
...
# Check for debootstrap or checkout debootstrap from its
# upstream SCM and run it from there.
...
# Finally check to see binfmt_misc is setup
BINFMT_DIR=/proc/sys/fs/binfmt_misc
if [ ! -e $BINFMT_DIR ]; then
...
${FAKEROOT} ${DEBOOTSTRAP} --variant=buildd --foreign --
arch=$DEB_ARCH $DEB_TYPE . http://httpredir.debian.org/debian ||
exit 1
exit 0
15 . 2

44. Budapest 2017
DebianUserDocker le
# Create Debian Bootstrap Image
#
# This is intended to be pre-poluated by:
# - a first stage debootstrap (see debian-bootstrap.pre)
# - a native qemu-$arch that binfmt_misc will run
FROM scratch
# Add everything from the context into the container
ADD . /
# Patch all mounts as docker already has stuff set up
RUN sed -i 's/in_target mount/echo not for docker in_target
mount/g' /debootstrap/functions
# Run stage 2
RUN /debootstrap/debootstrap --second-stage
# At this point we can install additional packages if we want
# Duplicate deb line as deb-src
RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >>
/etc/apt/sources.list
RUN apt-get update
RUN apt-get -y build-dep qemu
15 . 3