This document discusses the integration of LLVM and Clang into Buildroot. It provides an overview of LLVM and Clang, the objectives of integrating them into Buildroot, and the initial integration work done. It then covers cross-compiling Linux with Clang and some conclusions. Some key points include: - LLVM is an open source compiler infrastructure that includes the Clang C/C++ compiler. It uses an intermediate representation and focuses on compile time and code generation performance. - The initial integration into Buildroot included LLVM, Clang, and Mesa 3D driver support. It aimed to enable OpenCL and add new packages. - Cross-compiling Linux and userspace applications with Clang

1. LLVM/Clang integration to Buildroot
Romain Naour
romain.naour@smile.fr
September 22, 2019
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 1/35

2. Overview
1 Introduction
2 LLVM and Clang
3 Objectives
4 LLVM/Clang initial integration into Buildroot
5 Cross-compiling Linux with Clang
6 Conclusions
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 2/35

3. Plan
1 Introduction
2 LLVM and Clang
3 Objectives
4 LLVM/Clang initial integration into Buildroot
5 Cross-compiling Linux with Clang
6 Conclusions
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 3/35

4. Disclaimer
Purpose of the talk is to present the current state of
llvm/clang integration into Buildroot.
I am not a llvm/clang developer. Not pretending to know
everything about llvm/clang.
Experience gained from experimenting llvm/clang in the
context of Buildroot.
I am only interested in its integration in Buildroot to enable
new features.
Focused on AArch64 and x86_64 target.
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 4/35

5. Buildroot at a glance
From ELCE 2017 "Buildroot: What’s New?"1
Is an embedded Linux build system, builds from
source:
cross-compilation toolchain
root filesystem with many libraries/applications,
cross-built
kernel and bootloader images
More than 2500 packages available
Generates filesystem images, not a distribution
1
https://www.youtube.com/watch?v=D6zO4nMX9KY
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 5/35

6. Plan
1 Introduction
2 LLVM and Clang
3 Objectives
4 LLVM/Clang initial integration into Buildroot
5 Cross-compiling Linux with Clang
6 Conclusions
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 6/35

7. LLVM
Open source project started in 2000. LLVM 1.0 released in
October 2003
Several subprojects: LLVM Core, Clang, lldb, compiler-rt, libclc,
lld
Provides a compiler infrastructure written in C++11 (C++14)
Designed as an API from the beginning
Focusing on compile time and performance of the generated
code
Well structured and documented
Some existing backends:
ARM, ARM64, Hexagon, Mips, Mipsel, NVIDIA PTX 32/64,
PowerPC 32/64, AMDGPU, Sparc, Thumb, x86, x86-64, XCore
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 7/35

8. LLVM - Three-phase approach
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 8/35

9. LLVM - Internal aspects
Intermediate Representation (IR)
Mostly architecture-independent instruction set (RISC)
Strongly typed
Unlimited number of virtual registers in SSA
IR Code example:
define i32 @main() #0 {
entry:
%retval = alloca i32, align 4
%c = alloca i32, align 4
store i32 0, i32* %retval, align 4
%0 = load i32, i32* @a, align 4
%1 = load i32, i32* @b, align 4
%add = add nsw i32 %0, %1
store i32 %add, i32* %c, align 4
ret i32 0
}
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 9/35

10. Clang
Frontend C/C++, Objective C/C++ and OpenCL C for LLVM
Clear and concise diagnostics (error and warning messages)
Natively a cross-compiler: -target <triple>
Sanitizers
Goals
Designed to be highly compatible with GCC
C++11 supported since Clang 3.3
C++14 supported since Clang 3.4
C++17 supported since Clang 5
C++2a implementation started since Clang 6
Performance vs GCC ?2 3
2
http://www.phoronix.com/vr.php?view=25742
3
https://www.phoronix.com/vr.php?view=28232
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 10/35

11. Toolchain components
Component LLVM GNU
C/C++ Compiler clang gcc
Assembler cc1as as
Linker lld ld
Runtime compiler-rt libgcc
Debugger lldb gdb
Unwinder libunwind libgcc_s
C++ library libc++abi, libc++ libsupc++ libstdc++
Tools llvm-ar, llvm-as etc. ar, objdump, etc.
C library - libc (Glibc, Musl)
There is a proposal from Google about an LLVM libc4
.
4
https://lists.llvm.org/pipermail/llvm-dev/2019-June/133269.html
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 11/35

12. Plan
1 Introduction
2 LLVM and Clang
3 Objectives
4 LLVM/Clang initial integration into Buildroot
5 Cross-compiling Linux with Clang
6 Conclusions
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 12/35

13. Objectives
Initial integration by Valentin Korenblit (2018)
Preliminary study of LLVM/Clang.
LLVM/Clang integration to Buildroot for the target.
llvmpipe for Mesa 3D.
AMDGPU backend.
OpenCL implementations, most of them rely on LLVM
Enable OpenCL for AMD GPUs (Clover) and Broadcom
Videocore IV (VC4CL).
OpenCL support for already existing packages in Buildroot.
Integration of new packages that can benefit from OpenCL:
image processing (i.e. Darktable), simulation, cryptography.
Integration of new packages that can benefit from libLLVM or
libclang.
Use Clang as a cross-compiler (2019/2020?).
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 13/35

14. Plan
1 Introduction
2 LLVM and Clang
3 Objectives
4 LLVM/Clang initial integration into Buildroot
5 Cross-compiling Linux with Clang
6 Conclusions
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 14/35

15. Available hardware for testing
.
Platform 1 - x86_64 (HP ProBook)
Processor: AMD A4-3300M Dual Core @ 1.9 GHz.
GPU: AMD Radeon Dual Graphics (HD6480G + HD7450M).
Ease testing.
Platform 2 - ARM (Raspberry Pi 2 Model B).
Processor: ARMv7 Cortex-A7 Quad Core @ 900 MHz.
GPU: Broadcom Videocore IV.
Platform 3 - ARM/AArch64 (Raspberry Pi 3 Model B).
Processor: ARMv8 Cortex-A53 Quad Core @ 1.2 GHz.
GPU: Broadcom Videocore IV.
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 15/35

16. LLVM packaging for Buildroot
CMake-based project
Buildroot provides a CMake infrastructure
Plenty of options, some with misleading names
LLVM_TARGETS_TO_BUILD
LLVM_TARGET_ARCH
LLVM_DEFAULT_TARGET_TRIPLE
LLVM_HOST_TRIPLE
Difficult to be cross-compiled
At least llvm-tblgen must be compiled for the host first
It requires a modern and fully-featured toolchain (C++11 -> C++14)
Takes a lot of time to compile (host + target)
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 16/35

17. LLVM packaging for Buildroot
LLVM 5.0.1 is the first packaged version
Only LLVM libraries are needed (libLLVM.so), no tools
The main problem: llvm-config not giving the desired output
It’s a compiled program. Normally ”config” programs are scripts.
llvm-config compiled for the host must be installed to the
target’s sysroot.
Some output depends on its location and some other is
contained in the binary
Solution: do a full installation of LLVM for the host using the
same5
configuration options as for the target and link LLVM
tools with libLLVM (needed for AMDGPU backend).
5
Tools are not built for the target
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 17/35

18. LLVM support for Mesa 3D drivers
Gallium llvmpipe driver: software rasterizer that uses LLVM to
do runtime code generation.
OpenCL using Gallium R600 and RadeonSI
Both use AMDGPU LLVM backend
Some benchmarks (LLVM 5.0.1):
Gallium Driver GLMark2 GLMark2-es2
R600 on HD6480 156 156
softpipe on AMD A4-3300M 3 3
softpipe on Cortex-A53 (32-bit) NS 0
softpipe on Cortex-A53 (64-bit) NS 0
llvmpipe on AMD A4-3300M 47 52
llvmpipe on Cortex-A53 (32-bit) NS 11
llvmpipe on Cortex-A53 (64-bit) NS 13
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 18/35

19. Selecting llvmpipe
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 19/35

20. Clang packaging for Buildroot
LLVM and Clang should be version bumped together.
It is normally built inside LLVM source tree (llvm/tools) but
Buildroot uses per-package build directories.
Some more tweaks are needed.
Binaries, headers and some scripts must be removed from the
target.
Only libclang.so is necessary on the target.
Buildroot does not provide a standard infrastructure to install a
compiler on the target (you’re on your own).
Clang, Clang++ binaries can’t be easily disabled from the build
system.
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 20/35

21. OpenCL: Clover: Computing Language over Gallium
Clover needs libclang and libclc.
Valentin added Mesa3d OpenCL support.
Added AMDGPU LLVM backend support, libclc and clinfo
packages.
Tested VC4CL for Broadcom Videocore IV.
Runtime testing using Piglit.
See Linuxembedded blog6
for more details on OpenCL.
6
http:
//www.linuxembedded.fr/2018/07/llvmclang-integration-into-buildroot
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 21/35

22. LLVM/Clang in Buildroot - Current State
Commited to Buildroot’s master:
LLVM package (2018.05)
LLVM support for Mesa 3D (2018.05)
Clang package (2018.05)
Libclc package (2018.11)
libopencl, clinfo packages (2018.11)
Piglit package (2019.08)
Keep LLVM/Clang at the latest version
Buildroot LTS 2019.02.x use LLVM/Clang 7.0.1
TODO & WIP
Add OpenCL support for already existing packages (i.e opencv)
Add new packages that depend on LLVM/Clang or OpenCL
Use Clang as a cross-compiler
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 22/35

23. LLVM/Clang in Buildroot - Work in progress
Chromium7
& lld8
by Joseph Kogut
Chromium needs Clang as compiler.
Lld used as GNU ld drop-in replacement.
Compiler-rt & Libfuzzer9
by Matt Weber (Rockwell Collins)
Compiler-rt to run the code with sanitizer instrumentation.
Libfuzzer: a library for coverage-guided fuzz testing.
BCC (BPF Compiler Collection)10
by Jugurtha Belkalem (Smile)
BCC use libclang to build eBPF programs to do debugging11
.
7
https://github.com/jakogut/buildroot/commits/chromium-v7-wip
8
http://patchwork.ozlabs.org/project/buildroot/list/?series=121282
9
http://patchwork.ozlabs.org/project/buildroot/list/?series=105601
10
http://patchwork.ozlabs.org/project/buildroot/list/?series=106280
11
http://www.linuxembedded.fr/2019/05/bcc-integration-into-buildroot
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 23/35

24. Plan
1 Introduction
2 LLVM and Clang
3 Objectives
4 LLVM/Clang initial integration into Buildroot
5 Cross-compiling Linux with Clang
6 Conclusions
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 24/35

25. Compiling a Linux system with Clang
Until recently, Linux kernel expected to use some GCC
behavior that are not supported by Clang:
Variable Length Arrays inside structures
(VLA removed since 4.20)
Nested functions
Explicit register variables
LLVM assembler cannot be used to build the kernel
Support for asm goto in LLVM & Clang for x86_6412
LLVMLinux Project: Kernel 4.4 and 4.9 built with Clang for
x86_64 and ARM64 (patches applied)13
Dependency on GNU as and ld
Glibc14
? Musl15
?
12
https://bugs.llvm.org/show_bug.cgi?id=9295
13
https://lwn.net/Articles/734071/
14
https://sourceware.org/glibc/wiki/GlibcMeetsClang
15
https://wiki.musl-libc.org/supported-platforms.html
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 25/35

26. Compiling a Linux distribution with Clang
2019 EuroLLVM Developers’ Meeting: Switching a Linux
distribution’s main toolchains to LLVM/Clang16
- Bernhard
Rosenkränzer
From the OpenMandriva Wiki17
"Over 98% of packages in our
main repository are now built with LLVM/clang."
32757 Debian packages have been rebuild with Clang 8.
Among them, 1322 (4 %) failed18
16
https://www.youtube.com/watch?v=QinoajSKQ1k
17
https://wiki.openmandriva.org/en/4.0/Release_Notes#LLVM.2Fclang
18
https://clang.debian.net/status.php?version=8svn
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 26/35

27. Cross-compiling with Clang with Buildroot
In Buildroot we take advantage of all previous upstream work
to build an embedded system with Clang.
Objectives:
Build vanilla Linux kernels.
Build most of userspace applications.
Select GCC or Clang globally in the defconfig.
Limitations:
Keep toolchain components and bootloaders build with GCC
Rebuild LLVM/Clang for each defconfig build.
We expects build and runtime issue due to cross-compiling
with Clang.
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 27/35

28. Cross-compiling with Clang with Buildroot
What’s remain to do to use Clang as a cross-compiler into
Buildroot?
Fix Clang’s libgcc and crt*.o search path to build userspace
applications19
.
Add a toolchain wrapper as for GCC to executes the real
compiler with a number of arguments (sysroot) hardcoded.
Improve the package infrastructure where GCC is explicitely
used (generic packages, meson, cmake etc).
Add an option to select Clang or GCC in the defconfig.
Fixes most build issues on the long term.
Still a lot of warning due to GCC compiler flags used with Clang.
19
http://patchwork.ozlabs.org/patch/1159304/
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 28/35

29. Current issues with Clang
Busybox (segfault with applets (i.e init))20
.
Bootloader & firmware (uboot, barebox, grub2, ATF).
uboot: use global register variables.
grub2: inline asm constructs (.addrsig etc.)
Python3 build issue with _ctype module.
rng-tools: Outdated version (from 2004) using a VLA.
Valgrind: __builtin_longjmp not supported in LLVM/Clang for
Aarch64 target21
.
20
http://lists.busybox.net/pipermail/busybox/2019-June/087337.html
21
https://bugs.kde.org/show_bug.cgi?id=369723
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 29/35

30. Qemu runtime testing: AArch64 & x86_64
qemu_aarch64_virt_defconfig
qemu_x86_64_defconfig
Using the latest kernel version (5.2.7)
Using LLVM/Clang 9.0.0rc3 for x86_64 (asm-goto support)
Welcome to Buildroot
# uname −a
Linux buildroot 5.2.7 #1 SMP Thu Sep 5 17:33:00 CEST 2019 aarch64 GNU/Linux
# dmesg | head −n 2
Booting Linux on physical CPU 0x0000000000 [0x410fd034]
Linux version 5.2.7 (clang version 8.0.1 (tags/RELEASE_801/final)) #1 SMP Thu Sep 5 17:33:00 CEST
# cat /etc/os−release
NAME=Buildroot
VERSION=2019.11−git−00175−g0cc6fbbdf5
ID=buildroot
VERSION_ID=2019.11−git
PRETTY_NAME="Buildroot 2019.11−git"
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 30/35

31. Available hardware for testing
.
Aarch64 (Le Potato AML-S905X-CC).
Processor: ARM Cortex-A53 Quad Core @ 1.512GHz.
GPU: ARM Mali-450 @ 750MHz.
Potato board received two years ago at Kernel recipes (Thanks!)
Live Demo (kernel built with Clang)
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 31/35

32. Plan
1 Introduction
2 LLVM and Clang
3 Objectives
4 LLVM/Clang initial integration into Buildroot
5 Cross-compiling Linux with Clang
6 Conclusions
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 32/35

33. Conclusions
LLVM/Clang is already available for the target.
Clang can now build vanilla Linux Kernel.
It’s possible to use Clang as cross-compiler to build a Linux
embedded system.
Initial patch series to review:
"Add the support for Clang cross-compiler"22
Not yet ready for production, needs more build and runtime
testing.
Compared to Debian and OpenMandriva, Buildroot only have
2500+ packages.
Cross-compilation with clang remains complicated.
Many projects don’t test building with anything but native GCC
(x86 or x86_64).
22
http://patchwork.ozlabs.org/project/buildroot/list/?series=129565
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 33/35

34. Questions
Thanks for your attention!
Questions?
Romain Naour
romain.naour@smile.fr
Slides under CC-BY-SA 3.0
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 34/35

35. References
http://www.linuxembedded.fr/2018/07/
llvmclang-integration-into-buildroot
http://www.linuxembedded.fr/2019/08/
my-first-linux-kernel-built-with-clang-compiler
https://www.lowrisc.org/blog/2019/07/
large-scale-risc-v-llvm-testing-with-buildroot
https:
//www.phoronix.com/scan.php?page=news_item&px=LLVM-Asm-Goto-Merged
https://www.youtube.com/watch?v=QinoajSKQ1k
https://clang.debian.net/status.php?version=8svn
https://bootlin.com/pub/conferences/2016/elce/
petazzoni-toolchain-anatomy
www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 35/35