The digest of "Software Build System" Chap6

1. Make
@brownylin

2. Outline
• The GNU Make Programming Lang.
• Real-World Build System Scenarios

3. Introduction
• A rule (providing all the inter-file
dependency information)
• Time-stamp driven commands
rule
target sources
myprog:
prog.c
lib.c
gcc
–o
myprog
prog.c
lib.c
commands

4. The GNU Make Prog. Lang.
• 3 components
‣ File
dependencies:
GNU-­‐Make
performs
a
pa1ern-­‐matching
opera8on
to
decide
which
rule
to
evaluate
next
‣ Shell
commands:
A
list
of
shell
commands
encapsulated
within
each
rule,
to
be
executed
if
the
target
of
the
rule
is
out-­‐of-­‐date
‣ String
processing:
A
language
for
manipula8ng
GNU
Make
variables
(Func8onal
programming
paradigm)

5. 3 Sublanguages
• File dependencies
myprog:
prog.c
lib.c
• Shell commands
cp
myfile
yourfile
&&
cp
myfile1
yourfile1
md5
<
myfile
>>yourfile
touch
yourfile.done
• String processing
VARS
:=
$(sort
$(filter
srcs-­‐%
cflags-­‐%,
$(.VARIABLES)))

6. Programming Paradigm
• Imperative programming (C++, Java, OOP)
‣ Computa8on:
Execu8ng
statements
to
change
states
‣ Program:
Consists
of
a
sequence
of
commands
• Functional programming (Lisp, Haskell,
Erlang)
‣ Computa8on:
Evalua8on
of
expression
‣ Expression:
Formed
by
using
func8on
to
combine
basic
values

7. Create Sandwich
• Imperative
1. Take
a
bread
2. Spread
bread
with
bu1er
3. Put
cheese
on
the
bread
4. return
sandwich
• Functional
‣ return
put(
cheese,
spread(bu1er,
bread)
)

8. Simple Makefike
calculator:
add.o
calc.o
mult.o
sub.o
gcc
-­‐g
-­‐o
calculator
add.o
calc.o
mult.o
sub.o
add.o:
add.c
numbers.h
gcc
-­‐g
-­‐c
add.c
calc.o:
calc.c
numbers.h
gcc
-­‐g
-­‐c
calc.c
mult.o:
mult.c
numbers.h
gcc
-­‐g
-­‐c
mult.c
sub.o:
sub.c
numbers.h
gcc
-­‐g
-­‐c
sub.c

9. Makefile Rule Types (1/3)
• Rules with multiple targets
file1.o
file2.o:
source1.c
source2.c
source3.c
...
commands
go
here
...
• Rules with no prerequisites
.PHONY:
help
help:
@echo
“Usage:
make
all
ARCH=[i386|mips]”
@echo
“
make
clean”
the PHONY directive make GNU Make always execute the rule
(even a file named help exists)

10. Makefile Rule Types (2/3)
• Rules with patterns in the filename
%.o:
%.c
...
commands
go
here
...
• Rules that apply only to certain files
a.o
b.o:
%.o:
%.c
echo
This
rule
is
for
a.o
and
b.o
c.o
d.o:
%.o:
%.c
echo
This
rule
is
for
c.o
and
d.o

11. Makefile Rule Types (3/3)
• Multiple rules with the same target
‣ Only
one
of
these
rules
can
contain
a
set
of
shell
commands
chunk.o:
chunk.c
gcc
–c
chunk.c
chunk.o:
chunk.h
list.h
data.h

12. Makefile Variables
• The rules
‣ Variables
are
given
a
value
by
an
assignment
statement X
:=
5
‣ Variable
values
are
referenced
using
the
syntax $(X)
‣ All
variables
are
of
string
type.
No
mechanism
exists
for
declaring
variables
before
they’re
used
‣ Variables
have
global
scope
(within
a
single
makefile)

13. Immediate Evaluation
• Using the := operator
FIRST
:=
Hello
there
SECOND
:=
World
#
comments
go
here
MESSAGE
:=
$(FIRST)
$(SECOND)
FILES
:=
add.c
sub.c
mult.c
$(info
$(MESSAGE)
–
The
files
are
$(FILES))
• Output
Hello
there
World
–
The
files
are
add.c
sub.c
mult.c

14. Deferred Evaluation
• Using = instead of :=
CC
:=
gcc
CFLAGS
:=
-­‐g
CCOMP
=
$(CC)
$(CFLAGS)
$(info
Compiler
is
$(CCOMP))
CC
:=
i386-­‐linux-­‐gcc
$(info
Compiler
is
$(CCOMP))
• CCOMP variable isn’t evaluated until the
variable is actually used
$
gmake
Compiler
is
gcc
–g
Compiler
is
i386-­‐linux-­‐gcc
-­‐g

15. Conditional Assignment
• Assign a value if the variable doesn’t
already have one
CFLAGS
:=
-­‐g
CFLAGS
?=
-­‐O
$(info
CFLAGS
is
$(CFLAGS))
• Useful when you include one makefile
from within another (default value)

16. Built-In Variables and Rules
(1/2)
• automatic variables
‣ $@:
Contains
the
filename
of
the
current
rule’s
target
‣ $<:
Represents
the
first
prerequisite
of
a
rule
‣ $^:
Similar
to
$<,
but
it
evaluates
to
the
complete
list
of
prerequisites
in
the
rule,
with
spaces
between
them
‣ If
the
target
is
/home/john/work/src.c,
$(@D)
evaluates
to
/home/john/work;
$(@F)evaluates
to
src.c

17. Built-In Variables and Rules
(2/2)
• build-in rules
‣ show
it
use
gmake
–p
• built-in rule for C compilation
COMPILE.c
=
$(CC)
$(CFLAGS)
$(CPPFLAGS)
$(TARGET_ARCH)
–c
OUTPUT_OPTION
=
-­‐o
$@
%.o:
%.c
$(COMPILE.c)
$(OUTPUT_OPTION)
$<

18. Calculator Example (1/3)
calculator:
add.o
calc.o
mult.o
sub.o
gcc
-­‐g
-­‐o
calculator
add.o
calc.o
mult.o
sub.o
add.o:
add.c
numbers.h
gcc
-­‐g
-­‐c
add.c
calc.o:
calc.c
numbers.h
gcc
-­‐g
-­‐c
calc.c
mult.o:
mult.c
numbers.h
gcc
-­‐g
-­‐c
mult.c
sub.o:
sub.c
numbers.h
gcc
-­‐g
-­‐c
sub.c

19. Calculator Example (2/3)
• Use build-in C compilation rule
calculator:
add.o
calc.o
mult.o
sub.o
gcc
-­‐g
-­‐o
calculator
add.o
calc.o
mult.o
sub.o
add.o
calc.o
mult.o
sub.o:
numbers.h

20. Calculator Example (3/3)
SRCS
=
add.c
calc.c
mult.c
sub.c
PROG
=
calculator
CC
=
gcc
CFLAGS
=
-­‐g
OBJS
=
$(SRCS:.c=.o)
$(PROG):
$(OBJS)
$(CC)
$(CFLAGS)
-­‐o
$@
$^
$(OBJS):
numbers.h

21. Data Structures and Functions
• All of GNU Make’s variables are of
string type
• How to store complex data?
‣ As
long
as
you
have
a
mechanism
for
extrac8ng
specific
items
out
of
a
list,
you
can
treat
this
variable
like
a
structured
data
type
PROG_NAME
:=
my-­‐calculator
LIST_OF_SRCS
:=
calc.c
main.c
math.h
lib.c
COLORS
:=
red
FF0000
green
00FF00
blue
0000FF
purple
FF00FF
ORDERS
:=
100
green
cups
200
blue
plates

22. Common functions (1/4)
• words: Given a list as input, returns
the number of space-separated words
in that list
NUM_FILES
:=
$(words
$(LIST_OF_SRCS))
#
$(NUM_FILES)
evaluates
to
4
• word: Given a list, extracts the nth
word from that list
SECOND_FILE
:=
$(word
2,
$(LIST_OF_SRCS))
#
$(SECOND_FILE)
evaluates
to
main.c

23. Common functions (2/4)
• filter: Returns the words from a list,
which match a specific pattern
C_SRCS
:=
$(filter
%.c,
$(LIST_OF_SRCS))
#
all
C
source
files
• patsubst: For each word in a list,
replaces any that match a specific pat-
tern with a replacement pattern
OBJECTS
:=
$(patsubst
%.c,%.o,
$(C_SRCS))
#
similar
to
the
$(C_SRCS:.c=.o)

24. Common functions (3/4)
• addprefix: For each word in a list,
prepends an additional string
OBJ_LIST
:=
$(addprefix
objs/,
$(OBJECTS))
#
$(OBJ_LIST)
evaluates
to
objs/calc.o
objs/main.o
objs/lib.o
• foreach: Visits each word in a list and
constructs a new list containing the
“mapped” values
OBJ_LIST_2
:=
$(foreach
file,
$(OBJECTS),objs/$(file))
#
$(OBJ_LIST_2)
evaluates
to
objs/calc.o
objs/main.o
objs/lib.o

25. Common functions (4/4)
• dir/notdir: Given a file’s pathname,
returns the directory name component
or the filename component
DEFN_PATH
:=
src/headers/idl/interface.idl
DEFN_DIR
:=
$(dir
$(DEFN_PATH))
#
src/headers/idl/
DEFN_BASENAME
:=
$(notdir
$(DEFN_PATH))
#
interface.idl
• shell: Executes a shell command and
returns the command’s output as a
string
PASSWD_OWNER
:=
$(word
3,
$(shell
ls
-­‐l
/etc/passwd))

26. Concept of a macro (1/2)
• Associate a name with a complex GNU
Make expression and to pass arguments
into that expression
file_size
=
$(word
5,
$(shell
ls
-­‐l
$(1)))
PASSWD_SIZE
:=
$(call
file_size,/etc/passwd)
#
use
the
$(1)
syntax
to
reference
the
first
parameter
of
the
$(call)
expression

27. Concept of a macro (2/2)
• Define a canned sequence of shell
commands by using the define directive
define
start-­‐banner
@echo
==============
@echo
Starting
build
@echo
==============
endef
.PHONY:
all
all:
$(start-­‐banner)
$(MAKE)
-­‐C
lib1

28. Understanding Program Flow
• Parsing a makefile
‣ Reading
the
makefile
to
build
the
dependency
graph
‣ Execu8ng
the
compila8on
commands
• Controlling the parsing process
‣ Include
a
submakefile,
or
condi8onally
compile
parts
of
the
makefile
• Executing the rules
‣ The
order
in
which
rules
are
applied
and
the
corresponding
shell
commands
are
executed

29. Parsing a makefile (1/2)
1. Parsing phase
• All
rules
are
scanned,
all
variable
assignments
are
performed,
and
all
variables
and
func8ons
are
evaluated
2. Execution phase
• GNU
Make
examines
the
8me
stamps
on
all
the
files
to
determine
which
files
(if
any)
are
out
of
date
• The
appropriate
shell
commands
are
executed
to
bring
those
targets
up-­‐to-­‐date

30. Parsing a makefile (2/2)
X
:=
Hello
World
print:
echo
X
is
$(X)
X
:=
Goodbye
$
gmake
print
X
is
Goodbye
• Build system’s functionality can be
implemented by
‣ Using
GNU
Make
func8ons
(processed
during
the
first
phase)
‣ As
part
of
a
shell
script
(processed
during
the
second
phase)

31. Controlling the Parsing
Process
• File inclusion
FILES
:=
src1.c
src2.c
include
prog.mk
#
content
of
prog.mk
textually
#
inserted
here
src1.o
src2.o:
src.h
• Conditional compilation
CFLAGS
:=
-­‐DPATH="/usr/local"
ifdef
DEBUG
CFLAGS
+=
-­‐g
#
debug
case
if
DEBUG
is
defined
else
CFLAGS
+=
-­‐O
#
non-­‐debug
case
if
DEBUG
not
defined
endif

32. Executing the Rules (1/3)
1. Invokes GNU Make (gmaike) must specify
which target to build (default: first target
listed in the makefile)
2. Locates a rule to generate the target file, it
examines each of the prerequisites listed in
that rule and treats them recursively as
targets
Example:
Before linking add.o and calc.o into the calculator
executable program, GNU Make recursively searches
for rules that have add.o or calc.o on the left side

33. Executing the Rules (2/3)
3. If a rule is found for the target you’re
trying to satisfy
a. If
the
target
file
for
the
rule
doesn’t
yet
exist,
the
rule’s
shell
command
sequence
is
executed
and
the
file
is
created
for
the
first
8me
b. If
the
target
file
already
exists
on
the
disk,
the
8me
stamp
on
each
of
the
prerequisite
files
is
examined
to
see
if
any
are
newer
than
the
target
file

34. Executing the Rules (3/3)
4. If step 3 fails, meaning that the makefile
doesn’t contain a suitable rule to
generate a target file
a. If
the
target
file
exists
on
the
disk
(but
there’s
no
rule
to
regenerate
it),
GNU
Make
can
only
assume
that
this
is
a
source
file
that
was
handwri1en
by
the
developer
b. If
the
target
file
doesn’t
exist
on
the
disk,
GNU
Make
aborts
with
an
error
and
the
build
fails

35. Build System Scenarios
1. Source Code in a Single Directory
2. Multiple Directories
a. Source
Code
in
Mul8ple
Directories
b. Recursive
Make
over
Mul8ple
Directories
c. Inclusive
Make
over
Mul8ple
Directories
3. Defining New Compilation Tools
4. Building with Multiple Variants
5. Cleaning a Build Tree
6. Debugging Incorrect Builds

36. 1. Source Code in a Single
Directory
SRCS
=
add.c
calc.c
mult.c
sub.c
PROG
=
calculator
CC
=
gcc
CFLAGS
=
-­‐g
OBJS
=
$(SRCS:.c=.o)
$(PROG):
$(OBJS)
$(CC)
$(CFLAGS)
-­‐o
$@
$^
$(OBJS):
numbers.h
• Newly added source file didn’t actually
include numbers.h?
• Additional header files were added?

37. 1. Source Code in a Single
Directory
1. Automatically generate a new
dependency information file (with .d
suffix)
-­‐include
$(SRCS:.c=.d)
%.d:
%.c
@$(CC)
-­‐MM
$(CPPFLAGS)
$<
|
sed
's#(.*).o:
#1.o
1.d:
#g'
>
$@
2. Uses the makedepend command.

38. these cases, the so
2-a. Source Code in Multiple but are instead sp
Directories shows the tree for
• Problem
‣ Harder
dependency
genera8on
‣ Developer
conten8on
on
the
single
makefile
‣ Inability
to
subdivide
the
program
• How? Divide the build description
across more than one makefile
Figure 6.1 The
SRCS
=
libmath/clock.c
libmath/letter.c
libmath/number.c
sou
libprint/banner.c
libprint/center.c
libprint/normal.c
calc/calc.c
... For the first at
gram, but the SRC

39. source directory
2-b. Recursive Make over cursively invokin
Multiple Directories tory tree, with ea
• Recursive Make
‣ A
different
makefile
in
each
source
directory,
with
the
high-­‐level
makefile
SRCS
=
clock.c
letter.c
number.c
LIB
=
libmath.a
CC
=
gcc
CFLAGS
=
-­‐g
OBJS
=
$(SRCS:.c=.o)
$(LIB):
$(OBJS)
$(AR)
cr
$(LIB)
$(OBJS)
$(OBJS):
math.h
Figure 6.2 Multid

40. 2-b. Recursive Make over
Multiple Directories
libmath/Makefile libprint/Makefile
SRCS
=
clock.c
letter.c
number.c
SRCS
=
banner.c
center.c
normal.c
LIB
=
libmath.a
LIB
=
libprint.a
CC
=
gcc
CC
=
gcc
CFLAGS
=
-­‐g CFLAGS
=
-­‐g
OBJS
=
$(SRCS:.c=.o)
OBJS
=
$(SRCS:.c=.o)
$(LIB):
$(OBJS)
$(LIB):
$(OBJS)
$(AR)
cr
$(LIB)
$(OBJS)
$(AR)
cr
$(LIB)
$(OBJS)
$(OBJS):
math.h $(OBJS):
printers.h
SRCS
=
banner.c
center.c
normal.c
LIB
=
libprint.a
include
lib.mk
$(OBJS):
printers.h

41. 2-b. Recursive Make over
Multiple Directories
calc/Makefile
SRCS
=
calc.c
PROG
=
calculator
LIBS
=
../libmath/libmath.a
../libprint/libprint.a
CC
=
gcc
CFLAGS
=
-­‐g
OBJS
=
$(SRCS:.c=.o)
$(PROG):
$(OBJS)
$(LIBS)
$(CC)
-­‐o
$@
$^

42. 2-b. Recursive Make over
Multiple Directories
src/Makefile
.PHONY:
all
all:
$(MAKE)
-­‐C
libmath
$(MAKE)
-­‐C
libprint
$(MAKE)
-­‐C
calc
• all target has no prerequisites, each of the
recursive calls to $(MAKE) happens every
time
• Not the most efficient solution available

43. 2-b. Recursive Make over
Multiple Directories
• Problem?
‣ Hundred
files?
Build
everything
in
the
correct
order
becomes
an
impossible
task
‣ If
the
source
code
in
the
libmath
directory
started
to
use
the
libprint.a
?
(libmath.a
compiled
first,
so
it
uses
old
libprint.a)
‣ If
you
want
to
build
only
part
of
the
program
• If you started in the calc subdirectory and typed
gmake, it doesn’t attempt to rebuild any of those files
even if they are out-of-date (calc/Makefile
doesn’t know how to build libprint.a)

44. Figure 6.3 illustr
2-c. Inclusive Make over ple because a two-l
solution.
Multiple Directories
• One main makefile, at the top of
the source tree (make/
framework.mk)
• Software developers are only
encouraged to view and edit
Files.mk files
• GNU Make complexity is hidden
inside the make/framework.mk
file (non-guru software engineers
don’t attempt to change the build
mechanism by mistake)

45. 2-c. Inclusive Make over
Multiple Directories
• Files.mk files
‣ Designed
to
be
readable
and
editable
by
so`ware
developers
‣ Contain
only
variables
that
developers
care
about
src/Files.mk
src/libraries/math/Files.mk
SUBDIRS
:=
libraries
application
SRC
:=
add.c
mult.c
sub.c
SRC
:=
main.c
CFLAGS
:=
-­‐DBIG_MATH
CFLAGS
:=
-­‐g
src/libraries/Files.mk
SUBDIRS
:=
math
protocols
sql
widgets
#
none
of
the
source
files
from
that
directory
would
be
included

46. 2-c. Inclusive Make over
Multiple Directories
src/Makefile
_subdirs
:=
_curdir
:=
FRAMEWORK
:=
$(CURDIR)/make/framework.mk
include
Files.mk
include
$(FRAMEWORK)
VARS
:=
$(sort
$(filter
srcs-­‐%
cflags-­‐%,
$(.VARIABLES)))
$(foreach
var,
$(VARS),
$(info
$(var)
=
$($(var))))
.PHONY:
all
all:
@#
do
nothing

47. 2-c. Inclusive Make over
Multiple Directories
_subdirs
src/Files.mk
libraries applications src/libraries/Files.mk
applications libraries/math libraries/protocols
libraries/sql libraries/widgets

48. 2-c. Inclusive Make over
Multiple Directories
• Main algorithm for traversing the build tree and
collecting the values from each Files.mk fragment
srcs-­‐$(_curdir)
:=
$(addprefix
$(_curdir),$(SRC))
cflags-­‐$(_curdir)
:=
$(CFLAGS)
_subdirs
:=
$(_subdirs)
$(addprefix
$(_curdir),
$(SUBDIRS))
ifneq
($(words
$(_subdirs)),0)
_curdir
:=
$(firstword
$(_subdirs))/
_subdirs
:=
$(wordlist
2,
$(words
$(_subdirs)),
$(_subdirs))
SUBDIRS
:=
SRC
:=
CFLAGS
:=
include
$(_curdir)Files.mk
include
$(FRAMEWORK)
endif make/framework.mk

49. 2-c. Inclusive Make over
Multiple Directories
• Summary
‣ Using
one
instance
of
the
GNU
Make
process
enables
you
to
have
a
single
unified
dependency
graph
‣ Not
an
easy
system
to
create

50. Pros & Cons
• Pros • Cons
‣ Wide
support ‣ Inconsistent
language
design
‣ Vary
fast
tool
‣ No
standard
‣ Portable
syntax framework
‣ Fully
featured
‣ Lack
of
portability
‣ The
first
tool
‣ Challenging
debugging
‣ Not
easy
to
use

51. Evaluation
• Convenience (Poor)
• Correctness (Poor)
• Performance (Excellent)
• Scalability (Excellent)
• General Rule
‣ Using
GNU
Make
for
legacy
so`ware
that
already
uses
a
Make-­‐based
build
system
‣ First
consider
using
SCons
or
CMake
when
wri8ng
a
new
build
system

52. Thank you :)