In many projects, lexical preprocessors are used to manage different configurations of the project with conditional compilation. Unfortunately, while being a simple way to implement variability, conditional compilation and lexical preprocessor macros hinder automatic analysis, even though such analysis is urgently needed to combat variability-induced complexity. To analyze code with its variability, we need to parse it without preprocessing it. However, current parsing solutions use unsound heuristics, support only a subset of the language, or suffer from exponential explosion. We introduce a novel variability-aware parser that can parse unpreprocessed code without heuristics. On top of parsing, which detects syntax errors in all configurations, we have constructed a variability-aware type system and module system that additionally detect other compiler-time errors. With this infrastructure, we are in the process of checking the entire Linux kernel with 10000 compile-time configuration options and, hence, up to 2^10000 configurations for syntax and type errors.

1. Parsing and Type checking
all 210000 configurations
of the Linux kernel
10,000 features, 6 million lines of C code
Christian Kästner

3. Feature-Oriented
Product Lines

4. Database
Engine

5. Printer
Firmware

6. Linux
Kernel

7. 7

8. Software Product Lines
Boeing
Bosch Group in Industry
Cummins, Inc.
Ericsson
General Dynamics
General Motors
Hewlett Packard
Lockheed Martin
Lucent
NASA
Nokia
Philips
Siemens
…

9. Variability ~ Complexity

10. 33 features
optional, independent
a unique configuration for every
person on this planet

11. 320 features
optional, independent
more configurations than estimated
atoms in the universe

12. Correctness?

13. Product-Line Implementation

14. static int _rep_queue_filedone( Excerpt from
DB_ENV *dbenv, Oracle’s Berkeley DB
REP *rep,
__rep_fileinfo_args *rfp) {
#ifdef NO_QUEUE
COMPQUIET(rep, NULL);
COMPQUIET(rfp, NULL);
return (__db_no_queue_am(dbenv));
#else
db_pgno_t first, last;
u_int32_t flags;
int empty, ret, t_ret;
#ifdef DIAGNOSTIC
DB_MSGBUF mb;
#endif
// over 100 lines of add. code
}
#endif
Conditional Compilation

15. static int _rep_queue_filedone( Excerpt from
DB_ENV *dbenv, Oracle’s Berkeley DB
REP *rep,
__rep_fileinfo_args *rfp) {
#ifdef NO_QUEUE
COMPQUIET(rep, NULL);
COMPQUIET(rfp, NULL);
return (__db_no_queue_am(dbenv));
#else
db_pgno_t first, last;
u_int32_t flags;
int empty, ret, t_ret;
#ifdef X
#ifdef DIAGNOSTIC
DB_MSGBUF mb;void foo();
#endif #endif
// over 100 lines of add. code
} void bar() {
#endif foo();
}
Conditional Compilation

16. Objections / Criticism
Designed in the 70th and hardly evolved since
“#ifdef considered harmful”
“#ifdef hell”
“maintenance becomes a ‘hit or miss’ process”
“is difficult to determine if the code being
viewed is actually compiled into the system”
“incomprehensible source texts”
“programming errors are easy “CPP makes maintenance difficult”
to make and difficult to detect”
“source code rapidly
“preprocessor diagnostics are poor” becomes a maze”

17. [ICSE’08,ASE’08,Tools‘09,GPCE’09,EASE‘11,..]
Views
Visual representation
Disciplined mapping
Consistency checking
Refactorings
…
open source, fosd.net/cide
Virtual Separation of Concerns

18. 10,000 features, 6 million lines of C code

19. [ICSE’10, AOSD‘11]
40 Open-Source C Projects
apache, berkely db,
cherokee, clamav, dia,
emacs, freebsd, gcc,
Number of features
ghostscript, gimp, glibc,
gnumeric, gnuplot, irssi,
libxml, lighttpd, linux, lynx,
minix, mplayer, mpsolve,
openldap, opensolaris,
openvpn, parrot, php, pidgin,
postgresql, privoxy, python,
sendmail, sqlite, subversion,
sylpheed, tcl, vim, xfig,
xine-lib, xorg-server, xterm
Lines of code

20. Correctness?

21. Printer
Firmware

22. Checking Products
2000 Features
100 Printers
30 New Printers per Year
Printer
Firmware

23. Checking Products
10000 Features
210000 Configurations
Linux
Kernel

24. Checking Product Line
Implementation with 10000 Features
+ Generator
Linux
Kernel

25. Variability-Aware Analysis
Parser
Type System
Static Analysis
Bug Finding
Testing
Model Checking
Theorem Proving
…

26. Product Generation
Variability-Aware Conventional
Analysis Analysis

27. We aim for a sound and complete approach

28. Variability-Aware Analysis
Parser
Type System
Static Analysis
Bug Finding
Testing
Model Checking
Theorem Proving
…

30. References
Conflicts

31. Presence Conditions
WORLD
BYE
true
true

32. Reachability: pc(caller) -> pc(target)
Conflicts: ¬(pc(def1) ˄ pc(def2))
¬ (WORLD ˄ BYE)
true -> (WORLD v BYE)
true -> true

33. Reachability: pc(caller) -> pc(target)
Conflicts: ¬(pc(def1) ˄ pc(def2))
¬ (WORLD ˄ BYE)
Found 2 type errors:
- [WORLD & BYE] file hello.c:8:8v BYE)
true -> (WORLD
redefinition of msg
true [!WORLD & !BYE] file hello.c:11:8
- -> true
msg undeclared

34. P
Variability Model:
WORLD BYE
VM ->¬ (WORLD ˄ BYE)
VM -> (true -> (WORLD v BYE))
VM -> (true -> true)

35. AST with Variability Information
true -> true
greet.c
…
printf VWORLD VBYE main
msg ε msg ε printf
¬ (WORLD ˄ BYE)
true -> (WORLD v BYE)
msg
WORLD
BYE 35
Extended Lookup Mechanism

36. [ASE’08, TOSEM‘11]
Formalization: CFJ
Theorem (Product Generation Preserves Typing): All
products that are generated for valid feature selections
from a well-typed product line are well typed.

37. Product Generation
Variability-Aware Conventional
Analysis Analysis

38. Surface Complexity
Inherent
Complexity
SAT
Problem

39. [TOSEM‘11]
Product Line LOC Features Products Time per Time f.
Product entire
(sec) Product
Line (sec)
MobileMedia 5700 14 2784 0.3 2
Mobile RSS 20 000 14 2048 1 8
Reader
Lampiro 45 000 11 2048 2 19
Berkeley DB 70 000 42 3.6 billion 3 21

40. [ICSE’10, AOSD‘11]
40 Open-Source C Projects
apache, berkely db,
cherokee, clamav, dia,
variable code in C files (in %)
emacs, freebsd, gcc,
ghostscript, gimp, glibc,
gnumeric, gnuplot, irssi,
libxml, lighttpd, linux, lynx,
minix, mplayer, mpsolve,
openldap, opensolaris,
openvpn, parrot, php, pidgin,
postgresql, privoxy, python,
sendmail, sqlite, subversion,
sylpheed, tcl, vim, xfig,
xine-lib, xorg-server, xterm
Lines of code

42. Parse Type Check
greet.
c
… VWORL
printf VBYE main
.c
D
msg ε msg ε printf
Linker checks
msg
greet.
c
… VWORL
printf VBYE main
.c
D
msg ε msg ε printf
msg
greet.
c
… VWORL
printf VBYE main
D
.c msg ε msg ε printf
msg

43. Challenges
Real-world C code
C preprocessor
Huge size
Module system / linker checks

44. Variability-Aware Analysis
Parser
Type System
Static Analysis
Bug Finding
Testing
Model Checking
Theorem Proving
…

45. [OOPSLA‘11]
greet.c
…
printf VWORLD VBYE main
msg ε msg ε printf
msg
AST with Variability Information

46. Parsing C without Preprocessing

47. Macro expansion Undisciplined
needed for parsing annotations Alternative macros
?
? ?
greet.c
+ printf VWORLD VBYE + main
+ msg ε + msg ε printf
msg

50. Previous Solutions
Disciplined Subset
Requires Code Preparation
Heuristics and Partial Analysis
Inaccurate, False Positives
Brute Force
Infeasible Effort

51. TypeChef
( +
2
Variability-Aware * Variability-Aware
3 * VA
Lexer )
+
Parser
4A
5¬A 2 3 4 5
Variability-Aware
Analysis
https://github.com/ckaestne/TypeChef

52. [OOPSLA‘11]
4A (¬A 4¬A˄B +¬A˄B 6¬A )¬A true
( 3 + )
4¬A˄B +¬A˄B 6¬A
4A (¬A )¬A
+ 4¬A˄B +¬A˄B 6¬A
3 VA
4 VB
Library of
+ 6 Variability-Aware
Parser Combinators
4 6
in Scala

53. 7665 C files (x86)
0
353 included header files per C file
0
8590 distinct macros per C file
0
72 % conditional
0
30 seconds per file (median)
0
2.6.33.3
X86
0 syntax errors

54. Type Checking
20 seconds per file
2.6.33.3
X86

55. 511 files
260.000 lines of C code
811 features
51 minutes parsing
6 minutes type checking
4 seconds linker checking
Type Checking BusyBox

56. //… skipped 260 lines
struct globals {
double cur_time;
//… skipped 11 lines
#if ENABLE_FEATURE_NTPD_SERVER
int listen_fd;
#endif
unsigned verbose;
//… skipped 73 lines
};
//… skipped 1761 lines
int ntpd_main(int argc UNUSED_PARAM, char **argv)
{
#undef G
struct globals G;
//… skipped 81 lines
if (i > (ENABLE_FEATURE_NTPD_SERVER && G.listen_fd != -1)) {
…
ntpd.c: 2128
}
…
} [CONFIG_NTPD && !CONFIG_FEATURE_NTPD_SERVER]
field listen_fd unknown in struct globals

57. FUTURE DIRECTIONS

58. Correctness?

59. Product Generation
Variability-Aware Conventional
Analysis Analysis

60. Variability-Aware Analysis
Parser
Type System
Static Analysis
Bug Finding
Testing
Model Checking
Theorem Proving
…

61. Product-Line
Evolution
615 trillion config. 553 quintillion config.

62. [GPCE’09; Grant Prop.]
Reengineering Variability
#ifdef plug-ins
parameters feature modules
branches in VCS aspects
clones disciplined annotations
domain knowledge runtime variability
Disciplined
Legacy System refactoring Variability
Implementation

63. Compositional Approaches [ICSE’09, J.ASE’10, SCP’10, TSE’12]
Base / Platform
class Stack {
void push(Object o) {
elementData[size++] = o;
}
...
}
class Stack {
void push(Object o) {
Lock l = lock(o);
Feature: Queue elementData[size++] = o;
refines class Stack { Composition }
l.unlock();
...
void push(Object o) {
Lock l = lock(o); }
Super.push(o);
l.unlock();
}
...
}
Module
Components
Feature: Diagnostic Frameworks, Plug-ins
aspect Diagnostics {
... Feature-Modules / Mixin Layers / …
}
Aspects / Subjects, Hyper/J, Deltas

64. Predicting [SPLC’11, SQJ’11, ICSE’12]
Nonfunctional Properties

65. [AOSD’11 ESEM’11, EASE’11]
Empirical methods
& human factors

66. Domain-
Specific
Languages:
SugarJ
[GPCE’11, OOPSLA’11]
Runtime
Updates
for Java
[APSEC’08, J. SP&E’11]

67. Parsing and Type Checking
all 210000 Configurations
of the Linux Kernel
greet.c
4 ( 4¬A˄B +¬A˄B 6¬A )¬A
true
…
( 3 + ) printf VWORLD VBYE main
4¬A˄B +¬A˄B 6¬A
4 ( )¬A
4¬A˄B +¬A˄B 6¬A
msg ε msg ε printf
msg
https://github.com/ckaestne/TypeChef

68. TypeChef
(Analyzing Real-World C Code)
Virtual Separation of Concerns
(Tool Support for Annotation-Based
Variability Implementation)
Aspect-Oriented
Decomposition
of Berkeley DB
AOP
Compiler
Extensions
M.Sc.
Ph.D. Thesis Post Doc
Thesis
1982… 2006 2007 2008 2009 2010 2011 2012
Austin, Texas Magdeburg, Germany Marburg, Germany