Weitere ähnliche Inhalte
Ähnlich wie Building Large Java Projects Faster: Multicore javac and Makefile integration (20)
Kürzlich hochgeladen (20)
Building Large Java Projects Faster: Multicore javac and Makefile integration
- 1. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 131
- 2. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 132
Building Large Java Projects
Faster: Multicore javac and
Makefile integration
Fredrik Öhrström
Principal Engineer & Patent Liaison
- 3. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.3
The following is intended to outline our general product direction. It is intended
for information purposes only, and may not be incorporated into any contract.
It is not a commitment to deliver any material, code, or functionality, and should
not be relied upon in making purchasing decisions. The development, release,
and timing of any features or functionality described for Oracle’s products
remains at the sole discretion of Oracle.
- 4. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.4
What is the output from javac?
From a single Java source file, javac can generate:
Zero, to an infinite number of classes with unpredictable file names.
(package-info.java, inner classes and package private classes
not matching the file name)
If annotation processors are in use, then anything can happen.
(New sources, new packages, new xml-files, png-files etc)
- 5. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.5
How javac (re)compiles code
Javac requires explicit source files on the command line (or @file)
The explicit sources are always compiled and linking dependencies
outside the explicit sources, are read from the classpath.
(eg java.lang.Object, org.antlr.runtime.CharStream, etc)
If the to be linked dependencies are not yet compiled,
then use -sourcepath srcroot to link to source code.
- 6. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.6
How javac (re)compiles code
Unfortunately, the default behavior for to be linked sources found through
the -sourcepath, is to emit classes for these sources. Unless:
If -implicit:none is supplied on the command line
Or the corresponding class in the (boot)classpath or -d directory is newer than
the source.
If you develop jaxp (which is part of the jdk), then a jdk installed after
your source was written, will override your source in the sourcepath,
even though your source is probably newer.
- 7. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.7
Compiling, Headers and Linking in the Java world.
At runtime, linking of imports is done lazily when the class is loaded.
At compile time, usage of imports are checked against the class files
in the (boot)classpath.
At compile time, usage of imports are checked against the source
files in the sourcepath or explicit sources to be compiled.
But these checks will trigger a full compile of the referenced
classes. Not only their public apis.
This makes it very easy to create circular dependencies.
- 8. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.8
How javac (re)compiles code
Implicit generation of classes and accidentally not recompiling your
own source code, is usually not a problem. Because most developers
do not develop code that resides in the jdk.
But implicitly generated classes contribute to the confusion of what
javac outputs!
- 9. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.9
How a makefile works
imperative statements
goal: requirements
action that generates goal
# A makefile is executed imperatively to create
# a dependency tree from goals to requirements
- 10. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.10
How a makefile works
FLAGS += Os
app: app.c app.h
gcc $(FLAGS) o $@ $<
# Then a sequence of actions are calculated
# to generate the goal given on the command line.
# A goal with a newer timestamp than its
# requirements is considered up to date.
- 11. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.11
How a makefile works, macro calls
$(eval $(call SetupJavaCompilation,BUILD_JAXP,
SETUP:=GENERATE_NEWBYTECODE,
SRC:=jaxp/src,
BIN:=output/classes,
SRCZIP:=output/src.zip),
JAR:=output/classes.jar))
all: output/src.zip output/classes.jar
- 12. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.12
A javac command line
javac bootclasspath classes d classes src/java/lang/Object.java
src/java/lang/CharacterData.java src/java/lang/OutOfMemoryError.java
src/java/lang/Character.java src/java/lang/Override.java
src/java/lang/CharacterName.java src/java/lang/packageinfo.java
src/java/lang/CharSequence.java src/java/lang/Package.java
src/java/lang/ClassCastException.java src/java/lang/ProcessBuilder.java
src/java/lang/ClassCircularityError.java src/java/lang/Process.java
src/java/lang/ClassFormatError.java src/java/lang/Readable.java
src/java/lang/Class.java src/java/lang/ClassLoader.java
src/java/lang/ClassNotFoundException.java
src/java/lang/ReflectiveOperationException.java
src/java/lang/ClassValue.java src/java/lang/Runnable.java
src/java/lang/Cloneable.java .....
and ca 8000 more files....
You cannot really use the command line for large projects.
An @-file can be used to work around the command line length problem.
We need to use an external tool.
- 13. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.13
An ant build file
<project name="MyProject" default="dist" basedir=".">
<property name="src" location="src"/>
<property name="build" location="build"/>
<property name="dist" location="dist"/>
<target name="init">
<tstamp/>
<mkdir dir="${build}"/>
</target>
<target name="compile" depends="init" description="compile it>
<javac srcdir="${src}" destdir="${build}"/>
</target>
<target name="dist" depends="compile" description=”generate jar”>
<mkdir dir="${dist}/lib"/>
<jar jarfile="${dist}/lib/MyProject${DSTAMP}.jar"
basedir="${build}"/>
</target>
</project>
- 14. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.14
An attempt at makefile
SRC:=.../src
BUILD:=.../build
DIST:=.../dist
DSTAMP:=$(shell date +'%Y%m%d')
SOURCES:=$(shell find $(SRC) name ”*.java”)
$(DIST)/lib/MyProject$(DSTAMP).jar : $(BUILD)/the_build
mkdir p $(@D)
(cd $(BUILD) ; jar cf $@ *)
$(BUILD)/the_build : $(SOURCES)
mkdir p $(@D)
javac $(SOURCES) d $(BUILD)
touch $@
No incremental compile
No incremental jar update
Limit on the number of source files!
- 15. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.15
The major obstacles to writing makefiles for Java
unpredictable output from javac
for any decent size Java project, the filenames of the Java sources
do not fit on the command line!
- 16. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.16
Whither ant? Whither make?
It does not matter if you prefer make or ant. You cannot do proper
dependency tracking and incremental builds in Java because:
Javac lacks the necessary features! This cannot be fixed, lest you
write your own Java compiler, which is a lot of work.
It does not matter if you prefer make or ant. You cannot get proper
multi core support for speeding up compilation times for large Java
projects, because:
Javac lacks the necessary features! This cannot be fixed externally.
- 17. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.17
Whither ant? Whither make?
But what about ant depend?
Limited to what it can see in class files, eg cannot see where final
statics came from.
But what about ant fork=yes?
Does not work within a single javac task. Manual splitting of sources
is needed.
- 18. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.18
Whither IDEs?
IDEs like Eclipse and Netbeans do keep a database of the source
dependencies in memory. Needed for quick and efficient incremental
compiles while editing!
But very limited support for multi core compiles.
We need ant/make for batch compiles on build servers anyway.
- 19. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.19
A first attempt at improving javac
added a server to javac
added -XDdepfile=......
added -XDnativeapifile=.....
added -XDpubapifile=....
Relied on make (sic!) to split compile into multiple cores with a javac
invocation for each package.
This worked, sort of, but lets look at the makefile.
- 20. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.20
A horribly complex (tiny) part of a makefile
$2 : $$($2_PACKAGE_SRCS)
$(MKDIR) p $$($2_PACKAGE_BDIR)
$(RM) $2.tmp
$$(call ListPathsSafely,$2_PACKAGE_SRCS,n, >> $2.tmp)
$(ECHO) $$($2_PACKAGE_BDIR)*.class | $(GREP) v *.class | $(TR) ' ' 'n' >
$$($2_PACKAGE_BDIR)_the.package.prev
$(RM) $$($2_PACKAGE_BDIR)*.class $$($2_PACKAGE_BDIR)*.notify $$($2_PACKAGE_BDIR)*.deleted
$(ECHO) Compiling `$(WC) $2.tmp | $(TR) s ' ' | $(CUT) f 2 d ' '` files in package $(patsubst $4/%/,%,
$(dir $2.tmp))
$9 $$($2_REMOTE) $$($2_DEPS) $$($2_PUBAPI) $$($2_NATIVEAPI) $(10) implicit:none sourcepath
"$$($2_SRCROOTSC)" d $4 @$2.tmp
$(ECHO) $$($2_PACKAGE_BDIR)*.class | $(GREP) v *.class | $(TR) ' ' 'n' >
$$($2_PACKAGE_BDIR)_the.package.now
($(GREP) xvf $$($2_PACKAGE_BDIR)_the.package.now $$($2_PACKAGE_BDIR)_the.package.prev >
$$($2_PACKAGE_BDIR)_the.package.deleted;true)
$(ECHO) $1_CLASSES += `$(CAT) $$($2_PACKAGE_BDIR)_the.package.now` |
$(SED) 's/$$$$/$$$$$$$$/g' > $$($2_PACKAGE_BDIR)_the.package.d
$(ECHO) $1_JAVAS += $$($2_PACKAGE_SRCS) >> $$($2_PACKAGE_BDIR)_the.package.d
$(ECHO) $2_NOTIFIED:=true > $$($2_PACKAGE_BDIR)_the.package.notify
$$($2_APPEND_DEPS)
$$($2_COPY_FILES)
$(MV) f $2.tmp $2
- 21. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.21
A better way, move the makefile logic into a smart
javac wrapper (sjavac) written in Java.
sjavac -src sources -d classes
Compiles all sources below sources. (Expects standard package file
structure)
Creates a classes/javac_state file that contains the necessary
information for incremental compiles.
(timestamps, artifacts, dependencies and public apis)
Forces -implicit:none
Supports multiple cores.
javac must provide hooks for this wrapper to work.
- 22. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.22
A much better part of a makefile (plain javac)
(common/makefiles/JavaCompilation.gmk)
$$($1_BIN)/_the.batch: $$($1_SRCS) $$($1_DEPENDS)
$(MKDIR) p $$(@D)
$$(call ListPathsSafely,$1_SRCS,n, >> $$($1_BIN)/_the.batch.tmp)
$(ECHO) Compiling $1
$$($1_JVM) $$($1_JAVAC)
@$$($1_BIN)/_the.batch.tmp
$$($1_FLAGS)
d $$($1_BIN) $$($1_HEADER_ARGS)
Apropos the ListPathsSafely macro.... my birthday present wish is a new function in
GNU make:
$(write_to_file the.batch.tmp $(SOURCES))
Such a function can be used to solve any command line length limitation!
- 23. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.23
A much better part of a makefile (sjavac)
(common/makefiles/JavaCompilation.gmk)
$$($1_BIN)/javac_state: $$($1_SRCS) $$($1_DEPENDS)
$(MKDIR) p $$(@D)
$$(call ListPathsSafely,$1_SRCS,n, >> $$($1_BIN)/_the.batch.tmp)
$(ECHO) Compiling $1
$$($1_JVM) $$($1_SJAVAC)
$$($1_REMOTE) $$($1_SJAVAC_ARGS) permitunidentifiedartifacts
mfl $$($1_BIN)/_the.batch.tmp
$$($1_FLAGS)
d $$($1_BIN) $$($1_HEADERS_ARG)
Apropos the ListPathsSafely macro.... my birthday present wish is a new function in
GNU make:
$(write_to_file the.batch.tmp $(SOURCES))
Such a function can be used to solve any command line length limitation!
- 24. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.24
Your future makefile
SJAVAC:=java jar javac.jar com.sun.tools.sjavac.Main
SOURCES:=$(shell find src name ”*.java”)
bin/javac_state : $(SOURCES)
$(SJAVAC) src src h jniheaders d bin
Very fast source file timestamp scan, courtesy make
Proper incremental build, including removal of class files.
Proper generation and removal of C-header files for JNI
Multi core support
Oops, jar is not incremental....yet.
- 25. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.25
Your future ant build.xml
just replace the task javac with sjavac
Not so fast source file timestamp scan, but ok.
- 26. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.26
Your future ant build.xml
R src src d bin
P com.sun.java_cup.internal.runtime
S src/com/sun/java_cup/internal/runtime/Scanner.java
1347955522000
D java.lang
D java.util
I TYPE com.sun.java_cup.internal.runtime.Scanner
I METHOD public abstract
com.sun.java_cup.internal.runtime.Symbol next_token(
) throws java.lang.Exception
I VAR public int sym
A bin/com/sun/java_cup/internal/runtime/Scanner.class
1349217684000
- 27. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.27
Why is it hard to make javac multi core?
- 28. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.28
Why is it hard to make javac multi core?
- 29. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.29
Why is it hard to make javac multi core?
- 30. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.30
Why is it hard to make javac multi core?
- 31. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.31
One way to do it.....
- 32. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.32
Optimistic splitting compiling
3 cores give a 20% speedup.
Object.java is always compiled thrice....
Obvious improvements:
Share already compiled classes
Share loaded classes from classpath
Do emit on all cores
- 33. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.33
Optimistic splitting compiling
Need a new warning -Xlint:auxiliaryclass
Catches references to classes hidden inside sourcefiles
in such a way that they cannot be found on the
through the -sourcepath. Since this breaks multi core
compilation (and is generally a nuisance).
- 34. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.34
Lets investigate the new build system for the OpenJDK
Which was in fact the cause for developing sjavac
- 35. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.35
Status quo april 2011
Too speed up the OpenJDK build we faced a formidable problem!
Javac quirks and limitations
The Java language tendency for circular dependencies
Undocumented side effects of compiles
No working dependency propagation Java --> C++ --> Java
Lack of proper dependency propagation makes it hard to
parallelize the build!
Of course all the above are relevant for incremental builds too!
- 36. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.36
The OpenJDK is a complex product
It has a Java compiler
It has a JVM containing C++ code and Java code.
It has the JDK, containing C/C++ code and Java code.
It has complex intra-dependencies:
C++ code depends on C-headers generated from Java code.
Java code generated from C-headers from the OS.
Generated Java/C and C++ code from several different sources.
The new Java code sometimes needs the new Java compiler
The new Java compiler sometimes needs the new JDK
- 37. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.37
The Old OpenJDK build system was troublesome
Controlled through environment variables and command line
variables
No working dependency tracking
Slow compilation because of inefficient usage of build computer
Multiple almost identical copies of the same Makefiles
The source code has platform specific files in share/classes
and vice versa
A huge technical debt
- 38. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.38
The Old OpenJDK build system was troublesome
Each and every developer created their own build setup script.
This knowledge was shared through a human peer to peer network.
We encoded this knowledge into a configure script.
The configure api is a well known standard.
- 39. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.39
How to configure and build using the new build system.
hg clone http://hg.openjdk.java.net/buildinfra/jdk8
cd jdk8
sh get_source.sh
cd common/makefiles
bash ../autoconf/configure –enablesjavac
make
...........a lot of output........
cd ../..
./build/linuxx86normalserverrelease/jdk/bin/java version
make images
ls ./build/linuxx86normalserverrelease/images/j2sdkimage
make install
- 40. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.40
Contents of the generated spec.gmk file.
SRC_ROOT:=/work/jdk8
...
OUTPUT_ROOT:=/work/jdk8/build/linuxx86normalserverrelease
...
BOOT_JDK:=/usr/lib/jvm/java7openjdki386
...
CC:=CCACHE_COMPRESS=1 CCACHE_SLOPPINESS=time_macros
/usr/local/bin/ccache /usr/bin/cc
...
USE_EXTERNAL_LIBJPEG:=true
USE_EXTERNAL_LIBGIF:=false
USE_EXTERNAL_LIBZ:=false
...
SYS_ROOT:=/
- 41. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.41
The examples of the configure api.
configure help
withbootjdk=/../jdk7
enablesjavac
disablestaticlinkstdc++
host=i686unknownlinuxgnu
enabledebug
withjvmvariants=client,server
- 42. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.42
The first build of the OpenJDK with the new
build system and multi core sjavac
....
...
.
Compiling 2985 files in 181 packages (com.oracle.net to java.util.concurrent)
Compiling 2973 files in 168 packages (java.util.concurrent.atomic to sun.nio.ch.sctp)
Compiling 3022 files in 559 packages (sun.nio.cs to sun.util.xml)
....
...
.
Use “make VERBOSE=” to see all command lines.
- 43. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.43
The actual command line to compile the JDK!
sjavac –server:portfile=javacservers/server.port,poolsize=8,
id=GENERATE_JDKBYTECODE,javac=sjavac
x com.sun.pept.* x com.sun.tools.example.trace.*
xf *SolarisFileStore.java xf *SolarisFileSystem.java
... many more filter rules ...............
src jdk/src/share/classes:jdk/src/solaris/classes:gensrc
permitunidentifiedartifacts
mfl ...._the.batch.tmp
bootclasspath jdk/classes
source 7 target 7 encoding ascii
XDignore.symbol.file=true
Xlint:all,....,auxiliaryclasses
implicit:none
d jdk/classes
h jdk/gensrc_headers
- 44. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.44
An incremental build of the OpenJDK
touch ../../jdk/src/share/classes/com/sun/tools/jdi/SDE.java
make
########################################################################
########################################################################
##### Entering jdk for target(s) all #####
########################################################################
Compiling BUILD_JDK
Compiling com.sun.tools.jdi(80)
Note:
/home/fohrstro/jdk8/jdk/src/share/classes/com/sun/tools/jdi/EventRequestMa
nagerImpl.java uses unchecked or unsafe operations.
Note: Recompile with Xlint:unchecked for details.
########################################################################
##### Leaving jdk for target(s) all #####
########################################################################
##### Build time 00:00:08 jdk for target(s) all #####
########################################################################
- 45. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.45
An incremental build of the OpenJDK
emacs nw ../../jdk/src/share/classes/javax/net/SocketFactory.java
make
...
..
.
Compiling javax.net(2)
Package javax.net pubapi has changed!
Compiling com.sun.jndi.ldap(46)
Compiling javax.net.ssl(37) javax.rmi.ssl(2)
...
..
.
- 46. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.46
Old
New +sjavac
no ccache
New no sjavac
primed ccache
New +sjavac
empty ccache
New +sjavac
primed ccache
Langtools 37s 30s 29s 31s 31s
Corba 1m32s 7s 16s 7s 7s
Jaxp 18s 7s 15s 7s 7s
Jaxws 23s 9s 19s 9s 9s
Hotspot 3m01s 3m05s 1m10s 5m18s 1m7s
Jdk 9m52s 1m23s 2m9s 1m25s 50s
Total 15m43s 5m21s 4m38s 7m37s 2m51s
OpenJDK Build times: GNU/Linux 64bit 8cores 16GB
sjavac doubles the compilation speed! This is due to the server and multi core
support. There are several opportunities for improving the multi core support!
We have only skimmed the surface so far.
- 47. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.47
Summary
Initial support for multi core javac, more to come.
Incremental Java builds from the command line with proper
dependency tracking.
The new javac -h option to generate C-headers for JNI
A new lint warning -Xlint:auxiliaryclass
A much more inviting development environment for every new
developer who wants to contribute to the OpenJDK!
- 48. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.48
TODO
Push into tl/jdk8/langtools
now it is inside build-infra/jdk8/langtools
Incremental jar updates
Write sjavac task for ant
Add public api tracking for classes and jars
- 49. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 1349