https://www.eclipsecon.org/europe2018/sessions/parallel-builds-eclipse-ide-workspace-value-making-and-optimal-usages Eclipse Platform 4.8 introduced support for parallel builds in the workspace. This feature is placed at the lowest level of the workspace so it can easily be profitable to any adopter without specific adoption effort, besides respecting usual good practices. This can in theory turn the Eclipse workspace into one of the fastest polyglot build engines in developer world. In this presentation, we’ll explain what is the value to expect from this parallel builds (mostly performance gain and faster availability of project output). Then we’ll also get into the details of how all that is implemented, reminding ourselves some basics of dependency resolution, graph processing and scheduling rules. We’ll remind the important best practices of builders and scheduling in the Eclipse resource model that are necessary to reach best results. And we’ll go through a few case studies of project builders to highlight how those may be improved to take advantage of parallel builds, and get into deep details of how this can be enabled in JDT and other important Eclipse projects. After this presentation, plugin developers will be empowered to make their project builders take advantage of parallel builds in Eclipse IDE, and may get ideas about really using this feature as an alternative build engine to speed up their headless builds as well.

1. Parallel builds in Eclipse Workspaces
Value, making-of and optimal usages
Mickael Istria - @mickaelistria CC0 – No rights reserved

2. Demo Tool
For the demo/talk we use
https://github.com/mickaelistria/eclipse-ide-parallel-builds-demo
which
– Defines the WaitBuilder
– Allows to generate build Gantt chart
– Allows to generate signature of build
output (compare output)
– Allows to control scheduling rule for JDT
and M2E builders

3. Demo scenario
6 projects with WaitBuilder waiting for some time (Thread.sleep())
– Projects deps directly configured in .project
– Builder duration configured in waitBuilder.properties (default 1000ms)
– Builder schedulingRule configured in waitBuilder.properties (default current
project) can be set to workspace root or some other project.

4. The basics: legacy (pre-Photon)
Builds can only be sequential

5. The basics: Paralleling
● Create 1 job per project build
● Throttle on limited amount of threads
using JobGroup API
● Preference used by good old
Workspace.build() API
● Default in Photon and 2018-09 is
disabled (nbThreads == 1)

6. The basics: Parallelized
No dependencies, isolated scheduling rules

7. The basics: Benefits
● Shorter full time to completion (& less energy consumed)
● Faster availability of each project
– A long project or chain of projects build doesn’t block other
projects for too long
– If you have handlers on project builds completion, they’re
called way earlier
● Applies to any workspace-build solutions, including non-
IDE offerings like JDT-LS

8. Relying on jobs→Usable progress View

9. Dependency Graph: no parallel
Dependency graph is computed, and turned
into a sequence (via topological sort)
p3,p1,p2,p5,p4,p6

10. Dependency graph: Parallel
Graph is used directly and jobs are scheduled
for builds as soon as all ancestors are built

11. About Scheduling Rules
● 2 operations using conflicting scheduling rules
rule1.isConflicting(rule2) can NOT run in parallel
● Scheduling rules are usually Resource Path which conflict when one path is
a parent of the other:
– a/b and a/b/c are conflicting and those scheduling rules prevent jobs for
running in parallel
– a/b and a/c are not conflicting and those schduling rules can allow jobs to run in
parallel
● Builders are expected to implement
IncrementalProjectBuilder.getRule(…)
Default is workspace.getRoot()
Here, p1 build rule conflicts with all other build rules → p1 won’t build in // of other projects

12. Scheduling rules don’t change
sequential scheduling
No jobs → no issue about conflicting scheduling
rules

13. So why scheduling rules on builders?
“Safety”: Scheduling rule are here to prevent some
other thread to change critical content while builder is
running. The scheduling rule defines what must NOT
be changed externally while builder is running.
GOLDEN RULE: To minimize conflicts and allow
more parallelism, set the narrowest rule your
builder can deal with.

14. (Bad) State of art
Many builders do not specify a scheduling rule
(defaulting to workspace root, conflicting with
every workspace operation)
→ Building a project locks the whole workspace!
Builder uses root as SR, conflicts with whole WS
Change file in another
unrelated Project + Save

15. (Bad) examples
● JDT Builder https://bugs.eclipse.org/bugs/show_bug.cgi?id=531554
● M2E Builder https://bugs.eclipse.org/bugs/show_bug.cgi?id=538461
● PDE Builder https://bugs.eclipse.org/bugs/show_bug.cgi?id=531555
…
Use default workspace scheduling rule,
preventing parallel execution and blocking user
actions in unrelated files!

16. With finer scheduling rule...
More specific scheduling rule, only necessary
content is locked by builder.
For example, just using getProject() as
scheduling rule can often be enough.
Builder uses project as SR, conflicts with only project content
Change file in another
unrelated Project + Save File just saved!

17. GOLDEN RULE:
To minimize conflicts and allow more parallelism,
set the narrowest rule your builder can deal with.

18. 2018-09, a new hope...
JDT and m2e have a (hidden) preference to control
the builder’s scheduling rule and let it be null.
Hidden, but available for
testing with the test wizard,
or with typical preference
customization strategy (API,
plugin_customization.ini…)
in your Eclipse-based
package (like JDT-LS).
Try them!

19. Scheduling rules and parallel builds
All rules conflicting → No parallel builder execution

20. Current impl does not pass rules to schedule jobs
and use rules at job exec. Some jobs are started for
nothing ( ) and can block other builds.
But could do a bit better

21. Now you got it
● 1 project build → 1 job
● Throttling with JobGroup
● Scheduled according to dependency graph
● Honoring scheduling rules
GOLDEN RULE: To minimize conflicts and allow
more parallelism, set the narrowest rule your
builder can deal with.

22. Highlighted cases
● Long build chain vs standalone projet
● Dependency cycles
● 1 project builder uses workspace root as
scheduling rule

23. Dependency chain vs standalone
project
1 thread
4 threads
p6 is available much much sooner
Some project output available
much sooner

24. With cycles, sequential
Compute a best sequential order (using “Strongly
connected component”) and apply it
p1,p2,p3,p4,p5,p6

25. With cycles, parallel
1. Process graph until no node is ready (symptom of cycle)
2. Then process the 1st
waiting node in computed sequential order
3. Then back to regular graph processing.
1 2 3

26. Workspace root scheduling rule
● For backward compatibility, if any builder uses workspace
root scheduling rule, disable parallel builds (as the
scheduler will use this rule in its thread)
● Anyway, workspace root would conflict with about
everything…
● So most builders currently prevents parallel builds

27. GOLDEN RULE:
To minimize conflicts and allow more parallelism,
set the narrowest rule your builder can deal with.

28. Case studies
1. dotnet build builder
2. JavaBuilder (JDT) and MavenBuilder
(m2e)

29. Case Study #1
6 independent C# projects building with dotnet
build (builder scheduling root == current project)
Sequential build takes 39s
Parallel on 4 threads takes 26s
Actual gain: 33% (cool, but...)
Expected gain: 66%
?

30. Case study #1 - lesson learned
Most real world operation have bottlenecks (I/O,
network…) making concurrent executions slower.
Sequential
4 threads
But still a major win!

31. Case study #2: Java (JDT & m2e)
JavaBuilder and MavenBuilder do use default
workspace root scheduling rule by default →
Disable/prevents parallel builds

32. GOLDEN RULE:
To minimize conflicts and allow more parallelism,
set the narrowest rule your builder can deal with.

33. Case study #2: Java (JDT & m2e)
Applying the GOLDEN RULE on
JavaBuilder and
MavenBuilder shows that null
might be a good scheduling rule
https://bugs.eclipse.org/bugs/show_bug.cgi?id=531554#c22 . null
is the perfect value to avoid
conflict and enabled parallelism.

34. Effect of null scheduling rule for
JDT and m2e effect: story #1
Legacy
Scheduling
Rule
null
Scheduling
Rule
Build with 4
threads

35. Effect of null scheduling rule for
JDT and m2e effect: story #2
Legacy
Scheduling
Rule
null
Scheduling
Rule
Build with 4
threads

36. Mickael Istria - @mickaelistria CC0 – No rights reserved
And remember GOLDEN RULE:
To minimize conflicts and allow more parallelism,
set the narrowest rule your builder can deal with.
Questions?