Slides for Introducing Adept @ scala exchange 2013 Fredrik Ekholdt Dependency management should be something nobody should need to relate to, yet a surprising amount of time is spent on this aspect which should simply work. Adept is a new dependency management system in Scala, here to change the way it is done on the JVM. This talk will be about what Adept is, why you should care and how you can help.

2. Dependency managers
“a dependency manager automates the process of
using and upgrading software libraries in a
consistent manner.”
Finds the libraries you need, based on the libraries you want

3. A note on dependency
management
It does not have any intrinsic value
But saves some time/effort
It is not interesting (boring)
But sometime takes time (especially debugging)
Useful because it shares a decision you made
ʹI want lib A version 2ʹ
We should update our libraries more often!
So it should be easy to do

4. Dependency managmenet
‑ the afghan war of the JVM?

5. Introducing Adept
“a person who is skilled or proficient at
something.”
A dependency manager
Reliable
Fast
Knows what works with what
Only a dependency manager, nothing else
Platform independent

6. State = identity(time,value)
Identity: the library and the artifacts to be used (jars in junit,
spring, ...)
Value: the version to use (4.11, 3.2.5, ...)
Time: the library as it is ʹright nowʹ (lib A 5.0‑SNAPSHOT
today VS tomorrow)
State: the version of a library at a given time

7. Reliability
Same result every time
On the JVM, dependency managers are sort of reliable
Will state (the versions of the libraries at any given time)
never change?

8. Versioned meta‑data
Meta‑data: defines the data (dependencies, names, ...)
We want:
Immutable meta‑data
That is easy to change
Atomically change the entire meta‑data
In other words: Git

9. Distributed repositories
Download all meta‑data for each repository
Benefits
You are in control of changes (pull)
You can go back in time (checkout)
Speed
Git is supringsly good at compressing
Maven central: 150 mbs of compressed data (once)
No round trips to fetch meta‑data
No ʹinvalidʹ meta‑data cache
Parellel download of artifacts

10. Artifacts and meta‑data
Artifacts: represents the actual files used (jars)
When you want some libraries...
..the dependency manager looks up the meta‑data
Dependency manager finds the libraries you need...
...based on the meta‑data
The artifacts you must download are therefore...
...defined in the meta‑data ‑ RIGHT?

11. Artifacts ‑ mark 1
WRONG: Choosing a library in Ivy/Maven
Means a new lookup to some repository (or cache)
Artifact = file that matches the same repository pattern (or
cache)
Artifacts that can be chosen are:
The ones supposed to be used based on some repository
meta‑data
The ones that happens to cached
A new artifact, but using cached meta‑data
A cached artifact, but using new meta‑data

12. Artifacts ‑ mark 2
Cleanly seperated from meta‑data and repository
Meta‑data points to the SHA‑256s
Artifact lookup files: SHA‑256 and locations
This means:
New artifact means new meta‑data
New meta‑data can safely use new/old artifact
Benfits:
You are sure you got the right one
Easy to reason about
Very easy to cache
Secure

13. Speed
When I say ʹa8dac15ʹ...
... I do not care where it comes from
Means I can:
Use url to download jar from file server (nothing new)
Check cache if jar is in the cache (nothting new)
Use different urls to download the same jar (in pieces)
Use bittorrent to download one jar
Use bittorrent to download many jars
...

14. State & Adept
Summary:
You ask for the commit hash of...
The repositories you need...
And the libraries you require
Which points a stable set of SHA‑256
No changes in state:
For a given input, same artifacts every time
QED: Adept is reliable

15. So...
{Performance, reliabilty} does it really matter?
Why hasnʹt this been fixed before?

16. A killer feature?

17. Compatibility matrix
Dependencies maps to a compatibility matrix

18. Ivy/Maven
Automatically resolves to the ʹhighestʹ/ʹlatestʹ/... version
Compatiblity is not factored in
There is no... matrix
Users must know how libraries work together:
Must specify version(s) (ranges) on how libraries are
compatible
Override/exclude if there is a problem
Programs are good at this
Dos not communicate intent

19. Authors declares how their
libraries are compatible

20. Example: sbt plugins
Imagine if sbt:
In addition to ʹversionʹ (0.12.0, 0.12.1, ...)
Defines an extra attribute: ʹbinary‑versionʹ (0.12, ...)
A plugin (e.g. sbteclipse) that requires sbt specifies the
ʹbinary‑versionʹ
stcis vrin10rqie stbnr-eso 01
belpe eso . eurs b iayvrin .2
stcis vrin11rqie stbnr-eso 01
belpe eso . eurs b iayvrin .2
stcis vrin20rqie stbnr-eso 01
belpe eso . eurs b iayvrin .3
The user/build tool:
I use/am sbt 0.12.0, what are the sbteclipse versions I can
use
(Then chose the ʹbestʹ version)

21. Attributes
Adept allows resolution based on:
version
binary‑version
But also:
Snapshot/prerelease
QA department ʹseal of approvalʹ
...
Any attribute is a first‑class citizen

22. Adept values
Ivy/Maven: ʹversionʹ is the ʹvalueʹ of the library
Adept: ʹversionʹ is a common attribute
Variant == value in Adept
Example:
sbt { version 0.12.0, binary‑version 0.12, qa‑tested false }
sbt { version 0.12.0, binary‑version 0.12, qa‑tested true }

23. Variants
Id
com.typesafe.play/play
Artifacts
Attributes
ʹversion = 2.2.0ʹ
ʹbinary‑version = 2.2ʹ
...
Dependencies
Id
com.typesafe.akka/akka‑actors
Constraints
ʹbinary‑version = 2.2ʹ
...

24. Adept resolution
3 states
Resolved (only one variant for each library)
Under‑constrained (too many variants)
Over‑constrained (conflicts, or non‑existing Id)
Naive algorithm
Grab Ids of dependencies ...
...and all constraints for Id
...for each resolved: repeat...

25. Resolved
Variants:
saalbay{..,21.,21.}
cl-irr 293 .02 .03
ak 205dpnso saa293
ka .. eed n cl ..
ak 220dpnso saa21
ka .. eed n cl .0
ak 221dpnso saa21
ka .. eed n cl .0
pa 220dpnso {ka22 saa21}
ly .. eed n ak ., cl .0
pa 221dpnso {ka22 saa21}
ly .. eed n ak ., cl .0
Dependencies: play 2.2.1 + akka 2.2.0 + scala‑library 2.10.3

26. Over‑constrained
Variants:
saalbay{..,21.,21.}
cl-irr 293 .02 .03
ak 205dpnso saa293
ka .. eed n cl ..
ak 220dpnso saa21
ka .. eed n cl .0
ak 221dpnso saa21
ka .. eed n cl .0
pa 220dpnso {ka22 saa21}
ly .. eed n ak ., cl .0
pa 221dpnso {ka22 saa21}
ly .. eed n ak ., cl .0
Dependencies: play 2.2.1 + akka 2.2.0 + scala‑library 2.9.3
Or: foobar 2.2.1 + akka 2.2.0 + scala‑library 2.10.3

27. Under‑constrained???
Variants:
saalbay{..,21.,21.}
cl-irr 293 .02 .03
ak 205dpnso saa293
ka .. eed n cl ..
ak 220dpnso saa21
ka .. eed n cl .0
ak 221dpnso saa21
ka .. eed n cl .0
pa 220dpnso {ka22 saa21}
ly .. eed n ak ., cl .0
pa 221dpnso {ka223 saa21}
ly .. eed n ak .., cl .0
!!^!!
!
!
Dependencies: akka 2.2.3 + scala‑library 2.10.3 + play (any
version)
Resolves: there is only exactly one possible play version: 2.2.1

28. Under‑constrained
Variants:
saalbay{..,21.,21.}
cl-irr 293 .02 .03
ak 205dpnso saa293
ka .. eed n cl ..
ak 220dpnso saa21
ka .. eed n cl .0
ak 221dpnso saa21
ka .. eed n cl .0
pa 220dpnso {ka22 saa21}
ly .. eed n ak ., cl .0
pa 221dpnso {ka22 saa21}
ly .. eed n ak ., cl .0
Dependencies: akka 2.2.1 + play 2.2.1 + any 2.10 scala‑library?
Or: akka 2.2.1 + scala‑library 2.10.3 + any 2.2 play?

29. Architecture: Extensions
Adept does not do much!
Ideal: I declare only binary‑versions and commit,
Build tools gives me the ʹhighestʹ versions compatible
Build tools needs Adept + some extensions
Small core = minimal changes
Different extensions, but consistent resolution

30. Extensions:
Conventions/attribute names and meaning
Version, binary‑version
Version ordering
Overrides
Exclusions
Imports (Ivy/Maven/...)
...

31. Overrides/exclusions
Need to resolve ʹbenignʹ conflicts
(Versions differ, but not binary‑versions)
Creates a new variant with:
Different dependencies
Unique ʹoverridesʹ/ʹexcludesʹ attributes
The social dependency manager?
I can figure how my library is used

32. Other usecases
Snapshots #1
Use ʹprereleaseʹ/ʹsnapshotʹ/... attribute
Overwrite variant + commit new version
Update meta‑data = new snapshot
Flexible ...
...but you know exactly which artifact/meta‑data that was
used

33. Other usecases
Snapshots #2
CI & nightlies:
We know which version we used:
Safely update metadata after passing tests
User use same constraints (even when stable is released)
But build tool can give warning
No SNAPSHOTs after a release, and...
...no failures because a ʹversionʹ has been removed

34. Where are we
at?

35. What works
Resolution
ʹEqualityʹ constraint
binary‑version 1.0 == 1.0
Extensions:
Overrides, excludes
Version ordering
Ivy imports
Tests (test framework)

36. Help!
Final design!
Now is the time to set things straight
Handle libraries that defines the same interface
Attribute types & comparable constraints? (types, !=, >, <, ...)
Meta‑data is not (de)serialized
Artifacts files are not (de)serialized
More tests/use cases

37. Help!!
Git is not implemented
No CLI (useful for testing)
No built tool plugins (we need: sbt, gradle, SBuild, ant,
maven?)
Resolution is concise (but could even more concise?)
Resolution is fast (but could be faster?)

38. In other words:
We NEED you!

39. Exciting stuff because
We can get it right this
time!

40. But...
We have to build a new
ecosystem :(

41. Adepthub

42. Ideal workflow
1. Figure the library you want
2. Choose only version(s) you want to be compatible with
3. Repeat

43. Adepthub #1
Online resolution using Adept
Hosts meta‑data
Workflow:
1. Go to adepthub.com
2. Search and add new (compatible) libraries
3. Save (creates a perma link)
4. User/plugin/build tool downloads one file with:
Perma link, if you want to update
Locations (urls) and some human readable meta‑data
for all artifacts

44. Adepthub benefits
For you:
Uses Adept and hosts Adept repositories
Search across all repositories using:
Keywords on Id
Project info: github url, author, company, ...
No need to download meta‑data
However, it is possible to do it if you want
Import (meta‑data enhancement)
Resolution is done online...
...but can be consumed by maven, ivy, ...
= no plugins required

45. Adepthub benefits
For library authors:
Easy to publish
Github interface
Public & private repositories
Can have nice stats:
Who is using version X anyways? Now we know!
Trophies: show your awesomeness
For me:
Easier to build plugins
Read file + download artifacts
Launches:
When it is done, hopefully February/March

46. Feedback
Is search.maven.org good enough?
Please show support by signing up for beta:
http://adepthub.com/#beta

47. Interested?
https://github.com/adept‑dm/adept
(README links to: mailing list/spec)

48. Thanks
freekh+adept ʹatʹ gmail.com
@ekhfre