1.
Software Ecosystems as
Networks: the FASTEN project
Paolo Boldi
Università degli Studi di
Milano
Italy

2.
Programmers and sharing

3.
Programmers and sharing
❖ The fundamental act of
friendship among
programmers is the sharing of
programs
The GNU manifesto (1985)

4.
Sharing through software libraries

5.
Sharing through software libraries
❖ One form of sharing is providing libraries

6.
Sharing through software libraries
❖ One form of sharing is providing libraries
❖ Today, libraries are made available in the Internet

7.
Sharing through software libraries
❖ One form of sharing is providing libraries
❖ Today, libraries are made available in the Internet
❖ on forges (SourceForge, GitHub, BitBucket, …)

8.
Sharing through software libraries
❖ One form of sharing is providing libraries
❖ Today, libraries are made available in the Internet
❖ on forges (SourceForge, GitHub, BitBucket, …)
❖ or repositories (Maven, PyPi, CPAN, …)

9.
Sharing through software libraries
❖ One form of sharing is providing libraries
❖ Today, libraries are made available in the Internet
❖ on forges (SourceForge, GitHub, BitBucket, …)
❖ or repositories (Maven, PyPi, CPAN, …)
❖ Internet made the dream of collaborative development a
reality

10.
Industrial revolution
at the harbour of software development

11.
Industrial revolution
at the harbour of software development
❖ All trades, arts, and handiworks have gained by
division of labour, namely, when, instead of one
man doing everything, each confines himself to a
certain kind of work distinct from others in the
treatment it requires, so as to be able to perform it
with greater facility and in the greatest
perfection. Where the different kinds of work are
not distinguished and divided, where everyone is
a jack-of-all-trades, there manufactures remain
still in the greatest barbarism.
Immanuel Kant
Groundwork for the Metaphysics
of Morals (1785)

12.
With libraries come dependencies…

13.
With libraries come dependencies…
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<groupId>foo-bar</groupId>
<artifactId>NeoImport</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.neo4j.driver</groupId>
<artifactId>neo4j-java-driver</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.26</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
</dependencies>
</project>

14.
With libraries come dependencies…
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<groupId>foo-bar</groupId>
<artifactId>NeoImport</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.neo4j.driver</groupId>
<artifactId>neo4j-java-driver</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.26</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
</dependencies>
</project>
Library

15.
With libraries come dependencies…
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<groupId>foo-bar</groupId>
<artifactId>NeoImport</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.neo4j.driver</groupId>
<artifactId>neo4j-java-driver</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.26</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
</dependencies>
</project>
Library
Dependencies

16.
With libraries come dependencies…
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<groupId>foo-bar</groupId>
<artifactId>NeoImport</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.neo4j.driver</groupId>
<artifactId>neo4j-java-driver</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.26</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
</dependencies>
</project>
Library
Dependencies
+version

17.
With libraries come dependencies…
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<groupId>foo-bar</groupId>
<artifactId>NeoImport</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.neo4j.driver</groupId>
<artifactId>neo4j-java-driver</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.26</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
</dependencies>
</project>
Library
Dependencies
+version
+version constraints

18.
…and transitive dependencies…

19.
…and transitive dependencies…
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<groupId>foo-bar</groupId>
<artifactId>NeoImport</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.neo4j.driver</groupId>
<artifactId>neo4j-java-driver</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.26</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
</dependencies>
</project>

20.
…and transitive dependencies…
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<groupId>foo-bar</groupId>
<artifactId>NeoImport</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.neo4j.driver</groupId>
<artifactId>neo4j-java-driver</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.26</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
</dependencies>
</project>
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>it.unimi.dsi</groupId>
<artifactId>webgraph</artifactId>
<version>3.6.1</version>
</dependency>
<dependency>
<groupId>org.eclipse.jgit</groupId>
<artifactId>org.eclipse.jgit</artifactId>
<version>5.2.1.201812262042-r</version>
</dependency>
</dependencies>
</project>
Depends

21.
Dependency graphs

22.
Dependency graphs
❖ Library+versions and their
dependencies form (complex,
huge) dependency networks

23.
Dependency graphs
❖ Library+versions and their
dependencies form (complex,
huge) dependency networks
❖ Version constraints make these
networks more complicated
than simple graphs

24.
Dependency graphs
❖ Library+versions and their
dependencies form (complex,
huge) dependency networks
❖ Version constraints make these
networks more complicated
than simple graphs
❖ Package manager will finally
determine which version is
chosen for each library

25.
The dependency heaven

26.
The dependency heaven
❖ Relying on an
ecosystem of easy-to-
use well written
libraries made the
dream of code reuse a
reality

27.
The dependency hell

28.
The dependency hell
❖ A bug or security
breach or legal issue
concerning one single
piece…
❖ …can make the whole
tower fall!

29.
Recent dependency nightmares

30.
Recent dependency nightmares
❖ The leftpad incident (2016): millions of websites
affected

31.
Recent dependency nightmares
❖ The leftpad incident (2016): millions of websites
affected
❖ The Equifax breach (2017): costed 4B$

32.
Epidemics in dependency graphs
Lib A, vers 1.0
Lib B, vers 2.5
Lib C, vers 1.5Lib D, vers 3.0

33.
Epidemics in dependency graphs
Lib A, vers 1.0
Lib B, vers 2.5
Lib C, vers 1.5Lib D, vers 3.0
A vulnerability alert
is issued
about Lib D, vers 3.0

34.
Epidemics in dependency graphs
Lib A, vers 1.0
Lib B, vers 2.5
Lib C, vers 1.5Lib D, vers 3.0
A vulnerability alert
is issued
about Lib D, vers 3.0
All libraries in this
graph are infected!

35.
GitHub security alerts
But is this enough?

36.
The FASTEN Project
❖ Fine-Grained Analysis of SofTware Ecosystems as Networks
❖ Part of the EU H2020-ICT-2018-2020 Program
❖ Consortium

37.
Epidemics in dependency graphs
Lib A, vers 1.0
Lib B, vers 2.5
Lib C, vers 1.5Lib D, vers 3.0

38.
Epidemics in dependency graphs
A.f0
A.f2
A.f3
B.f1
B.f2
B.f3
C.f1
C.f2
D.f1
D.f2
D.f3

39.
Epidemics in dependency graphs
A.f0
A.f2
A.f3
B.f1
B.f2
B.f3
C.f1
C.f2
D.f1
D.f2
D.f3
A vulnerability alert
is issued
about Lib D, vers 3.0,
function f3

40.
Epidemics in dependency graphs
A.f0
A.f2
A.f3
B.f1
B.f2
B.f3
C.f1
C.f2
D.f1
D.f2
D.f3
A vulnerability alert
is issued
about Lib D, vers 3.0,
function f3

41.
Epidemics in dependency graphs
A.f0
A.f2
A.f3
B.f1
B.f2
B.f3
C.f1
C.f2
D.f1
D.f2
D.f3
A vulnerability alert
is issued
about Lib D, vers 3.0,
function f3
Much more informative!

42.
Examples

43.
Examples
❖ Fully precise change impact analysis: “How many libraries
are affected if I remove/modify a certain method/interface?”

44.
Examples
❖ Fully precise change impact analysis: “How many libraries
are affected if I remove/modify a certain method/interface?”
❖ Fully precise license compliance: “Is my library compliant
with the licenses of the libraries that I depend from (directly or
indirectly)? (e.g., am I linking any GPL code?)”

45.
Examples
❖ Fully precise change impact analysis: “How many libraries
are affected if I remove/modify a certain method/interface?”
❖ Fully precise license compliance: “Is my library compliant
with the licenses of the libraries that I depend from (directly or
indirectly)? (e.g., am I linking any GPL code?)”
❖ Fully precise risk profiling: “Does this vulnerability affect my
code?”

46.
Examples
❖ Fully precise change impact analysis: “How many libraries
are affected if I remove/modify a certain method/interface?”
❖ Fully precise license compliance: “Is my library compliant
with the licenses of the libraries that I depend from (directly or
indirectly)? (e.g., am I linking any GPL code?)”
❖ Fully precise risk profiling: “Does this vulnerability affect my
code?”
❖ Centrality analysis: “What methods/functions are more central
within a given ecosystem? are there bottlenecks? critical points?”

47.
The FASTEN toolchain

48.
The FASTEN toolchain
Project information
Security
alerts
Repositories

49.
The FASTEN toolchain
Project information
Security
alerts
Repositories
publish
Data stream
publish
publish

50.
The FASTEN toolchain
Project information
Security
alerts
Repositories
publish
Data stream
FASTEN
server
publish
publish

51.
The FASTEN toolchain
Project information
Security
alerts
Repositories
publish
Data stream
FASTEN
server
Call-graph
construction
publish
publish

52.
The FASTEN toolchain
Project information
Security
alerts
Repositories
publish
Data stream
FASTEN
server
Call-graph
construction
Storage
layer
publish
publish

53.
The FASTEN toolchain
Project information
Security
alerts
Repositories
publish
Data stream
FASTEN
server
Call-graph
construction
Storage
layer
Analysis
layer
publish
publish

54.
The FASTEN toolchain
Project information
Security
alerts
Repositories
publish
Data stream
FASTEN
server
Call-graph
construction
Storage
layer
Analysis
layer
RESTApi
publish
publish

55.
The FASTEN toolchain
Project information
Security
alerts
Repositories
publish
Data stream
FASTEN
server
Call-graph
construction
Storage
layer
Analysis
layer
RESTApiWebUI
publish
publish

56.
The FASTEN toolchain
Project information
Security
alerts
Repositories
publish
Data stream
FASTEN
server
Call-graph
construction
Storage
layer
Analysis
layer
RESTApiWebUI
publish
publish
Continuous
integration server

57.
The FASTEN toolchain
Project information
Security
alerts
Repositories
publish
Data stream
FASTEN
server
Call-graph
construction
Storage
layer
Analysis
layer
RESTApiWebUI
publish
publish
Continuous
integration server

58.
The FASTEN toolchain
Project information
Security
alerts
Repositories
publish
Data stream
FASTEN
server
Call-graph
construction
Storage
layer
Analysis
layer
RESTApiWebUI
publish
publish
Continuous
integration server
Developer

59.
Updating a library

60.
Updating a library
# Check outdated dependencies
$ pip list --outdated
Package Version Latest Type
---------- ------- ------ -----
Pygments 2.2.0 2.3.1 wheel
Before

61.
Updating a library
# Check outdated dependencies
$ pip list --outdated
Package Version Latest Type
---------- ------- ------ -----
Pygments 2.2.0 2.3.1 wheel
Before # Check outdated dependencies
$ pip list --outdated
Package Version Latest Type
---------- ------- ------ -----
Pygments 2.2.0 2.3.1 wheel
Updating Pygments will affect:
foo.py: function colorize
bar.py: function parse
After

62.
Updating a library
# Check outdated dependencies
$ pip list --outdated
Package Version Latest Type
---------- ------- ------ -----
Pygments 2.2.0 2.3.1 wheel
Before # Check outdated dependencies
$ pip list --outdated
Package Version Latest Type
---------- ------- ------ -----
Pygments 2.2.0 2.3.1 wheel
Updating Pygments will affect:
foo.py: function colorize
bar.py: function parse
After
# Estimate update impact
$ pip install --dry-run Pygments
Function Pygments.Formatter.format[formatter.py]
changed ->
check <your_app> at colorize[foo.py]:32

63.
Information about a library

64.
Information about a library
# Checking info about the library
$ pip show tornado
Name: tornado
Version: 5.0
Summary: Tornado is a Python web framework ...
Home-page: http://www.tornadoweb.org/
Author: Facebook
Author-email: ...
License: http://www.apache.org/licenses/
LICENSE-2.0
Location: ...
Requires: backports-abc, futures,
singledispatch
Required-by:
Before

65.
Information about a library
# Checking info about the library
$ pip show tornado
Name: tornado
Version: 5.0
Summary: Tornado is a Python web framework ...
Home-page: http://www.tornadoweb.org/
Author: Facebook
Author-email: ...
License: http://www.apache.org/licenses/
LICENSE-2.0
Location: ...
Requires: backports-abc, futures,
singledispatch
Required-by:
Before
# Checking info about the library
$ pip show tornado
Name: tornado
Version: 5.0
License: http://www.apache.org/licenses/
LICENSE-2.0
...
Maintainers: 3
Community size: 15
Used by: 145 on PyPI, 34433 on GitHub
Latest vulnerability: 13 months ago
All known vulnerabilities: 25 (best 10%)
License rating: Compatible
After

66.
Who uses my function?

67.
Who uses my function?
# Check uses of function pkg.list() in dependents
$ pip query --uses pkg.list
depA(v1.2).parse()
depA(v1.2).test()
depB(0.0.2).foo()
depC(1.2.1).calculate()X

68.
Who uses my function?
# Check uses of function pkg.list() in dependents
$ pip query --uses pkg.list
depA(v1.2).parse()
depA(v1.2).test()
depB(0.0.2).foo()
depC(1.2.1).calculate()X
# Rank them by importance (aka centrality)
$ pip query --uses --rank pkg.list
depC(1.2.1).calculate()
depB(0.0.2).foo()
depA(v1.2).parse()
depA(v1.2).test()

69.
Who uses my function?
# Check uses of function pkg.list() in dependents
$ pip query --uses pkg.list
depA(v1.2).parse()
depA(v1.2).test()
depB(0.0.2).foo()
depC(1.2.1).calculate()X
# Estimate "damage" if pkg.list will be updated
$ pip query —total pkg.list
3 direct and 223 indirect dependencies will be affected)
# Rank them by importance (aka centrality)
$ pip query --uses --rank pkg.list
depC(1.2.1).calculate()
depB(0.0.2).foo()
depA(v1.2).parse()
depA(v1.2).test()

70.
Who uses my function?
# Check uses of function pkg.list() in dependents
$ pip query --uses pkg.list
depA(v1.2).parse()
depA(v1.2).test()
depB(0.0.2).foo()
depC(1.2.1).calculate()X
# Estimate "damage" if pkg.list will be updated
$ pip query —total pkg.list
3 direct and 223 indirect dependencies will be affected)
# Notify direct dependencies of upcoming breakage
$ pip query --uses pkg.list |
cut -f 1 -d '(' |
xargs -I {} pip show {} |
grep Author-email: | cut -f 2 -d ':' |
xargs mail -s 'MyProject update will break yours!'
# Rank them by importance (aka centrality)
$ pip query --uses --rank pkg.list
depC(1.2.1).calculate()
depB(0.0.2).foo()
depA(v1.2).parse()
depA(v1.2).test()

71.
FASTEN challenges

72.
FASTEN challenges
❖ Sound call-graph generation

73.
FASTEN challenges
❖ Sound call-graph generation
❖ Scalability (huge graphs!)

74.
FASTEN challenges
❖ Sound call-graph generation
❖ Scalability (huge graphs!)
❖ Real-time responsiveness

75.
FASTEN challenges
❖ Sound call-graph generation
❖ Scalability (huge graphs!)
❖ Real-time responsiveness
❖ Data integration

76.
FASTEN challenges
❖ Sound call-graph generation
❖ Scalability (huge graphs!)
❖ Real-time responsiveness
❖ Data integration
❖ Tool development for end users

77.
Network analysis will be the next
step for the future of
software development

78.
Network analysis will be the next
step for the future of
software development

79.
Questions?
Paolo Boldi
Università degli Studi di
Milano
Italy
paolo.boldi@unimi.it