This presentation was given by Paolo Boldi, Milano University, online. Abstract:The goal of the EU project FASTEN is being able to perform a more sophisticated analysis of security-vulnerability propagation, licensing compliance, and dependency risk profiles (among others) by relying on the call-level dependency network of the whole software ecosystem. We outline the purpose and structure of the project, and present some preliminary results.

1. Software Ecosystems as
Networks: Advances on the
FASTEN project
Paolo Bold
i
Università degli Studi di
Milan
o
Italy

2. Programmers and sharing

3. Programmers and sharing
❖ The fundamental act of
friendship among
programmers is the sharing of
programs
The GNU manifesto (1985)

4. Why FASTEN?

5. Sharing through software libraries

6. Sharing through software libraries
❖ One form of sharing is providing libraries

7. Sharing through software libraries
❖ One form of sharing is providing libraries
❖ Today, libraries are made available in the Internet

8. Sharing through software libraries
❖ One form of sharing is providing libraries
❖ Today, libraries are made available in the Internet
❖ on repositories (SourceForge, GitHub, BitBucket, …)

9. Sharing through software libraries
❖ One form of sharing is providing libraries
❖ Today, libraries are made available in the Internet
❖ on repositories (SourceForge, GitHub, BitBucket, …)
❖ or forges (Maven, PyPi, CPAN, …)

10. Sharing through software libraries
❖ One form of sharing is providing libraries
❖ Today, libraries are made available in the Internet
❖ on repositories (SourceForge, GitHub, BitBucket, …)
❖ or forges (Maven, PyPi, CPAN, …)
❖ Internet made the dream of collaborative development a
reality

11. Industrial revolution
at the harbour of software development

12. Industrial revolution
at the harbour of software development
❖ All trades, arts, and handiworks have gained by
division of labour, namely, when, instead of one
man doing everything, each con
fi
nes himself to a
certain kind of work distinct from others in the
treatment it requires, so as to be able to perform it
with greater facility and in the greatest
perfection. Where the different kinds of work are
not distinguished and divided, where everyone is
a jack-of-all-trades, there manufactures remain
still in the greatest barbarism.
Immanuel Kan
t
Groundwork for the Metaphysic
s
of Morals (1785)

13. With libraries come dependencies…

14. With libraries come dependencies…
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<groupId>foo-bar</groupId>
<artifactId>NeoImport</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.neo4j.driver</groupId>
<artifactId>neo4j-java-driver</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.26</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
</dependencies>
</project>

15. With libraries come dependencies…
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<groupId>foo-bar</groupId>
<artifactId>NeoImport</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.neo4j.driver</groupId>
<artifactId>neo4j-java-driver</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.26</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
</dependencies>
</project>
Library

16. With libraries come dependencies…
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<groupId>foo-bar</groupId>
<artifactId>NeoImport</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.neo4j.driver</groupId>
<artifactId>neo4j-java-driver</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.26</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
</dependencies>
</project>
Library
Dependencies

17. With libraries come dependencies…
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<groupId>foo-bar</groupId>
<artifactId>NeoImport</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.neo4j.driver</groupId>
<artifactId>neo4j-java-driver</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.26</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
</dependencies>
</project>
Library
Dependencies
+version

18. With libraries come dependencies…
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<groupId>foo-bar</groupId>
<artifactId>NeoImport</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.neo4j.driver</groupId>
<artifactId>neo4j-java-driver</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.26</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
</dependencies>
</project>
Library
Dependencies
+version
+version constraints

19. …and transitive dependencies…

20. …and transitive dependencies…
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<groupId>foo-bar</groupId>
<artifactId>NeoImport</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.neo4j.driver</groupId>
<artifactId>neo4j-java-driver</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.26</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
</dependencies>
</project>

21. …and transitive dependencies…
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<groupId>foo-bar</groupId>
<artifactId>NeoImport</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.neo4j.driver</groupId>
<artifactId>neo4j-java-driver</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.26</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
</dependencies>
</project>
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>it.unimi.dsi</groupId>
<artifactId>webgraph</artifactId>
<version>3.6.1</version>
</dependency>
<dependency>
<groupId>org.eclipse.jgit</groupId>
<artifactId>org.eclipse.jgit</artifactId>
<version>5.2.1.201812262042-r</version>
</dependency>
</dependencies>
</project>
Depends

22. Dependency graphs

23. Dependency graphs
❖ Library+versions and their
dependencies form (complex,
huge) dependency networks

24. Dependency graphs
❖ Library+versions and their
dependencies form (complex,
huge) dependency networks
❖ Version constraints make these
networks more complicated
than simple graphs

25. Dependency graphs
❖ Library+versions and their
dependencies form (complex,
huge) dependency networks
❖ Version constraints make these
networks more complicated
than simple graphs
❖ Package manager will
fi
nally
determine which version is
chosen for each library

26. The dependency heaven

27. The dependency heaven
❖ Relying on an
ecosystem of easy-to-
use well written
libraries made the
dream of code reuse a
reality

28. The dependency hell

29. The dependency hell
❖ A bug or security
breach or legal issue
concerning one single
piece
…
❖ …can make the whole
tower fall!

30. Recent dependency nightmares

31. Recent dependency nightmares
❖ The leftpad incident (2016): millions of websites
affected

32. Recent dependency nightmares
❖ The leftpad incident (2016): millions of websites
affected
❖ The Equifax breach (2017): costed 4B$

33. Epidemics in dependency graphs
Lib A, vers 1.0
Lib B, vers 2.5
Lib C, vers 1.5Lib D, vers 3.0

34. Epidemics in dependency graphs
Lib A, vers 1.0
Lib B, vers 2.5
Lib C, vers 1.5Lib D, vers 3.0
A vulnerability aler
t
is issue
d
about Lib D, vers 3.0

35. Epidemics in dependency graphs
Lib A, vers 1.0
Lib B, vers 2.5
Lib C, vers 1.5Lib D, vers 3.0
A vulnerability aler
t
is issue
d
about Lib D, vers 3.0
All libraries in this
graph are infected!

36. GitHub security alerts
But is this enough?

37. The FASTEN Project
❖ Fine-Grained Analysis of SofTware Ecosystems as Network
s
❖ Part of the EU H2020-ICT-2018-2020 Progra
m
❖ Consortium

38. Epidemics in dependency graphs
Lib A, vers 1.0
Lib B, vers 2.5
Lib C, vers 1.5Lib D, vers 3.0

39. Epidemics in dependency graphs
A.f0
A.f2
A.f3
B.f1
B.f2
B.f3
C.f1
C.f2
D.f1
D.f2
D.f3

40. Epidemics in dependency graphs
A.f0
A.f2
A.f3
B.f1
B.f2
B.f3
C.f1
C.f2
D.f1
D.f2
D.f3
A vulnerability aler
t
is issue
d
about Lib D, vers 3.0
,
function f3

41. Epidemics in dependency graphs
A.f0
A.f2
A.f3
B.f1
B.f2
B.f3
C.f1
C.f2
D.f1
D.f2
D.f3
A vulnerability aler
t
is issue
d
about Lib D, vers 3.0
,
function f3

42. Epidemics in dependency graphs
A.f0
A.f2
A.f3
B.f1
B.f2
B.f3
C.f1
C.f2
D.f1
D.f2
D.f3
A vulnerability aler
t
is issue
d
about Lib D, vers 3.0
,
function f3
Much more informative!

43. Examples

44. Examples
❖ Fully precise change impact analysis: “How many libraries
are affected if I remove/modify a certain method/interface?”

45. Examples
❖ Fully precise change impact analysis: “How many libraries
are affected if I remove/modify a certain method/interface?”
❖ Fully precise license compliance: “Is my library compliant
with the licenses of the libraries that I depend from (directly or
indirectly)? (e.g., am I linking any GPL code?)”

46. Examples
❖ Fully precise change impact analysis: “How many libraries
are affected if I remove/modify a certain method/interface?”
❖ Fully precise license compliance: “Is my library compliant
with the licenses of the libraries that I depend from (directly or
indirectly)? (e.g., am I linking any GPL code?)”
❖ Fully precise risk pro
fi
ling: “Does this vulnerability affect my
code?”

47. Examples
❖ Fully precise change impact analysis: “How many libraries
are affected if I remove/modify a certain method/interface?”
❖ Fully precise license compliance: “Is my library compliant
with the licenses of the libraries that I depend from (directly or
indirectly)? (e.g., am I linking any GPL code?)”
❖ Fully precise risk pro
fi
ling: “Does this vulnerability affect my
code?”
❖ Centrality analysis: “What methods/functions are more central
within a given ecosystem? are there bottlenecks? critical points?”

48. The FASTEN toolchain

49. The FASTEN toolchain
Project information
Securit
y
alerts
Repositories

50. The FASTEN toolchain
Project information
Securit
y
alerts
Repositories
publish
Data stream
publish
publish

51. The FASTEN toolchain
Project information
Securit
y
alerts
Repositories
publish
Data stream
FASTE
N
server
publish
publish

52. The FASTEN toolchain
Project information
Securit
y
alerts
Repositories
publish
Data stream
FASTE
N
server
Call-graph
construction
publish
publish

53. The FASTEN toolchain
Project information
Securit
y
alerts
Repositories
publish
Data stream
FASTE
N
server
Call-graph
construction
Storage
layer
publish
publish

54. The FASTEN toolchain
Project information
Securit
y
alerts
Repositories
publish
Data stream
FASTE
N
server
Call-graph
construction
Storage
layer
Analysis
layer
publish
publish

55. The FASTEN toolchain
Project information
Securit
y
alerts
Repositories
publish
Data stream
FASTE
N
server
Call-graph
construction
Storage
layer
Analysis
layer
RESTApi
publish
publish

56. The FASTEN toolchain
Project information
Securit
y
alerts
Repositories
publish
Data stream
FASTE
N
server
Call-graph
construction
Storage
layer
Analysis
layer
RESTApiWebUI
publish
publish

57. The FASTEN toolchain
Project information
Securit
y
alerts
Repositories
publish
Data stream
FASTE
N
server
Call-graph
construction
Storage
layer
Analysis
layer
RESTApiWebUI
publish
publish
Continuous
integration server

58. The FASTEN toolchain
Project information
Securit
y
alerts
Repositories
publish
Data stream
FASTE
N
server
Call-graph
construction
Storage
layer
Analysis
layer
RESTApiWebUI
publish
publish
Continuous
integration server

59. The FASTEN toolchain
Project information
Securit
y
alerts
Repositories
publish
Data stream
FASTE
N
server
Call-graph
construction
Storage
layer
Analysis
layer
RESTApiWebUI
publish
publish
Continuous
integration server
Developer

60. Preliminary results

61. Server-side highlights

62. Dataflow example: CG generation
Done

63. Universal function identifiers
How to uniquely reference a function in a global namespace?
fasten://
/mvn
/org.slf4j.slf4j-api
/1.2.3
/org.slf4j.helpers
/BasicMarkerFactory.getDetachedMarker
(%2Fjava.lang%2FString)
%2Forg.slf4j%2FMarker
scheme
forge
artifact
version
namespace
function
argument(s)
return type
Done

64. Universal function identifiers
How to uniquely reference a function in a global namespace?
fasten://
/mvn
/org.slf4j.slf4j-api
/1.2.3
/org.slf4j.helpers
/BasicMarkerFactory.getDetachedMarker
(%2Fjava.lang%2FString)
%2Forg.slf4j%2FMarker
scheme
forge
artifact
version
namespace
function
argument(s)
return type
Generic format +
Java
Python
C
Done

65. Call graph transport
{
"product": "foo",
"forge": "mvn",
"depset": [
[
{ "product": "a", "forge": "mvn", "constraints": ["[1.2..1.5]", "[2.3..]"] },
{ "product": "b", "forge": "mvn", "constraints": ["[2.0.1]"] }
]
],
"version": "3.10.0.7",
"cha": {
"/name.space/A": {
"methods": {
"0": "/name.space/A.A()%2Fjava.lang%2FVoidType",
"1": "/name.space/A.g(%2Fjava.lang%2FString)%2Fjava.lang%2FInteger"
},
"superInterfaces": [ "/java.lang/Serializable" ],
"sourceFile": "filename.java",
"superClasses": [ "/java.lang/Object" ]
}
},
"graph": {
"internalCalls": [
[ 0, 1 ]
],
"externalCalls": [
[ "1", "///their.package/TheirClass.method()Response", { "invokeinterface": "1" } ]
]
},
"timestamp": 123
}
Done

66. Call graph transport
{
"product": "foo",
"forge": "mvn",
"depset": [
[
{ "product": "a", "forge": "mvn", "constraints": ["[1.2..1.5]", "[2.3..]"] },
{ "product": "b", "forge": "mvn", "constraints": ["[2.0.1]"] }
]
],
"version": "3.10.0.7",
"cha": {
"/name.space/A": {
"methods": {
"0": "/name.space/A.A()%2Fjava.lang%2FVoidType",
"1": "/name.space/A.g(%2Fjava.lang%2FString)%2Fjava.lang%2FInteger"
},
"superInterfaces": [ "/java.lang/Serializable" ],
"sourceFile": "filename.java",
"superClasses": [ "/java.lang/Object" ]
}
},
"graph": {
"internalCalls": [
[ 0, 1 ]
],
"externalCalls": [
[ "1", "///their.package/TheirClass.method()Response", { "invokeinterface": "1" } ]
]
},
"timestamp": 123
}
Generic format +
Java
Python
C
Done

67. Language-dependent
call graph generation
Done

68. Language-dependent
call graph generation
❖ Java: Based on tools from the OPAL project (stg-tud/opal
)
❖ Python: New static analysis tool: PyCG (Submitted ICSE 2020)
❖ C: CScout for static call graphs; gprof, callgrind for dynamic calls
Done

69. Current CG results
Language /
Ecosystem
Total
Packages
Results
Packages Nodes Edges Success
Rate
C / Debian
Buster
7.380 (757
analyzed) *
531 491.721 579.253 70%
Java / Maven 2.7M artifacts 2.4M ~5B+ ~56B+ 89.13%
Python / PyPI ~740 K ~520K ~211M ~310M 70%
In
progress
* Technical issues prohibited us from downloading the rest of the packages.

70. Call graph stitching
How to scale call graph processing to 10^6 package versions?
In
progress

71. Call graph stitching
❖ Idea: Decouple package resolution from call graph
generation
How to scale call graph processing to 10^6 package versions?
In
progress

72. Call graph stitching
❖ Idea: Decouple package resolution from call graph
generation
❖ Build and store call graphs per package version, incl.:
How to scale call graph processing to 10^6 package versions?
In
progress

73. Call graph stitching
❖ Idea: Decouple package resolution from call graph
generation
❖ Build and store call graphs per package version, incl.:
❖ unresolved calls
How to scale call graph processing to 10^6 package versions?
In
progress

74. Call graph stitching
❖ Idea: Decouple package resolution from call graph
generation
❖ Build and store call graphs per package version, incl.:
❖ unresolved calls
❖ class hierarchies (Java, Python)
How to scale call graph processing to 10^6 package versions?
In
progress

75. Call graph stitching
❖ Idea: Decouple package resolution from call graph
generation
❖ Build and store call graphs per package version, incl.:
❖ unresolved calls
❖ class hierarchies (Java, Python)
❖ Call graph stitching: Resolve unresolved
calls given a dependency tree
How to scale call graph processing to 10^6 package versions?
In
progress

76. The database schema
Done

77. Examples of queries:
largest packages (# of functions)
select p.package_name, pv.version, count(*)
from package_versions pv
join packages p on pv.package_id = p.id
join modules m on m.package_version_id = pv.id
join callables c on c.module_id = m.id
group by p.package_name, pv.version
order by count(*) desc
limit 10;

78. Examples of queries:
Packages depending on vulnerable package
SELECT package_version_id, p.package_name, pv.version
FROM dependencies d
JOIN package_versions pv ON pv.id = d.package_version_id
JOIN packages p ON p.id = pv.package_id
WHERE d.dependency_id =
(SELECT id
FROM packages
WHERE package_name = 'com.google.guava:guava')
AND '20.0' = ANY(d.version_range);

79. Graph analytics
(results shown refer to Java CG’s)
In
progress

80. Graph analytics
(results shown refer to Java CG’s)
❖ Graph stored using WebGraph (UMIL
)
❖ For 1.1M graphs (2.3B nodes, 18B edges)
:
❖ 3.6 bits per edge, plus global ID storage for each node
(9.0 bits per edge overall
)
❖ DB size: 38GB → we can
fi
t the whole of Maven in
RAM
In
progress

81. Graph storage
In
progress

82. Client-side highlights

83. Vulnerability Plugin
❖ Injecting vulnerability information at
package and callable leve
l
❖ Introducing a normalizing
Vulnerability Object de
fi
nition
among the different sources of
informatio
n
❖ Continuously pulling updates for new
information and storing the results
In
progress

84. REST API
❖ Implementation of endpoints to expose canned queries
from the metadata databas
e
❖ In development
:
❖ Full DB entity suppor
t
❖ Custom extension points
In
progress

85. Analysis plug-ins
RAPID: Risk Analysis and Propagation Inspection for Security and
Maintainability risk
s
❖ Plugin for code maintainability analysis
V1 deployed, processed 126K Maven coordinates to dat
e
❖ Plugin for security vulnerability propagatio
n
❖ User application to model and present risk
s
❖ Vulnerability data integratio
n
❖ Clearly De
fi
ned, NVD,
…
❖ Association at the function and package level
In
progress

86. License and Compliance analysis
❖ QMSTR Plugin consists of 3 steps
:
1. Build graph generation, consisting of information
about all the generated artifacts that will be
distributed together with the source code
;
2. Execution of static analysis tools that augment the
build graph with license and compliance metadata
;
3. Generation of a report with package's relevant license
and authorship metadata that is
fi
nally distributed
In
progress

87. Use cases
❖ Endocod
e
❖ Integration of
fi
rst version of license and compliance analyse
r
❖ Achieved the validation of `SPDX validator` use cas
e
❖ SI
G
❖ Integration of code quality analyser into FASTEN Serve
r
❖ XWIK
I
❖ Risk validation in the dependencies at Maven build tim
e
❖ Risk validation in the installed extensions of an XWiki instanc
e
❖ Filter out available compatible extensions for an XWiki instanc
e
❖ Discoverability of XWiki components in available extensions
In
progress

88. Future timeline

89. The future
End 2020
Q1 2021
Q2 2021
Q3 2021
REST API,
fi
rst full version of knowledge base, CG enrichment,
build graph integration,
fi
rst public announcement
Impact analysis, integration with MVN / PyPI;
fi
rst external user
Q4 2021
Q1 2022 FASTEN 2?
Industrial use cases integrated;
fi
rst external adoption
Licensing and security fully integrated;
Data-driven API evolution
Project
fi
nished; external integrations

90. Network analysis will be the next
step for the future of
software development

91. Network analysis will be the next
step for the future of
software development

92. Questions?
Paolo Bold
i
Università degli Studi di
Milan
o
Ital
y
paolo.boldi@unimi.it