Presentation at CHAOSSCon Europe 2020 about the generic technical lag software measurement framework. Technical lag measures the increasing difference between deployed software components and the ideal upstream software components.
For more information, see https://doi.org/10.1002/smr.2157
5. chaoss.community
Online survey
What would be the most appropriate (i.e., ideal)
version of a software library to depend on?
• 17 respondents
Highly educated with an average of 3 years of
development experience
• Responses:
5
★ Most stable (14)
★ Latest available (9)
★ Most documented (7)
★ Most secure (5)
6. chaoss.community
Idea: Technical Lag
“The increasing difference between deployed software
packages and the ideal available upstream packages.”
6
Ideal
• stability, security, functionality, recency, etc.
Difference
• time, version updates, bugs, vulnerabilities, features, …
J. Gonzalez-Barahona, P. Sherwood, G. Robles, D. Izquierdo (2017)
"Technical lag in software compilations: Measuring how outdated a software
deployment is.” IFIP International Conference on Open Source Systems. Springer
7. chaoss.community
Importance of Technical Lag
Semi-structured interviews:
2019
5 highly educated software practitioners with
an average of 10 years of experience
7
Technical Lag is important, especially if we mix
between the benefits of updating and the effort
required to do that.
9. chaoss.community
Measuring Technical Lag
A technical lag framework F is a tuple (C, L, ideal, delta, agg) with
• C a set of component releases
• L a set of possible lag values
• ideal: C → C computes the “ideal” (upstream) component release
for a given (deployed) release
• delta: C x C → L computes the difference between two
component releases
• agg: 2L → L aggregates the results of a set of lags
A formal framework for measuring technical lag in component repositories – and
its application to npm. A. Zerouali, T. Mens, J. Gonzalez-Barahona, A. Decan, E.
Constantinou, G. Robles. Wiley Journal on Software Evolution and Process, 2019
10. chaoss.community
Measuring Technical Lag
Given a technical lag framework F, we define
techlagF(c) = delta(c,ideal(c))
for any deployed component c
aggLagF(D) = agg( {techlagF(c) | c in D} )
for any set of deployed components D
A formal framework for measuring technical lag in component repositories – and
its application to npm. A. Zerouali, T. Mens, J. Gonzalez-Barahona, A. Decan, E.
Constantinou, G. Robles. Wiley Journal on Software Evolution and Process, 2019
11. chaoss.community
Technical Lag - Example
Time-based measurement of technical lag
(ideal = most recent release; delta = time difference)
1.0.1 1.1.0 2.0.01.2.0 2.0.1
deployed
package
upstream
package
Time lag
date(2.0.1) - date(1.1.0)
12. chaoss.community
Technical Lag - Example
Version-based measurement of technical lag
(ideal = highest release; delta = version difference)
1.0.1 1.1.0 2.0.12.0.0 1.2.0
deployed
package
1 major
upstream
package
1 patch
Version lag
1 major + 1 patch
13. chaoss.community
Technical Lag - Example
Vulnerability-based measurement of technical lag
(ideal = least vulnerable release; delta = #vulnerabilities)
1.0.1 1.1.0 2.0.01.2.0 2.0.1
deployed
package
upstream
package
Security lag
1 vulnerability fix behind
14. chaoss.community
Technical Lag - Example
Bug-based measurement of technical lag
(ideal = least known bugs; delta = #known bugs)
1.0.1 1.1.0 2.0.0
deployed
package
upstream
package
1.2.0 2.0.1
Dependency needs to be downgraded to
be able to use most stable version…
Bug lag
1 more bug than
most stable version
15. chaoss.community
Case study 1: Technical lag in npm
distribution of JavaScript packages
Credits: https://exploring-data.com/vis/npm-packages-dependencies/
A. Decan, T. Mens, E. Constantinou (2018)
On the evolution of technical lag in the npm
package dependency network. IEEE Int’l
Conf. Software Maintenance and Evolution
+20M
dependencies
18. chaoss.community
Technical Lag – Example
18
Time-based technical lag for deployed release debug 2.6.9
ideal (debug 2.6.9) = debug 3.1.0
timeLag(debug 2.6.9) = 26-09-2017 - 22-09-2017 = 4 days
versionLag(debug 2.6.9) = 1 major + 1 minor + 1 patch
19. chaoss.community
Technical Lag – Example
19
Time-based technical lag for deployed release ms 2.0.0
ideal (ms 2.0.0) = ms 2.1.1
timeLag(ms 2.0.0) = 30-11-2017 - 16-05-2017 = 198 days
versionLag(ms 2.0.0) = 1 minor + 1 patch
20. chaoss.community
Technical Lag – Example
20
Aggregated transitive time lag for deployed release youtube-player 5.5.0
agglag({debug 2.6.9, ms 2.0.0}) = max(4 days, 198 days) = 198 days
22. chaoss.community
Case study 2: Technical lag in
Debian-based Docker containers
A. Zerouali, T. Mens, G. Robles, J. Gonzalez-Barahona (2019). On the relation between
outdated Docker containers, security vulnerabilities, and bugs. IEEE In’tl Conf. SANER
23. chaoss.community
Case study 2: Technical lag in
Debian-based Docker containers
Important issues faced when deploying Docker containers:
• Security vulnerabilities
• Dependence on external software packages
• Presence of bugs in third-party software
• Outdated third-party software
24. chaoss.community
1.0.1 1.1.0 2.0.01.2.1 2.1.0
Docker
container C
Technical lag
∆ versions (freshness)
∆ vulnerabilities (security)
∆ bugs (stability)
« ideal »
release
deployed
container
included
Debian
package
release
upstream
releases of the
Debian package
Technical Lag in
Debian-based Docker containers
26
26. chaoss.community
Summary
Technical Lag is a very useful generic measure for assessing to which extent deployed
software is outdated w.r.t. upstream releases.
• Different ways to measure (time, version, bugs, vulnerabilities, …)
and aggregate (max, sum, …) technical lag
• It can be operationalized in different contexts (package dependency management,
container deployment, …)
Suggestion:
• Include this measure as part of the CHAOSS Metrics and Tooling
Open Challenges:
• How to measure effort required to update?
• How to combine multiple dimensions of technical lag?
• How to assess whether updates do not cause breaking changes?
30
27. chaoss.community
New proposed CHAOSS project metrics
• Dependencies
• Number of / List of; Direct or transitive
• Dependency depth
• Outdated dependencies
• List of / Number of / Ratio of
• Vulnerable dependencies
• List of / Number of / Ratio of
• Dependents (i.e. reverse dependencies)
• Number of / List of; Direct or transitive
• Dependency lag
• aggregated dependency-based technical lag of a project
• Deployment lag
• Aggregated lag of set of deployed components w.r.t. upstream
28. chaoss.community
SoHeal, May 2020 http://soheal.github.io
3rd Int’l ICSE Workshop on Software Health
What?
• Focus on the health of software projects, communities and ecosystems
• Discuss about technical, social, legal and business aspects related to
project effectiveness, success, longevity, growth, resilience, survival,
diversity, sustainability, popularity, inclusiveness, ...
Who?
• Open Source Community Members, Industry and Academia
Why?
• Raise awareness on software health
• Present tools, methods, practical experiences
• Advance body of knowledge on software health
Seoul, South Korea – May 2020
@iw_soheal
29. chaoss.community
SoHeal 2020 http://soheal.github.io/cft.html
Extended call for submissions
Are you a involved in software projects or ecosystems,
and have something to say about software health?
Submit a short paper or talk proposal on
Open source and industrial experiences from individual, team or community level
Relation between software health and social, technical, legal, process
and business aspects
Tools, dashboards and models to enable, assess, predict and
recommend software health
Guidelines and lessons learned
Submission deadline: Friday, February 7, 2019
Hinweis der Redaktion
So the goal and the focus of this thesis was to answer the following question:
[read slide]
we performed semi structured interviews with 5 software practitioners during the event of FOSDEM in Brussels,
In these interviews, we explained the technical lag and asked the interviewees about their opinion regarding the technical lag.
We found that throughout the interviews, the importance of technical lag was stressed. However,
participants acknowledged that a mix between what is missing (i.e., benefits) and the
effort (i.e., cost) needed to update would be even better than only knowing what is missing.
The benefits here can refer to many things, like a new feauture, a bug fix, or a vulnerability fix, etc. So the notion of ideal is related to what we want to have as benefits and since there are many benefits, there should be many ideals also
So, since the ideal of a version is not a specific version with some specific characteristic and
since the difference between the ideal and the deployed software isnt also a specific metric, we needed to create an extensible framework that supports all these variants.
Moreover, we needed a framework that should not only support one deployed software component, but it should support a set of deployed components. Since in most of the cases, a deployed software will come with its dependencies.
so indeed, we presented a technical lag framework as a tuple of :
[ Read slide]
so indeed, we presented a technical lag framework as a tuple of :
[ Read slide]
In other words, to compute the technical lag or the time lag for the time-based instatiation, we use the difference in days between the releases dates of the deployed version and the ideal version.
and then to compute the technical lag in terms of missing updates, we compute how many versions are between the deployed and the ideal versions while checking their version type.
For example here, from here to here the minor version number changes,
and then to compute the technical lag in terms of missing updates, we compute how many versions are between the deployed and the ideal versions while checking their version type.
For example here, from here to here the minor version number changes,
and then to compute the technical lag in terms of missing updates, we compute how many versions are between the deployed and the ideal versions while checking their version type.
For example here, from here to here the minor version number changes,
For example, in the case of npm packages, to specify a dependency, developers need to put a dependency constraint that refers to the range of allowed versions to be installed.
Let’s take a concrete example of the youtube-player npm package,
In the figure, we can see that this package makes use of three dependencies.
However, to specify which versions of these packages to use, the package youtube-player make use of dependency constraints.
So In many cases, the use of these constraints can lead to the use of outdated or not ideal package version.
Concrete example:
here. the version 5.5.0 of the youtube player package makes use of three direct dependencies (...).
Using different dependency constraints. While the debug package version makes use of the ms package using the constraint 2.0.0
this dependency here is considered as a transitive dependency for the youtube player package version.
Because of the use of these constraints, the debug and ms dependencies could not be installed with the latest available version at the release date of the youtube player package and thus they were inducing a technical lag
So while this situation is not necessarily a problem, it can sometimes come with issues. For example, if the outdated package version here was suffering from a bug or vulnerability, then this bug will propagate and debug will suffer from it and also youtube player will suffer from it.
Concrete example:
here. the version 5.5.0 of the youtube player package makes use of three direct dependencies (...).
Using different dependency constraints. While the debug package version makes use of the ms package using the constraint 2.0.0
this dependency here is considered as a transitive dependency for the youtube player package version.
Because of the use of these constraints, the debug and ms dependencies could not be installed with the latest available version at the release date of the youtube player package and thus they were inducing a technical lag
So while this situation is not necessarily a problem, it can sometimes come with issues. For example, if the outdated package version here was suffering from a bug or vulnerability, then this bug will propagate and debug will suffer from it and also youtube player will suffer from it.
Concrete example:
here. the version 5.5.0 of the youtube player package makes use of three direct dependencies (...).
Using different dependency constraints. While the debug package version makes use of the ms package using the constraint 2.0.0
this dependency here is considered as a transitive dependency for the youtube player package version.
Because of the use of these constraints, the debug and ms dependencies could not be installed with the latest available version at the release date of the youtube player package and thus they were inducing a technical lag
So while this situation is not necessarily a problem, it can sometimes come with issues. For example, if the outdated package version here was suffering from a bug or vulnerability, then this bug will propagate and debug will suffer from it and also youtube player will suffer from it.
Concrete example:
here. the version 5.5.0 of the youtube player package makes use of three direct dependencies (...).
Using different dependency constraints. While the debug package version makes use of the ms package using the constraint 2.0.0
this dependency here is considered as a transitive dependency for the youtube player package version.
Because of the use of these constraints, the debug and ms dependencies could not be installed with the latest available version at the release date of the youtube player package and thus they were inducing a technical lag
So while this situation is not necessarily a problem, it can sometimes come with issues. For example, if the outdated package version here was suffering from a bug or vulnerability, then this bug will propagate and debug will suffer from it and also youtube player will suffer from it.
so, as a conclusion statement :
The technical lag framework, the actionable results and the tools developed in this thesis can be used to help open source software developers and deployers to keep their software in a healthy shape.
So, starting with the first case, we focused on images that are based on Debian.
and to compute the technical lag, we relied on the same standards, which is the ideal version and then the difference between the used and ideal version.
In this case analysis, we considered three different ideal versions. For developers interested in freshness we considered the latest available version as ideal version.
For developers interested in security, we considered, the most secure software version as the idea version,
and for For developers interested in stability, we considered, the most stable software version as the idea version,
and so to compute the technical lag, we relied respectively on three measurement units
So, starting with the first case, we focused on images that are based on Debian.
and to compute the technical lag, we relied on the same standards, which is the ideal version and then the difference between the used and ideal version.
In this case analysis, we considered three different ideal versions. For developers interested in freshness we considered the latest available version as ideal version.
For developers interested in security, we considered, the most secure software version as the idea version,
and for For developers interested in stability, we considered, the most stable software version as the idea version,
and so to compute the technical lag, we relied respectively on three measurement units
So, starting with the first case, we focused on images that are based on Debian.
and to compute the technical lag, we relied on the same standards, which is the ideal version and then the difference between the used and ideal version.
In this case analysis, we considered three different ideal versions. For developers interested in freshness we considered the latest available version as ideal version.
For developers interested in security, we considered, the most secure software version as the idea version,
and for For developers interested in stability, we considered, the most stable software version as the idea version,
and so to compute the technical lag, we relied respectively on three measurement units
So, starting with the first case, we focused on images that are based on Debian.
and to compute the technical lag, we relied on the same standards, which is the ideal version and then the difference between the used and ideal version.
In this case analysis, we considered three different ideal versions. For developers interested in freshness we considered the latest available version as ideal version.
For developers interested in security, we considered, the most secure software version as the idea version,
and for For developers interested in stability, we considered, the most stable software version as the idea version,
and so to compute the technical lag, we relied respectively on three measurement units
So, starting with the first case, we focused on images that are based on Debian.
and to compute the technical lag, we relied on the same standards, which is the ideal version and then the difference between the used and ideal version.
In this case analysis, we considered three different ideal versions. For developers interested in freshness we considered the latest available version as ideal version.
For developers interested in security, we considered, the most secure software version as the idea version,
and for For developers interested in stability, we considered, the most stable software version as the idea version,
and so to compute the technical lag, we relied respectively on three measurement units
so, as a conclusion statement :
The technical lag framework, the actionable results and the tools developed in this thesis can be used to help open source software developers and deployers to keep their software in a healthy shape.