Design For Accessibility: Getting it right from the start
Select, manage, and backport the long term stable kernels
1. Select, Manage, and Backport the
LongTerm Stable Kernels
林上智 (SZ LIN) szlin@debian.org
蔡鎮宇 wens@csie.org
1
Date: 2021/7/31
2. 2
Disclaimer
• The information contained in the slide deck
represents the views and opinions of the
presenters and does not necessarily represent
the views or opinions of presenters’ past,
current, and future employers.
3. > 80 %
> 75 % 100 %
> 95 %
img src: https://kernel.org
src: https://www.linuxfoundation.org/about/
of the top one
million domains run
with Linux
of cloud-enabled
enterprises report
using Linux as their
primary cloud
platform
of new smartphones
sold run Android,
which is based on
the Linux kernel
of the top 500
supercomputers in
the world run on
Linux
6. Mainline Kernel [1]
Mainline tree is maintained by Linus Torvalds. It's the tree where all new features are introduced
and where all the exciting new development happens. New mainline kernels are released every
2-3 months.
Mainline Kernel [1]
Mainline tree is maintained by Linus Torvalds. It's the tree where all new features are introduced
and where all the exciting new development happens. New mainline kernels are released every
2-3 months.
7. Stable Kernel [1]
After each mainline kernel is released, it is considered "stable." Any bug fixes for a stable kernel
are backported from the mainline tree and applied by a designated stable kernel maintainer.
There are usually only a few bugfix kernel releases until next mainline kernel becomes available -
- unless it is designated a "longterm maintenance kernel." Stable kernel updates are released on
as-needed basis, usually once a week.
Stable Kernel [1]
After each mainline kernel is released, it is considered "stable." Any bug fixes for a stable kernel
are backported from the mainline tree and applied by a designated stable kernel maintainer.
There are usually only a few bugfix kernel releases until next mainline kernel becomes available -
- unless it is designated a "longterm maintenance kernel." Stable kernel updates are released on
as-needed basis, usually once a week.
8. Longterm Kernel [1]
There are usually several "longterm maintenance" kernel releases provided for the purposes of
backporting bugfixes for older kernel trees. Only important bugfixes are applied to such kernels
and they don't usually see very frequent releases, especially for older trees.
Longterm Kernel [1]
There are usually several "longterm maintenance" kernel releases provided for the purposes of
backporting bugfixes for older kernel trees. Only important bugfixes are applied to such kernels
and they don't usually see very frequent releases, especially for older trees.
9. End of LTS
9
Linux Kernel Releases
Mainline
Longterm
v4.4
Longterm
6+? years
v4.5 v4.19 v5.x
EOL
v4.4.x v4.4.y v4.4.z
v4.19.a v4.19.b
img src: https://en.wikipedia.org/wiki/Linux_kernel_version_history
End of LTS
6+? years
v4.4 v4.5 v4.19
Stable
10. 27.8
60-90 Day 66,492 3,386,347
21,074
Mainline Kernel
Release Cycle
Million Lines Files Lines of New Codes
in 2019
Different Authors
10
src: https://www.phoronix.com/scan.php?page=news_item&px=Linux-Git-Stats-EOY2019
img src: https://kernel.org
11. LTS: Long Term Stable Kernel [3]
Extend software uptime for stable kernel
• Only accept bug fixes and security fixes
img: https://www.kernel.org/category/releases.html
Retrieved 7th July
11
12. SoC Board Support Package Kernel
• Kernel version depends on SoC vendors
– Well made but not well maintained
• Contain lots of in-house patches
– Errata patches
– Specific feature patches
– …
• Different SoC might use different versions of kernel
• The lifetime is unsure
12
13. LTSI: Long Term Support Initiative [6]
• Linux Foundation collaborative project
– Based on LTS
– Add another chance to include further patches on top of LTS
– Auto Test framework
– Same lifetime with LTS (yearly release and 2 years life time)
13
14. CIP (Civil Infrastructure Platform) [7]
• Linux Foundation collaborative project
– Support kernel and core package
– Auto Test framework
– Maintenance period
• 10 years and more (10-20 years)
14
More info: CIP Kernel Team Activities to Accomplish Super Long Term Support
Embedded Linux Conference, 2020 [8]
15. Linux Kernel Source Comparison Table
Version
Maintenance
Period
(years)
Features
Latest
Version
Supported Real-
time kernel
Maintainer
SoC
BSP kernel
? Bug fixes ? N SoC vendor kernel team
LTS
kernel
2 ~ ?
• Bug fixes
• Security fixes
5.10 N Kernel.org
LTSI kernel 2 ~ ?
• Bug fixes
• Security fixes
• Specific features
• New features
4.14 N
LTSI
(Linux Foundation Projects)
CIP
kernel
10 +
• Bug fixes
• Security fixes
• Specific features
• New features
5.10 Y
CIP
(Linux Foundation Projects)
15
16. Use Release Version Instead of Rolling Version
v4.4.1
Security fixes
Security fixes
Bug fixes
Bug fixes
Upstream
rolling version
16
v4.4.2 v4.4.3
20. Supply Chain Risk Management Practices
for Federal Information Systems and
Organizations
Special Publication 800-161 [4]
SM-9: Security requirements for
externally provided components
ISA/ IEC 62443-4-1 [5] NERCCIP-010-2 [6]
Configuration Change Management and
Vulnerability Assessments
img src: https://pixabay.com/illustrations/policies-standards-compliance-4720824/
20
23. 23
Proactive Management
- Monitor the status of stable kernel by subscribing
the mailing list
• http://vger.kernel.org/vger-lists.html#stable
From Greg Kroah-Hartman <>
Subject [PATCH 5.10 000/243] 5.10.52-rc1 review
Date Mon, 19 Jul 2021 16:50:29 +0200
This is the start of the stable review cycle for the 5.10.52 release.
There are 243 patches in this series, all will be posted as a response
to this one. If anyone has any issues with these being applied, please
let me know.
Responses should be made by Wed, 21 Jul 2021 14:47:42 +0000.
Anything received after that time might be too late.
The whole patch series can be found in one patch at:
https://www.kernel.org/pub/linux/kernel/v5.x/stable-review/patch-5.10.52-rc1.gz
or in the git tree and branch at:
git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-5.10.y
and the diffstat can be found below.
thanks,
greg k-h
-------------
Pseudo-Shortlog of commits:
Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Linux 5.10.52-rc1
Dan Carpenter <dan.carpenter@oracle.com>
scsi: scsi_dh_alua: Fix signedness bug in alua_rtpg()
…
From Greg Kroah-Hartman <>
Subject [PATCH 5.10 000/243] 5.10.52-rc1 review
Date Mon, 19 Jul 2021 16:50:29 +0200
This is the start of the stable review cycle for the 5.10.52 release.
There are 243 patches in this series, all will be posted as a response
to this one. If anyone has any issues with these being applied, please
let me know.
Responses should be made by Wed, 21 Jul 2021 14:47:42 +0000.
Anything received after that time might be too late.
The whole patch series can be found in one patch at:
https://www.kernel.org/pub/linux/kernel/v5.x/stable-review/patch-5.10.52-rc1.gz
or in the git tree and branch at:
git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-5.10.y
and the diffstat can be found below.
thanks,
greg k-h
-------------
Pseudo-Shortlog of commits:
Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Linux 5.10.52-rc1
Dan Carpenter <dan.carpenter@oracle.com>
scsi: scsi_dh_alua: Fix signedness bug in alua_rtpg()
…
34. • The project shares its results with the upstream
• The project fulfills longer time maintenance and
security fixes
• The project develops their code very quickly
• The project faces difficulties to backport upstream
patches due to conflicts as time goes by
34
36. 36
Suitable changes in LongTerm Stable Kernels [10]
1. It must be obviously correct and tested.
2. It cannot be bigger than 100 lines, with context.
3. It must fix only one thing.
4. It must fix a real bug that bothers people (not a, “This could be a problem…” type thing).
5. It must fix a problem that causes a build error (but not for things marked CONFIG_BROKEN), an oops, a hang,
data corruption, a real security issue, or some “oh, that's not good” issue. In short, something critical.
6. Serious issues as reported by a user of a distribution kernel may also be considered if they fix a notable
performance or interactivity issue. As these fixes are not as obvious and have a higher risk of a subtle
regression they should only be submitted by a distribution kernel maintainer and include an addendum linking
to a bugzilla entry if it exists and additional information on the user-visible impact.
7. New device IDs and quirks are also accepted.
8. No “theoretical race condition” issues, unless an explanation of how the race can be exploited is also provided.
9. It cannot contain any “trivial” fixes in it (spelling changes, whitespace cleanups, etc).
10. It must follow the Documentation/Submitting Patches rules.
11. It or an equivalent fix must already exist in Linus' tree (upstream).
In practice, exceptions are sometimes made to rules 2, 3, 8 and 9, and we will likewise treat those as should rather than must rules.
37. 37
Option 1. To have the patch automatically included in
the stable tree, add the tag
Subject: USB: serial: digi_acceleport: fix write-wakeup deadlocks
The driver must not call tty_wakeup() while holding its private lock as
line disciplines are allowed to call back into write() from
write_wakeup(), leading to a deadlock.
Also remove the unneeded work struct that was used to defer wakeup in
order to work around a possible race in ancient times (see comment about
n_tty write_chan() in commit 14b54e3 ("USB: serial: remove
changelogs and old todo entries")).
Fixes: 1da177e ("Linux-2.6.12-rc2")
Cc: stable@vger.kernel.org
Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Johan Hovold <johan@kernel.org>
Subject: USB: serial: digi_acceleport: fix write-wakeup deadlocks
The driver must not call tty_wakeup() while holding its private lock as
line disciplines are allowed to call back into write() from
write_wakeup(), leading to a deadlock.
Also remove the unneeded work struct that was used to defer wakeup in
order to work around a possible race in ancient times (see comment about
n_tty write_chan() in commit 14b54e3 ("USB: serial: remove
changelogs and old todo entries")).
Fixes: 1da177e ("Linux-2.6.12-rc2")
Cc: stable@vger.kernel.org
Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Johan Hovold <johan@kernel.org>
src: https://github.com/torvalds/linux/commit/5098e77962e7c8947f87bd8c5869c83e000a522a
Procedure for submitting patches to the -stable tree [9]
38. 38
Option 2. After the patch has been merged to Linus’
tree, send an email
to stable@vger.kernel.org containing the subject of
the patch, the commit ID, why you think it should be
applied, and what kernel version you wish it to be
applied to.
Subject: request for 4.4-stable: 08474cc1e6ea7 ("bridge: Propagate vlan
add failure to user")
From: SZ Lin (林上智) <sz.lin@xxxxxxxx>
Date: Wed, 22 Aug 2018 20:07:14 +0800
Hi Greg,
This patch is not marked for 4.4-stable, but it's already in 4.9 and 4.14
stable.
Please apply to 4.4-stable.
This patch fixed the kernel oops issue when executing bridge tool, please
see reproducible steps with kernel 4.4 in below.
# brctl addbr br0
# brctl addif br0 eth0
[ 1073.390028] device eth0 entered promiscuous mode
[ 1073.395022] cpsw 4a100000.ethernet eth0: failed to initialize vlan
filtering on this port
# brctl addif br0 eth1
[ 1092.205685] net eth1: promiscuity not disabled as the other interface is
still in promiscuity mode
[ 1092.217541] device eth1 entered promiscuous mode
[ 1092.222460] cpsw 4a100000.ethernet eth1: failed to initialize vlan
filtering on this port
# ifconfig br0 192.168.127.253 netmask 255.255.254.0
[ 1112.412740] br0: port 1(eth0) entered forwarding state
[ 1112.417975] br0: port 1(eth0) entered forwarding state
...
Subject: request for 4.4-stable: 08474cc1e6ea7 ("bridge: Propagate vlan
add failure to user")
From: SZ Lin (林上智) <sz.lin@xxxxxxxx>
Date: Wed, 22 Aug 2018 20:07:14 +0800
Hi Greg,
This patch is not marked for 4.4-stable, but it's already in 4.9 and 4.14
stable.
Please apply to 4.4-stable.
This patch fixed the kernel oops issue when executing bridge tool, please
see reproducible steps with kernel 4.4 in below.
# brctl addbr br0
# brctl addif br0 eth0
[ 1073.390028] device eth0 entered promiscuous mode
[ 1073.395022] cpsw 4a100000.ethernet eth0: failed to initialize vlan
filtering on this port
# brctl addif br0 eth1
[ 1092.205685] net eth1: promiscuity not disabled as the other interface is
still in promiscuity mode
[ 1092.217541] device eth1 entered promiscuous mode
[ 1092.222460] cpsw 4a100000.ethernet eth1: failed to initialize vlan
filtering on this port
# ifconfig br0 192.168.127.253 netmask 255.255.254.0
[ 1112.412740] br0: port 1(eth0) entered forwarding state
[ 1112.417975] br0: port 1(eth0) entered forwarding state
...
39. Subject: [PATCH 4.4 1/3] ovl: rename is_merge to is_lowest
From: SZ Lin (林上智) <sz.lin@xxxxxxxx>
Date: Wed, 22 Aug 2018 20:07:14 +0800
From: Miklos Szeredi
commit 56656e960b555cb98bc414382566dcb59aae99a2 upstream
The 'is_merge' is an historical naming from when only a single lower layer
could exist. With the introduction of multiple lower layers the meaning of
this flag was changed to mean only the "lowest layer" (while all lower
layers were being merged).
So now 'is_merge' is inaccurate and hence renaming to 'is_lowest'
Signed-off-by: Miklos Szeredi
Signed-off-by: SZ Lin (林上智)
...
Subject: [PATCH 4.4 1/3] ovl: rename is_merge to is_lowest
From: SZ Lin (林上智) <sz.lin@xxxxxxxx>
Date: Wed, 22 Aug 2018 20:07:14 +0800
From: Miklos Szeredi
commit 56656e960b555cb98bc414382566dcb59aae99a2 upstream
The 'is_merge' is an historical naming from when only a single lower layer
could exist. With the introduction of multiple lower layers the meaning of
this flag was changed to mean only the "lowest layer" (while all lower
layers were being merged).
So now 'is_merge' is inaccurate and hence renaming to 'is_lowest'
Signed-off-by: Miklos Szeredi
Signed-off-by: SZ Lin (林上智)
...
39
Option 3. Send the patch, after verifying that it follows
the above rules, to stable@vger.kernel.org. You must
note the upstream commit ID in the changelog of your
submission, as well as the kernel version you wish it to
be applied to.
40. 40
Option 1 is strongly preferred, is the easiest and most common. Option 2 and Option 3 are
more useful if the patch isn’t deemed worthy at the time it is applied to a public git tree
(for instance, because it deserves more regression testing first). Option 3 is especially
useful if the patch needs some special handling to apply to an older kernel (e.g., if API’s
have changed in the meantime).
Note that for Option 3, if the patch deviates from the original upstream patch (for example
because it had to be backported) this must be very clearly documented and justified in the
patch description. [9]
note: another entry point of patch is AUTOSEL bot, which is a neural network for finding fixes [11].
41. 41
commit d733f7542ad47cf73e033c90cf55158587e1d060 upstream.
If an emac node has a phy-handle property that points to something
which is not a phy, then a segmentation fault will occur when the
interface is brought up. This is because while phy_connect() will
return ERR_PTR() on failure, of_phy_connect() will return NULL.
The common error check uses IS_ERR(), and so missed when
of_phy_connect() fails. The NULL pointer is then dereferenced.
Also, the common error message referenced slave->data->phy_id,
which would be empty in the case of phy-handle. Instead, use the
name of the device_node as a useful identifier. And in the phy_id
case add the error code for completeness.
Fixes: 9e42f715264f ("drivers: net: cpsw: add phy-handle parsing")
Signed-off-by: David Rivshin
Signed-off-by: David S. Miller
[SZ Lin (林上智): Tweak the patch to use original print function of dev_info()]
Signed-off-by: SZ Lin (林上智)
Signed-off-by: Greg Kroah-Hartman
commit d733f7542ad47cf73e033c90cf55158587e1d060 upstream.
If an emac node has a phy-handle property that points to something
which is not a phy, then a segmentation fault will occur when the
interface is brought up. This is because while phy_connect() will
return ERR_PTR() on failure, of_phy_connect() will return NULL.
The common error check uses IS_ERR(), and so missed when
of_phy_connect() fails. The NULL pointer is then dereferenced.
Also, the common error message referenced slave->data->phy_id,
which would be empty in the case of phy-handle. Instead, use the
name of the device_node as a useful identifier. And in the phy_id
case add the error code for completeness.
Fixes: 9e42f715264f ("drivers: net: cpsw: add phy-handle parsing")
Signed-off-by: David Rivshin
Signed-off-by: David S. Miller
[SZ Lin (林上智): Tweak the patch to use original print function of dev_info()]
Signed-off-by: SZ Lin (林上智)
Signed-off-by: Greg Kroah-Hartman
Elaborate the difference from
the original source
42. 42
Review cycle [9]
• When the -stable maintainers decide for a review cycle, the patches will be sent to
the review committee, and the maintainer of the affected area of the patch
(unless the submitter is the maintainer of the area) and CC: to the linux-kernel
mailing list.
• The review committee has 48 hours in which to ACK or NAK the patch.
• If the patch is rejected by a member of the committee, or linux-kernel members
object to the patch, bringing up issues that the maintainers and members did not
realize, the patch will be dropped from the queue.
• At the end of the review cycle, the ACKed patches will be added to the latest -
stable release, and a new -stable release will happen.
• Security patches will be accepted into the -stable tree directly from the security
kernel team, and not go through the normal review cycle. Contact the kernel
security team for more details on this procedure.