Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Linux Kernel Live Patching
Ruslan Bilovol
Agenda
2
• Why?
• History
• From Kprobes to LivePatch
• LivePatch
• Limitations
• Current status in mainline Linux kernel
...
Why?
3
• High cost of downtime caused by reboot
• Apply hotfixes without reboot
• Maximize uptime
• Scheduled downtime (de...
Linux Kernel Live Patching: History
• 2008: ksplice
– university research, acquired by Oracle, source is closed
• 2014: kP...
Commercial use
5
• Oracle Ksplice is available for Oracle Linux and RHEL servers with an
Oracle Linux Premier Support lice...
From Kprobes to LivePatch
Kprobes
7
• Enables you to dynamically break into any kernel routine
• Can trap at almost any kernel code address
– you ca...
Kprobes
8
Ftrace
• "Kernel Function Tracer"
• Enables the kernel to trace every kernel function
• Uses 'gcc -pg' option to insert “m...
Dynamic Ftrace
• "Dynamic Kernel Function Tracer"
• mcount points to a simple return
– Starting with gcc verson 4.6, the -...
Dynamic Ftrace
• Then NOP sequence can be dynamically patched into a tracer call
• Modules are processed as they are loade...
LivePatch?
12
• So we can replace functions in running kernel?
• ...
• Live Kernel Patching!
LivePatch
13
• Need to be more clever
– SMP synchronization
– task may sleep in a to-be-patched function
– function semant...
The LivePatch
LivePatch
15
• Initially implemented for x86-64 (kernel v4.0)
– extended to PPC64 and s390
• Is a .ko module
– just like a...
LivePatch
16
Hybrid Consistency Model
17
• Avoid calling a new function from old and vice versa
• A thread calls either all old functio...
Hybrid Consistency Model
18
ORC unwinder
19
• replaces GCC frame pointers approach
– much simpler and faster
– performance improvement (5-10% for some...
Shadow variables
20
• associate additional "shadow" data with existing data structures
– allocated separately
• based on g...
Modules live patching
21
• patch can be inserted before module loading
• LivePatch core gets notification on module load/u...
Patch/unpatch callbacks
22
• execute callback functions when a kernel object is (un)patched
• considered a "power feature"...
LivePatch example
23
static int livepatch_cmdline_proc_show(struct seq_file *m, void *v) { seq_printf(m, "%sn", "this has ...
LivePatch Limitations
24
• useful for small and critical fixes
– e.g. can’t do kernel upgrade to a new version
• Kprobes i...
How it looks like?
25
Current status in mainline Linux kernel
Initial LivePatch support (x86-64)
26
v4.0
s390 architecture support v4.1
PPC64 ar...
kpatch Set of Tools
Why need automatic patches generation?
28
• Human can produce errors
• Inlining
– inlined buggy function to other function...
Kpatch
29
• Red Hat project of live kernel patching
– includes few userspace tools
• Used in production (RH relies on it)
...
kpatch-build
30
• build original kernel
– with -ffunction-sections -fdata-sections flags
– use custom script as cross-comp...
kpatch limitations
31
• can’t do .data section changes (don’t remove static variables)
• no cross-compilation support (pat...
LivePatch writer recommendations
32
• make changes as small as possible
– e.g. this is “hot fix”, not a kernel upgrade to ...
Demo
Questions?
Sources
35
kernel.org
https://assets.ubuntu.com/v1/ac3aa269-DS_Canonical_Livepatch_Service_sc
reen-AW_08.17.pdf
https://ww...
Linux Kernel Live Patching
Nächste SlideShare
Wird geladen in …5
×

Linux Kernel Live Patching

1.738 Aufrufe

Veröffentlicht am

This presentation is about a methodology which allows patching of a running Linux kernel, its technical details, limitations as well as kpatch tools.

The talk was delivered by Ruslan Bilovol (Associate Manager, Consultant, GlobalLogic) at GlobalLogic Embedded Career Day #2 on February 10, 2018.

More about GlobalLogic Embedded Career Day #2: https://www.globallogic.com/ua/events/globallogic-kyiv-embedded-career-day-2-materials

Veröffentlicht in: Ingenieurwesen
  • Als Erste(r) kommentieren

Linux Kernel Live Patching

  1. 1. Linux Kernel Live Patching Ruslan Bilovol
  2. 2. Agenda 2 • Why? • History • From Kprobes to LivePatch • LivePatch • Limitations • Current status in mainline Linux kernel • Kpatch • LivePatch writer recommendations • Demo • Questions
  3. 3. Why? 3 • High cost of downtime caused by reboot • Apply hotfixes without reboot • Maximize uptime • Scheduled downtime (delay reboot)
  4. 4. Linux Kernel Live Patching: History • 2008: ksplice – university research, acquired by Oracle, source is closed • 2014: kPatch (Red Hat) – stopping the kernel for live patching – manual/automatic patches creation • 2014: kGraft (SUSE) – lazy kernel live patching – only manual patches creation • 2015: Livepatch (Linux kernel 4.0+) – combined consistency model of kGraft + kPatch 4
  5. 5. Commercial use 5 • Oracle Ksplice is available for Oracle Linux and RHEL servers with an Oracle Linux Premier Support license – $2299/node/year • RHEL Livepatching requires a Red Hat Enterprise Linux Server Premium Subscription – $1299/node/year • SUSE Livepatching is available as an add-on to SUSE Linux Enterprise Server 12 Priority Support subscription – $1,499/node/year • Canonical Livepatching service
  6. 6. From Kprobes to LivePatch
  7. 7. Kprobes 7 • Enables you to dynamically break into any kernel routine • Can trap at almost any kernel code address – you can probe even interrupt handlers • Probed instruction replaced with a breakpoint instruction • On trap executes "pre_handler", replaced instruction and "post_handler"
  8. 8. Kprobes 8
  9. 9. Ftrace • "Kernel Function Tracer" • Enables the kernel to trace every kernel function • Uses 'gcc -pg' option to insert “mcount” function call placed at the start of every kernel function • Big impact on performance (13%) 9
  10. 10. Dynamic Ftrace • "Dynamic Kernel Function Tracer" • mcount points to a simple return – Starting with gcc verson 4.6, the -mfentry has been added for x86, which calls " fentry " instead of "mcount" • Build-time tool parses ELF to find these location • On boot dynamic ftrace updates all the locations into NOPs 10
  11. 11. Dynamic Ftrace • Then NOP sequence can be dynamically patched into a tracer call • Modules are processed as they are loaded and before they are executed • No measurable performance degradation 11
  12. 12. LivePatch? 12 • So we can replace functions in running kernel? • ... • Live Kernel Patching!
  13. 13. LivePatch 13 • Need to be more clever – SMP synchronization – task may sleep in a to-be-patched function – function semantic changes – data structure changes – what if patch contains multiple functions that depend on each other
  14. 14. The LivePatch
  15. 15. LivePatch 15 • Initially implemented for x86-64 (kernel v4.0) – extended to PPC64 and s390 • Is a .ko module – just like any other kernel module • Uses dynamic FTRACE to replace original function • Creates /sys/kernel/livepatch dir – to monitor live patches status
  16. 16. LivePatch 16
  17. 17. Hybrid Consistency Model 17 • Avoid calling a new function from old and vice versa • A thread calls either all old functions or all new ones • Per-thread migration – stack checking of sleeping tasks – new thread inherits its parent patched state – interrupt handler inherits interrupted thread state • Per-patched-set migration • Stack checking – need reliable stack information – frame pointers are not fast enough • Interrupt handler inherits the patched state of interrupted task
  18. 18. Hybrid Consistency Model 18
  19. 19. ORC unwinder 19 • replaces GCC frame pointers approach – much simpler and faster – performance improvement (5-10% for some workloads) – 3.2% kernel .text savings • objtool for stack’s metadata validity on .o files and ORC data generation • disadvantage: needs more memory to store the ORC unwind tables • enabled by default since v4.14
  20. 20. Shadow variables 20 • associate additional "shadow" data with existing data structures – allocated separately • based on global in-kernel hashtable • associates pointers to parent objects and a numeric identifier of the shadow data – there may be multiple shadow variables attached to parent object
  21. 21. Modules live patching 21 • patch can be inserted before module loading • LivePatch core gets notification on module load/unload – patches module before it get executing – unpatches module on module unload • module can be inserted-removed multiple times
  22. 22. Patch/unpatch callbacks 22 • execute callback functions when a kernel object is (un)patched • considered a "power feature" – safe updates to global data – "patches" to init and probe functions – patching otherwise unpatchable code (i.e. assembly) – ability refuse to load livepatch or a module • originally livepatch didn’t have such ability
  23. 23. LivePatch example 23 static int livepatch_cmdline_proc_show(struct seq_file *m, void *v) { seq_printf(m, "%sn", "this has been live patched"); return 0; } static struct klp_func funcs[] = { { .old_name = "cmdline_proc_show", .new_func = livepatch_cmdline_proc_show, }, { } }; static struct klp_object objs[] = { { /* name being NULL means vmlinux */ .funcs = funcs, }, { } }; static struct klp_patch patch = { .mod = THIS_MODULE, .objs = objs, }; static int livepatch_init(void) { int ret; ret = klp_register_patch(&patch); if (ret) return ret; ret = klp_enable_patch(&patch); if (ret) { WARN_ON(klp_unregister_patch(&patch)); return ret; } return 0; } static void livepatch_exit(void) { WARN_ON(klp_unregister_patch(&patch)); } module_init(livepatch_init); module_exit(livepatch_exit);
  24. 24. LivePatch Limitations 24 • useful for small and critical fixes – e.g. can’t do kernel upgrade to a new version • Kprobes in the original function are ignored • can’t do semantics changes • can’t do data structure changes • inability to patch hand-written ASM – e.g. no fentry – can be hacked by patch/unpatch callbacks feature • vDSO can’t be patched (-pg is removed)
  25. 25. How it looks like? 25
  26. 26. Current status in mainline Linux kernel Initial LivePatch support (x86-64) 26 v4.0 s390 architecture support v4.1 PPC64 architecture support v4.7 New consistency model (allows removal of livepatch modules) v4.12 ORC unwinder v4.14 Shadow variables v4.15 Patch/unpatch callbacks v4.15 Removed “immediate” feature v4.16 Atomic replace feature in progress (v4.17?) AArch64 architecture support (depends on GCC 8.x) in progress (v5.???)
  27. 27. kpatch Set of Tools
  28. 28. Why need automatic patches generation? 28 • Human can produce errors • Inlining – inlined buggy function to other functions • GCC can do non-trivial optimization by default • Various IPA (inter-procedural optimization) – transform arguments passed by reference to arguments passed by value – create variants of a function with fewer arguments – optimize functions if values passed to them are constants • several clones of a function are possible
  29. 29. Kpatch 29 • Red Hat project of live kernel patching – includes few userspace tools • Used in production (RH relies on it) • Now compatible with upstream Livepatch • Consists of 3 parts – kpatch/Livepatch kernel module stub – run-time patches management on target • systemd service to load livepatches on boot • ‘kpatch’ tool to manage patches – kpatch-build set of tools for automatic patches generation
  30. 30. kpatch-build 30 • build original kernel – with -ffunction-sections -fdata-sections flags – use custom script as cross-compiler to collect .o files • build patched kernel – watch for changed objects • analyze changed objects – compare the objects at a section level – apply various rules to determine any object local sections that are dependencies of the changed section – use a lot of additional magic – create a diff of changed objects • attach changed functions to LivePatch kernel stub – finally generate .ko file
  31. 31. kpatch limitations 31 • can’t do .data section changes (don’t remove static variables) • no cross-compilation support (patch is in upstream) • no out-of-src build • patching of original sources • supports only RHEL/Fedora/OracleLinux and Ubuntu/Debian – Yocto support is on the way • every patch must also be analyzed in-depth by a human • can’t generate patches with shadow variables, load/unload hooks • no out-of-tree modules support
  32. 32. LivePatch writer recommendations 32 • make changes as small as possible – e.g. this is “hot fix”, not a kernel upgrade to a new version • be careful with data structure changes • be careful with synchronization changes – you can create deadlock situation • watch for inlined functions • FUNC , line preprocessor macros – resulting code may be changed unexpectedly • code removal – Livepatch replaces function, not removes it • static variables – don’t remove them, keep unused (kpatch-build tracks .data changes) • https://github.com/dynup/kpatch/blob/master/doc/patch-author-guide.md
  33. 33. Demo
  34. 34. Questions?
  35. 35. Sources 35 kernel.org https://assets.ubuntu.com/v1/ac3aa269-DS_Canonical_Livepatch_Service_sc reen-AW_08.17.pdf https://www.linuxplumbersconf.org/2014/ocw/system/presentations/1773/origi nal/ftrace-kernel-hooks-2014.pdf https://osseu17.sched.com/event/BxJF/kernel-live-patching-current-state-and- future-development-miroslav-benes-suse https://github.com/dynup/kpatch

×