1.
1
Linux kernel auto test
by using ktest
Version1.1.6
Satoru Takeuchi
satoru.takeuchi@gmail.com

2.
2
Introduction
● Purpose
– Learn how to test linux kernel automatically by
using ktest
– It's not a ktest reference, but a quick ktest learning
guide
● Target readers
– Linux kernel developers who have x86_64 linux
system and the KVM guest system on it (*1)
*1) Although ktest itself can be used for many environments, this slide focuses
on this specific combination for simplicity.

3.
3
Table of contents
1. What is ktest?
2. System requirement of ktest
3. System environment of this document
4. Initial configuration
5. How to test
6. Conclusion
7. References

4.
4
1. What is ktest?
● A kernel automated test tool which is distributed as part of
linux kernel
– Author: Steven Rostedt
● It keeps working even if test kernel has panic/hangup
problems.
● This document covers the following features
– Basic tests for a kernel
● build test→ install test→ boot test→ user defined test
– Test each patch of a patchset
– Bisect to find the problematic commit from a series of commits

5.
5
2. System requirement of ktest
● The following two system(VM is OK)
– Host: To build test kernel and to watch the progress of test
– Target: Run test with test kernel
● Can be poweron/off from Host
● Can be connected via console from Host
● Can be connected via ssh as root from Host
● Kernel source
– It should be git repository to use some advanced features

6.
6
3. System environment of this document
● I don't explain how to setup this environment here.
● If you try to do the subsequent steps, please replace each expression with
appropriate one depend on your environment.
Host Target
● System: debian/jessy x86_64
● hostname/user: host/sat
● Linux kernel's git repository under
/home/sat/src/linux
● “make mrproper” clean
● kernel config file for test kernel
can be boot on Target.
● /home/sat/kernel.config
● System: debian/jessy or CentOS7
x86_64 guest in Host
● Hostname/domain name: target
● Bootloader: grub2
● “grub-reboot” works fine
● Can be connected via “virsh
console” from Host
● Can be connected via “ssh
root@target” without passphrase

7.
7
4. Initial configuration (1/2): Make workspace
$ mkdir -p /home/sat/ktest
$ cd /home/sat/ktest
$ mkdir output
$ cp /home/sat/src/linux/tools/testing/ktest/ktest.pl .
● Run the following commands on Host.

8.
8
4. Initial Configuration (2/2): Prepare the grub2 entry
● Add the following entry in /boot/grub/grub.conf, the grub2's configuration file, on
Target.
menuentry 'ktest' {
...
linux /boot/ktest-vmlinuz … console=ttyS0,115200n8
...
initrd /boot/ktest-initramfs.img
}

9.
9
5. How to test
● Learn how to run the following tests one by one.
1. Build test.
2. Install test.
3. Boot test.
4. User defined test.
5. Test patchset.
6. Bisect

10.
10
5.1. Build test (1/2): Configuration
● Create /home/sat/ktest/ktest.conf as follows. All
pathnames should be full path here.
MACHINE = target # hostname of Target
LOG_FILE = /home/sat/ktest/ktest.log # The logfile of ktest
BUILD_DIR = /home/sat/src/linux # your kernel source path
OUTPUT_DIR = /home/sat/ktest/output # Output dir of kernel build
LOCALVERSION = -ktest # local version of the test kernel
BUILD_TARGET = arch/x86/boot/bzImage # The name of bzImage
TARGET_IMAGE = /boot/ktest-vmlinuz # Full path of vmlinuz image on Target
MIN_CONFIG = /home/sat/kernel.config # The base of .config
BUILD_TYPE= olddefconfig # configuration target of kernel build
BUILD_OPTIONS = -j8 # (optional) kernel build option
TEST_START
TEST_TYPE = build

11.
11
5.1. Build test (2/2): Run&Verify
● Run ktest as follows.
$ ./ktest.pl
● Then ktest builds your test kernel automatically.
●
If succeeds, test finished with the following log.
...
******************************************
*******************************************
KTEST RESULT: TEST 1 SUCCESS!!!! **
*******************************************
*******************************************
1 of 1 tests were successful
● If fails, diagnose the problem with the logfile.
● Failed during build → Your kernel would have some failed-to-build bug.
● Failed in other place → ktest.conf would be wrong.

12.
12
5.2. Install test
● Modify ktest.conf
+ SSH_USER = root
+ POST_INSTALL = ssh ${SSH_USER}@${MACHINE} /usr/sbin/mkinitramfs -o /boot/ktest-initramfs.img
$KERNEL_VERSION # (optional) The code run after installation on Host.
- TEST_TYPE = build
+ TEST_TYPE = install
● If your system is CentOS7, replace POST_INSTALL line as
follows.
● Run ktest
● Then ktest builds and installs your test kernel automatically.
● If fails, ktest.conf would be wrong.
$ ./ktest.pl
+ POST_INSTALL = ssh {SSH_USER}@${MACHINE} /usr/sbin/dracut -f
/boot/ktest-initramfs.img $KERNEL_VERSION

13.
13
5.3. Boot test (1/2): Configuration
● Modify ktest
+ REBOOT_TYPE = grub2
+ GRUB_FILE = /boot/grub/grub.cfg
+ GRUB_MENU = ktest
+ GRUB_REBOOT = grub-reboot # The command to boot your test kernel only once on Target
+ POWER_CYCLE = virsh destroy ${MACHINE}; sleep 5; virsh start ${MACHINE} # The commands from
Host to force reboot Target.
+ REBOOT = ssh ${SSH_USER}@${MACHINE} reboot; : # reboot command which ktest uses
+ REBOOT_ON_ERROR = 1 # (optional) Set if you want to reboot Target on failed-to-boot.
+ CONSOLE = virsh console ${MACHINE} # Command to connect from Host to Target
+ CLOSE_CONSOLE_SIGNAL = KILL # Signal to kill the console process.
...
- TEST_TYPE = install
+ TEST_TYPE = boot
● Modify ktest.conf
● If your guest is CentOS7, replace GRUB_FILE and GRUB_REBOOT
lines as follows
+ GRUB_FILE = /boot/grub2/grub.cfg
+ GRUB_REBOOT = grub2-reboot

14.
14
5.3. Boot test (2/2): Run&Verify
● Run ktest
● Then ktest builds, installs, boot your test kernel , and reboot
with the original kernel automatically.
● If fails, your kernel has failed-to-boot bugs or ktest.conf
would be wrong.
$ ./ktest.pl

15.
15
5.4. User defined test
1. Make test. Please note that the test is run under Host.
2. Modify ktest.conf
3. Run ktest.
● Then ktest builds, installs, boot your test kernel, run the user defined test with this
kernel, reboot with the original kernel automatically.
● If failed…
● on user defined test → Your kernel or your test would have some bugs.
● on other place → ktest.conf would be wrong.
$ ./ktest.pl
- TEST_TYPE = boot
+ TEST_TYPE = test
+ TEST = /home/sat/ktest/hello.test
$ cat hello.test
ssh 'root@target echo hello ktest!'

16.
16
5.5. Test patchset (1/4): Assumption
● You made the patchset, consists of the following ten patches, for 3.14-rc6.
– 7 th patch has panic-on-boot bug (of course you don't know it beforehand).
● You'd like to run boot test for each patch before submitting it to LKML.
● $ git log --oneline -11
● 3d02e41 test commit 10
● f3c04a3 test commit 9
● 3fec210 test commit 8
● 0641787 test commit 7 # It has a critical bug!
● a172eb6 test commit 6
● 63d9f91 test commit 5
● d6edbe0 test commit 4
● a350e10 test commit 3
● 52a2a97 test commit 2
● 8644b5e test commit 1
● fa389e2 Linux 3.14-rc6

17.
17
5.5. Test patchset (2/4): Configuration&Run
1. Modify ktest.conf
2. Run ktest
● Then ktest runs boot test for each patch with the folloing log.
- TEST_TYPE = test
- TEST = /home/sat/ktest/hello.test
+ TEST_TYPE = patchcheck
+ PATCHCHECK_TYPE = boot # Run boot test for each patch.
+ PATCHCHECK_START = 8644b5e # The first patch of the patchset
+ PATCHCHECK_END = 3d02e41 # The last patch of the patchset
$ ./ktest.pl
Processing commit 8644b5e0896b075c19f9d52aea1e9a22d7fdc89d test commit 1
…
Processing commit 52a2a973c29dce4d85e3c0d570f4351b248ab311 test commit 2
...

18.
18
5.5. Test patchset (3/4): Verify
● Ktest detects panic during the test of 7 th patch with the following log, and
finishes.
Processing commit 0641787fc60d4926b6d286a84417e4b6bb5eca3b test commit 7
...
[ 0.024472] Kernel panic - not syncing: Attempted to kill the idle task!
bug timed out after 1 seconds
Test forced to stop after 60 seconds after failure
CRITICAL FAILURE... failed - got a bug report
REBOOTING
ssh root@debian sync ... FAILED!
ssh root@debian reboot ... FAILED!
virsh destroy debian; sleep 5; virsh start debian ... SUCCESS
See /home/sat/ktest/ktest.log for more info.
failed - got a bug report
...

19.
19
5.5. Test patchset (4/4): Further work
● Modify PATCHCHECK_TYPE option if you want to change the test for each patch.
● The same value as TEST_TYPE option can be set.
● build, install, boot, test (You should also set TEST option here)
● ex) Run your own test, hello.test, for each patches.
- PATCHCHECK_TYPE = boot
+ PATCHCHECK_TYPE = test
+ TEST = /home/sat/ktest/hello.test

20.
20
5.6 Bisect (1/5): Assumption
● 3.14-rc6 boots without any problem.
● 3.14-rc6 with the following ten commits get panic on boot.
● 7 th commit has panic-on-boot bug (of course you don't know it beforehand).
● You want to find the problematic commit and report it to LKML.
● $ git log --oneline -11
● 3d02e41 test commit 10
● f3c04a3 test commit 9
● 3fec210 test commit 8
● 0641787 test commit 7 # It has a critical bug!
● a172eb6 test commit 6
● 63d9f91 test commit 5
● d6edbe0 test commit 4
● a350e10 test commit 3
● 52a2a97 test commit 2
● 8644b5e test commit 1
● fa389e2 Linux 3.14-rc6

21.
21
5.6. bisect (2/5): Configuration
● Modify ktest.conf
- TEST_TYPE = patchcheck
- PATCHCHECK_TYPE = boot
- PATCHCHECK_START = 8644b5e
- PATCHCHECK_END = 3d02e41
+ TEST_TYPE = bisect
+ BISECT_TYPE = boot
+ BISECT_GOOD = b28a960 # One before the first of suspect commits
+ BISECT_BAD = 3d02e41 # The last of suspect commits

22.
22
5.6. bisect (3/5): Run
● Run ktest
● Then ktest bisects the suspect commits automatically.
git bisect start ... SUCCESS
git bisect good fa389e220254c69ffae0d403eac4146171062d08 ... SUCCESS
git bisect bad 3d02e41fd18ddda74352beb35aa952266b34b92f ... Bisecting: 4 revisions left to test
after this (roughly 2 steps)
…
git bisect good ... Bisecting: 2 revisions left to test after this (roughly 1 step)
…
git bisect bad ... Bisecting: 0 revisions left to test after this (roughly 0 steps)
...
●
$ ./ktest.pl

23.
23
5.6. Bisect (4/5): Verify
● Finally ktest detects 7th commit from 3.14-rc6 is the bad one
git bisect good ... 0641787fc60d4926b6d286a84417e4b6bb5eca3b is the first bad commit
commit 0641787fc60d4926b6d286a84417e4b6bb5eca3b
Author: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Date: Sun Feb 16 22:48:02 2014 +0900
test commit 7
...
*******************************************
*******************************************
KTEST RESULT: TEST 1 SUCCESS!!!! **
*******************************************
*******************************************
...

24.
24
5.6. Bisect (5/5): Further work
● Modify BISECT_TYPE option if you want to change the test for each
bisect.
● The same value as TEST_TYPE option can be set.
● build, install, boot, test (you should also set TEST option here)
● ex) Run your own test, hello.test, for each bisect.
- PATCHCHECK_TYPE = boot
+ PATCHCHECK_TYPE = test
+ TEST = /home/sat/ktest/hello.test

25.
25
6. Conclusion
● Now you can do the automated kernel test. Congratulations!
● It can reduce plenty of your precious time.
● For more information, please read the references attached at the next
page.
● Feel free to ask me if you have any questions/comments.

26.
26
7. References
● The files under <kernel source>/testing/tools/ktest:
● ktest.pl: The ktest program
● sample.conf: The document for all options.
● examples/*: Sample configuration files for some Targets
● examples/kvm.conf is for the KVM Target
● ktest tutorial session by Steven Rostedt:
– http://video.linux.com/videos/automated-testing-with-ktestpl
● Elkdat: Prepare ktest environment by one command
● https://github.com/satoru-takeuchi/elkdat