SFO15-303: Linux Kernel Performance (LKP) Project on ARM Speaker: Chase Qi Date: September 23, 2015 ★ Session Description ★ There has been recent work upstream to enable the LKP tests to build and execute on ARM platforms. The QA team has been involved with the integration of these tests into Linaro’s Automated Validation Architecture, and this presentation will provide a brief overview of the LKP results that have been obtained. It will also demonstrate useful way to compare the data. ★ Resources ★ Video: Presentation: http://www.slideshare.net/linaroorg/sfo15303-linux-kernel-performance-lkp-project-on-arm Etherpad: pad.linaro.org/p/sfo15-303 Pathable: https://sfo15.pathable.com/meetings/302935 ★ Event Details ★ Linaro Connect San Francisco 2015 - #SFO15 September 21-25, 2015 Hyatt Regency Hotel http://www.linaro.org http://connect.linaro.org

1. Presented by
Date
Event
SFO15-303: LKP on ARM
Chase Qi, Milosz WasilewskiChase Qi
Wednesday 23 September 2015
SFO15

2. Overview
- LKP Project
- Enable LKP on ARM
- Steps to run LKP
- Run LKP in LAVA
- Result Parsing in LAVA
- Result comparison with LAVA
- Error Margin
- Next steps
- Q&A

3. lkp-tests Project
lkp-tests git repo: https://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git
Linux Kernel Performance: https://01.org/lkp
■ Avoid kernel performance regression
■ Enhance the Linux kernel performance
Contact via email: lkp@lists.01.org
Mail list: https://lists.01.org/mailman/listinfo/lkp

4. Problems with initial implementation
■ Test run setup failures
○ root file system dependencies
○ package dependencies
○ Some features are x86 specific
○ Lack of arm/arm64 support
○ Other compile, setup issues
■ Designed for high-performance system
○ Powerful processor
○ Fast memory architecture
○ Massive storage systems

5. Enable LKP on ARM
Author Q4 2014 Q1 2015 Q2 2015 Q3 2015 Total
Fengguang Wu 172 192 245 8 617
Wanlong Gao 49 47 78 7 181
Philip Li 0 0 74 55 129
Huang Ying 0 17 58 0 75
Yuanhan Liu 16 12 39 3 70
Zexi Li 0 4 36 5 45
Yao Weiqi 0 13 27 0 40
Huang, Ying 0 0 22 8 30
Riku Voipio 0 0 9 0 9
huangqh 0 1 5 2 8
Helia Correia 3 4 0 0 7
Aaron Lu 0 4 2 0 6
Tian Ye 3 0 0 0 3
Li, Zexi 0 2 0 0 2
Wenzhong Sun 0 0 0 2 2
Chase Qi 0 0 0 1 1
weiqi.yao 1 0 0 0 1
Total 244 296 595 91 1226
Commits per author per quarter to LKP upstream

6. lkp-tests Structure
Steps to write a test case; Tools for test run(in blue)
lkp-tests/
├── allot
├── bin # b. setup and test run
├── cluster
├── daemon
├── distro
├── doc
├── etc
├── filters
├── hosts
├── jobs # 4. job files in yaml format to define test parameters
├── lib
├── lkp-exec
├── monitors
├── pack
├── params
├── pkg # 1. package maker script
├── plot
├── repo
├── rootfs # 2. define package dependence
├── sbin # a. job split
├── setup
├── stats # 5. result paser
├── tests # 3. main test case script
├── tmp
└── tools

7. Steps to run LKP
Set PATH: cd lkp-tests
export LKP_SRC=$(pwd)
export PATH=$PATH:$LKP_SRC/bin
Set-up a job: lkp install $LKP_SRC/jobs/hackbench.yaml
Split the job: lkp split-job $LKP_SRC/jobs/hackbench.yaml
options:
-o, --output PATH output path
-c, --config CONFIG test kernel config
-k, --kernel COMMIT test kernel commit
--no-defaults do not load the defaults headers
Run the job: lkp run ./hackbench-50%-threads-socket.yaml
options:
-o, --output RESULT_ROOT dir for storing all results
-s, --set 'KEY: VALUE' add YAML hash to job
Check Results: lkp result hackbench

8. Run LKP in LAVA
Test definition
yaml file https://git.linaro.org/qa/test-definitions.git/blob/HEAD:/ubuntu/lkp.yaml
test script https://git.linaro.org/qa/test-definitions.git/blob/HEAD:/common/scripts/lkp.py
Supported params in the test definition
# Used to specify jobs located in 'lkp-tests/jobs/'.
JOB: 'hackbench'
# Used to specify all 40 commit hash characters.
COMMIT: 'commit'
# Set LOOPS to the times to run in the same lava_test_shell.
LOOPS: 1
# Set MONITORS to '--no-defaults' to disable the default monitors.
MONITORS: 'default'
Main steps of the test script
# lkp-tests/bin/setup-local ./jobs/hackbench.yaml
# lkp-tests/sbin/split-job MONITORS --kernel COMMIT --output $(pwd)/hackbench ./jobs/hackbench.yaml
# lkp-tests/bin/run-local --set 'compiler: CompilerVersion' $(pwd)/hackbench_sub_test.yaml
Parse test results.

9. Result Parsing in LAVA
structure of result_root:
/result/
hackbench/ # benchmark name
50%-process-socket/ # sub-testcase name
linaro-developer/ # hostname
ubuntu/ # rootfs
defconfig/ # config
gcc-4.9/ # compiler version
1efdb5f0a9243ca8f3460a5ce1b407b06a021f02/ # commit hash
0/ # test run
hostname, rootfs, config, compiler version and commit are
changeable with run-local or split-job options
Tree view of the first test run

10. Results in LAVA -single run
Decode job.json, collect results with lava-test-case. In Results Bundle, it looks like:
Attach raw data for post comparison between commits or test runs.
lkp-result-hackbench.tar.xz

11. Results in LAVA -multiple runs
{
"command": "lava_test_shell",
"parameters":
{
"testdef_repos": [
{
"git-repo": "url/test-
definitions",
"parameters":
{
"COMMIT": "commit1",
"JOB": "hackbench",
"LOOPS": "13"
},
"testdef": "ubuntu/lkp.yaml"
}],
"timeout": 1080000
}
},

12. LKP comparison with LAVA
Manual comparison
■ Download test results from LAVA
■ Compare a specific field with lkp compare tool
# lkp-tests/sbin/compare -d commit -f hackbench.throughput
commit1_result_root commit2_result_root
7d9071a095023cd1 1efdb5f0a9243ca8f3460a5ce1 testbox/testcase/testparams
---------------- -------------------------- ---------------------------
%stddev %change %stddev
|
21801 -2.6% 21236 hackbench/50%-process-pipe/linaro-developer
21801 -2.6% 21236 GEO-MEAN hackbench.throughput
Auto LKP comparison tools
https://git.linaro.org/people/chase.qi/lkp-comparison.git
Tools: lavadownload.py resultcompare.py
LAVA result download:
lavadownload.py -n lkp-result -l https://validation.linaro.org/RPC2/ -i lava_job_id
Compare scores produced by benchmark:
python resultcompare.py --job1 jobID1 --job2 jobID2
Compare a specific field:
python resultcompare.py --job1 jobID1 --job2 jobID2 -f time.system_time

13. LKP Comparison Output
v4.2-9700-g7d9071a09502 VS v4.2-rc6-176-g1efdb5f0a924
7d9071a0/1efdb5f0 measurement-testcase
-29.5% hackbench.throughput-1600%-process-pipe
+4.3% hackbench.throughput-1600%-process-socket
-25.7% hackbench.throughput-1600%-threads-pipe
+8.2% hackbench.throughput-1600%-threads-socket
-2.6% hackbench.throughput-50%-process-pipe
-27.8% hackbench.throughput-50%-process-socket
-3.9% hackbench.throughput-50%-threads-pipe
-24.2% hackbench.throughput-50%-threads-socket
-30.1% netpipe.bigger_5M_Mbps.avg-tcp
+4.0% netpipe.less_8K_usec.avg-tcp
+0.7% pbzip2.throughput-100%-500K
+0.4% pbzip2.throughput-100%-900K
+0.6% pbzip2.throughput-25%-500K
-0.5% pbzip2.throughput-25%-900K
+18.0% perf-bench-sched-pipe.ops_per_sec-10000000ops-process
+23.3% perf-bench-sched-pipe.ops_per_sec-10000000ops-threads
+0.1% pigz.throughput-100%-128K
+0.1% pigz.throughput-100%-512K
-0.2% pigz.throughput-25%-128K
+2.7% pigz.throughput-25%-512K
-0.1% pixz.throughput-100%
+0.3% pixz.throughput-25%
+0.1% plzip.throughput-100%
-0.1% plzip.throughput-25%
-0.3% pxz.throughput-100%
-0.2% pxz.throughput-25%
+20.4% unixbench.score-context1
+0.1% unixbench.score-dhry2reg
-1.8% unixbench.score-execl
-10.1% unixbench.score-fsbuffer
-9.2% unixbench.score-fsdisk
-8.7% unixbench.score-fstime
-16.4% unixbench.score-pipe
+0.1% unixbench.score-shell1
+9.8% unixbench.score-shell8
+0.1% unixbench.score-spawn
+2.6% unixbench.score-syscall
+0.0% unixbench.score-whetstone-double

14. Error Margin
v4.2-rc6-176-g1efdb5f0a924, two test runs on the same board.
1efdb5f0/1efdb5f0 measurement-testcase
+0.3% hackbench.throughput-1600%-process-pipe
-0.3% hackbench.throughput-1600%-process-socket
+0.5% hackbench.throughput-1600%-threads-pipe
+0.0% hackbench.throughput-1600%-threads-socket
-0.4% hackbench.throughput-50%-process-pipe
-0.5% hackbench.throughput-50%-process-socket
+0.0% hackbench.throughput-50%-threads-pipe
+0.1% hackbench.throughput-50%-threads-socket
-0.0% netpipe.bigger_5M_Mbps.avg-tcp
-0.1% netpipe.less_8K_usec.avg-tcp
-0.0% pbzip2.throughput-100%-500K
+0.9% pbzip2.throughput-100%-900K
+0.4% pbzip2.throughput-25%-500K
+1.2% pbzip2.throughput-25%-900K
-1.5% perf-bench-sched-pipe.ops_per_sec-10000000ops-process
-0.2% perf-bench-sched-pipe.ops_per_sec-10000000ops-threads
-0.1% pigz.throughput-100%-128K
+0.1% pigz.throughput-100%-512K
+0.3% pigz.throughput-25%-128K
+1.8% pigz.throughput-25%-512K
-0.1% pixz.throughput-100%
+0.5% pixz.throughput-25%
-0.0% plzip.throughput-100%
-0.0% plzip.throughput-25%
+0.0% pxz.throughput-100%
+0.2% pxz.throughput-25%
+0.3% unixbench.score-context1
+0.0% unixbench.score-dhry2reg
-1.1% unixbench.score-execl
+0.5% unixbench.score-fsbuffer
+0.3% unixbench.score-fsdisk
+0.8% unixbench.score-fstime
+0.4% unixbench.score-pipe
+0.9% unixbench.score-shell1
+0.3% unixbench.score-shell8
-0.1% unixbench.score-spawn
+0.1% unixbench.score-syscall
-0.1% unixbench.score-whetstone-double
1efdb5f0/1efdb5f0 measurement-testcase
+0.0% dbench.clients-100%
-15.8% dbench.max_latency-100%
+0.0% dbench.procs-100%
-0.8% dbench.throughput-MB/sec-100%

15. Next steps
■ Store results to database (kernelci.org)
■ Polish comparison tools
■ Add benchmarks to kernel (LSK) test plan
■ Investigate issues in lkp-tests

16. Q&A
Any questions?