SlideShare a Scribd company logo
1 of 59
Download to read offline
WWW.ANDESTECH.COM
Driving Innovations™
Experience on Porting
Perf & Ftrace to RISC-V
Experience on Porting
Perf & Ftrace to RISC-V
高魁良 (Alan Kao)
Andes Technology
2018/08/11
Confientia Driving Innovations™2
Driving Innovations™
About meAbout me
●
Alan (Quey-Liang) Kao
●
2007~2017 NTHU
●
BS, MS, P
●
2017.10~ Andes Technology
●
software engineer
●
RISC-V Linux
●
Interests in Linux
●
virtualization, container, tracing, perf
Confientia Driving Innovations™3
Driving Innovations™
OutlineOutline
●
Takeaway: The Big Picture
●
Ftrace
●
Basics
●
Perf
●
Basics
●
Porting Experiences
●
Conclusion
Confientia Driving Innovations™4
Driving Innovations™
The Big Picture
Confientia Driving Innovations™5
virtualization
networking
device manager
scheduler
file system
memory management
Driving Innovations™
一般使用者的日常一般使用者的日常
User Space
Kernel Space
Kernel
Confientia Driving Innovations™6
virtualization
networking
device manager
scheduler
file system
memory management
Driving Innovations™
一般使用者的日常一般使用者的日常
User Space
Kernel Space
Kernel
根本是黑盒子
Confientia Driving Innovations™7
virtualization
networking
device manager
scheduler
file system
memory management
Driving Innovations™
一般使用者的日常一般使用者的日常
User Space
Kernel Space
Kernel
如薄紗般的質感
透著一點神秘
根本是黑盒子
Confientia Driving Innovations™8
virtualization
networking
device manager
scheduler
file system
memory management
Driving Innovations™
一般使用者的日常一般使用者的日常
User Space
Kernel Space
Kernel
如薄紗般的質感
透著一點神秘
根本是黑盒子
若隱若現的是
人們沒親眼見過的遠古傳說
Confientia Driving Innovations™9
virtualization
networking
device manager
file system
memory management
Driving Innovations™
開發者的異常開發者的異常
User Space
Kernel Space
Kernel
scheduler
程式邏輯穿透 x1
中階元件理解 x1高級元件理解 x1
技能:系統監控
技能:善用工具
技能:資料分析
Confientia Driving Innovations™10
virtualization
networking
device manager
file system
memory management
Driving Innovations™
工具?工具?
User Space
Kernel Space
Kernel
scheduler
perf
ftrace
Confientia Driving Innovations™11
virtualization
networking
device manager
file system
memory management
Driving Innovations™
工具?工具?
User Space
Kernel Space
Kernel
scheduler
perf
ftrace
read, write
special file
Confientia Driving Innovations™12
virtualization
networking
device manager
file system
memory management
Driving Innovations™
kernel 提供工具,
工具監控 kernel 。
kernel 提供工具,
工具監控 kernel 。
User Space
Kernel Space
Kernel
scheduler
perf
ftrace
Confientia Driving Innovations™13
virtualization
networking
device manager
file system
memory management
Driving Innovations™
User Space
Kernel Space
Kernel
scheduler
perf
ftrace
就把「監控」想像成 ...
if ( 有人要我監控 A)
printf(“%d %dn”,A.a, A.b)
Confientia Driving Innovations™14
virtualization
networking
device manager
file system
memory management
Driving Innovations™
User Space
Kernel Space
Kernel
scheduler
perf
ftrace
就把「監控」想像成 ...
if ( 有人要我監控 A)
printf(“%d %dn”,A.a, A.b)
Probe Efect
被監控的系統的行為本身
會因為受到介入而改變
Confientia Driving Innovations™15
virtualization
networking
device manager
file system
memory management
Driving Innovations™
監控過程中產生資料監控過程中產生資料
User Space
Kernel Space
Kernel
scheduler
perf
ftrace
iata
Confientia Driving Innovations™16
virtualization
networking
device manager
file system
memory management
Driving Innovations™
監控過程中產生更多資料監控過程中產生更多資料
User Space
Kernel Space
Kernel
scheduler
perf
ftrace
iataiata
iata
iata
Confientia Driving Innovations™17
virtualization
networking
device manager
file system
memory management
Driving Innovations™
然後 kernel 恢復日常運作然後 kernel 恢復日常運作
User Space
Kernel Space
scheduler
iata
Kernel
iataiata
iata
iata
Confientia Driving Innovations™18
virtualization
networking
device manager
file system
memory management
Driving Innovations™
然後可以分析效能瓶頸然後可以分析效能瓶頸
User Space
Kernel Space
scheduler
iata
Kernel
iataiata
iata
iata
OS: 做投影片的效能瓶頸
是想梗做動畫啊!
Confientia Driving Innovations™19
Driving Innovations™
「劇本」與「角色」「劇本」與「角色」
1. 開發者使用效能監控工具
2. 工具改變 kernel 的部份狀態,進入監控模式
3. 產生監控所得資料
4. 離開監控模式
5. 開發者存取資料以作為分析之用
●
Ftrace
●
監控狀態時行為:
進入每個函數時,綁架正常的執
行流程
●
產生資料:
函數執行與離開的時間戳記
●
透過 tracefs 檔案系統取得
trace 內容
●
Perf
●
監控狀態時行為:
排程器在排入行程時啟動、行程
離開時移除 perf 機制
●
產生資料:
軟體事件、硬體 counter
●
perf 工具輸出
Confientia Driving Innovations™20
Driving Innovations™
Ftrace Overview
Confientia Driving Innovations™21
Driving Innovations™
IntroiuctionIntroiuction
●
Function Tracer
●
now serves as a framework, under the tracing subsystem
●
Originally a tracer in the PREEMPT_RT patchset
●
to measure latency
●
“Ftrace aiis little overheai to the system when
tracing.” -- Survey and Analysis of Kernel and
Userspace Tracers on Linux: Design, Implementation,
and Overhead, ACM Computing Survey 2018 June
Confientia Driving Innovations™22
Driving Innovations™
Basic UsageBasic Usage
●
Example 0: go to the right place
# cd /sys/kernel/debug/tracing
●
Example 1: start function tracer
# cat available_tracers
function function_graph ...
# echo function_graph > current_tracer && 
echo 1 > tracing_on
●
Example 2: set white/black list to trace
# cat available_filter_functions
... // some compiler-modified names here
# echo foo bar > set_ftrace_filter // white list
# echo foo bar > set_ftrace_notrace // black list (superior)
●
Example 3: check the trace
●
# cat trace // the blocking alternative: trace_pipe
Confientia Driving Innovations™23
Driving Innovations™
角色: Ftrace角色: Ftrace
1. 開發者使用效能監控工具
2. 工具改變 kernel 的部份狀態,進入監控模式
3. 產生監控所得資料
4. 離開監控模式
5. 開發者存取資料以作為分析之用
●
Ftrace
●
監控狀態時行為:
進入每個函數時,綁架正常的執
行流程
●
產生資料:
函數執行與離開的時間戳記
●
透過 tracefs 檔案系統取得
trace 內容
●
Perf
●
監控狀態時行為:
排程器在排入行程時啟動、行程
離開時移除 perf 機制
●
產生資料:
軟體事件、硬體 counter
●
perf 工具輸出
要如何綁架
函式的入口與出口?
Ftrace
porting 重點
Confientia Driving Innovations™24
Driving Innovations™
編譯 kernel 要使用 gcc -pg編譯 kernel 要使用 gcc -pg
●
without -pg
<foo>:
00: addi sp, sp, -16
04: sd s0, 0(sp)
08: sd ra, 8(sp)
0c: addi s0, sp, 16
10: mv a0, ra
...
●
with -pg
<foo>:
00: addi sp, sp, -16
04: sd s0, 0(sp)
08: sd ra, 8(sp)
0c: addi s0, sp, 16
10: call _mcount
14: mv a0, ra
...Prologue
進入函式時處理 stack 的階段
儲存返回位址 (ra) 很重要!
Epilogue
離開函式的階段
載回返回位址 (ra), ret
Confientia Driving Innovations™25
Driving Innovations™
<f2>:
00: addi sp, sp, -8
08: sd ra, 0(sp)
10: call _mcount
18: ld ra, 0(sp)
20: addi sp, sp, 8
28: ret
<f1>:
...
40: call f2
...
SP=0x10000100, PC=0x00SP=0x10000100, PC=0x00
stack:
100000D0
100000D8
100000E0
100000E8
100000F0
100000F8
10000100
剛從 f1 呼叫過來
ra = 0x48
呼叫時的下一行
是為返回位址
Confientia Driving Innovations™26
Driving Innovations™
SP=0x100000F8, PC=0x08SP=0x100000F8, PC=0x08
stack:
100000D0
100000D8
100000E0
100000E8
100000F0
100000F8
10000100
<f2>:
00: addi sp, sp, -8
08: sd ra, 0(sp)
10: call _mcount
18: ld ra, 0(sp)
20: addi sp, sp, 8
28: ret
<f1>:
...
40: call f2
...
ra = 0x48
Confientia Driving Innovations™27
Driving Innovations™
SP=0x100000F8, PC=0x10SP=0x100000F8, PC=0x10
stack:
100000D0
100000D8
100000E0
100000E8
100000F0
100000F8 0x48
10000100
<f2>:
00: addi sp, sp, -8
08: sd ra, 0(sp)
10: call _mcount
18: ld ra, 0(sp)
20: addi sp, sp, 8
28: ret
<f1>:
...
40: call f2
...
ra = 0x48
Confientia Driving Innovations™28
Driving Innovations™
SP=0x100000F8, PC=???SP=0x100000F8, PC=???
stack:
100000D0
100000D8
100000E0
100000E8
100000F0
100000F8 0x48
10000100
<_mcount>:
綁架首部曲:入口の侵蝕
按下第一次碼表
<f2>:
00: addi sp, sp, -8
08: sd ra, 0(sp)
10: call _mcount
18: ld ra, 0(sp)
20: addi sp, sp, 8
28: ret
<f1>:
...
40: call f2
...
腦內設定:
沒有什麼特別的意味
隱藏資料結構 :
0.1
ra = 0x18
請原諒我使用
psuedo code 和日文
Confientia Driving Innovations™29
<f2>:
00: addi sp, sp, -8
08: sd ra, 0(sp)
10: call _mcount
18: ld ra, 0(sp)
20: addi sp, sp, 8
28: ret
<f1>:
...
40: call f2
...
Driving Innovations™
SP=0x100000F8, PC=???SP=0x100000F8, PC=???
stack:
100000D0
100000D8
100000E0
100000E8
100000F0
100000F8 0x80
10000100
<_mcount>:
...
綁架二部曲:回傳の支配
隱藏資料結構 :
0.1 0x48
「難道說 ...
我們兩個人 ...
返回位址交換了???」
---- 你的啦 (ra)
ra = 0x18
Confientia Driving Innovations™30
<f2>:
00: addi sp, sp, -8
08: sd ra, 0(sp)
10: call _mcount
18: ld ra, 0(sp)
20: addi sp, sp, 8
28: ret
<f1>:
...
40: call f2
...
Driving Innovations™
SP=0x100000F8, PC=???SP=0x100000F8, PC=???
stack:
100000D0
100000D8
100000E0
100000E8
100000F0
100000F8 0x80
10000100
<_mcount>:
...
綁架三部曲:本體の回歸
ret
隱藏資料結構 :
0.1 0x48
ra = 0x18
Confientia Driving Innovations™31
<f2>:
00: addi sp, sp, -8
08: sd ra, 0(sp)
10: call _mcount
18: ld ra, 0(sp)
20: addi sp, sp, 8
28: ret
<f1>:
...
40: call f2
...
Driving Innovations™
SP=0x100000F8, PC=0x18SP=0x100000F8, PC=0x18
stack:
100000D0
100000D8
100000E0
100000E8
100000F0
100000F8 0x80
10000100
隱藏資料結構 :
0.1 0x48
ra = 0x18
Confientia Driving Innovations™32
<f2>:
00: addi sp, sp, -8
08: sd ra, 0(sp)
10: call _mcount
18: ld ra, 0(sp)
20: addi sp, sp, 8
28: ret
<f1>:
...
40: call f2
...
Driving Innovations™
SP=0x100000F8, PC=0x20SP=0x100000F8, PC=0x20
stack:
100000D0
100000D8
100000E0
100000E8
100000F0
100000F8 0x80
10000100
隱藏資料結構 :
0.1 0x48
ra = 0x80
Confientia Driving Innovations™33
<f2>:
00: addi sp, sp, -8
08: sd ra, 0(sp)
10: call _mcount
18: ld ra, 0(sp)
20: addi sp, sp, 8
28: ret
<f1>:
...
40: call f2
...
Driving Innovations™
SP=0x10000100, PC=0x28SP=0x10000100, PC=0x28
stack:
100000D0
100000D8
100000E0
100000E8
100000F0
100000F8 0x80
10000100
隱藏資料結構 :
0.1 0x48
ra = 0x80
Confientia Driving Innovations™34
<f2>:
00: addi sp, sp, -8
08: sd ra, 0(sp)
10: call _mcount
18: ld ra, 0(sp)
20: addi sp, sp, 8
28: ret
<f1>:
...
40: call f2
...
Driving Innovations™
SP=0x10000100, PC=0x80SP=0x10000100, PC=0x80
stack:
100000D0
100000D8
100000E0
100000E8
100000F0
100000F8 0x80
10000100
隱藏資料結構 :
0.1 0x48
<_mcount_2>:
綁架四部曲:出口の污染
80: 按下第二次碼表 ...0.23
88: 讀取隱藏資料結構
90: 時間差計算
98: 恢復回傳位址
A0:ret
ra = 0x80
Confientia Driving Innovations™35
<f2>:
00: addi sp, sp, -8
08: sd ra, 0(sp)
10: call _mcount
18: ld ra, 0(sp)
20: addi sp, sp, 8
28: ret
<f1>:
...
40: call f2
...
Driving Innovations™
SP=0x10000100, PC=0x88SP=0x10000100, PC=0x88
stack:
100000D0
100000D8
100000E0
100000E8
100000F0
100000F8 0x80
10000100
隱藏資料結構 :
0.1 0x48
<_mcount_2>:
80: 按下第二次碼表 ...
88: 讀取隱藏資料結構
90: 時間差計算
98: 恢復回傳位址
A0:ret
ra = 0x80
Confientia Driving Innovations™36
<f2>:
00: addi sp, sp, -8
08: sd ra, 0(sp)
10: call _mcount
18: ld ra, 0(sp)
20: addi sp, sp, 8
28: ret
<f1>:
...
40: call f2
...
Driving Innovations™
SP=0x10000100, PC=0x90SP=0x10000100, PC=0x90
stack:
100000D0
100000D8
100000E0
100000E8
100000F0
100000F8 0x80
10000100
隱藏資料結構 :
0x48
<_mcount_2>:
80: 按下第二次碼表 ...
88: 讀取隱藏資料結構
90: 時間差計算 ... 0.13
98: 恢復回傳位址
A0:ret
ra = 0x80
Confientia Driving Innovations™37
<f2>:
00: addi sp, sp, -8
08: sd ra, 0(sp)
10: call _mcount
18: ld ra, 0(sp)
20: addi sp, sp, 8
28: ret
<f1>:
...
40: call f2
...
Driving Innovations™
SP=0x10000100, PC=0x98SP=0x10000100, PC=0x98
stack:
100000D0
100000D8
100000E0
100000E8
100000F0
100000F8 0x80
10000100
隱藏資料結構 :
0x48
<_mcount_2>:
80: 按下第二次碼表 ...
88: 讀取隱藏資料結構
90: 時間差計算
98: 恢復回傳位址
A0:ret
ra = 0x80
Confientia Driving Innovations™38
<f2>:
00: addi sp, sp, -8
08: sd ra, 0(sp)
10: call _mcount
18: ld ra, 0(sp)
20: addi sp, sp, 8
28: ret
<f1>:
...
40: call f2
...
Driving Innovations™
SP=0x10000100, PC=0xA0SP=0x10000100, PC=0xA0
stack:
100000D0
100000D8
100000E0
100000E8
100000F0
100000F8 0x80
10000100
ra = 0x48
<_mcount_2>:
80: 按下第二次碼表 ...
88: 讀取隱藏資料結構
90: 時間差計算
98: 恢復回傳位址
A0:ret
Confientia Driving Innovations™39
<f2>:
00: addi sp, sp, -8
08: sd ra, 0(sp)
10: call _mcount
18: ld ra, 0(sp)
20: addi sp, sp, 8
28: ret
<f1>:
...
40: call f2
...
Driving Innovations™
SP=0x10000100, PC=0x48SP=0x10000100, PC=0x48
stack:
100000D0
100000D8
100000E0
100000E8
100000F0
100000F8 0x80
10000100
綁架最終曲:
就算這樣子走過一輪期待沒摸過
的人一次看懂是不是搞錯了什麼
ra = 0x48
Confientia Driving Innovations™40
Driving Innovations™
My frst PATCH!My frst PATCH!
commit 10626c32e3827bca560217966f5bd586c4e91584
Author: Alan Kao <alankao@andestech.com>
Date: Mon Dec 18 17:52:48 2017 +0800
riscv/ftrace: Aii basic support
This patch contains basic ftrace support for RV64I platform. Specifically, function
tracer, function graph tracer (HAVE_FUNCTION_GRAPH_TRACER), and ... are
implemented following the instructions in Documentation/trace/ftrace-design.txt.
Note that the functions in both ftrace.c and setup.c should not be hooked with the
compiler's -pg option: to prevent infinite self-referencing for the former, and to ignore
early setup stuf for the latter.
Signed-of-by: Alan Kao <alankao@andestech.com>
Signed-of-by: Palmer Dabbelt <palmer@sifive.com>
Confientia Driving Innovations™41
Driving Innovations™
[PATCHv4 3/6][PATCHv4 3/6]
commit bc1a4c3a842556852bb02039887d73899d513532
Author: Alan Kao <alankao@andestech.com>
Date: Tue Feb 13 13:13:18 2018 +0800
riscv/ftrace: Aii iynamic function graph tracer support
Once the function_graph tracer is enabled, a filtered function has the
following call sequence:
* ftracer_caller ==> on/of by ftrace_make_call/ftrace_make_nop
* ftrace_graph_caller
* ftrace_graph_call ==> on/of by ftrace_en/disable_ftrace_graph_caller
* prepare_ftrace_return
Considering the following DYNAMIC_FTRACE_WITH_REGS feature, it would be
more extendable to have a ftrace_graph_caller function, instead of
calling prepare_ftrace_return directly in ftrace_caller.
Cc: Greentime Hu <greentime@andestech.com>
Signed-of-by: Alan Kao <alankao@andestech.com>
Signed-of-by: Palmer Dabbelt <palmer@sifive.com>
Confientia Driving Innovations™42
Driving Innovations™
Perf Overview
Confientia Driving Innovations™43
Driving Innovations™
IntroiuctionIntroiuction
●
Perf
●
Profiling tool since 2.6.x
●
User space: perf CLI tool
●
perf stat: counting
●
perf record/report: sampling
●
...
●
Kernel space: perf event
●
kernel/event/core.c
●
include/linux/perf_event.h
●
arch/xxxx/yyyy/perf_event.c
●
CONFIG_PERF_EVENT
Confientia Driving Innovations™44
Driving Innovations™
Basic UsageBasic Usage
●
Example 0: list events
# perf list
●
Example 1: collect counters
# perf stat -e faults,cycles ls ~
●
Example 2: profiling the system
# perf record -e faults ls ~
# perf report
# Overhead Command Shared Object Symbol
# ........ ....... ................. ...................
14.29% ls [kernel.kallsyms] [k] __asm_copy_to_user
14.29% ls busybox [.] 0x00000000000c1d0e
14.29% ls busybox [.] 0x00000000000eeed2
14.29% ls busybox [.] 0x0000000000118634
11.43% ls busybox [.] 0x00000000000c56ae
8.57% ls busybox [.] 0x00000000001039da
5.71% ls busybox [.] 0x00000000000eec90
2.86% ls [kernel.kallsyms] [k] __clear_user
Confientia Driving Innovations™45
Driving Innovations™
角色: Perf角色: Perf
1. 開發者使用效能監控工具
2. 工具改變 kernel 的部份狀態,進入監控模式
3. 產生監控所得資料
4. 離開監控模式
5. 開發者存取資料以作為分析之用
●
Ftrace
●
監控狀態時行為:
進入每個函數時,綁架正常的執
行流程
●
產生資料:
函數執行與離開的時間戳記
●
透過 tracefs 檔案系統取得
trace 內容
●
Perf
●
監控狀態時行為:
排程器在排入行程時啟動、行程
離開時移除 perf 機制
●
產生資料:
軟體事件、硬體 counter
●
perf 工具輸出
Perf
porting 重點
Confientia Driving Innovations™46
Driving Innovations™
Perf Counting EventPerf Counting Event
task6 task1 task4 task3 task2
task1
... ...
event_init
白話:我想要監控
cache miss 數量
sched_in
將 cache miss 的硬體
counter 值記錄在 prev
變數中
sched_out
讀取 cache miss counter
與 prev 取差值
儲存至 count 變數
exit
回傳 count 即為所求
Confientia Driving Innovations™47
Driving Innovations™
[PATCH 1/2] Coies[PATCH 1/2] Coies
commit d9695108ebad944f32c72c90bc94142c4f44454f
Author: Alan Kao <alankao@andestech.com>
Date: Fri Apr 20 07:27:49 2018 +0800
perf: riscv: preliminary RISC-V support
This patch provide a basic PMU, riscv_base_pmu, which supports two
general hariware event, instructions and cycles. Furthermore, this
PMU serves as a reference implementation to ease the portings in
the future.
riscv_base_pmu shouli be able to run on any RISC-V machine that
conforms to the Priv-Spec. Note that the latest qemu model hasn't
fully support a proper behavior of Priv-Spec 1.10 yet, but work
around should be easy with very small fixes. Please check
https://github.com/riscv/riscv-qemu/pull/115 for future updates.
Confientia Driving Innovations™48
Driving Innovations™
[PATCH 2/2] Doc[PATCH 2/2] Doc
commit 6f2490406927492c392e1b51f5706a2a0b8e4a1b
Author: Alan Kao <alankao@andestech.com>
Date: Fri Apr 20 07:27:50 2018 +0800
perf: riscv: Aii Document for Future Porting Guiie
Reviewed-by: Alex Solomatnikov <sols@sifive.com>
Cc: Nick Hu <nickhu@andestech.com>
Cc: Greentime Hu <greentime@andestech.com>
Signed-of-by: Alan Kao <alankao@andestech.com>
Signed-of-by: Palmer Dabbelt <palmer@sifive.com>
Confientia Driving Innovations™49
Driving Innovations™
Porting Experience(s)
Confientia Driving Innovations™50
Driving Innovations™
1. 人在江湖,搞清楚遊戲規則1. 人在江湖,搞清楚遊戲規則
●
前置準備: patch第一個
●
媒介: Linux 社群用 Mailing List 散
佈、討論 patch
●
工具:英文練好、 git 用熟
●
品質:發送 patch 或是 patchset 之前
務必要檢查
●
應對: Maintainer 不會故意刁難,理
直氣和地討論即可
Confientia Driving Innovations™51
Driving Innovations™
2. 做好功課2. 做好功課
●
WHAT :你在意的程式片段是什麼?
●
在 source tree 找答案
●
HOW :那些程式碼是怎麼形成的?
●
用 git log/blame 找答案
●
WHY :為什麼變成現在這樣?
●
爬 mailing list history 找答案
●
沒有頭緒?整個 mailing list 訂下去
Confientia Driving Innovations™52
Driving Innovations™
3. 抄3. 抄
●
關於 arch-dep 移植 ...
●
各種 CPU 架構很多,你 ( 正 ) 煩惱的,
他們搞不好都解了
●
大家也都是抄來抄去的
●
注意遊戲規則,不能抄了又說是原創
●
不要抄 x86
Confientia Driving Innovations™53
Driving Innovations™
4. 觸類旁通4. 觸類旁通
●
雖然可以閉門造車做 arch-dep 移植 ...
●
但是多看 generic 部份、 userspace
部份都有幫助
●
提昇理解的深度
●
要是觸上癮了 ...
●
C 函式庫(一個 glibc 不夠,你試過 newlib
嗎?)
●
各種 Unix-like 表兄弟作業系統
Confientia Driving Innovations™54
Driving Innovations™
5. 自己的開發環境自己守護5. 自己的開發環境自己守護
●
工具鏈、核心、根目錄、模擬器 ...
●
RISC-V 天地初開時,真的很容易炸
●
不要以為 git 就無敵
●
遇到控制狂上游,天天 force update
●
遇到放置型上游,一整年都不更新
●
RISC-V 社群兩種都有
●
我自己都用 docker 定期存 image
●
容量肥大但好用
Confientia Driving Innovations™55
Driving Innovations™
Conclusion
Confientia Driving Innovations™56
Driving Innovations™
ConclusionConclusion
●
真的很抱歉把主題訂太大了
●
perf 和 ftrace 都是很實用的工
具,值得摸摸看
●
如果對這兩個主題或 Linux 貢獻有
興趣,歡迎聯絡我
●
alankao@andestech.com
●
Happy Hacking!
Confientia Driving Innovations™57
Driving Innovations™
Ftrace ReferencesFtrace References
●
Ftrace Kernel Hooks: More than just tracing
by Steven Rostedt
●
知乎軟件架構設計專題
Linux 2 ftrace在 下做性能分析 :
●
完成 RISC-V 環境架設:
busybear官方版本的 ;
outdated docker稍微有點 的 版本
Confientia Driving Innovations™58
Driving Innovations™
Perf ReferencesPerf References
●
perf wiki: tutorial
●
Tracing with Perf Tools by Namhyung Kim
●
Documentation/riscv/pmu.txt
Confientia Driving Innovations™59
Driving Innovations™
Thank You!Thank You!

More Related Content

Similar to 20180811 coscup

[CB19] MalConfScan with Cuckoo: Automatic Malware Configuration Extraction Sy...
[CB19] MalConfScan with Cuckoo: Automatic Malware Configuration Extraction Sy...[CB19] MalConfScan with Cuckoo: Automatic Malware Configuration Extraction Sy...
[CB19] MalConfScan with Cuckoo: Automatic Malware Configuration Extraction Sy...
CODE BLUE
 
Cray XT Porting, Scaling, and Optimization Best Practices
Cray XT Porting, Scaling, and Optimization Best PracticesCray XT Porting, Scaling, and Optimization Best Practices
Cray XT Porting, Scaling, and Optimization Best Practices
Jeff Larkin
 
Ein Stall voller Trüffelschweine - (PHP-)Profiling-Tools im Überblick
Ein Stall voller Trüffelschweine - (PHP-)Profiling-Tools im ÜberblickEin Stall voller Trüffelschweine - (PHP-)Profiling-Tools im Überblick
Ein Stall voller Trüffelschweine - (PHP-)Profiling-Tools im Überblick
renebruns
 

Similar to 20180811 coscup (20)

[CB19] MalConfScan with Cuckoo: Automatic Malware Configuration Extraction Sy...
[CB19] MalConfScan with Cuckoo: Automatic Malware Configuration Extraction Sy...[CB19] MalConfScan with Cuckoo: Automatic Malware Configuration Extraction Sy...
[CB19] MalConfScan with Cuckoo: Automatic Malware Configuration Extraction Sy...
 
Is your code ready for PHP 7 ?
Is your code ready for PHP 7 ?Is your code ready for PHP 7 ?
Is your code ready for PHP 7 ?
 
DEF CON 27 - HUBER AND ROSKOSCH - im on your phone listening attacking voip c...
DEF CON 27 - HUBER AND ROSKOSCH - im on your phone listening attacking voip c...DEF CON 27 - HUBER AND ROSKOSCH - im on your phone listening attacking voip c...
DEF CON 27 - HUBER AND ROSKOSCH - im on your phone listening attacking voip c...
 
Software Define your Current Storage with Opensource
Software Define your Current Storage with OpensourceSoftware Define your Current Storage with Opensource
Software Define your Current Storage with Opensource
 
180809
180809180809
180809
 
Accelerated .NET Memory Dump Analysis training public slides
Accelerated .NET Memory Dump Analysis training public slidesAccelerated .NET Memory Dump Analysis training public slides
Accelerated .NET Memory Dump Analysis training public slides
 
Cray XT Porting, Scaling, and Optimization Best Practices
Cray XT Porting, Scaling, and Optimization Best PracticesCray XT Porting, Scaling, and Optimization Best Practices
Cray XT Porting, Scaling, and Optimization Best Practices
 
Ubuntu core on bubblegum 96
Ubuntu core on bubblegum 96Ubuntu core on bubblegum 96
Ubuntu core on bubblegum 96
 
Ubuntu core on bubblegum 96
Ubuntu core on bubblegum 96Ubuntu core on bubblegum 96
Ubuntu core on bubblegum 96
 
44CON 2014 - Stupid PCIe Tricks, Joe Fitzpatrick
44CON 2014 - Stupid PCIe Tricks, Joe Fitzpatrick44CON 2014 - Stupid PCIe Tricks, Joe Fitzpatrick
44CON 2014 - Stupid PCIe Tricks, Joe Fitzpatrick
 
Intro to open source telemetry linux con 2016
Intro to open source telemetry   linux con 2016Intro to open source telemetry   linux con 2016
Intro to open source telemetry linux con 2016
 
Basic Linux kernel
Basic Linux kernelBasic Linux kernel
Basic Linux kernel
 
機器學習應用於蔬果辨識
機器學習應用於蔬果辨識機器學習應用於蔬果辨識
機器學習應用於蔬果辨識
 
4 Node.js Gotchas: What your ops team needs to know
4 Node.js Gotchas: What your ops team needs to know4 Node.js Gotchas: What your ops team needs to know
4 Node.js Gotchas: What your ops team needs to know
 
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...
 
PVS-Studio, a solution for resource intensive applications development
PVS-Studio, a solution for resource intensive applications developmentPVS-Studio, a solution for resource intensive applications development
PVS-Studio, a solution for resource intensive applications development
 
Ein Stall voller Trüffelschweine - (PHP-)Profiling-Tools im Überblick
Ein Stall voller Trüffelschweine - (PHP-)Profiling-Tools im ÜberblickEin Stall voller Trüffelschweine - (PHP-)Profiling-Tools im Überblick
Ein Stall voller Trüffelschweine - (PHP-)Profiling-Tools im Überblick
 
PHP QA Tools
PHP QA ToolsPHP QA Tools
PHP QA Tools
 
PGI Compilers & Tools Update- March 2018
PGI Compilers & Tools Update- March 2018PGI Compilers & Tools Update- March 2018
PGI Compilers & Tools Update- March 2018
 
Chicago Docker Meetup Presentation - Mediafly
Chicago Docker Meetup Presentation - MediaflyChicago Docker Meetup Presentation - Mediafly
Chicago Docker Meetup Presentation - Mediafly
 

Recently uploaded

Recently uploaded (20)

A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
 
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfA Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
 
What need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java DevelopersWhat need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java Developers
 
Agnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in KrakówAgnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in Kraków
 
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdfStrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
 
AI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in Michelangelo
 
A Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data MigrationA Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data Migration
 
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdfMicrosoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
 
AI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning Framework
 
SQL Injection Introduction and Prevention
SQL Injection Introduction and PreventionSQL Injection Introduction and Prevention
SQL Injection Introduction and Prevention
 
Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024
 
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product UpdatesGraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
 
INGKA DIGITAL: Linked Metadata by Design
INGKA DIGITAL: Linked Metadata by DesignINGKA DIGITAL: Linked Metadata by Design
INGKA DIGITAL: Linked Metadata by Design
 
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdf
 
5 Reasons Driving Warehouse Management Systems Demand
5 Reasons Driving Warehouse Management Systems Demand5 Reasons Driving Warehouse Management Systems Demand
5 Reasons Driving Warehouse Management Systems Demand
 
The Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion ProductionThe Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion Production
 
how-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdfhow-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdf
 
APVP,apvp apvp High quality supplier safe spot transport, 98% purity
APVP,apvp apvp High quality supplier safe spot transport, 98% purityAPVP,apvp apvp High quality supplier safe spot transport, 98% purity
APVP,apvp apvp High quality supplier safe spot transport, 98% purity
 
Crafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM IntegrationCrafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM Integration
 

20180811 coscup

  • 1. WWW.ANDESTECH.COM Driving Innovations™ Experience on Porting Perf & Ftrace to RISC-V Experience on Porting Perf & Ftrace to RISC-V 高魁良 (Alan Kao) Andes Technology 2018/08/11
  • 2. Confientia Driving Innovations™2 Driving Innovations™ About meAbout me ● Alan (Quey-Liang) Kao ● 2007~2017 NTHU ● BS, MS, P ● 2017.10~ Andes Technology ● software engineer ● RISC-V Linux ● Interests in Linux ● virtualization, container, tracing, perf
  • 3. Confientia Driving Innovations™3 Driving Innovations™ OutlineOutline ● Takeaway: The Big Picture ● Ftrace ● Basics ● Perf ● Basics ● Porting Experiences ● Conclusion
  • 4. Confientia Driving Innovations™4 Driving Innovations™ The Big Picture
  • 5. Confientia Driving Innovations™5 virtualization networking device manager scheduler file system memory management Driving Innovations™ 一般使用者的日常一般使用者的日常 User Space Kernel Space Kernel
  • 6. Confientia Driving Innovations™6 virtualization networking device manager scheduler file system memory management Driving Innovations™ 一般使用者的日常一般使用者的日常 User Space Kernel Space Kernel 根本是黑盒子
  • 7. Confientia Driving Innovations™7 virtualization networking device manager scheduler file system memory management Driving Innovations™ 一般使用者的日常一般使用者的日常 User Space Kernel Space Kernel 如薄紗般的質感 透著一點神秘 根本是黑盒子
  • 8. Confientia Driving Innovations™8 virtualization networking device manager scheduler file system memory management Driving Innovations™ 一般使用者的日常一般使用者的日常 User Space Kernel Space Kernel 如薄紗般的質感 透著一點神秘 根本是黑盒子 若隱若現的是 人們沒親眼見過的遠古傳說
  • 9. Confientia Driving Innovations™9 virtualization networking device manager file system memory management Driving Innovations™ 開發者的異常開發者的異常 User Space Kernel Space Kernel scheduler 程式邏輯穿透 x1 中階元件理解 x1高級元件理解 x1 技能:系統監控 技能:善用工具 技能:資料分析
  • 10. Confientia Driving Innovations™10 virtualization networking device manager file system memory management Driving Innovations™ 工具?工具? User Space Kernel Space Kernel scheduler perf ftrace
  • 11. Confientia Driving Innovations™11 virtualization networking device manager file system memory management Driving Innovations™ 工具?工具? User Space Kernel Space Kernel scheduler perf ftrace read, write special file
  • 12. Confientia Driving Innovations™12 virtualization networking device manager file system memory management Driving Innovations™ kernel 提供工具, 工具監控 kernel 。 kernel 提供工具, 工具監控 kernel 。 User Space Kernel Space Kernel scheduler perf ftrace
  • 13. Confientia Driving Innovations™13 virtualization networking device manager file system memory management Driving Innovations™ User Space Kernel Space Kernel scheduler perf ftrace 就把「監控」想像成 ... if ( 有人要我監控 A) printf(“%d %dn”,A.a, A.b)
  • 14. Confientia Driving Innovations™14 virtualization networking device manager file system memory management Driving Innovations™ User Space Kernel Space Kernel scheduler perf ftrace 就把「監控」想像成 ... if ( 有人要我監控 A) printf(“%d %dn”,A.a, A.b) Probe Efect 被監控的系統的行為本身 會因為受到介入而改變
  • 15. Confientia Driving Innovations™15 virtualization networking device manager file system memory management Driving Innovations™ 監控過程中產生資料監控過程中產生資料 User Space Kernel Space Kernel scheduler perf ftrace iata
  • 16. Confientia Driving Innovations™16 virtualization networking device manager file system memory management Driving Innovations™ 監控過程中產生更多資料監控過程中產生更多資料 User Space Kernel Space Kernel scheduler perf ftrace iataiata iata iata
  • 17. Confientia Driving Innovations™17 virtualization networking device manager file system memory management Driving Innovations™ 然後 kernel 恢復日常運作然後 kernel 恢復日常運作 User Space Kernel Space scheduler iata Kernel iataiata iata iata
  • 18. Confientia Driving Innovations™18 virtualization networking device manager file system memory management Driving Innovations™ 然後可以分析效能瓶頸然後可以分析效能瓶頸 User Space Kernel Space scheduler iata Kernel iataiata iata iata OS: 做投影片的效能瓶頸 是想梗做動畫啊!
  • 19. Confientia Driving Innovations™19 Driving Innovations™ 「劇本」與「角色」「劇本」與「角色」 1. 開發者使用效能監控工具 2. 工具改變 kernel 的部份狀態,進入監控模式 3. 產生監控所得資料 4. 離開監控模式 5. 開發者存取資料以作為分析之用 ● Ftrace ● 監控狀態時行為: 進入每個函數時,綁架正常的執 行流程 ● 產生資料: 函數執行與離開的時間戳記 ● 透過 tracefs 檔案系統取得 trace 內容 ● Perf ● 監控狀態時行為: 排程器在排入行程時啟動、行程 離開時移除 perf 機制 ● 產生資料: 軟體事件、硬體 counter ● perf 工具輸出
  • 20. Confientia Driving Innovations™20 Driving Innovations™ Ftrace Overview
  • 21. Confientia Driving Innovations™21 Driving Innovations™ IntroiuctionIntroiuction ● Function Tracer ● now serves as a framework, under the tracing subsystem ● Originally a tracer in the PREEMPT_RT patchset ● to measure latency ● “Ftrace aiis little overheai to the system when tracing.” -- Survey and Analysis of Kernel and Userspace Tracers on Linux: Design, Implementation, and Overhead, ACM Computing Survey 2018 June
  • 22. Confientia Driving Innovations™22 Driving Innovations™ Basic UsageBasic Usage ● Example 0: go to the right place # cd /sys/kernel/debug/tracing ● Example 1: start function tracer # cat available_tracers function function_graph ... # echo function_graph > current_tracer && echo 1 > tracing_on ● Example 2: set white/black list to trace # cat available_filter_functions ... // some compiler-modified names here # echo foo bar > set_ftrace_filter // white list # echo foo bar > set_ftrace_notrace // black list (superior) ● Example 3: check the trace ● # cat trace // the blocking alternative: trace_pipe
  • 23. Confientia Driving Innovations™23 Driving Innovations™ 角色: Ftrace角色: Ftrace 1. 開發者使用效能監控工具 2. 工具改變 kernel 的部份狀態,進入監控模式 3. 產生監控所得資料 4. 離開監控模式 5. 開發者存取資料以作為分析之用 ● Ftrace ● 監控狀態時行為: 進入每個函數時,綁架正常的執 行流程 ● 產生資料: 函數執行與離開的時間戳記 ● 透過 tracefs 檔案系統取得 trace 內容 ● Perf ● 監控狀態時行為: 排程器在排入行程時啟動、行程 離開時移除 perf 機制 ● 產生資料: 軟體事件、硬體 counter ● perf 工具輸出 要如何綁架 函式的入口與出口? Ftrace porting 重點
  • 24. Confientia Driving Innovations™24 Driving Innovations™ 編譯 kernel 要使用 gcc -pg編譯 kernel 要使用 gcc -pg ● without -pg <foo>: 00: addi sp, sp, -16 04: sd s0, 0(sp) 08: sd ra, 8(sp) 0c: addi s0, sp, 16 10: mv a0, ra ... ● with -pg <foo>: 00: addi sp, sp, -16 04: sd s0, 0(sp) 08: sd ra, 8(sp) 0c: addi s0, sp, 16 10: call _mcount 14: mv a0, ra ...Prologue 進入函式時處理 stack 的階段 儲存返回位址 (ra) 很重要! Epilogue 離開函式的階段 載回返回位址 (ra), ret
  • 25. Confientia Driving Innovations™25 Driving Innovations™ <f2>: 00: addi sp, sp, -8 08: sd ra, 0(sp) 10: call _mcount 18: ld ra, 0(sp) 20: addi sp, sp, 8 28: ret <f1>: ... 40: call f2 ... SP=0x10000100, PC=0x00SP=0x10000100, PC=0x00 stack: 100000D0 100000D8 100000E0 100000E8 100000F0 100000F8 10000100 剛從 f1 呼叫過來 ra = 0x48 呼叫時的下一行 是為返回位址
  • 26. Confientia Driving Innovations™26 Driving Innovations™ SP=0x100000F8, PC=0x08SP=0x100000F8, PC=0x08 stack: 100000D0 100000D8 100000E0 100000E8 100000F0 100000F8 10000100 <f2>: 00: addi sp, sp, -8 08: sd ra, 0(sp) 10: call _mcount 18: ld ra, 0(sp) 20: addi sp, sp, 8 28: ret <f1>: ... 40: call f2 ... ra = 0x48
  • 27. Confientia Driving Innovations™27 Driving Innovations™ SP=0x100000F8, PC=0x10SP=0x100000F8, PC=0x10 stack: 100000D0 100000D8 100000E0 100000E8 100000F0 100000F8 0x48 10000100 <f2>: 00: addi sp, sp, -8 08: sd ra, 0(sp) 10: call _mcount 18: ld ra, 0(sp) 20: addi sp, sp, 8 28: ret <f1>: ... 40: call f2 ... ra = 0x48
  • 28. Confientia Driving Innovations™28 Driving Innovations™ SP=0x100000F8, PC=???SP=0x100000F8, PC=??? stack: 100000D0 100000D8 100000E0 100000E8 100000F0 100000F8 0x48 10000100 <_mcount>: 綁架首部曲:入口の侵蝕 按下第一次碼表 <f2>: 00: addi sp, sp, -8 08: sd ra, 0(sp) 10: call _mcount 18: ld ra, 0(sp) 20: addi sp, sp, 8 28: ret <f1>: ... 40: call f2 ... 腦內設定: 沒有什麼特別的意味 隱藏資料結構 : 0.1 ra = 0x18 請原諒我使用 psuedo code 和日文
  • 29. Confientia Driving Innovations™29 <f2>: 00: addi sp, sp, -8 08: sd ra, 0(sp) 10: call _mcount 18: ld ra, 0(sp) 20: addi sp, sp, 8 28: ret <f1>: ... 40: call f2 ... Driving Innovations™ SP=0x100000F8, PC=???SP=0x100000F8, PC=??? stack: 100000D0 100000D8 100000E0 100000E8 100000F0 100000F8 0x80 10000100 <_mcount>: ... 綁架二部曲:回傳の支配 隱藏資料結構 : 0.1 0x48 「難道說 ... 我們兩個人 ... 返回位址交換了???」 ---- 你的啦 (ra) ra = 0x18
  • 30. Confientia Driving Innovations™30 <f2>: 00: addi sp, sp, -8 08: sd ra, 0(sp) 10: call _mcount 18: ld ra, 0(sp) 20: addi sp, sp, 8 28: ret <f1>: ... 40: call f2 ... Driving Innovations™ SP=0x100000F8, PC=???SP=0x100000F8, PC=??? stack: 100000D0 100000D8 100000E0 100000E8 100000F0 100000F8 0x80 10000100 <_mcount>: ... 綁架三部曲:本體の回歸 ret 隱藏資料結構 : 0.1 0x48 ra = 0x18
  • 31. Confientia Driving Innovations™31 <f2>: 00: addi sp, sp, -8 08: sd ra, 0(sp) 10: call _mcount 18: ld ra, 0(sp) 20: addi sp, sp, 8 28: ret <f1>: ... 40: call f2 ... Driving Innovations™ SP=0x100000F8, PC=0x18SP=0x100000F8, PC=0x18 stack: 100000D0 100000D8 100000E0 100000E8 100000F0 100000F8 0x80 10000100 隱藏資料結構 : 0.1 0x48 ra = 0x18
  • 32. Confientia Driving Innovations™32 <f2>: 00: addi sp, sp, -8 08: sd ra, 0(sp) 10: call _mcount 18: ld ra, 0(sp) 20: addi sp, sp, 8 28: ret <f1>: ... 40: call f2 ... Driving Innovations™ SP=0x100000F8, PC=0x20SP=0x100000F8, PC=0x20 stack: 100000D0 100000D8 100000E0 100000E8 100000F0 100000F8 0x80 10000100 隱藏資料結構 : 0.1 0x48 ra = 0x80
  • 33. Confientia Driving Innovations™33 <f2>: 00: addi sp, sp, -8 08: sd ra, 0(sp) 10: call _mcount 18: ld ra, 0(sp) 20: addi sp, sp, 8 28: ret <f1>: ... 40: call f2 ... Driving Innovations™ SP=0x10000100, PC=0x28SP=0x10000100, PC=0x28 stack: 100000D0 100000D8 100000E0 100000E8 100000F0 100000F8 0x80 10000100 隱藏資料結構 : 0.1 0x48 ra = 0x80
  • 34. Confientia Driving Innovations™34 <f2>: 00: addi sp, sp, -8 08: sd ra, 0(sp) 10: call _mcount 18: ld ra, 0(sp) 20: addi sp, sp, 8 28: ret <f1>: ... 40: call f2 ... Driving Innovations™ SP=0x10000100, PC=0x80SP=0x10000100, PC=0x80 stack: 100000D0 100000D8 100000E0 100000E8 100000F0 100000F8 0x80 10000100 隱藏資料結構 : 0.1 0x48 <_mcount_2>: 綁架四部曲:出口の污染 80: 按下第二次碼表 ...0.23 88: 讀取隱藏資料結構 90: 時間差計算 98: 恢復回傳位址 A0:ret ra = 0x80
  • 35. Confientia Driving Innovations™35 <f2>: 00: addi sp, sp, -8 08: sd ra, 0(sp) 10: call _mcount 18: ld ra, 0(sp) 20: addi sp, sp, 8 28: ret <f1>: ... 40: call f2 ... Driving Innovations™ SP=0x10000100, PC=0x88SP=0x10000100, PC=0x88 stack: 100000D0 100000D8 100000E0 100000E8 100000F0 100000F8 0x80 10000100 隱藏資料結構 : 0.1 0x48 <_mcount_2>: 80: 按下第二次碼表 ... 88: 讀取隱藏資料結構 90: 時間差計算 98: 恢復回傳位址 A0:ret ra = 0x80
  • 36. Confientia Driving Innovations™36 <f2>: 00: addi sp, sp, -8 08: sd ra, 0(sp) 10: call _mcount 18: ld ra, 0(sp) 20: addi sp, sp, 8 28: ret <f1>: ... 40: call f2 ... Driving Innovations™ SP=0x10000100, PC=0x90SP=0x10000100, PC=0x90 stack: 100000D0 100000D8 100000E0 100000E8 100000F0 100000F8 0x80 10000100 隱藏資料結構 : 0x48 <_mcount_2>: 80: 按下第二次碼表 ... 88: 讀取隱藏資料結構 90: 時間差計算 ... 0.13 98: 恢復回傳位址 A0:ret ra = 0x80
  • 37. Confientia Driving Innovations™37 <f2>: 00: addi sp, sp, -8 08: sd ra, 0(sp) 10: call _mcount 18: ld ra, 0(sp) 20: addi sp, sp, 8 28: ret <f1>: ... 40: call f2 ... Driving Innovations™ SP=0x10000100, PC=0x98SP=0x10000100, PC=0x98 stack: 100000D0 100000D8 100000E0 100000E8 100000F0 100000F8 0x80 10000100 隱藏資料結構 : 0x48 <_mcount_2>: 80: 按下第二次碼表 ... 88: 讀取隱藏資料結構 90: 時間差計算 98: 恢復回傳位址 A0:ret ra = 0x80
  • 38. Confientia Driving Innovations™38 <f2>: 00: addi sp, sp, -8 08: sd ra, 0(sp) 10: call _mcount 18: ld ra, 0(sp) 20: addi sp, sp, 8 28: ret <f1>: ... 40: call f2 ... Driving Innovations™ SP=0x10000100, PC=0xA0SP=0x10000100, PC=0xA0 stack: 100000D0 100000D8 100000E0 100000E8 100000F0 100000F8 0x80 10000100 ra = 0x48 <_mcount_2>: 80: 按下第二次碼表 ... 88: 讀取隱藏資料結構 90: 時間差計算 98: 恢復回傳位址 A0:ret
  • 39. Confientia Driving Innovations™39 <f2>: 00: addi sp, sp, -8 08: sd ra, 0(sp) 10: call _mcount 18: ld ra, 0(sp) 20: addi sp, sp, 8 28: ret <f1>: ... 40: call f2 ... Driving Innovations™ SP=0x10000100, PC=0x48SP=0x10000100, PC=0x48 stack: 100000D0 100000D8 100000E0 100000E8 100000F0 100000F8 0x80 10000100 綁架最終曲: 就算這樣子走過一輪期待沒摸過 的人一次看懂是不是搞錯了什麼 ra = 0x48
  • 40. Confientia Driving Innovations™40 Driving Innovations™ My frst PATCH!My frst PATCH! commit 10626c32e3827bca560217966f5bd586c4e91584 Author: Alan Kao <alankao@andestech.com> Date: Mon Dec 18 17:52:48 2017 +0800 riscv/ftrace: Aii basic support This patch contains basic ftrace support for RV64I platform. Specifically, function tracer, function graph tracer (HAVE_FUNCTION_GRAPH_TRACER), and ... are implemented following the instructions in Documentation/trace/ftrace-design.txt. Note that the functions in both ftrace.c and setup.c should not be hooked with the compiler's -pg option: to prevent infinite self-referencing for the former, and to ignore early setup stuf for the latter. Signed-of-by: Alan Kao <alankao@andestech.com> Signed-of-by: Palmer Dabbelt <palmer@sifive.com>
  • 41. Confientia Driving Innovations™41 Driving Innovations™ [PATCHv4 3/6][PATCHv4 3/6] commit bc1a4c3a842556852bb02039887d73899d513532 Author: Alan Kao <alankao@andestech.com> Date: Tue Feb 13 13:13:18 2018 +0800 riscv/ftrace: Aii iynamic function graph tracer support Once the function_graph tracer is enabled, a filtered function has the following call sequence: * ftracer_caller ==> on/of by ftrace_make_call/ftrace_make_nop * ftrace_graph_caller * ftrace_graph_call ==> on/of by ftrace_en/disable_ftrace_graph_caller * prepare_ftrace_return Considering the following DYNAMIC_FTRACE_WITH_REGS feature, it would be more extendable to have a ftrace_graph_caller function, instead of calling prepare_ftrace_return directly in ftrace_caller. Cc: Greentime Hu <greentime@andestech.com> Signed-of-by: Alan Kao <alankao@andestech.com> Signed-of-by: Palmer Dabbelt <palmer@sifive.com>
  • 42. Confientia Driving Innovations™42 Driving Innovations™ Perf Overview
  • 43. Confientia Driving Innovations™43 Driving Innovations™ IntroiuctionIntroiuction ● Perf ● Profiling tool since 2.6.x ● User space: perf CLI tool ● perf stat: counting ● perf record/report: sampling ● ... ● Kernel space: perf event ● kernel/event/core.c ● include/linux/perf_event.h ● arch/xxxx/yyyy/perf_event.c ● CONFIG_PERF_EVENT
  • 44. Confientia Driving Innovations™44 Driving Innovations™ Basic UsageBasic Usage ● Example 0: list events # perf list ● Example 1: collect counters # perf stat -e faults,cycles ls ~ ● Example 2: profiling the system # perf record -e faults ls ~ # perf report # Overhead Command Shared Object Symbol # ........ ....... ................. ................... 14.29% ls [kernel.kallsyms] [k] __asm_copy_to_user 14.29% ls busybox [.] 0x00000000000c1d0e 14.29% ls busybox [.] 0x00000000000eeed2 14.29% ls busybox [.] 0x0000000000118634 11.43% ls busybox [.] 0x00000000000c56ae 8.57% ls busybox [.] 0x00000000001039da 5.71% ls busybox [.] 0x00000000000eec90 2.86% ls [kernel.kallsyms] [k] __clear_user
  • 45. Confientia Driving Innovations™45 Driving Innovations™ 角色: Perf角色: Perf 1. 開發者使用效能監控工具 2. 工具改變 kernel 的部份狀態,進入監控模式 3. 產生監控所得資料 4. 離開監控模式 5. 開發者存取資料以作為分析之用 ● Ftrace ● 監控狀態時行為: 進入每個函數時,綁架正常的執 行流程 ● 產生資料: 函數執行與離開的時間戳記 ● 透過 tracefs 檔案系統取得 trace 內容 ● Perf ● 監控狀態時行為: 排程器在排入行程時啟動、行程 離開時移除 perf 機制 ● 產生資料: 軟體事件、硬體 counter ● perf 工具輸出 Perf porting 重點
  • 46. Confientia Driving Innovations™46 Driving Innovations™ Perf Counting EventPerf Counting Event task6 task1 task4 task3 task2 task1 ... ... event_init 白話:我想要監控 cache miss 數量 sched_in 將 cache miss 的硬體 counter 值記錄在 prev 變數中 sched_out 讀取 cache miss counter 與 prev 取差值 儲存至 count 變數 exit 回傳 count 即為所求
  • 47. Confientia Driving Innovations™47 Driving Innovations™ [PATCH 1/2] Coies[PATCH 1/2] Coies commit d9695108ebad944f32c72c90bc94142c4f44454f Author: Alan Kao <alankao@andestech.com> Date: Fri Apr 20 07:27:49 2018 +0800 perf: riscv: preliminary RISC-V support This patch provide a basic PMU, riscv_base_pmu, which supports two general hariware event, instructions and cycles. Furthermore, this PMU serves as a reference implementation to ease the portings in the future. riscv_base_pmu shouli be able to run on any RISC-V machine that conforms to the Priv-Spec. Note that the latest qemu model hasn't fully support a proper behavior of Priv-Spec 1.10 yet, but work around should be easy with very small fixes. Please check https://github.com/riscv/riscv-qemu/pull/115 for future updates.
  • 48. Confientia Driving Innovations™48 Driving Innovations™ [PATCH 2/2] Doc[PATCH 2/2] Doc commit 6f2490406927492c392e1b51f5706a2a0b8e4a1b Author: Alan Kao <alankao@andestech.com> Date: Fri Apr 20 07:27:50 2018 +0800 perf: riscv: Aii Document for Future Porting Guiie Reviewed-by: Alex Solomatnikov <sols@sifive.com> Cc: Nick Hu <nickhu@andestech.com> Cc: Greentime Hu <greentime@andestech.com> Signed-of-by: Alan Kao <alankao@andestech.com> Signed-of-by: Palmer Dabbelt <palmer@sifive.com>
  • 49. Confientia Driving Innovations™49 Driving Innovations™ Porting Experience(s)
  • 50. Confientia Driving Innovations™50 Driving Innovations™ 1. 人在江湖,搞清楚遊戲規則1. 人在江湖,搞清楚遊戲規則 ● 前置準備: patch第一個 ● 媒介: Linux 社群用 Mailing List 散 佈、討論 patch ● 工具:英文練好、 git 用熟 ● 品質:發送 patch 或是 patchset 之前 務必要檢查 ● 應對: Maintainer 不會故意刁難,理 直氣和地討論即可
  • 51. Confientia Driving Innovations™51 Driving Innovations™ 2. 做好功課2. 做好功課 ● WHAT :你在意的程式片段是什麼? ● 在 source tree 找答案 ● HOW :那些程式碼是怎麼形成的? ● 用 git log/blame 找答案 ● WHY :為什麼變成現在這樣? ● 爬 mailing list history 找答案 ● 沒有頭緒?整個 mailing list 訂下去
  • 52. Confientia Driving Innovations™52 Driving Innovations™ 3. 抄3. 抄 ● 關於 arch-dep 移植 ... ● 各種 CPU 架構很多,你 ( 正 ) 煩惱的, 他們搞不好都解了 ● 大家也都是抄來抄去的 ● 注意遊戲規則,不能抄了又說是原創 ● 不要抄 x86
  • 53. Confientia Driving Innovations™53 Driving Innovations™ 4. 觸類旁通4. 觸類旁通 ● 雖然可以閉門造車做 arch-dep 移植 ... ● 但是多看 generic 部份、 userspace 部份都有幫助 ● 提昇理解的深度 ● 要是觸上癮了 ... ● C 函式庫(一個 glibc 不夠,你試過 newlib 嗎?) ● 各種 Unix-like 表兄弟作業系統
  • 54. Confientia Driving Innovations™54 Driving Innovations™ 5. 自己的開發環境自己守護5. 自己的開發環境自己守護 ● 工具鏈、核心、根目錄、模擬器 ... ● RISC-V 天地初開時,真的很容易炸 ● 不要以為 git 就無敵 ● 遇到控制狂上游,天天 force update ● 遇到放置型上游,一整年都不更新 ● RISC-V 社群兩種都有 ● 我自己都用 docker 定期存 image ● 容量肥大但好用
  • 55. Confientia Driving Innovations™55 Driving Innovations™ Conclusion
  • 56. Confientia Driving Innovations™56 Driving Innovations™ ConclusionConclusion ● 真的很抱歉把主題訂太大了 ● perf 和 ftrace 都是很實用的工 具,值得摸摸看 ● 如果對這兩個主題或 Linux 貢獻有 興趣,歡迎聯絡我 ● alankao@andestech.com ● Happy Hacking!
  • 57. Confientia Driving Innovations™57 Driving Innovations™ Ftrace ReferencesFtrace References ● Ftrace Kernel Hooks: More than just tracing by Steven Rostedt ● 知乎軟件架構設計專題 Linux 2 ftrace在 下做性能分析 : ● 完成 RISC-V 環境架設: busybear官方版本的 ; outdated docker稍微有點 的 版本
  • 58. Confientia Driving Innovations™58 Driving Innovations™ Perf ReferencesPerf References ● perf wiki: tutorial ● Tracing with Perf Tools by Namhyung Kim ● Documentation/riscv/pmu.txt
  • 59. Confientia Driving Innovations™59 Driving Innovations™ Thank You!Thank You!