Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
<jinuk.kim@ifunfactory.com>
2018 iFunFactory Dev Day
(2013-)
• (C++, C#)
• (Python)
(2012-2013)
• (Python)
(2007-2012)
• (C++)
• (C++, C#, Python)
• (...
2018 iFunFactory Dev Day
Contents.
01
02
03
04
05
06
2018 iFunFactory Dev Day
01
2018 iFunFactory Dev Day
?
?
Linux eBPF + BCC
2018 iFunFactory Dev Day
:
02
2018 iFunFactory Dev Day
:
• (2017 )
• +
•
• DB /
• CPU
• (lock)
• (DB, redis, 3rd party API)
•
2018 iFunFactory Dev Day
:
2018 iFunFactory Dev Day
:
• / DB / CPU ( / / / )
• DB
• /
2018 iFunFactory Dev Day
•
.
• /
• / / (fiber) /
•
• /
•
•
2018 iFunFactory Dev Day
• / .
• .
•
•
• / /
2018 iFunFactory Dev Day
03
2018 iFunFactory Dev Day
• /
2018 iFunFactory Dev Day
: CPU
• GNU Compiler Collection (GCC)
• -pg (gprof)
• LLVM/Clang
• -fprofile-instr-generate
• Vis...
2018 iFunFactory Dev Day
• /
• /
•
• /
2018 iFunFactory Dev Day
GProf :
% cumulative self self total
time seconds seconds calls ms/call ms/call name
14.29 0.03 0...
2018 iFunFactory Dev Day
GProf :
2018 iFunFactory Dev Day
• gprof: 30% - 300%
• Valgrind: 400% - 4,000%
• 3rd (.dll, .so)
•
2018 iFunFactory Dev Day
.
/
• (CPU, , , )
•
•
2018 iFunFactory Dev Day
• IO 10ms ?
• ?
• ?
2018 iFunFactory Dev Day
04
2018 iFunFactory Dev Day
CPU
• .
• Linux perf (kernel v2.6+)
• Linux eBPF (kernel v3.18+; v4.9+)
• Google Performance Tool...
2018 iFunFactory Dev Day
• ( 50 - 200 )
• CPU
•
• /
2018 iFunFactory Dev Day
: CPU
Linux v4.13 eBPF + BCC profile:
Flamegraph: / 

https://github.com/brendangregg/FlameGraph
2018 iFunFactory Dev Day
2018 iFunFactory Dev Day
2018 iFunFactory Dev Day


OS
/
( ), OS
2018 iFunFactory Dev Day
OS
OS
OS !
• Linux v4.13 randstruct
• eBPF #define
2018 iFunFactory Dev Day
?
•Disk I/O ?
•CPU ?
• (lock) ?
• API ?
• ?
• ?
2018 iFunFactory Dev Day
Linux
•eBPF (kernel v3.18+; 4.9+ )
•perf (kernel v2.6+; )
Windows
•Event Tracing for Windows (ETW...
2018 iFunFactory Dev Day
?
/ ?
2018 iFunFactory Dev Day


2018 iFunFactory Dev Day
/
2018 iFunFactory Dev Day
Linux eBPF + BCC
• eBPF: /
• BCC: Python / lua / go eBPF
2018 iFunFactory Dev Day
eBPF
OS 

(x86-64/arm64 )
VM
• / ( )
•
• JIT
2018 iFunFactory Dev Day
eBPF: ?
•
• (= )
• ( )
•
( )
2018 iFunFactory Dev Day
BCC
• eBFP Compiler Collection
• Python / lua / go /
• C eBPF
• eBPF
• python / lua / go
2018 iFunFactory Dev Day
eBPF + BCC:
CPU, , I/O,
eBPF + BCC
( )
:
1. ,
2. eBPF + BCC
3.
2018 iFunFactory Dev Day
BCC :
BCC funclatency
/
2018 iFunFactory Dev Day
: eBPF (1)
2018 iFunFactory Dev Day
: eBPF (2)
2018 iFunFactory Dev Day
2018 iFunFactory Dev Day
: BCC + BPF
2018 iFunFactory Dev Day
:
2018 iFunFactory Dev Day
Linux Kernel v4.13+
randstruct .
BCC “bpf_text”
#define randomized_struct_fields_start struct {
#...
2018 iFunFactory Dev Day
05 HTTP API
2018 iFunFactory Dev Day
HTTP API
I/O, DB , ( )
/OS
: flaskr
• Crow HTTP
• CrowDB + MariaDB
• RESTful API
• https://github...
2018 iFunFactory Dev Day
HTTP :
URL / HTTP
HTTP ,

4 

2018 iFunFactory Dev Day
HTTP :
DB
JSON
HTTP
2018 iFunFactory Dev Day
HTTP :
MariaDB DB
HTTP DNS
2018 iFunFactory Dev Day
4 1000
250
≦ 4 ms
2018 iFunFactory Dev Day
siege: HTTP
siege -c 32

-r 16384

-b

http://example.com
URL 32 (-c32)
(-b)
16k (-r 16384)
2018 iFunFactory Dev Day
:
110 /
: 27.5 /
: 36 ms
2018 iFunFactory Dev Day
?
2018 iFunFactory Dev Day
#1: CPU
2018 iFunFactory Dev Day
: CPU ?
eBPF + BCC: profile
•
•
30 , 199 flaskr
sudo profile -p $(pgrep -nx flaskr)

-f

-F 199 3...
2018 iFunFactory Dev Day
(1): Flamegraph
2018 iFunFactory Dev Day
(2):
DNS C (10%)
• __GI_nss_files_servent
(OS )
• TCP (6.3%)
• (3%)
• (2%)
( ) !
2018 iFunFactory Dev Day
CPU ?
Linux htop (= top )
• CPU 42% CPU .
2018 iFunFactory Dev Day
#2:
2018 iFunFactory Dev Day
#2 ?
HTTP API
:
• DB
• SQL
• SQL
• etc.
2018 iFunFactory Dev Day
: DB ?
eBPF + BCC: mysql_qslower:
eBPF + BCC: funclatency
•
MariaDB (mysql_ )

sudo funclatency.p...
2018 iFunFactory Dev Day
: SQL
• : 0.5 ms - 2 ms
• 2 ms ≪ 36 ms ( )
2018 iFunFactory Dev Day
: SQL
• : 0.008 ms - 0.032 ms
• 0.032 ms ≪ 36 ms ( )
2018 iFunFactory Dev Day
2018 iFunFactory Dev Day
#3: DB
2018 iFunFactory Dev Day
:
?
?
• CPU
• OS CPU
:
• OS CPU
• CPU =
2018 iFunFactory Dev Day
:
eBPF + BCC: offcputime
•
•
2018 iFunFactory Dev Day
: Flamegraph
• poll() : C . 78%;
• epoll() : . 16%; boost::asio
2018 iFunFactory Dev Day
: poll() ?
2018 iFunFactory Dev Day
poll()
C++
( )
!
2018 iFunFactory Dev Day
?
•
• :
•
• /
2018 iFunFactory Dev Day
eBPF + BCC: offwaketime
:
•
•
offwaketime -p $(pgrep -nx flaskr) -f
2018 iFunFactory Dev Day
• UDP/IP poll
2018 iFunFactory Dev Day
: UDP ?
2018 iFunFactory Dev Day
UDP ?
HTTP TCP/IP
UDP :
• DHCP
• DNS
• RTP
• etc…
DNS . ( )
2018 iFunFactory Dev Day
: DNS ?
2018 iFunFactory Dev Day
DNS
DNS (C API )
• getnameinfo
• getaddrinfo
• gethostname
• etc…
eBPF + BCC: funclatency
2018 iFunFactory Dev Day
• getnameinfo
• 32ms - 134ms
• (36ms)
2018 iFunFactory Dev Day
: DNS
DNS DNS dnsmasq
: 1,567 /
• 390 / , 2.5 ms
: 14
• : 110 / vs. 1,567 /
2018 iFunFactory Dev Day
: MS Windows + ETW
GUI + /
ETW / Xperf
: The Lost Xperf Documentation on CPU Scheduling
2018 iFunFactory Dev Day
06 

/
2018 iFunFactory Dev Day
(runqlat)
• ( ?)
Disk I/O
• I/O (fileslower)
• (filetop)
• ext4slower, btrfsslower,
2018 iFunFactory Dev Day
(memleak)
• 30 ?
(deadlock_detector)
• mutex
2018 iFunFactory Dev Day
Android /
https://lwn.net/Articles/742363/
2018 iFunFactory Dev Day
Android /
eBPF
x86-64 / arm64
1. 

+ LLVM
2. Android
3. 

4. 

/ /
https://github.com/joelagnel/b...
2018 iFunFactory Dev Day
Android /
BPFd 

(https://github.com/joelagnel/bpfd)
eBPF + BCC (v4.6+; v4.9+)
• Android 8.0 Oreo...
2018 iFunFactory Dev Day

 

eBPF + BCC
Q&A
2018 iFunFactory Dev Day
2018 iFunFactory Dev Day
Thank You!
660, 1 B 606
+82-70-4923-6566 www.ifunfactory.cominfo@ifu...
Nächste SlideShare
Wird geladen in …5
×

[아이펀팩토리] 2018 데브데이 서버위더스 _04 리눅스 게임 서버 성능 분석

1.358 Aufrufe

Veröffentlicht am

2018 아이펀팩토리 데브데이 서버위더스
(iFunFactory DevDay Server,Withus)

제목 : 코드 수정 없이 리눅스 게임 서버 성능 평가
발표자 : 김진욱 CTO
일정 : 2018년 03월 28일
개요 : 서버에 코드 추가 없이도 고수준에서 서버 성능 분석이 가능하다!
실행 중인 서버를 외부에서 관찰하여 최대한 성능을 분석하고 디버깅 하는 법을 공개 합니다.

아래링크를 통해 아이펀팩토리의 더 많은 정보를 얻으실 수 있습니다.
*아이펀팩토리 홈페이지 : https://ifunfactory.com/
*아이펀팩토리 기술 블로그 : https://blog.ifunfactory.com/

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

[아이펀팩토리] 2018 데브데이 서버위더스 _04 리눅스 게임 서버 성능 분석

  1. 1. <jinuk.kim@ifunfactory.com>
  2. 2. 2018 iFunFactory Dev Day (2013-) • (C++, C#) • (Python) (2012-2013) • (Python) (2007-2012) • (C++) • (C++, C#, Python) • (C#)
  3. 3. 2018 iFunFactory Dev Day Contents. 01 02 03 04 05 06
  4. 4. 2018 iFunFactory Dev Day 01
  5. 5. 2018 iFunFactory Dev Day ? ? Linux eBPF + BCC
  6. 6. 2018 iFunFactory Dev Day : 02
  7. 7. 2018 iFunFactory Dev Day : • (2017 ) • + • • DB / • CPU • (lock) • (DB, redis, 3rd party API) •
  8. 8. 2018 iFunFactory Dev Day :
  9. 9. 2018 iFunFactory Dev Day : • / DB / CPU ( / / / ) • DB • /
  10. 10. 2018 iFunFactory Dev Day • . • / • / / (fiber) / • • / • •
  11. 11. 2018 iFunFactory Dev Day • / . • . • • • / /
  12. 12. 2018 iFunFactory Dev Day 03
  13. 13. 2018 iFunFactory Dev Day • /
  14. 14. 2018 iFunFactory Dev Day : CPU • GNU Compiler Collection (GCC) • -pg (gprof) • LLVM/Clang • -fprofile-instr-generate • Visual C++ instrumentation • Valgrind Callgrind
  15. 15. 2018 iFunFactory Dev Day • / • / • • /
  16. 16. 2018 iFunFactory Dev Day GProf : % cumulative self self total time seconds seconds calls ms/call ms/call name 14.29 0.03 0.03 207323 0.00 0.00 crow::json::escape 14.29 0.06 0.03 3059 0.01 0.02 crow::CrowApp::handle 9.52 0.08 0.02 2638 0.01 0.02 crow::json::dump_internal 4.76 0.09 0.01 307190 0.00 0.00 _Hashtable<…>::insert_unique 4.76 0.10 0.01 208356 0.00 0.00 _Hashtable<…>::_M_rehash 4.76 0.11 0.01 19129 0.00 0.00 vector<crow::json::wvalue>::_M_emplace_back_aux 4.76 0.12 0.01 11961 0.00 0.00 crow::Connection<crow::SocketAdaptor, crow::CrowApp>::cancel_deadline_timer
  17. 17. 2018 iFunFactory Dev Day GProf :
  18. 18. 2018 iFunFactory Dev Day • gprof: 30% - 300% • Valgrind: 400% - 4,000% • 3rd (.dll, .so) •
  19. 19. 2018 iFunFactory Dev Day . / • (CPU, , , ) • •
  20. 20. 2018 iFunFactory Dev Day • IO 10ms ? • ? • ?
  21. 21. 2018 iFunFactory Dev Day 04
  22. 22. 2018 iFunFactory Dev Day CPU • . • Linux perf (kernel v2.6+) • Linux eBPF (kernel v3.18+; v4.9+) • Google Performance Tools • Visual C++ : CPU
  23. 23. 2018 iFunFactory Dev Day • ( 50 - 200 ) • CPU • • /
  24. 24. 2018 iFunFactory Dev Day : CPU Linux v4.13 eBPF + BCC profile: Flamegraph: / 
 https://github.com/brendangregg/FlameGraph
  25. 25. 2018 iFunFactory Dev Day
  26. 26. 2018 iFunFactory Dev Day
  27. 27. 2018 iFunFactory Dev Day 
 OS / ( ), OS
  28. 28. 2018 iFunFactory Dev Day OS OS OS ! • Linux v4.13 randstruct • eBPF #define
  29. 29. 2018 iFunFactory Dev Day ? •Disk I/O ? •CPU ? • (lock) ? • API ? • ? • ?
  30. 30. 2018 iFunFactory Dev Day Linux •eBPF (kernel v3.18+; 4.9+ ) •perf (kernel v2.6+; ) Windows •Event Tracing for Windows (ETW; Win 8+) •Windows Event Logging ( ; Win 2k+)
  31. 31. 2018 iFunFactory Dev Day ? / ?
  32. 32. 2018 iFunFactory Dev Day 

  33. 33. 2018 iFunFactory Dev Day /
  34. 34. 2018 iFunFactory Dev Day Linux eBPF + BCC • eBPF: / • BCC: Python / lua / go eBPF
  35. 35. 2018 iFunFactory Dev Day eBPF OS 
 (x86-64/arm64 ) VM • / ( ) • • JIT
  36. 36. 2018 iFunFactory Dev Day eBPF: ? • • (= ) • ( ) • ( )
  37. 37. 2018 iFunFactory Dev Day BCC • eBFP Compiler Collection • Python / lua / go / • C eBPF • eBPF • python / lua / go
  38. 38. 2018 iFunFactory Dev Day eBPF + BCC: CPU, , I/O, eBPF + BCC ( ) : 1. , 2. eBPF + BCC 3.
  39. 39. 2018 iFunFactory Dev Day BCC : BCC funclatency /
  40. 40. 2018 iFunFactory Dev Day : eBPF (1)
  41. 41. 2018 iFunFactory Dev Day : eBPF (2)
  42. 42. 2018 iFunFactory Dev Day
  43. 43. 2018 iFunFactory Dev Day : BCC + BPF
  44. 44. 2018 iFunFactory Dev Day :
  45. 45. 2018 iFunFactory Dev Day Linux Kernel v4.13+ randstruct . BCC “bpf_text” #define randomized_struct_fields_start struct { #define randomized_struct_fields_end };
  46. 46. 2018 iFunFactory Dev Day 05 HTTP API
  47. 47. 2018 iFunFactory Dev Day HTTP API I/O, DB , ( ) /OS : flaskr • Crow HTTP • CrowDB + MariaDB • RESTful API • https://github.com/jinukkim/devday-2018-03
  48. 48. 2018 iFunFactory Dev Day HTTP : URL / HTTP HTTP ,
 4 

  49. 49. 2018 iFunFactory Dev Day HTTP : DB JSON HTTP
  50. 50. 2018 iFunFactory Dev Day HTTP : MariaDB DB HTTP DNS
  51. 51. 2018 iFunFactory Dev Day 4 1000 250 ≦ 4 ms
  52. 52. 2018 iFunFactory Dev Day siege: HTTP siege -c 32
 -r 16384
 -b
 http://example.com URL 32 (-c32) (-b) 16k (-r 16384)
  53. 53. 2018 iFunFactory Dev Day : 110 / : 27.5 / : 36 ms
  54. 54. 2018 iFunFactory Dev Day ?
  55. 55. 2018 iFunFactory Dev Day #1: CPU
  56. 56. 2018 iFunFactory Dev Day : CPU ? eBPF + BCC: profile • • 30 , 199 flaskr sudo profile -p $(pgrep -nx flaskr)
 -f
 -F 199 30

  57. 57. 2018 iFunFactory Dev Day (1): Flamegraph
  58. 58. 2018 iFunFactory Dev Day (2): DNS C (10%) • __GI_nss_files_servent (OS ) • TCP (6.3%) • (3%) • (2%) ( ) !
  59. 59. 2018 iFunFactory Dev Day CPU ? Linux htop (= top ) • CPU 42% CPU .
  60. 60. 2018 iFunFactory Dev Day #2:
  61. 61. 2018 iFunFactory Dev Day #2 ? HTTP API : • DB • SQL • SQL • etc.
  62. 62. 2018 iFunFactory Dev Day : DB ? eBPF + BCC: mysql_qslower: eBPF + BCC: funclatency • MariaDB (mysql_ )
 sudo funclatency.py -p $(pgrep -nx flaskr)
 -F /usr/lib/mariadb/libmariadb.so:mysql_*
  63. 63. 2018 iFunFactory Dev Day : SQL • : 0.5 ms - 2 ms • 2 ms ≪ 36 ms ( )
  64. 64. 2018 iFunFactory Dev Day : SQL • : 0.008 ms - 0.032 ms • 0.032 ms ≪ 36 ms ( )
  65. 65. 2018 iFunFactory Dev Day
  66. 66. 2018 iFunFactory Dev Day #3: DB
  67. 67. 2018 iFunFactory Dev Day : ? ? • CPU • OS CPU : • OS CPU • CPU =
  68. 68. 2018 iFunFactory Dev Day : eBPF + BCC: offcputime • •
  69. 69. 2018 iFunFactory Dev Day : Flamegraph • poll() : C . 78%; • epoll() : . 16%; boost::asio
  70. 70. 2018 iFunFactory Dev Day : poll() ?
  71. 71. 2018 iFunFactory Dev Day poll() C++ ( ) !
  72. 72. 2018 iFunFactory Dev Day ? • • : • • /
  73. 73. 2018 iFunFactory Dev Day eBPF + BCC: offwaketime : • • offwaketime -p $(pgrep -nx flaskr) -f
  74. 74. 2018 iFunFactory Dev Day • UDP/IP poll
  75. 75. 2018 iFunFactory Dev Day : UDP ?
  76. 76. 2018 iFunFactory Dev Day UDP ? HTTP TCP/IP UDP : • DHCP • DNS • RTP • etc… DNS . ( )
  77. 77. 2018 iFunFactory Dev Day : DNS ?
  78. 78. 2018 iFunFactory Dev Day DNS DNS (C API ) • getnameinfo • getaddrinfo • gethostname • etc… eBPF + BCC: funclatency
  79. 79. 2018 iFunFactory Dev Day • getnameinfo • 32ms - 134ms • (36ms)
  80. 80. 2018 iFunFactory Dev Day : DNS DNS DNS dnsmasq : 1,567 / • 390 / , 2.5 ms : 14 • : 110 / vs. 1,567 /
  81. 81. 2018 iFunFactory Dev Day : MS Windows + ETW GUI + / ETW / Xperf : The Lost Xperf Documentation on CPU Scheduling
  82. 82. 2018 iFunFactory Dev Day 06 
 /
  83. 83. 2018 iFunFactory Dev Day (runqlat) • ( ?) Disk I/O • I/O (fileslower) • (filetop) • ext4slower, btrfsslower,
  84. 84. 2018 iFunFactory Dev Day (memleak) • 30 ? (deadlock_detector) • mutex
  85. 85. 2018 iFunFactory Dev Day Android / https://lwn.net/Articles/742363/
  86. 86. 2018 iFunFactory Dev Day Android / eBPF x86-64 / arm64 1. 
 + LLVM 2. Android 3. 
 4. 
 / / https://github.com/joelagnel/bpfd
  87. 87. 2018 iFunFactory Dev Day Android / BPFd 
 (https://github.com/joelagnel/bpfd) eBPF + BCC (v4.6+; v4.9+) • Android 8.0 Oreo v4.4 AOSP 8.0 v4.10;
  88. 88. 2018 iFunFactory Dev Day 
 
 eBPF + BCC
  89. 89. Q&A 2018 iFunFactory Dev Day 2018 iFunFactory Dev Day Thank You! 660, 1 B 606 +82-70-4923-6566 www.ifunfactory.cominfo@ifunfactory.com

×