This talk approaches how to debug applications nodejs in production with low overhead

2. Node Performance Debug
@rafaelgss
https://blog.rafaelgss.com.br/node-cpu-profiler

3. Performance is a big area
Everything can be improved!
The tests depends of hardware!

4. What part is the bottleneck?
The bottleneck is internal
E.g:
● Synchronous operation
● Non-optimized operation
The bottleneck is external
E.g:
● Database
● External API
● Socket communication

5. Baby Steps
1 How slow is it?
2 Where is the slowness at?
3 Why is slow?
4 Improve
How slow
is it?
Where are
the slow?
Improve
5 Repeat

6. How slow is it?

7. To know if we reach our goals we
need metrics!
● Wrk
● Apache Benchmark (ab)
● ...

8. ● KafkaMeter
● Kafka-load-consumer - made with
in Jacareí
● Function-perf - made with in Jacareí
Event-Driven applications?

9. Test on node-bottleneck!
https://github.com/rafaelgss/node-bottleneck

10. Endpoint

11. Autocannon
How slow is it?

13. Our metrics
Http Router Requests/(10~12sec) Throughput(MB)
Express 148k 18.5

14. Where is the slowness at?

15. We need tools!
● 0x
● Clinic Js
● CDT (Chrome Dev Tools)
● Perf

16. About perf_events
● Profiling tool available only kernel linux, and can be used with eBPF
tools: kprobes, uprobes, etc.
● Measure how many CPU instructions happens in your
application/command. Try: “sudo perf stat ls”

17. Let’s check

18. Getting output by perf
root$ perf record …. write on /tmp/$PID.map (JIT)
root$ chown root /tmp/perf-3870.map // doesn’t exists
root$ perf script > output-perf

19. Node Profiling Options
--perf-basic-prof and --perf-basic-prof-only-function
replace most of functions V8::Function::Call to real
javascript function
Output the files in a format that existing perf tools can
consume.
Here you can see this flag on NodeJs world

20. Let’s check (again)
term1: root$ node --perf-basic-prof-only-functions index.js &
[1] 3870
term1:root$ perf record -F 99 -p 3870 -g -- sleep 20
term2:root$ autocannon -c 10 -d 10 http://localhost:3000/
term1:root$ chown root /tmp/perf-3870.map
term1:root$ perf script > output-perf
root$ cat output-perf

21. We have javascript functions!
But still hard to check

22. Too many traces

23. Solution: FlameGraphs

24. Generating FlameGraph
root$ google-chrome flame.svg
root$ git clone --depth 1
https://github.com/brendangregg/FlameGraph
root$ cd FlameGraph/
root$ ./stackcollapse-perf.pl < ../$PROJECT/output-
perf | ./flamegraph.pl --colors js > flame.svg

25. flame.svg

26. Why is slow?

27. Function that use most time of CPU
express
more than 60% from express

28. Understanding one line
● LazyCompile is an event generated by V8 Compiler, meaning that your
function is being compiled and optimized on-demand.
● Asterisk* This is good news meaning that your code was successfully
compiled to native code (fast) if you see a tilde (~) that means your code is
being interpreted (slow).
● Path & Line - This tells file and line.

29. Improve

30. Improving
Moving from express to fastify as HTTP Router
node-bottleneck-v2

31. Results on V2

32. Our metrics
Http Router Requests/(10~12sec) Throughput(MB)
Express 148k 18.5
Fastify 210k 30.8> 40%

33. What next?

34. Repeat!

35. References
- https://github.com/RafaelGSS/node-bottleneck
- https://github.com/RafaelGSS/kafka-load-consumer
- https://github.com/RafaelGSS/function-perf
- http://www.brendangregg.com/flamegraphs.html
- http://fastify.io/
- https://expressjs.com/
- https://perf.wiki.kernel.org/index.php/Main_Page
- https://github.com/mcollina/autocannon