Writing your application is one thing. Making the application to perform well is another. We usually forget there is somebody else on the other side of the screen, that becomes very frustrated and upset when he needs to wait until this one page finally loads. It requires a lot of experience to predict specific behaviour and to know what kind of things to avoid. And even with that there is so many different factors that can affect the end user experience. During this talk I will talk about tools and techniques you can use to measure and monitor your application performance.

1. Sebastian Marek, Internal Systems Technical Architect
MANAGING AND
MONITORING APPLICATION
PERFORMANCE
http://www.flickr.com/photos/wraith34/9377593973/

2. a Pole living in Sheffield
over 12 years in software
development
big fan of process automation
TDD and CI
occasionally contributes to open-
source projects
wants to be a knight @proofek
h"ps://joind.in/9301

3. HTTPS://WWW.CODECLUB.ORG.UK/
A nationwide network of volunteer-led after
school coding clubs for children aged 9-11
Volunteer Code Club Projects Venue Children
+ + +

4. PERFORMANCE MONITORING
Overview
network level
system level
application level
End-user level
http://www.flickr.com/photos/global-jet/483825828/

5. PERFORMANCE MONITORING
Network level
failed TCP connections and packet
losses
bandwidth usage
firewalls and loadbalancers
connectivity to external serviceshttp://www.flickr.com/photos/sbrown6079/2038502257/

6. PERFORMANCE MONITORING
System level
CPU load
memory consumption
disk space
http://www.flickr.com/photos/viagallery/3178699697/

7. PERFORMANCE MONITORING
Application level
new deployments
response time
key transactions
memory usage
http://www.flickr.com/photos/qualityfrog/3546657245/

8. PERFORMANCE MONITORING
End-user level
response time
user experience
http://www.flickr.com/photos/alesk/356136498/

9. remote debugging
code coverage
tracing
profiling
Xdebug

10. Installation
XDEBUG
# pecl install xdebug
zend_extension=”/usr/lib/php/extensions/xdebug.so”
configure in xdebug.ini
web server restart (optional)

11. Configuration
XDEBUG
;Profiling settings (always on)
xdebug.profiler_enable = 1
xdebug.profiler_output_dir = /var/log/xdebug/
;Profiling settings (triggered by GET/POST/COOKIE)
xdebug.profiler_enable = 0
xdebug.profiler_output_dir = /var/log/xdebug/
xdebug.profiler_enable_trigger = 1

12. Browser integration
XDEBUG

13. Profiling data
XDEBUG
fl=php:internal
fn=php::spl_autoload_register
881 3
fl=/Users/Sebastian/git/app/src/include/HTMLPurifier/HTMLPurifier.standalone.php
fn=HTMLPurifier_Bootstrap::registerAutoload
878 21
cfl=php:internal
cfn=php::spl_autoload_functions
calls=1 0 0
880 1
cfl=php:internal
cfn=php::spl_autoload_register
calls=1 0 0
881 3

14. Profiling tools
XDEBUG
KCacheGrind for *nix systems
qcachegrind for MacOsX (and apparently Windows)
MacCallGrind for MacOsX
WebGrind via browser

15. qCacheGrind
XDEBUG

16. WebGrind
XDEBUG

17. profiling
web interface
xhProf

18. Installation
XHPROF
# pecl install xhprof
extension=”/usr/lib/php/extensions/xhprof.so”
configure in xhprof.ini
web server restart (optional)

19. Configuration
XHPROF
; XHProf
extension = xhprof.so
xhprof.output_dir = /var/log/xhprof/

20. Profiling tools
XHPROF
default web interface
xhGui (old, also known as XH UI)
xhGui (new, with MongoDB backend)

21. Default web interface - Installation
XHPROF
git clone git@github.com:facebook/xhprof.git

22. Default web interface - Installation
XHPROF
;Webserver (VirtualHost)
<VirtualHost *:80>
DocumentRoot “/Users/Sebastian/git/xhprof/xhprof_html”
ServerName xhprof.localdomain
<Directory /Users/Sebastian/git/xhprof/xhprof_html>
Order allow,deny
Allow from all
</Directory>
ErrorLog “/private/var/log/apache2/xhprof-error_log”
CustomLog “/private/var/log/apache2/xhprof-access_log” common
</VirtualHost>

23. Default web interface - Installation
XHPROF
;auto_prepend file - enable xhprof
$xhProfHome = __DIR__ . ‘/..’;
$profiler_namespace = ‘myapp’; // namespace for your application
if (extension_loaded(‘xhprof’)) {
include_once “$xhProfHome/xhprof_lib/utils/xhprof_lib.php”;
include_once “$xhProfHome/xhprof_lib/utils/xhprof_runs.php”;
xhprof_enable(XHPROF_FLAGS_CPU | XHPROF_FLAGS_MEMORY);
}

24. Default web interface - Installation
XHPROF
;auto_prepend file - collect data
function xhprofShutdownFunction()
{
global $profiler_namespace;
$xhProfUrl = “http://xhprof.localdomain/”;
if (extension_loaded(‘xhprof’)) {
$xhprof_data = xhprof_disable();
$xhprof_runs = new XHProfRuns_Default();
$run_id = $xhprof_runs->save_run($xhprof_data, $profiler_namespace);
$profiler_url = sprintf(“$xhProfUrl/index.php?run=%s&source=%s”, $run_id, $profiler_namespace);
echo “<a href=’$profiler_url’ target=‘_blank’>Profiler output</a>”;
}
}
register_shutdown_function(‘xhprofShutdownFunction’);

25. Default web interface - Installation
XHPROF
;enable profiling in your application (VirtualHost)
<VirtualHost *:80>
DocumentRoot “/Users/Sebastian/git/my_app/src”
ServerName my_app.localdomain
<Directory /Users/Sebastian/git/my_app/src >
Order allow,deny
Allow from all
</Directory>
php_admin_value auto_prepend_file /Users/Sebastian/git/xhprof/xhprof_html/header.php
</VirtualHost>

26. Default web interface - Home page
XHPROF

27. Default web interface - Drilldown
XHPROF

28. Default web interface - Call graph
XHPROF

29. Default web interface - Full Call graph
XHPROF

30. XH UI - Installation
XHPROF
git clone git@github.com:preinheimer/xhprof.git

31. XH UI - Installation
XHPROF
;Webserver (VirtualHost)
<VirtualHost *:80>
DocumentRoot “/Users/Sebastian/git/xhui/xhprof_html”
ServerName xhui.localdomain
<Directory /Users/Sebastian/git/xhui/xhprof_html>
Order allow,deny
Allow from all
</Directory>
ErrorLog “/private/var/log/apache2/xhui-error_log”
CustomLog “/private/var/log/apache2/xhui-access_log” common
</VirtualHost>

32. XH UI - Configuration
XHPROF
# vim xhui/xhprof_lib/config.php
and reuse already provided xhui/external/header.php

33. XH UI - Installation
XHPROF
;enable profiling in your application (VirtualHost)
<VirtualHost *:80>
DocumentRoot “/Users/Sebastian/git/my_app/src”
ServerName my_app.localdomain
<Directory /Users/Sebastian/git/my_app/src >
Order allow,deny
Allow from all
</Directory>
php_admin_value auto_prepend_file /Users/Sebastian/git/xhui/external/header.php
</VirtualHost>

34. XH UI - Home page
XHPROF

35. XH UI - Transaction detail
XHPROF

36. XH UI - Transaction detail
XHPROF

37. XHG - Installation
XHPROF
git clone git@github.com:preinheimer/xhgui.git

38. XHG - Installation
XHPROF
;Webserver (VirtualHost)
<VirtualHost *:80>
DocumentRoot “/Users/Sebastian/git/xhgui/webroot”
ServerName xhgui.localdomain
<Directory /Users/Sebastian/git/xhgui/webroot>
Order allow,deny
Allow from all
AllowOverride All
</Directory>
ErrorLog “/private/var/log/apache2/xhgui-error_log”
CustomLog “/private/var/log/apache2/xhgui-access_log” common
</VirtualHost>

39. XHG - Requirements
XHPROF
MongoDB server
PHP MongoDB extension
PHP mcrypt extension

40. XHG - Installation
XHPROF
# cd xhgui
# php install.php

41. XHG - Installation
XHPROF
;enable profiling in your application (VirtualHost)
<VirtualHost *:80>
DocumentRoot “/Users/Sebastian/git/my_app/src”
ServerName my_app.localdomain
<Directory /Users/Sebastian/git/my_app/src >
Order allow,deny
Allow from all
</Directory>
php_admin_value auto_prepend_file /Users/Sebastian/git/xhgui/external/header.php
</VirtualHost>

42. XHG - Home page
XHPROF

43. XHG - Transaction trace
XHPROF

44. XHG - Transaction trace
XHPROF

45. XHG - Transaction trace
XHPROF

46. XHG - Transaction history
XHPROF

47. XHG - Transaction call graph
XHPROF

48. Glossary
PROFILING
call count - number of times a function/request was called
wt (Wall Time/Wall Clock Time) - time it took to execute a function/request
cpu - CPU time spent executing a function/request
mu - Amount of memory used during function/request execution
pmu - The peak amount of memory used during function/request execution
exclusive - total time spent excluding time to execute other functions
inclusive - total time spent including time to execute other functions

49. Process
PROFILING
Profile and record the results
Identify functions/requests using the most CPU time/memory (exclusive)
Find out what’s causing the issue
Refactor and optimize the code - look for external resources - database, web
services, filesystem
Profile and compare the results
start all over again

50. server monitoring
application monitoring
real user monitoring
NewRelic

51. Supported web applications
NEW RELIC

52. Supported PHP framework
NEW RELIC

53. Installation
NEW RELIC
# RHEL
$ rpm -Uvh http://yum.newrelic.com/pub/newrelic/el5/x86_64/newrelic-
repo-5-3.noarch.rpm
$ yum install newrelic-php5
$ /usr/bin/newrelic-install
# Ubuntu / Debian
$ wget -O - https://download.newrelic.com/548C16BF.gpg | sudo apt-key add -
deb http://apt.newrelic.com/debian/ newrelic non-free
$ apt-get install newrelic-php5
$ /usr/bin/newrelic-install

54. Server Monitoring
NEW RELIC

55. Server Monitoring - Main Dashboard
NEW RELIC
Portal01
App01
App02
Portal02
SugarCRM
SugarCRM
Support Centre

56. Server Monitoring - Overview
NEW RELIC
App02
App02
SugarCRM (App02)

57. Application Monitoring
NEW RELIC

58. Server Trace - Summary
NEW RELIC
App01(5.3.19+a2h(2.2.3):SugarCRM)

59. Server Trace - Details
NEW RELIC

60. Server Trace - SQL queries
NEW RELIC

61. Application Monitoring - Error Tracking
NEW RELIC

62. Real User Monitoring - App Dashboard
NEW RELIC

63. Understanding RUM
NEW RELIC
Web application - The time spent in the application code
Network - The time it takes for a request to make a round-trip
over the internet.
DOM processing - Time spent in the browser parsing and
interpreting the HTML.
Page rendering - Time spent in the browser displaying the
HTML, running in-line JavaScript and loading images.

64. Understanding RUM
NEW RELIC
Source: http://blog.newrelic.com

65. Real User Monitoring - Browser Page Load
NEW RELIC

66. Real User Monitoring - Browser Page Load
NEW RELIC

67. Apdex
NEW RELIC

68. Apdex
NEW RELIC
0 T F
Satisfied zone: < T
Tolerating zone: T - F (4T)
Frustrated zone: > F

69. Apdex score
NEW RELIC
Satisfied customers + (Tolerating customers/2)
Total number of customers

70. Apdex by regions
NEW RELIC

71. Apdex - baseline
NEW RELIC
check the average response time
set your Apdex to average response time
find areas that require performance fixes
when performance improves lower your Apdex

72. New Relic - Map
NEW RELIC
memcached.local
crm.local

73. Keeping on top of the changes
NEW RELIC

74. Server Monitoring - plugins
NEW RELIC

75. Server Monitoring - APC
NEW RELIC

76. Server Monitoring - MySQL
NEW RELIC

77. Server Monitoring - Apache
NEW RELIC

78. PERFORMANCE MONITORING
Summary
Xdebug - application profiling during development stage
xhProf - application profiling during development stage and on
live production platform
New Relic - end-to-end application monitoring on live
production platfrom

79. PERFORMANCE MONITORING
Alternatives
DataDog - http://www.datadoghq.com/product/
GraphDat - http://www.graphdat.com
Scout - https://scoutapp.com/
Nagios - http://www.nagios.org/
Graphite - http://graphite.wikidot.com

80. Resources
Xdebug - http://www.xdebug.org/docs/profiler
kCacheGrind - http://kcachegrind.sourceforge.net
WebGrind - https://github.com/jokkedk/webgrind
MacCallGrind for MacOsX - http://www.maccallgrind.com
xhProf - https://github.com/facebook/xhprof
xhUI - https://github.com/preinheimer/xhprof
xhGui - https://github.com/preinheimer/xhgui
New Relic - http://newrelic.com
New Relic Documentation - https://newrelic.com/docs
Apdex - http://apdex.org
http://techportal.inviqa.com/2009/12/01/profiling-with-xhprof/
http://techportal.inviqa.com/2013/10/01/profiling-php-applications-with-xhgui/

81. Q & A
h"ps://joind.in/9301