Dustin Whittle provides a performance testing crash course covering why performance matters, how to understand baseline performance, common performance testing tools like Apache Bench, Siege, and Bees with Machine Guns for load testing, and best practices for treating performance as a key feature throughout development and production. Key aspects include instrumenting all aspects of an application, automating performance testing, and monitoring performance both on the client and server sides.

1. Performance Tes-ng
Crash Course
Dus$n Whi*le

2. Dus-n Whi5le
• dus-nwhi5le.com
• @dus-nwhi5le
• San Francisco, California, USA
• Technologist, Traveler, Pilot, Skier, Diver, Sailor, Golfer

3. Why does performance ma5er?

5. Performance directly impacts
the bo5om line

7. How fast is fast enough?
§ Performance is key to a great user experience
- Under 100ms is perceived as reac$ng instantaneously
- A 100ms to 300ms delay is percep$ble
- 1 second is about the limit for the user's flow of thought to stay
uninterrupted
- Users expect a site to load in 2 seconds
- AMer 3 seconds, 40% will abandon your site.
- 10 seconds is about the limit for keeping the user's a*en$on
§ Modern applica-ons spend more -me in the browser than on the server-side

8. Login
Flight Status
Search Flight
Purchase
Mobile
Big data
SOA
NOSQL
Cloud
Agile
Web
Application complexity is exploding

9. Up-me is cri-cal for enterprises and consumers

10. Treat performance as a feature!

11. Tools of the trade for
performance tes-ng

12. Understand your
baseline performance

13. Sta-c
vs
Hello World
vs
Applica-ons

14. Apache Bench

15. ab -c 1 -t 10 -k
http://acmedemoapp.com/

16. Benchmarking acmedemoapp.com (be patient)
Finished 187 requests
Server Software: Apache-Coyote/1.1
Server Hostname: acmedemoapp.com
Server Port: 80
Document Path: /
Document Length: 5217 bytes
Concurrency Level: 1
Time taken for tests: 10.039 seconds
Complete requests: 187
Failed requests: 0
Keep-Alive requests: 187
Total transferred: 1021768 bytes
HTML transferred: 975579 bytes
Requests per second: 18.63 [#/sec] (mean)
Time per request: 53.687 [ms] (mean)

17. ab -c 10 -t 10 -k
http://acmedemoapp.com/

18. Benchmarking acmedemoapp.com (be patient)
Finished 659 requests
Server Software: Apache-Coyote/1.1
Server Hostname: acmedemoapp.com
Server Port: 80
Document Path: /
Document Length: 5217 bytes
Concurrency Level: 10
Time taken for tests: 10.015 seconds
Complete requests: 659
Failed requests: 0
Keep-Alive requests: 659
Total transferred: 3600776 bytes
HTML transferred: 3438003 bytes
Requests per second: 65.80 [#/sec] (mean)
Time per request: 151.970 [ms] (mean)
Time per request: 15.197 [ms] (mean, across all

19. Siege

20. siege -c 10 -b -t 10S http://acmedemoapp.com/

21. ** SIEGE 3.0.6
** Preparing 10 concurrent users for
battle.
The server is now under siege...
Lifting the server siege... done.
Transactions: 623 hits
Availability: 100.00 %
Elapsed time: 9.57 secs
Data transferred: 3.10 MB
Response time: 0.15 secs
Transaction rate: 65.10 trans/sec
Throughput: 0.32 MB/sec
Concurrency: 9.91
Successful transactions: 623
Failed transactions: 0
Longest transaction: 0.30
Shortest transaction: 0.10

22. Crawl the entire app to
discover all urls

23. sproxy -o ./urls.txt

24. SPROXY v1.02 listening on port 9001
...appending HTTP requests to: ./urls.txt
...default connection timeout: 120 seconds

25. wget -r -l 0 -t 1 --spider -w 1
-e "http_proxy = http://127.0.0.1:9001"
"http://acmedemoapp.com/"
sort -u -o urls.txt urls.txt

26. http://acmedemoapp.com/
http://acmedemoapp.com/about
http://acmedemoapp.com/cart
http://acmedemoapp.com/currency/change/EUR
http://acmedemoapp.com/currency/change/GBP
http://acmedemoapp.com/currency/change/USD
http://acmedemoapp.com/login
http://acmedemoapp.com/register/
http://acmedemoapp.com/t/brand/bookmania
http://acmedemoapp.com/t/category/books
http://acmedemoapp.com/t/category/mugs
http://acmedemoapp.com/t/category/stickers
http://acmedemoapp.com/terms-of-service

27. Benchmark traffic across
all unique urls with siege

28. siege -v -c 50 -i -t 3M -f urls.txt -d 10

29. ** SIEGE 3.0.6
** Preparing 10 concurrent users for battle.
The server is now under siege...
Lifting the server siege... done.
Transactions: 623 hits
Availability: 100.00 %
Elapsed time: 9.57 secs
Data transferred: 3.10 MB
Response time: 0.15 secs
Transaction rate: 65.10 trans/sec
Throughput: 0.32 MB/sec
Concurrency: 9.91
Successful transactions: 623
Failed transactions: 0
Longest transaction: 0.30
Shortest transaction: 0.10

31. Multi-Mechanize is an open source
framework for performance and load
testing

32. pip install multi-mechanize

33. multimech-newproject
demo

34. import requests
class Transaction(object):
def run(self):
r = requests.get(‘http://acmedemoapp.com/)
r.raw.read()

35. import mechanize
import time
class Transaction(object):
def run(self):
br = mechanize.Browser()
br.set_handle_robots(False)
start_timer = time.time()
resp = br.open(‘http://acmedemoapp.com/)
resp.read()
latency = time.time() - start_timer
self.custom_timers['homepage'] = latency
start_timer = time.time()
resp = br.open(‘http://acmedemoapp.com/cart')
resp.read()
latency = time.time() - start_timer
self.custom_timers['cart'] = latency
assert (resp.code == 200)

36. multimech-run demo

38. What about when you need
more than one machine?

39. Who lives in the cloud?

40. Bees with Machine
Guns

41. A utility for arming (creating)
many bees (micro EC2 instances)
to attack (load test)
targets (web applications)

42. pip install
beeswithmachineguns

43. # ~/.boto
[Credentials]
aws_access_key_id=xxx
aws_secret_access_key=xxx
[Boto]
ec2_region_name = us-west-2
ec2_region_endpoint = ec2.us-west-2.amazonaws.com

44. bees up -s 2 -g default -z us-west-2b -i
ami-bc05898c -k appdynamics-
dustinwhittle-aws-us-west-2 -l ec2-user

45. Connecting to the hive.
Attempting to call up 2 bees.
Waiting for bees to load their machine guns...
.
.
.
.
Bee i-3828400c is ready for the attack.
Bee i-3928400d is ready for the attack.
The swarm has assembled 2 bees.

46. bees report

47. Read 2 bees from the roster.
Bee i-3828400c: running @ 54.212.22.176
Bee i-3928400d: running @ 50.112.6.191

48. bees attack -n 1000 -c 50 -u
http://acmedemoapp.com/

49. Read 2 bees from the roster.
Connecting to the hive.
Assembling bees.
Each of 2 bees will fire 500 rounds, 25 at a time.
Stinging URL so it will be cached for the attack.
Organizing the swarm.
…
Offensive complete.
Complete requests: 1000
Requests per second: 306.540000 [#/sec] (mean)
Time per request: 163.112000 [ms] (mean)
50% response time: 151.000000 [ms] (mean)
90% response time: 192.000000 [ms] (mean)
Mission Assessment: Target crushed bee offensive.
The swarm is awaiting new orders.

50. bees attack -n 100000 -c 1000 -u
http://acmedemoapp.com/

51. Read 2 bees from the roster.
Connecting to the hive.
Assembling bees.
Each of 2 bees will fire 50000 rounds, 500 at a time.
Stinging URL so it will be cached for the attack.
Organizing the swarm.
…
Offensive complete.
Complete requests: 100000
Requests per second: 502.420000 [#/sec] (mean)
Time per request: 360.114000 [ms] (mean)
50% response time: 451.000000 [ms] (mean)
90% response time: 402.000000 [ms] (mean)
Mission Assessment: Target crushed bee offensive.
The swarm is awaiting new orders.

52. Read 2 bees from the roster.
Connecting to the hive.
Assembling bees.
Each of 2 bees will fire 50000 rounds, 500 at a time.
Stinging URL so it will be cached for the attack.
Organizing the swarm.
Bee 0 is joining the swarm.
Bee 1 is joining the swarm.
Bee 0 is firing his machine gun. Bang bang!
Bee 0 lost sight of the target (connection timed out).
Bee 1 lost sight of the target (connection timed out).
Offensive complete.
Target timed out without fully responding to 2 bees.
No bees completed the mission. Apparently your bees are
peace-loving hippies.
The swarm is awaiting new orders.

53. bees down

54. Read 2 bees from the roster.
Connecting to the hive.
Calling off the swarm.
Stood down 2 bees.

55. locust.io
https://github.com/locustio/locust

56. pip install locustio

60. There are many tools…
•Gatling.io
•Wrk
•Vegeta
•Tsung
•…

61. What about the client side?

62. In modern web applications more
latency comes from the client-side
than the server-side.

67. npm install psi

69. WBench

70. gem install wbench

71. wbench
http://dustinwhittle.com/

73. Automate client-side
performance tes-ng with
Grunt/Gulp

74. Use Bower (for dependencies),
Grunt/Gulp (for automa-on),
and Yeoman (for bootstrapping)

76. How many people understand exactly
how fast their site runs in produc-on?

77. Track performance in
development and produc-on

78. Instrument everything = code, databases,
caches, queues, third party services, and
infrastructure.

80. Statsd + Graphite
+ Grafana

82. Track performance of
end users

83. webpagetest.org

87. SiteSpeed.io

96. Load tes-ng services
from the cloud

100. Test for failures
• NetFlix Simian Army + Chaos Monkey
• What happens if you lose a caching layer?
• What happens if dependencies slow down?

101. Performance Ma*ers
• Treat performance as a feature
• Using the 14kb Rule for instant loading
• Markup management
• Elimina$ng excess AJAX calls
• Working with and around applica$on cache
• Developing a responsive design + image strategy
• Implemen$ng a good touch-first strategy
• Code management for good produc$on and development experiences
• Using task runners to build and deploy produc$on code

102. Best Prac$ces
• Treat performance as a feature
• Capacity plan and load test the server-side
• Op-mize and performance test the client-side
• Understand your star-ng point
• Instrument everything
• Monitor performance in development and produc-on
• Measure the difference of every change
• Automate performance tes-ng in your build and deployment process
• Understand how failures impact performance

103. The protocols are evolving
• The limita$ons of HTTP/1.X forced us to develop various applica$on
workarounds (sharding, concatena$on, spri$ng, inlining, etc.) to op$mize
performance. However, in the process we’ve also introduced numerous
regressions: poor caching, unnecessary downloads, delayed execu$on, and
more.
• HTTP/2 eliminates the need for these hacks and allows us to both simplify our
applica$ons and deliver improved performance.
• You should unshard, unconcat, and unsprite your assets
• You should switch from inlining to server push
• Read Ilya Grigorik awesome book on browser performance - h*p://hpbn.co/
h*p2

104. Integrate automated performance tes-ng
into con-nuous integra-on for server-side
and client-side

105. Understand the performance implica-ons
of every deployment and package upgrade

106. Monitor end user experience
from end to end in produc-on

107. Ques-ons?

108. Find these slides on SpeakerDeck
h5ps://speakerdeck.com/dus-nwhi5le

109. http://www.appdynamics.com/