1. Uncertainty is a recipe for failure!
TSUNG LOAD TESTING
Testing HTTP(S)/ Databases / LDAP/ XMPP..
<insert your plugin here>
January 2013 Steffen Larsen (slarsen@braintrust)
twitter: @zooldk
Saturday, January 26, 13

2. TSUNG INTRO
Introduction to Tsung
Walk through the phases of a TSUNG load test
Mainly talk about using TSUNG on the HTTP protocol
Show simple use cases (my installation is fcked - sorry!)
Drink a beer!
Saturday, January 26, 13

3. TSUNG INTRO The purpose of Tsung is to simulate users in order to test the scalability and performance of
IP based client/server applications.
Open Source (GPLv2)
Based on Erlang which implicitly defines it as:
Highly concurrent and blazingly fast..
Small memory fingerprint (512 bytes for an erlang process vs. 512k default for
a JVM stack for a thread)
Distributed by the nature of Erlang and easily done (so we can do DDOS!) :-)
Multi protocol: HTTP (REST, SOAP,WebDav etc), PostgreSQL, MySQL, LDAP,
XMPP/Jabber. Support for SSL and Cookie handling
Easy Setup! - no confusing GUI (vi and a DTD is enough). Xorg might not be
available on the servers.
Saturday, January 26, 13

4. TSUNG IN HTTP LAND
Supports both old HTTP 1.0 and 1.1 (Web Sockets on a
branch)
GET, POST, PUT, DELETE etc. Methods support
Cookies are automatically handled
WWW basic authentication
Basic support for SOAP using HTTP mode
Proxy mode to record sessions using a web browser..
Saturday, January 26, 13

5. TSUNG LIFE CYCLE
0.write your tsung.xml 1.run
2.tsung gen. data
XML Tsung server
3.tsung measures
reports
4.report generation
Saturday, January 26, 13

6. TSUNG PHASES..
Define load clients (localhost or multiple for massive/distributed testing)
Define servers you want to load test
Define the load scenario (num. users, arrivals etc, timing)
Define your monitors (servers you are load testing AND related systems)
Record or define your user scenarios (proxy and record or manually edit config)
Run! (tsung -f config.xml start)
Grab a beer.. (cheers!)
Generate Graphs (for the data reported by tsung demon)
Saturday, January 26, 13

7. DEMO
show: simple_example.xml config
run: tsung -f simple_example.xml start
see the log file (default $HOME/.tsung/log/<timestamp>
Saturday, January 26, 13

8. RECORDING..
You can either define you data by hand in the configuration
file or record it via. a proxy and a web browser.
tsung-recorder -L 9000 start (recorder will listen on port
9000)
Now start you browser, configure a new proxy and hit the
page you want to record.
Use the out put in your tsung xml file and voila!
Saturday, January 26, 13

9. WHY TSUNG?
Your own scripts with curl, wget.. - Stop that!
AB Apache Bench (slow as he..)
HTTPPerf (better but still not fast and not distributed)
JMeter (Java based) - quite good, Nice GUI, distributed! but
JVM mem and consumption !
Grinder (Java based)??
Gatling-Tools (JVM, scala, looks promising..using the akka
actor principle for concurrency - instead of normal threads -
similar to erlang)
Saturday, January 26, 13

10. BURNING SERVERS
0/1? load testing or load generation!
0 VS 1
Saturday, January 26, 13

11. MONITOR YOUR LOAD!
Default SNMP OIDS for CPU usage, network traffic, load,
memory etc. - No need to install a special agent on the tested
server
Define your own OIDs! (patch in 1.4.x done by my poor
erlang skills)
With this you can measure your JVM, disk, IO etc.
Network always the problem... there should be open for
SNMP and the ports your are testing!
Saturday, January 26, 13

12. GRAPHS!
Tsung have now generated data in the log directory (default
$HOME/.tsung/log.. Can be set different with -d option)
/usr/local/Cellar/tsung/1.4.2/lib/tsung/bin/tsung_stats.pl
<log>
Now HTML is ready and can be served and showed off
Saturday, January 26, 13

13. DEMO
tsung -f monitor_example.xml start
see the log file (default $HOME/.tsung/log/<timestamp>
Generate graphs from your monitors:
/usr/local/Cellar/tsung/1.4.2/lib/tsung/bin/
tsung_stats.pl <log>
Saturday, January 26, 13

14. TSUNG XML CONFIG
more advanced stuff
Many options: set browser agents, probability for hit
Get content from files (CVS or.. define your pattern)
Mix files or code into config with substitution
Use xpath or jsonpath to MATCH on the response and make
you action!
Use regular expressions as well.. (GNU RegExp)
Programmatically (use conditionals: for loops, if, repeats or
variables etc. for testing)
Saturday, January 26, 13

15. DISTRIBUTED TESTS
Tsung distributed load testing is based on SSH shell and
erlang distribution system.
Distribution is easy! just add more clients!
Remember to make network accessible for the controller
and workers - ssh login with a key from controller to all the
slave nodes (workers) should be available
Same version of tsung/erlang on EVERY machine
(controller/workers)
Saturday, January 26, 13

16. DISTRIBUTED TESTS
0.write your tsung.xml 1.run Tsung
generate/gather
worker
Tsung
XML
controller Tsung
server
worker
Tsung
worker
reports
multiple servers as workers
4.report generation
Saturday, January 26, 13

17. MORE ADVANCED
STUFF
Overlay/merge your graphs, useful for comparing (tsplot)
Besides SNMP we can monitor with Munin or Erlang agents
on the remote servers..
Choose another backend: text, json, rrdtool etc. (if you want
another graphing tool or what ever)
Measure in realtime or generate afterwards.
Saturday, January 26, 13

18. MORE ADVANCED
STUFF
The recording proxy also works for database protocols
(query proxy)
Mix your sessions types (mysq with HTTP or XMPP ...)
Saturday, January 26, 13

19. TSUNG GOTCHA’S
If you are new at Erlang, dumps of stack traces can be hard
if anything is wrong!
Remember to write the config in the correct order of the
DTD (its picky!)
The path to the DTD was wrong.. comes with funny
exceptions!
Turn off debugging when testing - it takes time to write.
Ensure that your versions is the same and compiled (if not a
package) in the same way if you are running distributed!
Saturday, January 26, 13

20. TSUNG GOTCHA’S
For massive testing
set sysctl.conf for many client connections..
set the hard limit of “max files open” larger than 1024.
Do that on every controller and node! (otherwise you limit
yourself!)
Do some tuning of the network stack! memory for TCP ,
read, write and backlog. Example:
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
# This gives the kernel more memory for TCP
# which you need with many (100k+) open socket connections
net.ipv4.tcp_mem = 50576 64768 98152
# Backlog
net.core.netdev_max_backlog = 2048
net.core.somaxconn = 1024
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_syncookies = 1
Saturday, January 26, 13

21. TESTING PLAN
GOTCHAS
Understand what you are testing
Prepare your self: write down your expectations, what went
wrong and what to improve.. Use a wiki (JIRA)
Store your test for later comparison (use tsplot etc.)
Test on the correct network. Plan which one. Take into
consideration that you have to reach the servers being tested
(both test port and SNMP)
Prepare your network a massive load!. Be prepared that a
switch will blow up or so..
Saturday, January 26, 13

22. LINKS ETC.
http://tsung.erlang-projects.org
http://dak1n1.com/blog/14-http-load-generate
http://programmingobsession.blogspot.dk/2012/09/tsung-
vs-jmeterj.html
Installation:
Mac (homebrew): brew install tsung
Linux: apt-get install tsung
Windows?
Compile your own -- a lot of dependencies..
Saturday, January 26, 13