1. © 2012 FuseSource Corp.All rights reserved.1
Apache
Ac(veMQ
Performance
Tuning
Chris(an
Posta
Senior
Consultant
and
Architect
Blog:
h6p://chris9anposta.com/blog/
Twi6er:
@chris9anposta
GitHub:
h6ps://github.com/chris9an-­‐posta

2. © 2012 FuseSource Corp.All rights reserved.2
ceposta
§ Chris9an
Posta
§ Blog:
h6p://chris9anposta.com/blog/
§ Email:
ceposta@apache.org
§ Twi6er:
@chris9anposta
§ GitHub:
h6ps://github.com/chris9an-­‐posta
§ Goolge+:
h6ps://plus.google.com/103810850748178017238
§ Senior
Consultant
and
Architect
at
Red
Hat
(formerly
FuseSource)
§ Commi6er
on:
Ac9veMQ,
Apollo
§ Author
of
Essen%al
Camel
Components
DZone
Refcard

3. © 2012 FuseSource Corp.All rights reserved.3
What
Am
I
Talking
About?
§ What
is
Ac9veMQ?
§ Performance/Benchmarking
Tools
§ Tuning

4. © 2012 FuseSource Corp.All rights reserved.4
Apache
Ac(veMQ
§ OpenSource
Messaging
Server
• Queues,
Topics,
persistent
messaging,
high
availability,
etc
§ Apache
v2.0
License
§ Ac9ve
community,
Mature,
Stable
§ Used
at
top
companies
• Insurance,
banking,
retail,
ecommerce,
health
care,
avia9on,
shipping,
et.
al!
§ 5.8.0
latest
release

5. © 2012 FuseSource Corp.All rights reserved.5
Apache
Ac(veMQ
§ High
performance
§ High
Availability
§ Light-­‐weight
§ Mul9-­‐protocol
§ JMS
compliant
§ Supported
in
Produc9on

6. © 2012 FuseSource Corp.All rights reserved.6
Accessible!
§ Java
§ C/C++
h6p://ac9vemq.apache.org/cms/
§ .NET
h6p://ac9vemq.apache.org/nms/
§ PHP
§ Python
§ Ruby
§ PHP
§ JavaScript
§ Telnet!
§ Any
that
can
make
a
TCP
connec9on
and
send
Text!

7. © 2012 FuseSource Corp.All rights reserved.7
Transports
§ TCP
§ UDP
§ SSL
§ NIO
§ HTTP/s
§ VM
§ WS
(Web
Sockets)
§ WSS
(Secure
Web
Sockets)

8. © 2012 FuseSource Corp.All rights reserved.8
Wire
Protocols
§ Openwire
h6p://ac9vemq.apache.org/openwire.html
§ Simple
Text
Oriented
Messaging
Protocol
(STOMP)
h6p://stomp.github.io
§ AMQP
1.0
h6p://www.amqp.org/resources/specifica9ons
§ MQTT
h6p://mq6.org
§ HTTP/REST

9. © 2012 FuseSource Corp.All rights reserved.9
Ac(veMQ
Features
§ Master/Slave
fault
tolerance
and
High
Availability
§ Broker
“networking”
or
clustering
§ Pluggable
persistence
(KahaDB,
LevelDB,
JDBC)
§ Broker
interoperability
(RabbitMQ,
HornetQ,
etc)
§ Virtual
Topics
§ Mirrored
Queues
§ JMX
Monitoring

10. © 2012 FuseSource Corp.All rights reserved.10
When
to
use
Messaging?
§ Asynchronous
communica9on/integra9on
§ Durability
§ Loose
coupling
§ Heterogenous
integra9on
§ Real-­‐9me
data

11. © 2012 FuseSource Corp.All rights reserved.11
Configura(on
Ac9veMQ
is
Highly
Configurable!!

12. © 2012 FuseSource Corp.All rights reserved.12
Performance/Benchmarking
Tools

13. © 2012 FuseSource Corp.All rights reserved.13
So
What
Are
We
Covering?
§ Tools
§ How
to
approach
performance
tuning
§ Areas
to
tweak!

14. © 2012 FuseSource Corp.All rights reserved.14
What
are
Your
Objec(ves!?
§ Please,
please,
please..
Know
what
you’re
trying
to
accomplish!
§ Know
your
use
cases!
§ Know
your
hardware/OS
§ Understand
all
of
the
broker
config
changes
you
make!
§ VALIDATE
YOUR
CHANGES!

15. © 2012 FuseSource Corp.All rights reserved.15
Broker
Benchmarking
Tools
§ Ac9veMQ
Performance
Module
§ jms-­‐benchmark
§ JMSTester
§ Apache
Jmeter
§ Grinder
§ …?

16. © 2012 FuseSource Corp.All rights reserved.16
Ac(veMQ
Performance
Module
§ Part
of
the
Ac9veMQ
Build
h6p://ac9vemq.apache.org/ac9vemq-­‐performance-­‐module-­‐users-­‐manual.html
§ Maven
plugin
§ Quick
and
easy
to
get
started
§ Flexible
§ Records
throughput,
basic
sta9s9cs,
and
CPU
metrics

17. © 2012 FuseSource Corp.All rights reserved.17
jms-­‐benchmark
§ Opensource
at
github,
from
Hiram
Chirino
h6p://github.com/chirino/jms-­‐benchmark
§ Comprehensive
benchmarks
§ Configurable
§ All-­‐on-­‐one
§ Mul9ple
brokers
§ Pre6y
graphs!

18. © 2012 FuseSource Corp.All rights reserved.18
JmsTester
§ From
FuseSource
h6p://jmstester.fusesource.org
§ My
branch:
h6ps://github.com/chris9an-­‐posta/jms-­‐tester
§ Allows
more
complicated
load
tes9ng
§ Distributed
§ Records
messaging
throughput
and
CPU,
Memory,
Network
IO,
Disk
IO,
et
al.
§ Thorough
documenta9on
§ My
blog
post:
h6p://www.chris9anposta.com/blog/?p=268

19. © 2012 FuseSource Corp.All rights reserved.19
JMeter
§ Swiss
army
knife
of
load
tes9ng
§ From
Apache
h6p://jmeter.apache.org
§ JMS,
HTTP,
TCP,
SOAP,
JDBC,
and
many
others
§ Many
aggregators
and
repor9ng
features
§ Mature
product,
well
known

20. © 2012 FuseSource Corp.All rights reserved.20
OS
Tools
§ Lest
we
forget…
rely
heavily
on
the
tools
your
Opera9ng
System
exposes!
§ Linux
• top
• iostat
• netstat
• vmstat
§ Windows
• Built
in
process,
resource
monitors

21. © 2012 FuseSource Corp.All rights reserved.21
Ac(veMQ
Performance
Module
§ Maven
plugin
§ Can
use
exis9ng
project:
h6ps://svn.apache.org/repos/asf/ac9vemq/sandbox/ac9vemq-­‐persest
§ Spin
up
producers,
consumers
§ Control
number
of
threads,
test
dura9ons,
etc
§ Stats
and
Repor9ng
• Throughput,
min/max,
broken
down
per
client
plugin
groupIdorg.apache.ac9vemq.tooling/groupId
ar9factIdac9vemq-­‐perf-­‐maven-­‐plugin/ar9factId
version${ac9vemq-­‐version}/version
/plugin

22. © 2012 FuseSource Corp.All rights reserved.22
Using
Performance
Module
§ Example:
§ Out
of
the
box
examples:
§ Other
out
of
the
box
profiles:
• Producer
– AMQ-­‐Prod-­‐1-­‐1-­‐queue-­‐nonpersistent.proper9es
– AMQ-­‐Prod-­‐10-­‐1-­‐queue-­‐nonpersistent.proper9es
– AMQ-­‐Prod-­‐10-­‐10-­‐topic-­‐persistent.proper9es
• Consumer
– AMQ-­‐Cons-­‐1-­‐1-­‐queue.proper9es
– AMQ-­‐Cons-­‐10-­‐1-­‐topic-­‐durable.proper9es
– AMQ-­‐Cons-­‐10-­‐10-­‐queue.proper9es
user@computer mvn activemq-perf:producer –Dfactory.brokerURL=tcp://localhost:61616
user@computer mvn activemq-perf:producer
–DsysTest.propsConfigFile=AMQ-prod-1-1-queue-persistent.properties

23. © 2012 FuseSource Corp.All rights reserved.23
What
Broker
to
Use?
§ Load
up
an
external
broker
• Exis9ng
broker
• No
maven
access
• Different
machine
(good
idea!)
§ Load
one
from
the
perf
module
• Tests
and
broker
config
all
in
one
• Simple
maven
command
user@computer mvn activemq-perf:broker –Durl=tcp://localhost:61616

24. © 2012 FuseSource Corp.All rights reserved.24
Defaults
§ Overall
• tp,cpu
reports
• 5
min
• 30s
warm
up
30s
cool
down
• 1s
sample
rate
§ Producer
• Non-­‐persistent
• Auto
ack
• 1K
message
size
§ Consumer
• Non
durable
subscrip9on
• Auto
ack
• No
tx

25. © 2012 FuseSource Corp.All rights reserved.25
Copyright
©
2011
Progress
Sosware
Corpora9on
and/or
its
subsidiaries
or
affiliates.
All
rights
reserved.
A
Progress
Sosware
Company
FuseSource
Confiden9al
25
Quick
Demo!

26. © 2012 FuseSource Corp.All rights reserved.26
Results
explana(on

27. © 2012 FuseSource Corp.All rights reserved.27
While
Running…
§ Use
JConsole!
§ A
more
powerful
toolkit?
YourKit?
§ Rely
on
OS
monitoring,
top,
iostat,
etc

28. © 2012 FuseSource Corp.All rights reserved.28
Copyright
©
2011
Progress
Sosware
Corpora9on
and/or
its
subsidiaries
or
affiliates.
All
rights
reserved.
A
Progress
Sosware
Company
FuseSource
Confiden9al
28
Tuning
Ac(veMQ

29. © 2012 FuseSource Corp.All rights reserved.29
Approach
§ Know
your
requirements
§ Use
real
hardware,
real
scenarios,
real
se}ngs
§ Know
what
config
changes
do!
§ Tune
§ Run
tests
§ Capture
results
§ Repeat
§ Change
only
one
se}ng
at
a
9me!

30. © 2012 FuseSource Corp.All rights reserved.30
Where
are
the
BoPlenecks?
§ Overall
system
§ Network
latencies
§ Disk
IO!
§ Threading
overheads
§ JVM
op9miza9ons

31. © 2012 FuseSource Corp.All rights reserved.31
Network
§ Mbits/Gbits
per
second
throughput
§ Topology
• Hops
• Switches
• Routes
• Gateways
§ Firewalls
§ Internal/External
clients

32. © 2012 FuseSource Corp.All rights reserved.32
Disk
§ Read/Write
speeds
§ Mechanical
vs
SSD
§ Shared
disks,
SAN,
NAS,
NFS,
etc
§ Ac9veMQ
DiskBenchmark:
Benchmarking:
/Users/cposta/temp/test.dat
Writes:
159232
writes
of
size
4096
wri6en
in
10.244
seconds.
15543.928
writes/second.
60.718468
megs/second.
Sync
Writes:
33213
writes
of
size
4096
wri6en
in
10.001
seconds.
3320.968
writes/second.
12.972531
megs/second.
Reads:
5160332
reads
of
size
4096
read
in
10.001
seconds.
515981.6
writes/second.
2015.5531
megs/second.
user@computer java -classpath lib/activemq-kahadb-store-5.8.0.jar
org.apache.activemq.store.kahadb.disk.util.DiskBenchmark
/Users/cposta/temp/test.dat

33. © 2012 FuseSource Corp.All rights reserved.33
Tune
What
Needs
Tuning!
The following aren’t things you must or need do…
they are things to consider and can be worth tuning.
Your
specific
use
cases
will
illuminate
what
needs
tuning

34. © 2012 FuseSource Corp.All rights reserved.34
Transport
Tuning
§ TCP
Tuning
• Timeouts
(Handshake,
Teardowns)
• Conges9on
avoidance,
sliding
windows
• Send/Receive
Buffers!!
• Default
64K,
but
increase
to
Bandwidth
Delay
Product
– Buffer
=
Bandwidth
*
RTT
h6p://www.speedguide.net/bdp.php
– See
h6p://en.wikipedia.org/wiki/TCP_tuning
§ Configure
the
socket:
§ transport.*
§ socket.*
transportConnector
name=openwire
uri=tcp://0.0.0.0:61616?maximumConnec9ons=1000
wireformat.maxFrameSize=104857600transport.socketBufferSize=131072
/

35. © 2012 FuseSource Corp.All rights reserved.35
Transport
Tuning…
§ Buffering
between
Socket
and
Protocol
Codec
§ IOBufferSize
• Default
8k
transportConnector
name=openwire
uri=tcp://0.0.0.0:61616?maximumConnec9ons=1000
wireformat.maxFrameSize=104857600transport.ioBufferSize=65536
/

36. © 2012 FuseSource Corp.All rights reserved.36
Wire
Format
Tuning
§ OpenWire
(default
protocol)
• wireFormat.tcpNoDelayEnabled
–
whether
or
not
to
use
Nagle’s
batching
algorithm
(default:
true
..that
means
no
Nagle)
• wireFormat.cacheSize
–
how
many
command
object
constants
to
cache
(default:
1024)
• wireFormat.cacheEnabled
–
turn
cache
on/off
(default:
true)
• wireFormat.9ghtEncoding
–
reduce
network
load
(default:
true)
• wireFormat.maxFrameSize
–
message
sizes
(default:
Long.MAX_LONG)
• wireFormat.maxInac9vityDura9on
–
monitors
period
of
inac9vity
(default:
30000ms,
set
to
0
to
disable))
transportConnector
name=openwire
uri=tcp://0.0.0.0:61616?wireFormat.tcpNoDelayEnabled=false
/

37. © 2012 FuseSource Corp.All rights reserved.37
Tuning
Clients
§ Compression
• ?jms.useCompression=true
• jms.*
configures
underlying
connec9on
factory
• Uses
java.u9l.zip.Deflater
§ Encoding
• JMS
Message
types
– Object
– Bytes
– Text
– Stream
– Map
transportConnector
name=openwire”
uri=tcp://0.0.0.0:61616?jms.useCompression=true
/

38. © 2012 FuseSource Corp.All rights reserved.38
Tuning
Producers
§ Producer
Flow
Control
• Persistent
vs
non
persistent
messaging
• Broker
vs
Client
• Broker
memory,
client
producer
window
• Default
sync
send
for
persistent,
async
send
for
non-­‐persistent
and
transac9ons
(except
for
commit/rollback
opera9on)
• En9re
connec9on
flow
control,
individual
des9na9on
flow
control,
TCP
flow
control
• Disable
flow
control
§ jms.alwaysSyncSend
§ jms.useAsyncSend
§ Send
fail
if
no
space
policyEntry
queue=“queueWildcard”
producerFlowControl=“false”
/

39. © 2012 FuseSource Corp.All rights reserved.39
Failover
§ Use
failover
transport
for
automa9c
reconnects
§ backup
§ priorityBackup
§ trackMessages
(default
false)
§ backOffMul9plier
§ maxReconnectA6empts
§ randomize
§ Can
automa9cally
get
updates
from
broker
when
cluster
situa9on
changes
failover:(tcp://primary:61616,tcp://secondary:61616)?randomize=false

40. © 2012 FuseSource Corp.All rights reserved.40
Other
Producer
Parameters
§ jms.watchTopicAdvisories=false
§ jms.copyMessageOnSend
§ Batch
using
transac9ons

41. © 2012 FuseSource Corp.All rights reserved.41
Tuning
Consumers
§ Prefetch
• Queue
–
1000
• Queue
Browser
–
500
• Topic
–
32K
• Durable
Topic
–
100
§ Broker
side
memory
§ Client
side
memory
§ Consumer
starva9on

42. © 2012 FuseSource Corp.All rights reserved.42
Prefetch
SeXngs
§ Broker
§ Consumer
§ Des9na9on
broker
...
des9na9onPolicy
policyMap
policyEntries
policyEntry
queue=queue.
queuePrefetch=”1”/
policyEntry
topic=topic.
topicPrefetch=”1000”/
/policyEntries
/policyMap
/des9na9onPolicy
/broker
jms.prefetchPolicy.queuePrefetch=100
new
Ac9veMQQueue(TEST.QUEUE?consumer.prefetchSize=10)

43. © 2012 FuseSource Corp.All rights reserved.43
Consumer
ACK
Modes
§ Session.AUTO_ACKNOWLEDGE
§ Session.CLIENT_ACKNOWLEDGE
§ Session.DUPS_OKAY_ACKNOWLEDGE
§ Session.SESSION_TRANSACTED
§ Ac9veMQSession.INDIVIDUAL_ACKNOWLEDGE
§ jms.op9mizedAcknowledge
with
AUTO_ACK
§ Note
the
differences
with
DUPS_OKAY_ACKNOWLEDGE
when
using
topics
or
queues.

44. © 2012 FuseSource Corp.All rights reserved.44
Other
Consumer
Parameters
§ jms.alwaysSessionAsync
reduce
context
switching
§ MessageListener
faster
§ Spring
DMLC:
Cache
consumers
and
connec9ons!

45. © 2012 FuseSource Corp.All rights reserved.45
Tuning
Broker
Core
§ Keep
an
eye
on
number
of
threads
• UseDedicatedTaskRunner
–
use
a
dedicated
dispatch
thread
for
queues
• op9mizedDispatch
–
don’t
take
the
penalty
of
context
switch,
just
let
the
transport
thread
do
the
dispatch
• asyncDispatch
–
don’t
take
the
penalty
of
context
switch,
let
the
dispatching
thread
send
to
the
consumer’s
transport
• NIO
connector
nio://localhost:61616

46. © 2012 FuseSource Corp.All rights reserved.46
Broker
Resource
SeXngs
§ MemoryUsage
se}ngs
for
messages
§ Caching
for
persistent
vs
non-­‐persistent
§ Policy
se}ngs,
memory
limits
for
des9na9ons,
cursor
high
water
mark,
etc
§ Temp
usage
for
offlining
non-­‐persistent
messages
§ Use
the
right
combina9on
of
memory,
producer
flow
control,
and
message
cursors

47. © 2012 FuseSource Corp.All rights reserved.47
Configura(on
of
Resources
!–
defaults
-­‐-­‐
systemUsage
systemUsage
memoryUsage
memoryUsage
limit=64
mb
/
/memoryUsage
storeUsage
storeUsage
limit=100
gb
/
/storeUsage
tempUsage
tempUsage
limit=10
gb
/
/tempUsage
/systemUsage
/systemUsage
policyEntry
queue=“queueWildcard”
memoryLimit=“10M”
/

48. © 2012 FuseSource Corp.All rights reserved.48
KahaDB
§ Current
messaging
database
§ Fast,
op9mized
for
messaging
§ Journal
+
Index
+
write
ahead
log
§ Fast
recovery
§ Fewer
file
descriptors

49. © 2012 FuseSource Corp.All rights reserved.49
Tuning
KahaDB
§ What
levels
of
reliability
must
you
have?
§ Durability
vs
throughput
§ fsync
vs
fflush
§ Areas
of
tuning
• Concurrent
store
and
dispatch
• Index
paging,
caching,
page
size,
sync,
checkpoin9ng
• Journal
sync,
file
length,
checkpoin9ng

50. © 2012 FuseSource Corp.All rights reserved.50
KahaDB
Journal
SeXngs
§ enableJournalDiskSyncs
(default:
true)
§ cleanupInterval
(default:
30000ms)
§ checkForCorruptJournalFiles
(default:
false)
§ journalSize
(default:
32MB)
broker
brokerName=broker
...
persistenceAdapter
kahaDB
directory=ac9vemq-­‐data
journalMaxFileLength=32mb”
cleanupInterval=“150000”
checkForCorruptJournalFiles=“true”/
/persistenceAdapter
...
/broker

51. © 2012 FuseSource Corp.All rights reserved.51
KahaDB
Index
SeXngs
§ How
osen
the
index
writes
updates
• checkpointInterval
(default:
5000),
indexWriteBatchSize
(default:
1000)
• Can
amor9ze
cost
across
small
writes?
Or
batch
them
up
for
larger
writes
§ Size
of
the
index
in
memory
• indexCacheSize
(default:
10000)
§ Sync
to
disk
• enableIndexDiskSyncs
(default:
true)
broker
brokerName=broker
...
persistenceAdapter
kahaDB
directory=ac9vemq-­‐data
journalMaxFileLength=32mb”
indexCacheSize=“40000”
enableIndexDiskSyncs=“false”/
/persistenceAdapter
...
/broker

52. © 2012 FuseSource Corp.All rights reserved.52
Tradeoffs
§ Point
to
point,
publish
subscribe
§ Levels
of
durability
§ Duplicate
messages
§ Security
§ Throughput
§ Priority

53. © 2012 FuseSource Corp.All rights reserved.53
Summary
§ Ac9veMQ
is
a
highly
tunable,
configurable
piece
of
server
sosware
§ Know
your
use
cases
and
requirements
§ Use
tools
to
not
only
load
the
broker,
but
monitor
the
bo6lenecks
§ Tune
for
producers/consumers,
broker
core,
network,
and
disk
IO
§ Ask
the
computer!

54. © 2012 FuseSource Corp.All rights reserved.54
Copyright
©
2011
Progress
Sosware
Corpora9on
and/or
its
subsidiaries
or
affiliates.
All
rights
reserved.
A
Progress
Sosware
Company
FuseSource
Confiden9al
54
Ques(ons?

55. © 2012 FuseSource Corp.All rights reserved.55
Useful
Links
§ Apache
Ac9veMQ
h6p://ac9vemq.apache.org/
§ JMS
Benchmarks
h6p://github.com/chirino/jms-­‐benchmark
§ Ac9veMQ
Performance
module
tools:
h6p://svn.apache.org/repos/asf/ac9vemq/sandbox/
ac9vemq-­‐persest
§ Lots
of
blogging
about
Ac9veMQ
J
h6p://chris9anposta.com/blog