One of the new challenges of IT today is the "Big Data", to solve this problem many solutions are available on the market and some new paradigms have appeared. In most of these new paradigms the Message Queue covers an important part, more than the past. This is a small introduction to the use of Messaging Middleware and an overview of the main open source products available.

1. Why do I have to use a Message Queue System ?
Fabrizio Manfred Furuholmen"
Beolink.org!

2. Agenda Beolink.org!
§ Introduction
§ History
§ Basic components
§ Message Queue
§ Usage type
§ Advantages
§ Implementation
§ Solution
§ Performance
§ Scalability/High Avaibility
§ Big Data
§ Distributed
§ Cloud Computing
2"
Europython 2012"

3. Introduction: Example Beolink.org!
Process Comunication!
Ack Message: !
2012082609000000-Serial-Type-OK.xml!
Process A! FTP Server! Process B!
Filesystem!
Message: ! Get Message!
2012082609000000-Serial-Type-ProcessB.xml!
!
!
More than 10 years ago!
3"
9/11/12"

4. Introduction: Definition Beolink.org!
"
"
"
“…message queueing is a method by which process (or
program instance) can exchange or pass data using an
interface to a system-managed queue of message...”"
4"
9/11/12"

5. Introduction: Components Beolink.org!
BROKER! CONSUMER!
PRODUCER!
Ack Message: !
2012082609000000-Serial-Type-OK.xml!
Process A! FTP Server! Process B!
Filesystem!
Message: ! Get Message!
2012082609000000-Serial-Type-ProcessB.xml!
!
!
Topic!
Queue!
5"
9/11/12"

6. Introduction: Broker Beolink.org!
Message-oriented middleware (MOM)
“…message broker is an architectural pattern for message
validation, message transformation and message routing.
It mediates communication amongst applications,
minimizing the mutual awareness that applications should
have of each other in order to be able to exchange
messages, effectively implementing decoupling…”
6
9/11/12

7. Beolink.org!
Is message queue
middleware only a temporary
storage ?
7
9/11/12

8. Message Queue Beolink.org!
q Asynchronous
communication
q Lock
q Concurrent Read/Write
P! C!
q Burst Message
Es.
q Decoupling
Multimedia Converter
q Reliability
SMS gateway
q Multi platform
8
9/11/12

9. Message Queue: Multi Processing Beolink.org!
q Parallel processing
C!
q Load Balancing P!
C!
q High Availability P! C!
q Elastic P! C!
Es.
q Maintenance operation
Image converter
Billing Event
User Provisioning
9
9/11/12

10. Message Queue: Pub/Sub Beolink.org!
q Sending messages to
many consumers at
once C!
P! X! C!
q Event Driven
C!
Es.
Push notification
Chat Room
10
9/11/12

11. Message Queue: Routing Beolink.org!
q Static with routing key
q Pattern base
q Pattern topic Queue A! C!
q Dynamic with header P! X! Queue B! C!
evaluation
Queue C!
C!
Es.
Logging collector
User Provisioning on
target System
Info Sync
11
9/11/12

12. Message Queue: RPC Beolink.org!
q Remote Procedure Call
q Single queue for Consumer
q One queue for each Producer
q Reply to options
Queue !
P! C!
Tmp Queue !
Es.
Distributed Scheduler
CallBack
WAN comunication
12
9/11/12

13. Message : More … Beolink.org!
q Persistent Message
q Queue
q Priority / Re ordering
q Message Group
q QOS / rating
q Deduplication
q Broker Network
q Cluster
q Load distribution over
WAN
q Message routing
13
9/11/12

14. Beolink.org!
“Message Queue scales to any number
of cores, avoids all locks, costs little
more than conventional single-threaded
programming, is easy to learn, and
does not crash in strange ways.
At least no more strangely than a
normal single-threaded program…”
http://msdn.microsoft.com/en-us/magazine/cc817398.aspx
http://www.zeromq.org/blog:_start/p/2
http://ulf.wiger.net/weblog/2008/02/06/what-is-erlang-style-concurrency/
14
9/11/12

15. Beolink.org!
Simple solution to a
complicated problem!
15
9/11/12

16. Implementation Beolink.org!
q Internal implementation
q Python (Queue), Perl (Thread::Queue) ...
q Nosql Based
q Redis, MongoDB, Memcache …
q Framework
q Generic application framework: Gearman
q Stomp Based: ActiveMQ, Apollo…
q AMQP Based:RabbitMQ, Qpid…
q Other : kafka…
q Alternative solutions
q Broker less (0MQ, Crossroads I/O)
q Services
16
9/11/12

17. Implementation:Message Format Beolink.org!
q Internal / Object
q STOMP
Simple (or Streaming) Text Oriented Message Protocol (STOMP) is a simple
text-based protocol, designed for working with Message Oriented
Middleware
q AMQP
Advanced Message Queuing Protocol is an application layer protocol,
designed to efficiently support a wide variety of messaging applications and
communication patterns.
q XMPP
Extensible Messaging and Presence Protocol
q JSON
JavaScript Object Notation, is a text-based
17
9/11/12

18. Implementation: NoSQL Beolink.org!
Redis Internal Function RestMQ
self.redis = redis.StrictRedison (…) The HTTP operation on url:
/queue/queuename
def send(self,queue,message):
self.redis.rpush(queue,message) Post Message
{
def recv(self,queue) cmd: add,
return self.redis.blpop(queue) queue: genesis,
value: abacab
}
Queue Name = KEY
Message = Value Get Message
Queue = List {
Notify = Event cmd: take,
queue: genesis
}
The message can be formatted as a json object, so
more complex data can be sent. It was inspired by
Amazon SQS
Demo sub/pub :https://gist.github.com/348262
18
9/11/12

19. Implementation: AMQP Beolink.org!
RabbitMQ
Producer Consumer
#!/usr/bin/env python #!/usr/bin/env python
import pika import pika
connection = connection =
pika.BlockingConnection(pika.ConnectionParameters( pika.BlockingConnection(pika.ConnectionParameters(
host='localhost')) host='localhost'))
channel = connection.channel() channel = connection.channel()
channel.queue_declare(queue=’myqueue') channel.queue_declare(queue=’myqueue')
channel.basic_publish(exchange='', print ' [*] Waiting for messages. To exit press CTRL+C'
routing_key=’myqueue',
body=’message 1 ') def callback(ch, method, properties, body):
print [x] Sent ’Message 1 print [x] Received %r % (body,)
connection.close()
channel.basic_consume(callback,
queue=’myqueue',
no_ack=True)
channel.start_consuming()
19
9/11/12

20. Implementation: Alternatives Beolink.org!
ZeroMQ
Producer Consumer
#!/usr/bin/env python #!/usr/bin/env python
import zmq import zmq
context = zmq.Context() context = zmq.Context()
socket = context.socket(zmq.REQ) socket = context.socket(zmq.REP)
socket.bind(tcp://127.0.0.1:5000) socket.bind(tcp://127.0.0.1:5000)
while True: while True:
msg =”my msg” msg = socket.recv()
socket.send(msg) print Got, msg
print ”Send, msg socket.send(msg)
msg = socket.recv()
20
9/11/12

21. Beolink.org!
…, but it is not fast
enough …
21
9/11/12

22. Performance: Redis Beolink.org!
The Linux box is running Linux
2.6, it's Xeon X3320 2.5 GHz.
Text executed using the
loopback interface (127.0.0.1).
22
9/11/12

23. Performance: RabbitMQ Beolink.org!
PowerEdge R610 with dual
Xeon E5530s and 40GB RAM!
23
9/11/12

24. Performance: ActiveMQ/Apollo Beolink.org!
! Producer
EC2 High-CPU Extra Large Instance EC2
xlarge !
!
7 GB of memory
20 EC2 Compute Units (8 virtual cores with 2.5
EC2 Compute Units each)
model name : Intel(R) Xeon(R) CPU
E5506 @ 2.13GHz
Consumer
OS: Amazon Linux 64bitLinux ip-10-70-206-42
2.6.35.14-97.44.amzn1.
5 Consumer
5 Producer
Apollo
ActiveMQ
24
9/11/12

25. Performance: kafka Beolink.org!
message size = 200 bytes
batch size = 200 messages
fetch size = 1MB
flush interval = 600 messages
25
9/11/12

26. Performance: zeroMQ Beolink.org!
Box 1:
8-core AMD Opteron 8356, 2.3GHz
Mellanox ConnectX MT25408 in 10GbE mode
Linux/Debian 4.0 (kernel version 2.6.24.7)
ØMQ version 0.3.1
Box 2:
8-core Intel Xeon E5440, 2.83GHz
Mellanox ConnectX MT25408 in 10GbE mode
Linux/Debian 4.0 (kernel version 2.6.24.7)
ØMQ version 0.3.1
Throughput gets to the maximum of 2.8 million messages !
per second for messages 8 bytes long!
26
9/11/12

27. Performance Beolink.org!
27
9/11/12

28. Performance Beolink.org!
q Persistence
message can fault down to hundreds of message
per Second
q Bandwidth
Message size and Acknowledge increase the usage
of bandwidth
q Topics
The routing based on the value of header, increase
the delay
q Queue
Number of queue increase the delay
q Cluster
Replication message increase the time for the
acknowledgement
28
9/11/12

29. Beolink.org!
Big Data
29
9/11/12

30. Big Data Beolink.org!
Big data spans three dimensions
q Volume
Enterprises are awash with ever-growing data of all types, easily
amassing terabytes—even petabytes—of information.
q Velocity
Sometimes 2 minutes is too late. For time-sensitive processes such
as catching fraud, big data must be used as it streams into your
enterprise in order to maximize its value.
q Variety
Big data is any type of data - structured and unstructured data such
as text, sensor data, audio, video, click streams, log files and more.
30
9/11/12

31. Big Data Beolink.org!
Big Data! Message Queue!
Volume! Load Balancing:
- with Multi Brokers Conf
- with Multi queues Conf
Velocity! Parallel Processing
- Balance base on time spent
- Increase capacity on demand
High Availability
Variety! Routing Key
Path
- Header analysis
- Topic
31
9/11/12

32. Big Data: Linkedin Beolink.org!
End User action Tracking
Operational Metrics
Forntend (Producer)= 100Mb/s
Bakend (Consumer) = 200Mb/s
Main Datacenter! Analysis Datacenter!
FD! FD! FD!
Load Balancer!
Broker! Broker!
Broker! Broker!
Broker! Broker!
Realtime! Realtime DWH! Hadoop
http://incubator.apache.org/kafka/design.html
http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf
32
9/11/12

33. Big Data: Soocial.com Beolink.org!
Synchronization
btw different
applications
Collect tracking
data
http://aws.typepad.com/aws/2008/12/
33
9/11/12

34. Big Data: CERN Beolink.org!
chitecture
Monitor reliability
and availability of
European
distributed
computing
Infrastructure
(GRID)
100K system
Monitored
Component drilldown
fusesource.com/collateral/download/82/
34
9/11/12

35. Big Data: OpenStack Beolink.org!
OpenStack
Coordination and
provisioning
RackSpace
Cloud Management
Tasks
35
9/11/12

36. Cloud Computing: Netflix Beolink.org!
Transcoding
media conversion
for different device
and channels
36
9/11/12

37. Cloud Computing Beolink.org!
“Everything fails all the time”
Werner Vogels
CTO of Amazon
37
9/11/12

38. Cloud Computing: SLA Beolink.org!
• Amazon’s EC2 availability SLA is
SLA 99.95% = 4.38 hours = 16680 sec
• Restarting time = 5 minutes = 300
RTO sec
• 55 Reboot per year
Event
No one declare the MTBF !!!!
38
9/11/12

39. Cloud Computing: Design Beolink.org!
Design your application architecture for
failure. Don’t look for alternatives
…split your applications into different components
and, make sure every component of your
application has redundancy with no common points
of failure…
39
9/11/12

40. Cloud Computing: Architecture Beolink.org!
Load Balancer!
cache
Auto Scaling!
FD! FD! FD!
Broker! Broker
MOM!
Auto Scaling!
BL! BL BL
Back End / Storage
40
9/11/12

41. Cloud Computing: Multi site Beolink.org!
LOCAL MOM! LOCAL MOM!
W QUEUE W QUEUE
SITE A SITE C
BROKERS!
!
WAN MOM!
SITE B W QUEUE
LOCAL MOM!
41
9/11/12

42. Cloud Computing: Design Beolink.org!
High Availability!
42
9/11/12

43. Cloud Computing: Scalability / HA Beolink.org!
q Master-Salve topology
queue is assigned to a master node, and all changes to the queue are also replicated to a salve node. If the
master has failed, the slave can take over. (e.g. Qpid and ActiveMQ, RabbitMQ).
q Queue Distribution
queues are created and live in a single node, and all nodes know about all the queues in the system. When a
node receives a request to a queue that is not available in the current node, it routes the request to the node that
has the queue. (e.g. RabbitMQ)
q Cluster Connections
Clients may define cluster connections giving a list of broker nodes, and messages are distributed across those
nodes based on a defined policy (e.g. Fault Tolerance Policy, Load Balancing Policy). It also supports message
redistribution, and it plays a minor role in this setup.
q Broker networks
The brokers are arranged in a topology, and subscriptions are propagated through the topology until messages
reach a subscriber. Usually, this uses Consumer priority mode where brokers that are close to the point of origin
are more likely to receive the messages. The challenge is how to load balance those messages. (e.g. ActiveMQ)
43
9/11/12

44. Cloud Computing: Cluster Beolink.org!
Client! Client! Client! Lookup!
- Defined IP
- Multicast
- BootStrap
- Agent
!
Node! Node! Node!
!
!
Inter node Comunication! !
Internal Message Routing! !
!
!
MOM!
Storage
!
Storage! Storage !
Cluster!
44
9/11/12

45. Cloud Computing: Cluster Beolink.org!
Configuration!
Two Cluster
with one node
Single Cluster
with two nodes
RabbitMQ:
http://skillsmatter.com/custom/presentations/talk4.rabbitmq_internals.pdf
45
9/11/12

46. Cloud Computing: Federation/
Shovel Beolink.org!
RabbitMQ:
http://skillsmatter.com/custom/presentations/talk4.rabbitmq_internals.pdf
46
9/11/12

47. Beolink.org!
Are you happy?!
47
9/11/12

48. Critical Points Beolink.org!
q Dimension
q Message Size E = (P/C-1)*T
q Number of Queue
q Persistence L = (P-C)*T
q Delay of the queue
q Persistence only when you need
q Cluster on client side or via
boostrap
q Acknowledge when you need
q Topic vs Queue
q Queue Length
q Performance Test
48
9/11/12

49. More complex Beolink.org!
ρ = λ / µ
Exponential probability density
All customers have the same value
Any arbitrary probability distribution
Processing Delay
Transmission Delay
Propagation Delay
49
9/11/12

50. Last but not .. Beolink.org!
Redhat key RedHat
Many Vendor
sponsor of acquired
Products
Qpid Fusesourse
Message Vmware
Cloud Queue as a
Service
acquired
RabbitMQ
50
9/11/12

51. Conclusion Beolink.org!
The science of programming:
“…make building blocks that people can understand and use
easily, and people will work together to solve the very largest
problems.”
51
9/11/12

52. I look forward to meeting you… Beolink.org!
XVII European AFS meeting 2012
University of Edinburgh
October 16th to 18th
Who should attend:
§ Everyone interested in deploying a globally accessible
file system
§ Everyone interested in learning more about real
world usage of Kerberos authentication in single
realm and federated single sign-on environments
§ Everyone who wants to share their knowledge and
experience with other members of the AFS and
Kerberos communities
§ Everyone who wants to find out the latest
developments affecting AFS and Kerberos
More Info: http://openafs2012.inf.ed.ac.uk/
52
9/11/12

53. Thank you
manfred@freemails.ch
http://www.beolink.org
#
Beolink.org!

54. Message Queue Length Beolink.org!
Which is the right size ?
…30% extra capacity?
54
9/11/12

55. Basic Formula Beolink.org!
Peak! Burst Time!
E = (P/C-1)*T
Elapsed!
Capacity!
55
9/11/12

56. Basic Formula Beolink.org!
Length!
Capacity!
L = (P-C)*T
Burst Time!
Peak!
56
9/11/12

57. Cloud Computing Beolink.org!
57
9/11/12