An Introduction to AMQP with Code Samples

What’s Messaging?
An Introduction

History of Message Queuing
Manual Telegraphy Machine Assisted Telegraphy
1920s1911 - 192019th Century 1900s
Telegrams sent using
“Store and Forward”
1900
1930s

Electronic Telegraphy
1950s1940s1950s1940s
Electronic Telegram
Machines, eg Plan 55-A
1948
IBM
M
ne Assisted Telegraphy
1920s1920 1930s

Telcos UseElectronic Telegraphy
1950s1940s 1960s
IBM System/360 with
BTAM & QTAM
Message Switching
1964
First Electronic Mail
Solutions
1965
Banking Users
1970s
IBM TCAM which is
the ﬁrst true solution
Retired 1990!
1971

l
Financial Trading UsersBanking Users FMCG & Utilities
1980s 1990s
Growth of SMTP
Origins of Tibco in
Stock Price Messaging
1980s
IBM Launch MQSeries
(now WebsphereMQ)
1992
1970s
IBM TCAM which is
the ﬁrst true solution
Retired 1990!
1971

l
FMCG & Utilities
1990s
MQSeries
phereMQ)
92
Corporates Large Websites YOU
Noughties Today
Sun Release Java JMS,
Reinvigorating
Enterprise Messaging
2001
AMQP Working
Group Formed by
Investment Banks
2006
Cloud Enables and
Drives StormMQ
Adoption
2009

The Integration Tag Soup
WS –File Transfer
!"#$%&'(#)#*#+%+
Message Queuing
SOAP
EMail
SMTP
FTP
RSYNC
NTFS
!','-
!).$%&'/$.0%&1$%+
234%5)6!%$7%$
SOA
Message
Orientated
Middleware
JMS
MQ
MQSeries
AMQP
Amazon SQS
Tibco
StormMQ
RabbitMQ
HTTP
TCP/IP
DCE / RPC
CORBA
DCOM
REST
RMI
XML-RPC
.NET Remoting
Remote Procedure Call

Under Examination, though
File Transfer
!"#$%&'(#)#*#+%+
Don’tWork in the Cloud

Under Examination, though
Deployment
Intimate System Knowledge
Conﬁguration
Admin
File Transfer
!"#$%&'(#)#*#+%+
Don’tWork in the Cloud
Scaling

Ideal: Message Queuing
Systems are “loosely-coupled”
Message
Queuing

Ideal: Message Queuing
Systems are “loosely-coupled”
Message
Queuing
Systems don’t
know each other

Why Use it: Loose Coupling
Billing
Catalogue
Shipping
S-a-a-S Inventory
How do we connect them, without one outage or system
change taking everything down like a pack of cards?

Why Use it: Loose Coupling
Billing
Catalogue
Shipping
S-a-a-S Inventory
How do we connect them, without one outage or system
change taking everything down like a pack of cards?
Message Queuing lets Systems and Components exchange
data, events, commands and actions with one another
with no explicit knowledge or need for them to be online

It should be ideal for the cloud

!
!
!
Until today, messaging is …
(ZeroMQ)
(SQS uses HTTP)
(MSMQ uses COM)
(memcached)
(Java JMS
uses
Source)
(MQSeries / WebsphereMQ)
(Talarian, Rendezvous, etc)
Platform Restricted BespokeProprietary

!
!
!
Until today, messaging is …
(ZeroMQ)
(SQS uses HTTP)
(MSMQ uses COM)
(memcached)
(Java JMS
uses
Source)
(MQSeries / WebsphereMQ)
(Talarian, Rendezvous, etc)

Why is it Hell?
You need more staff And moneyYou need bridge technology

A-MQPAdvanced Message Queue Protocol
However, there’s a solution

However, AMQP ﬁxes this
A common wire-level binary
format and protocol
An explicit deﬁnition of a
server (aka broker)’s
semantics
Open Means
Interoperable

That is good …
“AMQP will be to Messaging what HTTP was to theWeb”
MRG
Clients run on any Platform Vendors are Interoperable

That is good …
“AMQP will be to Messaging what HTTP was to theWeb”
MRG
Clients run on any Platform Vendors are Interoperable
64K

Quick Recap
For Beer
The ﬁfth male member
of the A-Team, Frankie,
was played by
Question
Message Queuing connects systems and
components. Is it ideal for the cloud?
Does Loose-Coupling make individual
systems more likely to suffer outages?
Yes No
AMQP is Open.
This makes it suitable for programming
in C and Javascript?

EddieVelez
Quick Recap
✓
The ﬁfth male member
of the A-Team, Frankie,
was played by
Question
Message Queuing connects systems and
components. Is it ideal for the cloud?
Does Loose-Coupling make individual
systems more likely to suffer outages?
Yes No
AMQP is Open.
This makes it suitable for programming
in C and Javascript?
!
✓

Common Terms and Jargon
Jargon

Messaging: Which Jargon?
EMail
(SMTP, POP3, IMAP)
VoIP
(VoiceMail, XMPP)
Texting
(SMS)
Instant Messaging
(ICQ, MSN, Jabber)
Twitter
Enterprise Service Bus
(ESB)
Dynamic OO
Languages
(eg Ruby)
Message Queuing
(MQ)
Message Queuing
(MQ)
What do we
mean by
Messaging?

Essential Terms
Message Queue
Behaves like a Queue
Is First-In First-Put (FIFO)
Elements are Messages

Essential Terms
Message Queue What’s a Message?
Envelope
Payload
Header

Essential Terms
*Strictly speaking
a receiver polls for messages
a consumer has messages pushed
≣ ≊Send Publish Enqueue
≣ ≊ DequeueConsume*Receive

So how can you use it?
Concepts
Useful Book:
Enterprise
Integration Patterns
Hohpe &Woolf

Store and Forward
Billing
“Could you send me a duplicate of my bill please”
REST

Store and Forward
Billing
REST
!

Store and Forward
Billing
REST
StormMQ
Messaging
Billing
!

Store and Forward
Billing
REST
StormMQ
Messaging
Billing
!
!

Fire and Forget
Shipping
“Too many orders.The website is running like a dog.”
SQL Push SQL Pull

Fire and Forget
Shipping
“Too many orders.The website is running like a dog.”
SQL Push SQL Pull
StormMQ
Messaging
Shipping

One-To-Many
“There’s a new widget in the Catalogue: Tell all the systems.”
Catalogue
Shipping
S-a-a-S Inventory
File Transfer
ETL

One-To-Many
“There’s a new widget in the Catalogue: Tell all the systems.”
Catalogue
Shipping
S-a-a-S Inventory
StormMQ
Messaging

Publish-Subscribe (“Topics”)
“Shipments Sent, Delivered and Returned”
Shipping
S-a-a-S Inventory
Billing
AnySent
Sent or
Returned

StormMQ
Messaging
Publish-Subscribe (“Topics”)
“Shipments Sent, Delivered and Returned”
Shipping
S-a-a-S Inventory
Billing
AnySent
Sent or
Returned

Round-Robin
Billing
“How do we easily scale a massive batch job like Billing?”
EMail BillGenerate

StormMQ
Messaging
Round-Robin
Billing
“How do we easily scale a massive batch job like Billing?”
EMail BillGenerate

Getting Technical
Delving into AMQP

AMQP Client AMQP ServerTCP / IP Network
Connection Virtual Host
Connections and Channels

Connection
AMQP Client AMQP ServerTCP / IP Network
Connection
Virtual Host
Connections and Channels
TLS
“Shielding”
Channels
Each Channel is Independent:
Effectively, aVirtual Connection

Basic AMQP: Connections
Basic AMQP: Connections
Open a Connection to aVirtual Host
Open a Channel
Send a Message
Receive a Message
Close Channel
Close Connection
You only need one channel!
"
#
$
%
&
'
How does
this look
in Code?

// The API is styled similarly to Functionally Orientated Programming (FOP)
// It makes extensive use of Constructor Injection and anonymous classes modelling 'blocks'
// Closes are handled implicitly by design with exceptions thrown, ie try-finally is internal to the API
public void demonstrateConnection()
{
final AmqpConnectionFactory amqpConnectionFactory =
amqpConnectionFactoryWhichVerifiesStormMQ(“mycompany”, “mysystem”, “development”, “user”, “password”);
// Creates and closes a connection
amqpConnectionFactory . useConnection
(
// Creates and closes a channel on a connection
new ChannelCreatingConnectionUser
(
// Adapts a channel to make it convenient to use
new ConvenientChannelCreatingConnectionUser
(
new ConvenientChannelUser()
(
public void use(final @NotNull ConvenientChannel convenientChannel)
{
// Send a message, or,
// Receive a message (or both)
}
);
)
)
)
}

// Sending a Message
{
// ConvenientChannel has static convenience methods to create messageHeaders (meta-data)
// Use them to signify a message's content’s MIME type or an unique id
// Use meta data sparingly; think of it like email headers. More later
final BasicProperties messageHeaders = emptyBasicProperties();
// All messages are byte arrays
final byte[] messageBody = "HelloWorld" . getBytes( forName("UTF-8") );
// Send the message
final String routingKey = "usually the destination message queue";
final boolean mandatory = false;
final boolean immediate = false;
convenientChannel . basicPublish( "", routingKey, mandatory, immediate, messageHeaders, messageBody );
}
/* Notes
* Mandatory (mandatory = true) messages are returned unless routed to a queue
* Immediate (mandatory = true) messages are returned unless there is another channel consuming
*/

// Receiving a Message using Polling
{
final boolean noAck = false;
convenientChannel . basicGet( "queue", noAck, DoNothingWhenMessageNotReceived, new MessageReceived()
{
public void messageReceived(final @NotNull GetResponse message)
{
final byte[] messageBody = message . getBody();
final BasicProperties messageHeaders = message . getProps();
// Acknowledge successfully received messages if noAck = false
// If not acknowledged, they are eventually placed back on the queue for re-delivery
convenientChannel . basicAck( message );
}
});
}
/* Notes
* DoNothingWhenMessageNotReceived is a singleton of MesageNotReceived
* Implement DoNothing to do something else, eg for debugging
* Polling is thread-safe but inefficient; using consumers (“push”) for more efficiency…
*/

// Consuming Messages using Push
{
final boolean noAck = true;
convenientChannel . basicConsume( "my queue", noAck, new ConvenientConsumer()
{
// Consumers run on a different thread
public void handleDelivery( final String consumerTag, final Envelope envelope, final BasicProperties
messageHeaders, final byte[] messageBody )
{
final byte[] messageBody = message . getBody();
final BasicProperties messageHeaders = message . getProps();
// No need to acknowledge message as noAck = true
}
});
}
/* Notes
* Consumers only exist as long as a channel is open
* Add a threading primitive (eg volatile boolean) and reference it at *
* It is possible to use multiple channels with several consumers…
*/
*

!
✓
Channels
Complex Applications
Consume on several threads
Send on several threads
Why Use More than One Channel?
Recommendation: Keep it Simple!
Simple Applications
Send on a queue
Receive on another
Transactional Applications
NeedTransactions on a queue
Don’t NeedTransactions on another
✓

Exchanges Route Messages
Exchanges route Messages to Message Queues
Exchange
MQ
A
MQ
B

Exchanges Route Messages
Exchanges route Messages to Message Queues
You send messages to an exchange, not a message queue
Exchange
MQ
A
MQ
B

How do Exchanges Route?
Exchange
MQ
A
The Exchange ﬁnds a Binding
matching the Routing Key
#
A Binding connects a Routing Key
to one or more Message Queues
$
Every sent message has a
Routing Key
"
The Exchange delivers the
Message to the Message Queue
%
A message queue can be
bound more than once to
one or more exchanges
(
“string”

Types of Exchange
More rarely used exchange types include amq.headers and extensions
Message Queue Name
direct
Like a ‘Map’:All MQs
bound with the
routing key receive
copies of the message
“” (blank)
amq.direct
fanout
Empty String
All MQs bound to the
Exchange receive
copies of the message
amq.fanout
topic
Dotted
Bindings use
globbing expressions
(wildcards) to route
messages to MQs
amq.topic
A message queue can be bound more than once to an exchange;
A message queue can be bound to more than one exchange
But a message queue will only receive a message sent once
Routing Key
Routing
Behaviour
Default
Deﬁnitions
Point – to – Point One – to – Many Publish – SubscribeTypical Use

// Declaring Exchanges
{
final boolean passivelyDeclare = false;
final boolean durable = true;
final boolean autoDelete = false;
// Just change the ExchangeType to one of direct, fanout or topic
convenientChannel . exchangeDeclare( "my exchange A", direct, passivelyDeclare, durable, autoDelete );
convenientChannel . exchangeDeclare( "my exchange B", fanout, passivelyDeclare, durable, autoDelete );
convenientChannel . exchangeDeclare( "my exchange C", topic, passivelyDeclare, durable, autoDelete );
// A passive declaration asserts that an exchange exists, causing a channel exception if it does not
convenientChannel . exchangeDeclare( "my exchange A", direct, true, durable, autoDelete );
}
/* Notes
* Always use durable = true with StormMQ as we are a hosted service
* Nearly always use autoDelete = false as exchanges should be long-lived
* Declaring an exchange twice is not an error; if parameters differ, new values are used if possible
* Passive declaration is interesting but typical deployment practices mitigate its usefulness
*/

// Declaring and Binding Message Queues
{
final boolean passivelyDeclare = false;
final boolean durable = true;
final boolean autoDelete = false;
final boolean exclusive = false;
// A declared queue is bound to the default ("") exchange with the routing key the queue name ("my queue")
convenientChannel . queueDeclare( "my queue", passivelyDeclare, durable, exclusive, autoDelete );
// A message queue can be bound to another exchange with whatever key you choose
convenientChannel . queueBind( "my queue", "another exchange", "some routing key" );
// When binding to a fanout exchange the routing key is the empty string ("")
convenientChannel . queueBind( "my queue", "a fanout exchange, eg amq.fanout", "" );
// When binding to a topic exchange, set the routing key as a wildcard
// Publish messages with specific routing keys eg "BONDS.USD.FTSE", "BONDS.USD.NYSE", "BONDS.GBP.FTSE"
convenientChannel . queueBind( "my queue", "a fanout exchange, eg amq.topic", "BONDS.USD.*" );
}
/* Notes
* Always use durable = true with StormMQ as we are a hosted service
* Nearly always use autoDelete = false as message queues should be long-lived
* Declaring a message queue twice is not an error; if parameters differ, new values are used if possible
* Exclusive message queues are for advanced usages and often aren’t needed
*/

AMQP: Introducing Users
but
StormMQ enforces that the same
Users and Permissions are in
all of a System’s Environments
Passwords are different
Passwords are securely generated
by StormMQ and are 512 bit keys
Conﬁguration Managers can prevent
revelation of passwords to different
system users, eg developers vs sysadmins
Aim: Separation of Environments
A Connection can use several
authentication mechanisms
which are perVirtual Host
but
Commonest is User - Password
andand
Aim: Separation of Responsibility
Users are intended to be system
accounts (robots), not sysadmins
Users have read, write and create
permissions using a wildcard syntax:
important to name queues, etc well

// Example of create-system.json used with API call stormmq-create-system
{
"companyName" : "usuallyYourUserName",
"systemName" : "anythingAlphabetic",
"environments" :
[{
"environmentName" : "development",
// Different StormMQ clusters have different characteristics (and costs)
"clusterName" : "free-1",
// These are the StormMQ user accounts to which passwords for AMQP users (below) can be revealed for
this environment (development)
"permittedStormMQUserNames" : [ "developerPeter", "developerJames", "developerJohn" ]
}],
// User permissions operate on Message Queues and Exchanges
"amqpUserPermissions" :
{
"queueReader" : { "create" : "^$", "read" : ".*", "write" : "^$" },
"queueWriter" : { "create" : "^$", "read" : "^$", "write" : ".*" },
"queueCreator" : { "create" : ".*", "read" : "^$", "write" : "^$" },
"onlyProcessingObjects" : { "create" : "^processing.*", "read" : "^processing.*", "write" : "^processing.*" }
}
}

Conﬁguring AMQP
“*Our REST API has Java, Ruby, PHP and .NET bindings”
AMQP StormMQ
✓ ActiveMessage Queues
Exchanges
Bindings
Users
User Permissions
Entire Systems
! Extra
! Extra
! Extra
✓ Active
✓ Active
✓ REST
✓ REST
✓ REST*
Programmatic Conﬁg
✓ Same
✓ Same
✓ Same

AMQP: Getting Statistics
“*Password revelation is restricted to speciﬁc individuals”
AMQP StormMQ
Message Queues
Exchanges
Bindings
Generated Passwords
Usage Patterns
Entire Systems
! Extra
! Extra
! Extra
✓ REST
✓ REST
✓ REST
Programmatic Info
! Extra ✓ REST
! Extra ✓ REST*
! Extra ✓ REST

// Getting Statistics from StormMQ using the REST API in Java
// If you need to use proxies or experience the pain of Java SSL choose a different method in ApiConfiguration
final ApiConfiguration apiConfiguration = directConnectionIfHavingProblemsWithSslVerification();
// If your Magic Secret Key is in an unusual place choose a different method in Api
final Api api = apiByGuessingUsingOsConfigurationFiles( apiConfiguration, stormMqUserName );
final List<AmqpQueue> amqpQueues = api . listQueues( companyName, systemName, environmentName );
final List<AmqpExchange> amqpExchanges = api . listExchanges( companyName, systemName, environmentName );
final List<AmqpBinding> amqpBindings = api . listBindings( companyName, systemName, environmentName );
final Map<AmqpUserName, AmqpUserPassword> passwords =
api . listAmqpUserNamesAndPasswords( companyName, systemName, environmentName );
final CompanyDetails companyDetails = api . describeCompany( companyName );
// Finding “what we’ve got”
final List<CompanyName> companyNames = api . listCompanies();
final List<ClusterName> clusterNames = api . listClusters( companyName );
final List<SystemName> systemNames = api . listSystems( companyName );
final SystemDetails systemDetails = api . describeSystem( companyName, systemName );
final Set<Environment> environments = systemDetails . getEnvironments();
/* Notes
* Also consider the command-line tools stormmq-list-queues, stormmq-list-exchanges and stormmq-list-bindings
* Further examples are in com.stormmq.api.Example (part of the url signer jar)
*/

Virtual Hosts
Virtual Hosts: Think Apache!
Virtual Hosts isolate Message Queues
Virtual Hosts isolate User Groups*
Virtual Hosts isolate Exchanges*
Virtual Hosts isolate Bindings*
Virtual Hosts isolate
Virtual Hosts: StormMQ
Virtual Hosts isolateYour Company
Virtual Hosts isolateYour Systems
Virtual Hosts isolateYour Environments
Enable Conﬁguration Management
Virtual Hosts stop Data Accidents
Widgets Ltd /
Invoicing System /
/
Testing
Development
Production

Message Properties
Server Understood Good Practice
“This is just the tip of messaging best practice with AMQP”
Persistent = 1
Priority = 0
Timestamp = 4568965345
MessageId = 567A-GH-20100709
Priority Queues are per-Message
Control life of a Message
Persistence is per-Message
Timestamps expire Messages
Completely Optional
MIME Content-Type
Unique MessageId
Schema &Version

// Message Properties: Examples of Good Practice
// Being explicit is good practice
ﬁnal BasicProperties messageHeaders = new BasicProperties();
{{
DeliveryMode . NonPersistent . setOn( this );
Priority . Zero . setOn( this );
setMessageId( "My unique ID, eg a GUID or ascending value" );
// Explicit MIME type including charset; use application/octect-stream for trully binary data
setContentType("application/json; charset=utf-8");
// Use compression for larger textual data; use the same values as HTTP headers; use "identity" for none
setContentEncoding( "gzip" );
// Timestamp each message
setTimestamp( new Date( System . currentTimeMillis() ) );
// Schema and version, perhaps using a XML namespace deﬁnition
setType( "syslog-message_1.0.4" );
// Optional: Use an expiration to discard old messages that are stale (milliseconds, example is 5 minutes)
setExpiration( Integer . toString( 5 * 60 * 1000) );
// Optional: Identify the sender for later debugging
setAppId( "sending application name" );
setUserId( "role of sender, or, user of machine" );
// Optional: Identify domain or environment (consumer could error if working in different one)
setClusterId( "production" );
}};

And there is more!
Transactions
More Message
Properties
Custom Message
Properties
Immediate Delivery
Additional Exchange
Types
Auto Deletion
Fine-Grained User
Permissions
Queue
Purging
QoS

Useful Book:
Enterprise
Integration Patterns
Hohpe &Woolf

An Introduction to AMQP with Code Samples

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie An Introduction to AMQP with Code Samples

Ähnlich wie An Introduction to AMQP with Code Samples (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

An Introduction to AMQP with Code Samples