SlideShare ist ein Scribd-Unternehmen logo
1 von 27
Downloaden Sie, um offline zu lesen
ZeroMQ: Super Sockets By James Dennis (@j2labs) http://j2labs.net
What is ZeroMQ? Message Passing You still design the payload Fault tolerant A message IS your payload even when broken into multiple packets Many messaging “patterns” offered
What is ZeroMQ? Super Sockets Multiple networking protocols(tcp, ipc, whatevz!) Language agnostic (bindings for: ruby, python, java, c, lua, lisp, scheme and more…) Feels like a scripting language for sockets Socket behaviors Round-robin routing Publishing to subscribers Sending job requests
What ZeroMQ is not… Persistent reasonably easy to add A message queue there are literally zero mq’s here An out-of-the-box solution instead, it’s easy to build any solution remember, they’re like scripting sockets
Socket Types (transports) tcp :: regular TCP sockets socket.connect(“tcp://…”) ipc :: interprocess communication (unix sockets) socket.connect(“ipc://…”) in-proc :: in process communication socket.connect(“in-proc://…”) pgm – pragmatic multicast (or epgmvia udp) socket.connect(“pgm://…”)
Socket Behaviors Actually called patterns They exist to make typical cases easy Request-Reply Publish-Subscribe Pipeline Exclusive pair you probably don’t want this
Patterns: PUSH/PULL Sends messages to multiple connected hosts Passes messages ROUND ROBIN to connections Blocks PUSH socket if no connections are present N hosts PULL messages from pusher Connect/Remove new hosts as necessary
Patterns: PUSH/PULL Server: Sends a message import zmq ctx = zmq.Context() s = ctx.socket(zmq.PUSH) s.bind("ipc://*:5678") while True: s.send(”blablabla") # blocks if no receivers     print “Sent one”
Patterns: PUSH/PULL Client: Prints messages from server socket import zmq ctx = zmq.Context() s = ctx.socket(zmq.PULL) s.connect("ipc://*:5678") while True: msg = s.recv()     print 'Got msg:', msg
Patterns: PUSH/PULL A typical worker pipeline  looks a bit like this. What’s that PUB/SUB stuff?
Patterns: PUB/SUB PUB socket sends topical payloads Can start streaming messages right away ZMQ discards messages if no subscribers exist SUB socket subscribes to “topic” receives all messages reads topic and discards accordingly happens under the hood.
Patterns: PUB/ SUB Server: sends messages to `whatevz` s = ctx.socket(zmq.PUB) s.bind("tcp://127.0.0.1:5566") topic = ’whatevz’ message = json.dumps({'msg': 'what up dude?'}) while True: s.send(topic, message) # send to topic     print ‘Sent one’
Patterns: PUB/SUB Client: demonstrates subscribing to `whatevz` s = ctx.socket(zmq.SUB) s.connect("tcp://127.0.0.1:5566") topic = "whatevz" s.setsockopt(zmq.SUBSCRIBE, topic) # set socket option while True: msg = s.recv()     print 'Got msg:’, json.loads(msg)
Patterns: REQ/REP One to one messaging Each REQ message must have a REP response Synchronous communication Will error if two REQ’s are sent before one REP
Patterns REQ/REP Client context.socket(zmq.REQ) # btw… REQ sockets are blocking! Worker(could say “server”) context.socket(zmq.REP) Multiple Connections One-to-one messaging is useful for workers getting work Basically, pull work request & reply when done Client may need to multiple workers Can’t just open more sockets… we need a new pattern…
Patterns: REQ/REP CLIENTsends a “Hello” SERVERsends a “world” back The REQ socket can send a message, but must not send anything else until it gets a response or ZMQ will error. Likewise, a REP socket cannot send a message unless it first received one.
Synchronized PUB/SUB Handling “no broker” (aka building one) More than one socket used Like having OneMQ now Hypothetical Communication Order: Subscriber sends sub REQ for topic Publisher REP - “listen <tcp://here>” Subscriber listens to PUB socket
Patterns: XREQ/XREP XREQ is a historical name for DEALER A DEALER asynchronously sends to *REP sockets It can send to XREP (aka: ROUTER) sockets too XREP is a historical name for ROUTER A ROUTER asynchronously handles *REQ sockets It can handle DEALER (aka: XREQ) sockets too.
Patterns: DEALER/ROUTER ,[object Object]
ROUTER processes message and sends to some worker via the DEALER
DEALERsends message to worker and waits for one response
WORKER processes message, sends response back
DEALER get message from WORKER, sends along to ROUTER
ROUTERgets message and sends back to CLIENT
CLIENTdoes something with it!,[object Object]
Broker Model: aka “The Titanic”
Broker Model with ZMQ Every connection is a REQ socket ROUTER socket to all connections LRU Queue (your broker code!) Up to the LRU Queue handle socket identities properly REQ/REP gets that for free.

Weitere ähnliche Inhalte

Was ist angesagt?

gRPC Design and Implementation
gRPC Design and ImplementationgRPC Design and Implementation
gRPC Design and ImplementationVarun Talwar
 
A whirlwind tour of the LLVM optimizer
A whirlwind tour of the LLVM optimizerA whirlwind tour of the LLVM optimizer
A whirlwind tour of the LLVM optimizerNikita Popov
 
HTTP/2 standard for video streaming
HTTP/2 standard for video streamingHTTP/2 standard for video streaming
HTTP/2 standard for video streamingHung Thai Le
 
Introduction To RabbitMQ
Introduction To RabbitMQIntroduction To RabbitMQ
Introduction To RabbitMQKnoldus Inc.
 
Faster packet processing in Linux: XDP
Faster packet processing in Linux: XDPFaster packet processing in Linux: XDP
Faster packet processing in Linux: XDPDaniel T. Lee
 
What is gRPC introduction gRPC Explained
What is gRPC introduction gRPC ExplainedWhat is gRPC introduction gRPC Explained
What is gRPC introduction gRPC Explainedjeetendra mandal
 
Discover Quarkus and GraalVM
Discover Quarkus and GraalVMDiscover Quarkus and GraalVM
Discover Quarkus and GraalVMRomain Schlick
 
Message queuing telemetry transport (mqtt) message format
Message queuing telemetry transport (mqtt) message formatMessage queuing telemetry transport (mqtt) message format
Message queuing telemetry transport (mqtt) message formatHamdamboy (함담보이)
 
Real Time UI with Apache Kafka Streaming Analytics of Fast Data and Server Push
Real Time UI with Apache Kafka Streaming Analytics of Fast Data and Server PushReal Time UI with Apache Kafka Streaming Analytics of Fast Data and Server Push
Real Time UI with Apache Kafka Streaming Analytics of Fast Data and Server PushLucas Jellema
 
HTTP2 and gRPC
HTTP2 and gRPCHTTP2 and gRPC
HTTP2 and gRPCGuo Jing
 
Accelerating Envoy and Istio with Cilium and the Linux Kernel
Accelerating Envoy and Istio with Cilium and the Linux KernelAccelerating Envoy and Istio with Cilium and the Linux Kernel
Accelerating Envoy and Istio with Cilium and the Linux KernelThomas Graf
 

Was ist angesagt? (20)

gRPC Design and Implementation
gRPC Design and ImplementationgRPC Design and Implementation
gRPC Design and Implementation
 
gRPC
gRPCgRPC
gRPC
 
A whirlwind tour of the LLVM optimizer
A whirlwind tour of the LLVM optimizerA whirlwind tour of the LLVM optimizer
A whirlwind tour of the LLVM optimizer
 
HTTP/2 standard for video streaming
HTTP/2 standard for video streamingHTTP/2 standard for video streaming
HTTP/2 standard for video streaming
 
Quarkus k8s
Quarkus   k8sQuarkus   k8s
Quarkus k8s
 
RabbitMQ.ppt
RabbitMQ.pptRabbitMQ.ppt
RabbitMQ.ppt
 
Introduction To RabbitMQ
Introduction To RabbitMQIntroduction To RabbitMQ
Introduction To RabbitMQ
 
gRPC Overview
gRPC OverviewgRPC Overview
gRPC Overview
 
Faster packet processing in Linux: XDP
Faster packet processing in Linux: XDPFaster packet processing in Linux: XDP
Faster packet processing in Linux: XDP
 
Power-up services with gRPC
Power-up services with gRPCPower-up services with gRPC
Power-up services with gRPC
 
What is gRPC introduction gRPC Explained
What is gRPC introduction gRPC ExplainedWhat is gRPC introduction gRPC Explained
What is gRPC introduction gRPC Explained
 
Discover Quarkus and GraalVM
Discover Quarkus and GraalVMDiscover Quarkus and GraalVM
Discover Quarkus and GraalVM
 
Message queuing telemetry transport (mqtt) message format
Message queuing telemetry transport (mqtt) message formatMessage queuing telemetry transport (mqtt) message format
Message queuing telemetry transport (mqtt) message format
 
RabbitMQ & Kafka
RabbitMQ & KafkaRabbitMQ & Kafka
RabbitMQ & Kafka
 
Real Time UI with Apache Kafka Streaming Analytics of Fast Data and Server Push
Real Time UI with Apache Kafka Streaming Analytics of Fast Data and Server PushReal Time UI with Apache Kafka Streaming Analytics of Fast Data and Server Push
Real Time UI with Apache Kafka Streaming Analytics of Fast Data and Server Push
 
HTTP2 and gRPC
HTTP2 and gRPCHTTP2 and gRPC
HTTP2 and gRPC
 
Accelerating Envoy and Istio with Cilium and the Linux Kernel
Accelerating Envoy and Istio with Cilium and the Linux KernelAccelerating Envoy and Istio with Cilium and the Linux Kernel
Accelerating Envoy and Istio with Cilium and the Linux Kernel
 
Mutiny + quarkus
Mutiny + quarkusMutiny + quarkus
Mutiny + quarkus
 
Building microservices with grpc
Building microservices with grpcBuilding microservices with grpc
Building microservices with grpc
 
gRPC - RPC rebirth?
gRPC - RPC rebirth?gRPC - RPC rebirth?
gRPC - RPC rebirth?
 

Andere mochten auch

Sistema de Mensajeria de Colas con ZeroMQ y Python
Sistema de Mensajeria de Colas con ZeroMQ y PythonSistema de Mensajeria de Colas con ZeroMQ y Python
Sistema de Mensajeria de Colas con ZeroMQ y PythonErnesto Crespo
 
Introduction to ZeroMQ - eSpace TechTalk
Introduction to ZeroMQ - eSpace TechTalkIntroduction to ZeroMQ - eSpace TechTalk
Introduction to ZeroMQ - eSpace TechTalkMahmoud Said
 
Software Architecture over ZeroMQ
Software Architecture over ZeroMQSoftware Architecture over ZeroMQ
Software Architecture over ZeroMQpieterh
 
Zeromq anatomy & jeromq
Zeromq anatomy & jeromqZeromq anatomy & jeromq
Zeromq anatomy & jeromqDongmin Yu
 
CurveZMQ, ZMTP and other Dubious Characters
CurveZMQ, ZMTP and other Dubious CharactersCurveZMQ, ZMTP and other Dubious Characters
CurveZMQ, ZMTP and other Dubious Characterspieterh
 
Build reliable, traceable, distributed systems with ZeroMQ
Build reliable, traceable, distributed systems with ZeroMQBuild reliable, traceable, distributed systems with ZeroMQ
Build reliable, traceable, distributed systems with ZeroMQRobin Xiao
 
Redis & ZeroMQ: How to scale your application
Redis & ZeroMQ: How to scale your applicationRedis & ZeroMQ: How to scale your application
Redis & ZeroMQ: How to scale your applicationrjsmelo
 
The promise of asynchronous php
The promise of asynchronous phpThe promise of asynchronous php
The promise of asynchronous phpWim Godden
 
Saltconf 2016: Salt stack transport and concurrency
Saltconf 2016: Salt stack transport and concurrencySaltconf 2016: Salt stack transport and concurrency
Saltconf 2016: Salt stack transport and concurrencyThomas Jackson
 
Revolutionary Open Source
Revolutionary Open SourceRevolutionary Open Source
Revolutionary Open Sourcepieterh
 
Software Architecture using ZeroMQ - techmesh 2012
Software Architecture using ZeroMQ - techmesh 2012Software Architecture using ZeroMQ - techmesh 2012
Software Architecture using ZeroMQ - techmesh 2012pieterh
 
Switch or broker
Switch or brokerSwitch or broker
Switch or brokerpieterh
 
Git Without Branches - Simple, Smooth, Scalable
Git Without Branches - Simple, Smooth, ScalableGit Without Branches - Simple, Smooth, Scalable
Git Without Branches - Simple, Smooth, Scalablepieterh
 
WTF Is Messaging And Why You Should Use It?
WTF Is Messaging And Why You Should Use It?WTF Is Messaging And Why You Should Use It?
WTF Is Messaging And Why You Should Use It?James Russell
 
Social architecture-101
Social architecture-101Social architecture-101
Social architecture-101pieterh
 
Fosdem 2009
Fosdem 2009Fosdem 2009
Fosdem 2009pieterh
 
Distributed Applications with Perl & Gearman
Distributed Applications with Perl & GearmanDistributed Applications with Perl & Gearman
Distributed Applications with Perl & GearmanIssac Goldstand
 
Introduction to ZeroMQ
Introduction to ZeroMQIntroduction to ZeroMQ
Introduction to ZeroMQYiHung Lee
 
Event Driven with LibUV and ZeroMQ
Event Driven with LibUV and ZeroMQEvent Driven with LibUV and ZeroMQ
Event Driven with LibUV and ZeroMQLuke Luo
 

Andere mochten auch (20)

Sistema de Mensajeria de Colas con ZeroMQ y Python
Sistema de Mensajeria de Colas con ZeroMQ y PythonSistema de Mensajeria de Colas con ZeroMQ y Python
Sistema de Mensajeria de Colas con ZeroMQ y Python
 
Introduction to ZeroMQ - eSpace TechTalk
Introduction to ZeroMQ - eSpace TechTalkIntroduction to ZeroMQ - eSpace TechTalk
Introduction to ZeroMQ - eSpace TechTalk
 
Software Architecture over ZeroMQ
Software Architecture over ZeroMQSoftware Architecture over ZeroMQ
Software Architecture over ZeroMQ
 
Zeromq anatomy & jeromq
Zeromq anatomy & jeromqZeromq anatomy & jeromq
Zeromq anatomy & jeromq
 
CurveZMQ, ZMTP and other Dubious Characters
CurveZMQ, ZMTP and other Dubious CharactersCurveZMQ, ZMTP and other Dubious Characters
CurveZMQ, ZMTP and other Dubious Characters
 
Build reliable, traceable, distributed systems with ZeroMQ
Build reliable, traceable, distributed systems with ZeroMQBuild reliable, traceable, distributed systems with ZeroMQ
Build reliable, traceable, distributed systems with ZeroMQ
 
Redis & ZeroMQ: How to scale your application
Redis & ZeroMQ: How to scale your applicationRedis & ZeroMQ: How to scale your application
Redis & ZeroMQ: How to scale your application
 
The promise of asynchronous php
The promise of asynchronous phpThe promise of asynchronous php
The promise of asynchronous php
 
Saltconf 2016: Salt stack transport and concurrency
Saltconf 2016: Salt stack transport and concurrencySaltconf 2016: Salt stack transport and concurrency
Saltconf 2016: Salt stack transport and concurrency
 
Revolutionary Open Source
Revolutionary Open SourceRevolutionary Open Source
Revolutionary Open Source
 
Software Architecture using ZeroMQ - techmesh 2012
Software Architecture using ZeroMQ - techmesh 2012Software Architecture using ZeroMQ - techmesh 2012
Software Architecture using ZeroMQ - techmesh 2012
 
Switch or broker
Switch or brokerSwitch or broker
Switch or broker
 
Git Without Branches - Simple, Smooth, Scalable
Git Without Branches - Simple, Smooth, ScalableGit Without Branches - Simple, Smooth, Scalable
Git Without Branches - Simple, Smooth, Scalable
 
WTF Is Messaging And Why You Should Use It?
WTF Is Messaging And Why You Should Use It?WTF Is Messaging And Why You Should Use It?
WTF Is Messaging And Why You Should Use It?
 
Social architecture-101
Social architecture-101Social architecture-101
Social architecture-101
 
Fosdem 2009
Fosdem 2009Fosdem 2009
Fosdem 2009
 
Distributed Applications with Perl & Gearman
Distributed Applications with Perl & GearmanDistributed Applications with Perl & Gearman
Distributed Applications with Perl & Gearman
 
Introduction to Heroku Postgres
Introduction to Heroku PostgresIntroduction to Heroku Postgres
Introduction to Heroku Postgres
 
Introduction to ZeroMQ
Introduction to ZeroMQIntroduction to ZeroMQ
Introduction to ZeroMQ
 
Event Driven with LibUV and ZeroMQ
Event Driven with LibUV and ZeroMQEvent Driven with LibUV and ZeroMQ
Event Driven with LibUV and ZeroMQ
 

Ähnlich wie ZeroMQ: Super Sockets - by J2 Labs

Socket programming with php
Socket programming with phpSocket programming with php
Socket programming with phpElizabeth Smith
 
ZeroMQ - Sockets on steroids!
ZeroMQ - Sockets on steroids!ZeroMQ - Sockets on steroids!
ZeroMQ - Sockets on steroids!Pedro Januário
 
Lindsay distributed geventzmq
Lindsay distributed geventzmqLindsay distributed geventzmq
Lindsay distributed geventzmqRobin Xiao
 
Zmq in context of openstack
Zmq in context of openstackZmq in context of openstack
Zmq in context of openstackYatin Kumbhare
 
Of the variedtypes of IPC, sockets arout and awaythe foremostcommon..pdf
Of the variedtypes of IPC, sockets arout and awaythe foremostcommon..pdfOf the variedtypes of IPC, sockets arout and awaythe foremostcommon..pdf
Of the variedtypes of IPC, sockets arout and awaythe foremostcommon..pdfanuradhasilks
 
TCP Sockets Tutor maXbox starter26
TCP Sockets Tutor maXbox starter26TCP Sockets Tutor maXbox starter26
TCP Sockets Tutor maXbox starter26Max Kleiner
 
Servers with Event Machine - David Troy - RailsConf 2011
Servers with Event Machine - David Troy - RailsConf 2011Servers with Event Machine - David Troy - RailsConf 2011
Servers with Event Machine - David Troy - RailsConf 2011David Troy
 
The Zen of High Performance Messaging with NATS
The Zen of High Performance Messaging with NATSThe Zen of High Performance Messaging with NATS
The Zen of High Performance Messaging with NATSApcera
 
The Zen of High Performance Messaging with NATS
The Zen of High Performance Messaging with NATS The Zen of High Performance Messaging with NATS
The Zen of High Performance Messaging with NATS NATS
 
The Zen of High Performance Messaging with NATS (Strange Loop 2016)
The Zen of High Performance Messaging with NATS (Strange Loop 2016)The Zen of High Performance Messaging with NATS (Strange Loop 2016)
The Zen of High Performance Messaging with NATS (Strange Loop 2016)wallyqs
 
4 - OOP - Taste of Smalltalk (VisualWorks)
4 - OOP - Taste of Smalltalk (VisualWorks)4 - OOP - Taste of Smalltalk (VisualWorks)
4 - OOP - Taste of Smalltalk (VisualWorks)The World of Smalltalk
 
Vert.x for Microservices Architecture
Vert.x for Microservices ArchitectureVert.x for Microservices Architecture
Vert.x for Microservices ArchitectureIdan Fridman
 
Atom The Redis Streams-Powered Microservices SDK: Dan Pipemazo
Atom The Redis Streams-Powered Microservices SDK: Dan PipemazoAtom The Redis Streams-Powered Microservices SDK: Dan Pipemazo
Atom The Redis Streams-Powered Microservices SDK: Dan PipemazoRedis Labs
 
TorqueBox: The beauty of Ruby with the power of JBoss. Presented at Devnexus...
TorqueBox: The beauty of Ruby with the power of JBoss.  Presented at Devnexus...TorqueBox: The beauty of Ruby with the power of JBoss.  Presented at Devnexus...
TorqueBox: The beauty of Ruby with the power of JBoss. Presented at Devnexus...bobmcwhirter
 

Ähnlich wie ZeroMQ: Super Sockets - by J2 Labs (20)

Socket programming with php
Socket programming with phpSocket programming with php
Socket programming with php
 
ØMQ - An Introduction
ØMQ - An IntroductionØMQ - An Introduction
ØMQ - An Introduction
 
ZeroMQ - Sockets on steroids!
ZeroMQ - Sockets on steroids!ZeroMQ - Sockets on steroids!
ZeroMQ - Sockets on steroids!
 
Python networking
Python networkingPython networking
Python networking
 
Lindsay distributed geventzmq
Lindsay distributed geventzmqLindsay distributed geventzmq
Lindsay distributed geventzmq
 
Zmq in context of openstack
Zmq in context of openstackZmq in context of openstack
Zmq in context of openstack
 
Of the variedtypes of IPC, sockets arout and awaythe foremostcommon..pdf
Of the variedtypes of IPC, sockets arout and awaythe foremostcommon..pdfOf the variedtypes of IPC, sockets arout and awaythe foremostcommon..pdf
Of the variedtypes of IPC, sockets arout and awaythe foremostcommon..pdf
 
TCP Sockets Tutor maXbox starter26
TCP Sockets Tutor maXbox starter26TCP Sockets Tutor maXbox starter26
TCP Sockets Tutor maXbox starter26
 
Servers with Event Machine - David Troy - RailsConf 2011
Servers with Event Machine - David Troy - RailsConf 2011Servers with Event Machine - David Troy - RailsConf 2011
Servers with Event Machine - David Troy - RailsConf 2011
 
The Zen of High Performance Messaging with NATS
The Zen of High Performance Messaging with NATSThe Zen of High Performance Messaging with NATS
The Zen of High Performance Messaging with NATS
 
The Zen of High Performance Messaging with NATS
The Zen of High Performance Messaging with NATS The Zen of High Performance Messaging with NATS
The Zen of High Performance Messaging with NATS
 
The Zen of High Performance Messaging with NATS (Strange Loop 2016)
The Zen of High Performance Messaging with NATS (Strange Loop 2016)The Zen of High Performance Messaging with NATS (Strange Loop 2016)
The Zen of High Performance Messaging with NATS (Strange Loop 2016)
 
4 - OOP - Taste of Smalltalk (VisualWorks)
4 - OOP - Taste of Smalltalk (VisualWorks)4 - OOP - Taste of Smalltalk (VisualWorks)
4 - OOP - Taste of Smalltalk (VisualWorks)
 
Tcpip
TcpipTcpip
Tcpip
 
Vert.x for Microservices Architecture
Vert.x for Microservices ArchitectureVert.x for Microservices Architecture
Vert.x for Microservices Architecture
 
Ømq & Services @ Chartboost
Ømq & Services @ ChartboostØmq & Services @ Chartboost
Ømq & Services @ Chartboost
 
10 Networking
10 Networking10 Networking
10 Networking
 
Multi user chat system using java
Multi user chat system using javaMulti user chat system using java
Multi user chat system using java
 
Atom The Redis Streams-Powered Microservices SDK: Dan Pipemazo
Atom The Redis Streams-Powered Microservices SDK: Dan PipemazoAtom The Redis Streams-Powered Microservices SDK: Dan Pipemazo
Atom The Redis Streams-Powered Microservices SDK: Dan Pipemazo
 
TorqueBox: The beauty of Ruby with the power of JBoss. Presented at Devnexus...
TorqueBox: The beauty of Ruby with the power of JBoss.  Presented at Devnexus...TorqueBox: The beauty of Ruby with the power of JBoss.  Presented at Devnexus...
TorqueBox: The beauty of Ruby with the power of JBoss. Presented at Devnexus...
 

ZeroMQ: Super Sockets - by J2 Labs

  • 1. ZeroMQ: Super Sockets By James Dennis (@j2labs) http://j2labs.net
  • 2. What is ZeroMQ? Message Passing You still design the payload Fault tolerant A message IS your payload even when broken into multiple packets Many messaging “patterns” offered
  • 3. What is ZeroMQ? Super Sockets Multiple networking protocols(tcp, ipc, whatevz!) Language agnostic (bindings for: ruby, python, java, c, lua, lisp, scheme and more…) Feels like a scripting language for sockets Socket behaviors Round-robin routing Publishing to subscribers Sending job requests
  • 4. What ZeroMQ is not… Persistent reasonably easy to add A message queue there are literally zero mq’s here An out-of-the-box solution instead, it’s easy to build any solution remember, they’re like scripting sockets
  • 5. Socket Types (transports) tcp :: regular TCP sockets socket.connect(“tcp://…”) ipc :: interprocess communication (unix sockets) socket.connect(“ipc://…”) in-proc :: in process communication socket.connect(“in-proc://…”) pgm – pragmatic multicast (or epgmvia udp) socket.connect(“pgm://…”)
  • 6. Socket Behaviors Actually called patterns They exist to make typical cases easy Request-Reply Publish-Subscribe Pipeline Exclusive pair you probably don’t want this
  • 7. Patterns: PUSH/PULL Sends messages to multiple connected hosts Passes messages ROUND ROBIN to connections Blocks PUSH socket if no connections are present N hosts PULL messages from pusher Connect/Remove new hosts as necessary
  • 8. Patterns: PUSH/PULL Server: Sends a message import zmq ctx = zmq.Context() s = ctx.socket(zmq.PUSH) s.bind("ipc://*:5678") while True: s.send(”blablabla") # blocks if no receivers print “Sent one”
  • 9. Patterns: PUSH/PULL Client: Prints messages from server socket import zmq ctx = zmq.Context() s = ctx.socket(zmq.PULL) s.connect("ipc://*:5678") while True: msg = s.recv() print 'Got msg:', msg
  • 10. Patterns: PUSH/PULL A typical worker pipeline looks a bit like this. What’s that PUB/SUB stuff?
  • 11. Patterns: PUB/SUB PUB socket sends topical payloads Can start streaming messages right away ZMQ discards messages if no subscribers exist SUB socket subscribes to “topic” receives all messages reads topic and discards accordingly happens under the hood.
  • 12. Patterns: PUB/ SUB Server: sends messages to `whatevz` s = ctx.socket(zmq.PUB) s.bind("tcp://127.0.0.1:5566") topic = ’whatevz’ message = json.dumps({'msg': 'what up dude?'}) while True: s.send(topic, message) # send to topic print ‘Sent one’
  • 13. Patterns: PUB/SUB Client: demonstrates subscribing to `whatevz` s = ctx.socket(zmq.SUB) s.connect("tcp://127.0.0.1:5566") topic = "whatevz" s.setsockopt(zmq.SUBSCRIBE, topic) # set socket option while True: msg = s.recv() print 'Got msg:’, json.loads(msg)
  • 14. Patterns: REQ/REP One to one messaging Each REQ message must have a REP response Synchronous communication Will error if two REQ’s are sent before one REP
  • 15. Patterns REQ/REP Client context.socket(zmq.REQ) # btw… REQ sockets are blocking! Worker(could say “server”) context.socket(zmq.REP) Multiple Connections One-to-one messaging is useful for workers getting work Basically, pull work request & reply when done Client may need to multiple workers Can’t just open more sockets… we need a new pattern…
  • 16. Patterns: REQ/REP CLIENTsends a “Hello” SERVERsends a “world” back The REQ socket can send a message, but must not send anything else until it gets a response or ZMQ will error. Likewise, a REP socket cannot send a message unless it first received one.
  • 17. Synchronized PUB/SUB Handling “no broker” (aka building one) More than one socket used Like having OneMQ now Hypothetical Communication Order: Subscriber sends sub REQ for topic Publisher REP - “listen <tcp://here>” Subscriber listens to PUB socket
  • 18. Patterns: XREQ/XREP XREQ is a historical name for DEALER A DEALER asynchronously sends to *REP sockets It can send to XREP (aka: ROUTER) sockets too XREP is a historical name for ROUTER A ROUTER asynchronously handles *REQ sockets It can handle DEALER (aka: XREQ) sockets too.
  • 19.
  • 20. ROUTER processes message and sends to some worker via the DEALER
  • 21. DEALERsends message to worker and waits for one response
  • 22. WORKER processes message, sends response back
  • 23. DEALER get message from WORKER, sends along to ROUTER
  • 24. ROUTERgets message and sends back to CLIENT
  • 25.
  • 26. Broker Model: aka “The Titanic”
  • 27. Broker Model with ZMQ Every connection is a REQ socket ROUTER socket to all connections LRU Queue (your broker code!) Up to the LRU Queue handle socket identities properly REQ/REP gets that for free.
  • 28. Flexibility Via Decentralization “ØMQ is like Git(essentially distributed) whereas AMQP is like SVN (essentially centralized)” – James Dennis (me) Welcome From AMQP: http://www.zeromq.org/docs:welcome-from-amqp
  • 29. Mongrel2 (Zed Shaw’s second web server) http://mongrel2.org
  • 30. Brubeck Mongrel2 handler in Python Modeled after Tornado with ideas from Flask / Bottle Uses Eventlet Coroutines! Nonblocking I/O! Uses ZeroMQ Write back end services in any language! Uses DictShield I wrote this too! Database agnostic modeling! ZeroMQ is language agnostic, right?! https://github.com/j2labs/brubeck
  • 31. Try it! https://github.com/j2labs/zmq_examples http://mongrel2.org https://github.com/j2labs/brubeck Read more! http://nichol.as/zeromq-an-introduction http://zguide.zeromq.org/
  • 32. Questions ? James Dennis (@j2labs) jdennis@gmail.com