Fundamental to any distributed system are communication patterns: point-to-point, request-reply, transactional queues, and publish-subscribe. Large distributed systems often employ two or more communication patterns. Using a single middleware that supports multiple communication patterns is a very cost-effective way of developing and maintaining large distributed systems. This talk will begin with an introduction of Data Distribution Service (DDS) – an OMG standard – that supports data-centric publish-subscribe communication for real-time distributed systems. DDS separates state management and distribution from application logic and supports discoverable data models. The talk will then describe how RTI Connext Messaging goes beyond vanilla DDS and implements various communication patterns including request-reply, command-response, and guaranteed delivery. You will also learn how these patterns can be combined to create interesting variations when the underlying substrate is as powerful as DDS. We’ll also discuss APIs for creating high-performance applications using the request-reply communication pattern.

1. Sumant Tambe, Ph.D.
Senior Software Research Engineer
Real-Time Innovations, Inc.
www.rti.com

2. Agenda
• Communication Models
• Data Distribution Service Standard
• Data-Centricity 101
– Demo
• Communication Patterns
– Request/Reply
– Guaranteed Delivery
10/11/2012 2

3. Middleware Communication Models
10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 3

4. RPC Vs. Pub-Sub/Messaging/Data-Distribution
• RPC (WebServices, CORBA, DCOM) offer a remote method abstraction
– Familiar OO programming model
– Results in a tightly-coupled system
• Forces synchronous invocations
• Imposes global object model
• Limited QoS(appearance of local method call)
• Lack robustness: cascading points of failure
– Typically built on top of TCP:
• Impacts scalability and time-determinism
– Best-suited to smaller, closely-coupled systems
• Pub-Sub (Messaging Data-Distribution) offer a queue-based and/or
replicated-data model
– Subsystems are decoupled in time, space, and synchronization
• Contracts established by verifying QoS compatibility
– Supports a variety of transports including multicast UDP
– Better suited for high-performance and real-time
10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 4

5. Queue Vs. Pub/Sub
• Queue
– Message sent to Queue
– Multiple readers can read from the queue
– Each message is delivered to ONLY one reader
• Readers “affect each other”
– Apps:
• Job Scheduling, Load Balancing, Collaboration
• Pub/Sub
– Message sent to Topic
– Multiple readers can subscribe to Topic with or
without filters
– Each message delivered to ALL subscribers that
pass filter
• Readers are decoupled
– Apps:
• Notifications, Information Distribution
10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 5

6. Pub/Sub Vs. Data-Distribution
• Pub-Sub
– Only messages no concept of data
– Each message is interpreted without context
– Messages must be delivered FIFO or according to
some “priority” attribute
– No Caching of data
– Simple QoS: filters, durability, lifespan
• Data-Distribution
– Messages represent update to data-objects
– Data-Objects identify by a key Data Bus
– Middleware maintains state of each object
– Objects are cached. Applications can read at
leisure
– Smart QoS
• Ownership, History (per key), Deadline
– Subsumes Pub-Sub
10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 6

7. DDS: Standards-based Integration Infrastructure for
Critical Applications
Streaming
Sensors Events
Data
Real-Time Enterprise
Actuators
Applications Applications
© 2009 Real-Time Innovations, Inc.

8. Systems that interact with the Real World
• Must adapt to changing environment
• Cannot stop processing the information
• Live within world-imposed timing
Beyond traditional interpretation of real-time
© 2010 Real-Time Innovations, Inc.

9. Real-Time Data Distribution
10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 9

10. Family of Specifications
2008 2009 2010 2010 2012 2013
UML Profile DDS for DDS DDS-STD-C++ DDS Security RPC over DDS
for DDS Lw CCM X-Types DDS-JAVA5
App 2004 App App
DDS Spec
DDS 2006 DDS DDS
Implementation DDS Implementation Implementation
Interoperablity
Network / TCP / UDP / IP
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL 10

11. Broad Adoption
• Vendor independent
Cross-vendor portability
– API for portability
– Wire protocol for interoperability
• Multiple implementations
– 10 of API DDS API
– 8 support RTPS
• Heterogeneous Middleware
– C, C++, Java, .NET (C#, C++/CLI)
– Linux, Windows, VxWorks, other DDS Real-Time
embedded & realtime Publish-Subscribe
Wire Protocol (RTPS)
• Loosely coupled
Cross-vendor interoperability
© 2009 Real-Time Innovations, Inc. 11

12. DDS adopted by key programs
• European Air Traffic Control
– DDS used to interoperate ATC centers
• UK Generic Vehicle Architecture
– Mandates DDS for vehicle comm.
– Mandates DDS-RTPS for interop.
• DISR (DoD Information Technology
Standards Registry)
– Mandates DDS for Pub-Sub API and Interop.
• US Navy Open Architecture
– Mandates DDS for Pub-Sub
• SPAWAR NESI
– Mandates DDS for Pub-Sub SOA © 2010 Real-Time Innovations, Inc.

13. Evolution of DataBus
Data-centricity basics

14. Everyday Example: Schedule Meeting via Emails
Alternative Process #1 (message-centric):
1. Email: “Meeting Monday at 10:00.”
2. Email: “Here’s dial-in info for meeting…”
3. Email: “Meeting moved to Tuesday”
4. You: “Where do I have to be? When?”
5. You: (sifting through email messages…)
14

15. Everyday Example: Schedule Meeting Using a Calendar
Alternative Process #2:
1. Calendar: (add meeting Monday at 10:00)
2. Calendar: (add dial-in info)
3. Calendar: (move meeting to Tuesday)
4. You: “Where do I have to be? When?”
5. You: (check calendar. Contains
consolidated-state)
The difference is state!
The infrastructure consolidates changes and maintains it
15

16. What’s the Difference? State.
• Objects have identity and “State” (“data”) is a
attributes
– The meeting will run 1:00– snapshot of those
2:00 in the conference room. attributes and
– My friend’s phone number is characteristics.
555-1234 his email is…
– The car is blue and is
traveling north from
Sunnyvale at 65 mph. If the infrastructure
• …whether they exist in maintains the state,
the real world, in the the application does
computer, or both not need to re-
• …whether or not we construct it…
observe or acknowledge
them

17. Why is it better to have the (data-centric)
middleware manage the state?
• Reconstructing the state of an object is hard
– Must infer based on all previous messages
– Maintaining all these messages is expensive
– Each app makes these inferences  duplicate effort
• Reconstructing state is not robust
– Many copies of state  may be different  bugs
vs. Uniform operations on state  fewer bugs
– Any missing change compromises integrity
• State awareness results in better performance
– Middleware can be smart about what to send and when
• Data-type awareness simplifies programming
– Middleware supports direct definition and instantiation of the data-
model

18. DDS Communication Model
10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 18

19. Demo: Publish-Subscribe
© 2009 Real-Time Innovations, Inc.

20. DDS Real-Time Quality of Service (QoS) Attributes
QoS Policy QoS Policy
DURABILITY USER DATA
User QoS
HISTORY TOPIC DATA
Volatility
READER DATA LIFECYCLE GROUP DATA
WRITER DATA LIFECYCLE PARTITION
Presentation
LIFESPAN PRESENTATION
Infrastructure
ENTITY FACTORY DESTINATION ORDER
RESOURCE LIMITS OWNERSHIP
Redundancy
RELIABILITY OWNERSHIP STRENGTH
Delivery
TIME BASED FILTER LIVELINESS
Transport
DEADLINE LATENCY BUDGET
CONTENT FILTERS TRANSPORT PRIORITY

21. Communication Patterns
Beyond DDS
10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 21

22. Why Patterns
• From app programming POV
– Patterns solve specific problems nicely
• From platform or systems POV
– Handling each pattern in independent ad hoc ways
results in brittle, poorly performing systems
• Solution: Build platform in terms of
fundamental patterns
– Derive high-level patterns from basic patterns;
expose those to apps
© 2012 RTI 22

23. Communication Pattern
Request/Reply

24. Introduction to Request/Reply
• Publish/Subscribe (Pub/Sub)
– Fundamental pattern in RTI Connext
– Pub/Sub shines where there is
• Push model
• Unidirectional stream of data
• One-to-many (multicast)
• Request/Reply
– Client-server model built on top of pub/sub
– Request/Reply shines where there is
• Pull model
• Bidirectional communication
• A need for remote code execution
– E.g., Get/set configuration, send commands
10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 24

25. Request-Reply
Request
Request Topic
Requestor (Foo) Replier
Reply
Topic Reply
(Bar)
© 2011 Real-Time Innovations, Inc.
COMPANY CONFIDENTIAL

26. Correlation
Correlation
Message ID
1 3 2 1
Requests Replier
1
1 2 3
Requestor
Replies
Correlation ID
© 2011 Real-Time Innovations, Inc.
COMPANY CONFIDENTIAL

27. How Does Correlation Work
Requester App Replier App
sample_id1
Data VGUID1 VSN1 Foo Data
Foo Write Writer Reader
(VGUID1)
Part of Foo
Processing
SampleInfo VSN1 VGUID1
Bar
Read
Data Data
Bar
Reader Writer
VGUID2 VSN2 VGUID1 VSN1 Bar
(VGUID2)
sample_id-id2 related_sample_id1
Index Condition
“@related_sample_id.VSN == VSN1”
Content-based Filter
“@related_sample_id.VGUID == VGUID1”
Filtering Implementation details
10/11/2012 © 2012 RTI RTI • COMPANY CONFIDENTIAL
© 2012 • COMPANY CONFIDENTIAL 27 27

28. Single-Request Multiple-Reply
Request
Requestor Replier
1 2 3
Replies
Sequence ID
© 2011 Real-Time Innovations, Inc.
COMPANY CONFIDENTIAL

29. Multiple Repliers
Reply
Replier A
Requester Replier B
Request
Replier C
© 2011 Real-Time Innovations, Inc.

30. Replier “Hello, World!” (Server-side)
using namespace connext;
DDS::DomainParticipantFactory factory = DDS::DomainParticipantFactory::get_instance();
DDS::DomainParticipant * participant = factory->create_participant(domain_id,
DDS::PARTICIPANT_QOS_DEFAULT,
NULL, DDS::STATUS_MASK_NONE);
std::auto_otr<Replier<Foo, Bar> > replier (new Replier<Foo, Bar>(participant, "TestService"));
Sample<Foo> request;
// Receive one request
bool received = replier->receive_request(request, MAX_WAIT);
if (!received) {
std::cout << "Requests not received" << std::endl;
return false;
}
// A WriteSample automatically initializes and finalizes
// the data using the TypeSupport (in this case BarTypeSupport)
WriteSample<Bar> reply;
if (request.info().valid_data) {
sprintf(reply.data().message, "Reply for %s", request.data().message);
// Send a reply for that request
replier->send_reply(reply, request.identity());
// Note: a replier can send more than one reply for the same request
}
10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 30

31. Requester “Hello, World!” (Client-side)
using namespace connext;
// Create a DomainParticipant
DDS::DomainParticipant * participant = ...;
// Create a Requester
std::auto_ptr<Requester<Foo, Bar> > requester
(new Requester<Foo, Bar>(participant, "TestService"));
// Send request
WriteSample<Foo> request;
strcpy(request.data().message, "A Request");
requester->send_request(request);
if (requester->wait_for_replies(2, MAX_WAIT, request.identity())) {
LoanedSamples<Bar> replies = requester->take_replies(request.identity());
// If there are no replies, the container is empty
for(LoanedSamples<Bar>::const_iterator it = replies.begin();
it != replies.end();
++it) {
if (it->info().valid_data) {
std::cout << "Received reply: " << it->data().message << std::endl;
}
}
}
10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 31

32. Implementing Request/Reply Pattern
Steps With Vanilla DDS API With BIGPINE API
Define request and reply types Manual Manual
Create Participant Manual Manual
Pollute application data model with a Manual Not necessary
correlation-id
Create publisher/subscriber Manual Automatic
Create datareader/datawriter Manual Automatic
Register request/reply types Manual Automatic
Create request/reply topics Manual Automatic
Setup CFT to filter unwanted replies Manual Automatic
Correlation of request/reply samples Manual Semi-Automatic
Memory management of samples Manual Automatic
Lifecycle management of entities Manual Automatic
Language friendly (idiomatic API) Partially Yes
Scalable Depends on impl Yes
10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 32

33. Salient Features of Connext Request/Reply
• New API, built on top of DDS
• Data-centric
• Scalable
• Simple to use, abstracts many DDS details
• Idiomatic, language-friendly API (C++, Java, and
C#)
• It’s not RPC, though can be the foundation for an
RPC solution
10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 33

34. Scalable Request-Reply
Requester 1
Reply 1
Replier Requester 2
Replies Reply 2
…
Thousands of clients! Requester N
Reply N
© 2011 Real-Time Innovations, Inc.
COMPANY CONFIDENTIAL

35. Combining Patterns
Subscriber
Wire Tap
Request
Request Topic
Requestor Replier
Reply
Topic Reply
Subscriber
© 2011 Real-Time Innovations, Inc.
COMPANY CONFIDENTIAL

36. Communication Pattern
Guaranteed Delivery

37. The Guaranteed Delivery Problem
• Delivery of samples from a Producer to the
Intended Consumers such that the delivery is
robust to multiple kind of failures
• Guaranteed delivery does not imply complete
certainty  Continuum of Delivery Guarantees
© 2012 RTI • COMPANY CONFIDENTIAL

38. The Guaranteed Delivery Scenario
• Producer ‘P’ publish a Sample ‘S’ that should
be received by Consumer ‘C’
Consumer process failure
Producer process failure (the process crashes after the sample is
(the producer process crashes after the
received by the middleware and before it is
sample has been written)
processed by the application)
Sample ‘S’
Producer ‘P’ Consumer ‘C’
(DataWriter) (DataReader)
Network and transport failures
(packets are lost due to temporary
network failures or Late joiner consumer
transport/middleware buffer overflows) (the consumer is not running when
the sample is written)
© 2012 RTI • COMPANY CONFIDENTIAL

39. Guaranteed Delivery
App-level ack
Publisher Subscriber
Message Message
Durable
Subscriber
Message
Message
Disk © 2011 Real-Time Innovations, Inc.
COMPANY CONFIDENTIAL

40. DDS Durability QoS Primer
DDS Durability controls data availability with respect to
late joining datareaders
• VOLATILE
– No need to keep data samples for late joiners
• TRANSIENT_LOCAL
– Data instances available for late joiners
– Data does not survive beyond datawriter
• TRANSIENT
– Data is not tied to the lifecycle of the datawriter.
– A separate service (persistence service) keep data in memory
• PERSISTENT
– Data is kept on permanent storage, so that they can outlive a
system session
10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 40

41. Beyond DDS: Required Subscriptions
• DataWriter can be configured with:
– a list of required subscriptions
• Required subscriptions are named subscriptions identified by a
role name
• DataWriter must store a sample until both:
– Acknowledged by all active reliable readers
– Acknowledged by all required subscriptions
• DataReader identifies itself:
– As serving a required subscription
– Uniquely via a virtual GUID

42. Beyond DDS: Durable Subscriptions
• Concept
– It is a required subscription with durability >= TRANSIENT
– Features that allows to keep data until received by durable
subscriptions for which readers may or may not exist at
any given time
– Data is not lost even if the DataWriter crashes
• Benefits and Use Cases
• In combination with other features durable subscriptions
provides guaranteed messaging
• Requires persistence service
– To persist data
– To persist existence of durable subscriptions and their stats

43. Beyond DDS: Collaborative DataWriters
• Concept:
– Multiple DataWriters can relay samples from a common
stream (data source). The DataReaders reconstruct the
original stream and deliver to the application the complete
set of samples
• Benefits & Use cases:
– High availability and fault tolerance
• Multiple DataWriters publishing the same samples
– Load balancing and data partition
• Multiple DataWriters publishing different sets of samples from the
same source
– Scalability
• Partition across different machines and cores

44. Delivery Models: Best-Effort to Guaranteed!
Use Case Features in RTI Connext Messaging 5.0.0
Periodic data from live producers Best effort reliability
Event data from live producers Reliable reliability
Last value cache from live producers (entire Transient local durability +
state of the datawriter cache) Reliable reliability
Durable last value cache (datawriter may Persistent durability +
come and go) Reliable reliability +
Persistence Service
Guaranteed messaging from live producers Transient local durability +
Reliable reliability +
Required subscriptions +
Application level ACK
Durable Guaranteed messaging Persistent durability +
Reliable reliability +
Collaborative DataWriters +
Persistence Service +
Durable subscriptions +
Application level ACK
© 2012 RTI • COMPANY CONFIDENTIAL

45. References
• Real-Time Innovations, Inc.
– www.rti.com
• RTI Connext DDS
– www.rti.com/products/dds
• RTI Connext Request-Reply API
– Just Google!
• OMG DDS Specifications
– www.omg.org/technology/documents/dds_spec_catalog.htm
10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 45

46. Thank you
© 2012 RTI • COMPANY CONFIDENTIAL