Building Reactive Applications with DDS

Building Reactive
Applications with DDS
Angelo Corsaro, PhD
Chief Technology Oﬃcer
OMG DDS SIG Co-Chair
angelo.corsaro@prismtech.com

PrismTech

• Reactive Systems were characteristic of “niche” applications domains such
as Industrial, Military, Telco, Finance and Aerospace

• To address the challenges imposed contemporary Internet Applications an
increasing movement of architects and developers is recognising the
importance of the techniques and technologies used for building Reactive
Systems

PrismTech

Copyright PrismTech, 2014

Reactive Systems Go Mainstream

Integrated Modular Vetronics

Air Trafﬁc Control & Management

PrismTech

Training & Simulation Systems

Unmanned Air Vehicles

Naval Combat Systems

Aerospace Applications


Defense and Aerospace

Autonomous Vehicle Systems

Train Control / Management Systems

PrismTech

Smart Grids / Power Generation

Smart Cities

Connected Medical Devices

Financial Services


Industrial Internet

Reactive Manifesto
• The techniques and

Responsive

Event-Driven

Resilient

• Recently, the Reactive

Manifesto was defined to
summarise the key traits of
“Reactive Applications”

Scalable
http://www.reactivemanifesto.org/

PrismTech


technologies traditionally
used in a class of Reactive
and Interactive Systems is
gaining quite a bit of
attention from the
mainstream developers
community

Reactive Manifesto
• Event-Driven: modular,
• Responsive: Timely react to
stimuli

• Resilient: Tolerate failures
functionally and ideally
temporally

• Scalable: Gracefully scale up
and down depending on
load demands

Event-Driven

Resilient

Scalable
http://www.reactivemanifesto.org/

PrismTech


Responsive

asynchronous

Reactive Systems

1

• A Reactive System designates a permanently operating system that
responds, reacts, to external stimuli at a speed determined by its
environment, e.g. an industrial control system.

[1] Nicolas Halbwachs, Synchronous Programming of Reactive Systems

PrismTech


Reactive Systems

Transformational Systems

• A Transformational System designates a system that terminates after
transforming its input into its output, e.g. a compiler

PrismTech


Transformational Systems

Interactive Systems

• An Interactive System designates a systems that permanently

communicates with its environment at its own speed, e.g. a computer
game

PrismTech


Interactive Systems

Reactive vs. Interactive Systems
Interactive

• react to an environment which

• react to an environment which

• response time needs to be

• response time needs to be

• distributed concurrent systems

• distributed concurrent systems

• data-centric

• data-centric

“cannot wait”, e.g. reacting late
has harsh consequences
deterministic

“doesn’t like to wait”, e.g. reacting
late induces QoS degradation
acceptable

The term reactive systems is increasingly used to denote interactive systems with stringent
response time, performance and scalability constraints.
PrismTech


Reactive

Reactive Systems Key Traits
Concurrency

• Aside from the concurrency between the system and its environment, it is natural to

Strict Temporal Requirements

• Reactive Systems have strict requirements with respect to the rate at which the need to
process input as well as the response time for their outputs

Determinism

• The output of these systems are generally determined by the input and the occurrence
time, e.g. no scheduling effects on the temporal properties of the output

Reliability

• Reactive Systems are often life/mission critical, as such reliability is of utmost importance
PrismTech


decompose these systems as an ensemble of concurrent components that cooperate
to achieve the intended behaviour

Architectural Styles for
Reactive Systems

Stream Processing
• Stream Processing is a commonly adopted architectural style for building
• Stream Processing is often applied in:
-

Interactive Systems

-

Signal Processing Systems

-

Functional Stream Programming

-

PrismTech

Reactive Systems

Data Flow Systems


systems that operate over continuous (theoretically infinite) streams of data

Stream Processing
for modelling systems that react to streams
of data produced by the cyber-phisical
world, such as data produced by sensors,
the stock market, etc.

• Stream Processing Systems operate in realtime over streams and generate in turns
other streams of data providing
information on what is happening or
suggesting actions to perform, such as by
stock X, raise alarm Y, or detected spatial
violation, etc.

PrismTech


• Stream Processing Architectures are ideal

Stream Processing
• Stream Processing Systems are typically modelled

• Modules usually play one of the following roles:
-

-

PrismTech

Filter

Sources: Injecting data into the System

Filter

Filter

Filters/Actors: Performing some computation over
sources
Sinks: Consuming the data produced by the system


as collection of concurrent modules
communicating via typed data channels

Source

Stream

Sink

Streams in DDS
infinite sequence of data samples
of a given type T

Filter

• As such, in DDS, a stream can be

naturally represented with a Topic

• Example:
-

PrismTech

Topic<PressureType>
Topic<MarketData>

Filter

Filter
Stream


• A stream represents a potentially

Topic
• A Topic defines a domain-wide information’s class


• A Topic is defined by means of a (name, type, qos)
tuple, where

type: is the programming language type associated
with the topic. Types are extensible and evolvable

Name

-

qos: is a collection of policies that express the nonfunctional properties of this topic, e.g. reliability,
persistence, etc.

Topic
Qo
S

-

PrismTech

name: identifies the topic within the domain

e
Typ

-

•As explained in the previous slide a topic defines a class/type of information
•Topics can be defined as Singleton or can have multiple Instances
•Topic Instances are identified by means of the topic key
•A Topic Key is identified by a tuple of attributes -- like in databases
•Remarks:
• A Singleton topic has a single domain-wide instance
• A “regular” Topic can have as many instances as the number of

different key values, e.g., if the key is an 8-bit character then the topic
can have 256 different instances

PrismTech


Topic and Instances

• Sources in DDS are mapped to

Filter

DataWriters for a given Topic

• Sinks in DDS are mapped to

Filter

Filter

DataReaders for a given Topic
Source

PrismTech


Source and Sinks in DDS

Sink

Filters in DDS
application business logic

• Each Filter has one or more input stream
and one or more output streams

• Obviously, a Filter consumes data from an
input stream through a DDS data reader
and pushed data into an output stream
through a DDS data writer

PrismTech

Filter
Filter

Filter


• Filters implement bit and pieces of the

Loose Coupling
• Anonymous Communication => No Spatial Coupling
• Non-Blocking and Asynchronous Interaction => No control or temporal
coupling

In DDS the only thing that matters is the kind of information a module is
interested in consuming or producing — who is producing the data or when
the data has been produced is not relevant

PrismTech


Stream Processing Architectures in general and DDS-based architecture in
particular promote loose coupling

Location Transparency
• Location Transparency is an important

architectural property as it makes it
possible to completely decouple the
logical decomposition of a system from
its actual physical deployment

• DDS Automatic Discovery brings

location transparency to the next level
by enabling systems that are zero-conf,
self-forming and self-healing

Module
Logical
Deployment Unit

Physical
Deployment Unit

Module

Module
PrismTech

Module

Module


Module

Temporal Decoupling
Volatile
Durability

decoupling by means of its Durability
Policy

• As such the data produced by a
Source can be:
-

-

-

PrismTech

Source

t

Sink

t

Sink

t

Volatile: available for those that are
around at the time at production
Transient: available as far as the
system/source is running
Durable: available forever

Transient
Durability
Source

t

Sink

t

Sink

t


• DDS provides full control over temporal

Failure Detection
traditional faults as well as performance
failures

• The Fault-Detection mechanism is

Source

TD

Sink

t

controlled by means of the DDS Liveliness
policy

• Performance Failures can be detected

using the Deadline Policy which allows to
receive notification when data is not
received within the expected delays

Fault
Notiﬁcation

Source

t

Sink
P

P

P

Performance
Failure
Notiﬁcation
PrismTech

t


• DDS provides mechanism for detecting

Fault-Masking
mechanism that allow to replicate
Sources and transparently switch
over when a failure occurs

• At any point in time the “active”

source is the one with the highest
strength. Where the strength is an
integer parameter controller by the
user

PrismTech

Source
Source

t

Sink

t


• DDS provides a built-in fault-masking

Performance
-

15-20 usec Inter-Core Latency

-

45-75 usec over GBps Ethernet

• Throughput
-

PrismTech

Up to 10+M msg/sec inter-core
Up to 5M msg/sec inter node


• Latency

Standard and Interoperable
introduced as an OMG standard in 2006 to
address Real-time and interoperable Data
Sharing in Network Centric Systems

• Key requirements for the standard were

high and predictable performance, and
scalability to ultra-large-scale deployments

• DDS is recommended by key administration
worldwide, e.g. DoD, MoD, EUROCAE, etc.
and widely adopted across application
domains, e.g., Automated Trading,
Simulations, SCADA, Telemetry, etc.

PrismTech


• The Data Distribution Service (DDS) was

Dealing with Events
environment

• Stimuli coming from the cyber-physical world are often represented as
events to be handled by the Reactive/Interactive System
-

e.g. think about the event from a GUI, or the data available event in DDS

• Events are often handled through some form of callbacks, such as listeners,
functors, etc. This stile of event handling leads to inversion of control

PrismTech


• Reactive and Interactive Systems continuously interact with their

The Callback Hell
• The Inversion of Control induced by callback-based code in known to
• As an mental exercise, think about the call-back style code you’d have to
write to in Java for drawing when the mouse is being dragged. Although
the logic is simple, it is far cry from having a “declarative style”

• The problem of callback management, infamously known as Callback Hell,
has become even more important due to the surge of Reactive Systems…

• Can we escape the Callback Hell?

PrismTech


make applications brittle and harder to understand

Reactive Programming
• Reactive Programming is a paradigm based on a relaxed form of
• The Reactive Programming is built around the notion of continuous timevarying values and propagation of change

• Reactive Programming facilitates the declarative development of non-

blocking event driven applications — in essence you express what should
be done and the runtime decides when to do it

• Reactive Programming was popularised in the context of Functional

Programming Languages by the seminal done by Elliot and Hudak in 1997
as part of Fran — a framework for composing richly interactive multimedia
animations

PrismTech


Synchronous Dataflow Programming

execute typical instruction
fetch from L1 cache memory
branch misprediction
fetch from L2 cache memory
Mutex lock/unlock
fetch from main memory
send 2K bytes over 1Gbps network
read 1MB sequentially from memory
fetch from new disk location (seek)
read 1MB sequentially from disk
send packet US to Europe and back

1/1,000,000,000 sec = 1 nanosec
0.5 nanosec
5 nanosec
7 nanosec
25 nanosec
100 nanosec
20,000 nanosec
250,000 nanosec
8,000,000 nanosec
20,000,000 nanosec
150 milliseconds = 150,000,000 nanosec
http://norvig.com/21-‐days.html

PrismTech


Time for Common Operations

Reactive Programming Libraries
There is an increasing number of Reactive Programming framework. The Rx
framework introduced by Erik Meijer is becoming a reference
.NET Rx

• Arguably the first framework that brought Reactive Programming to the
masses. Available for the .NET platform

RxJava (http://bit.ly/rxjava)

• An Open Source implementation of Rx for the JVM contributed by NetFlix, it
supports, Java, Scala, Kotlin, Clojure, Groovy and JRuby

PrismTech


!

RxJava Overview
for composing asynchronous and event-based programs by using
observable sequences.

• RxJava allow you to compose streams of data declaratively while

abstracting away low level concerns such as threading, synchronization,
concurrent data structures, and non-blocking I/O

• The two main concepts at the foundation of RxJava are Observables and
Observers

PrismTech


• RxJava is a Java implementation of .NET Reactive Extensions — a library

RxJava Observables
• RxJava Observables allow the composition of flows and sequences of
• You can think of Observables as some kinds of asynchronous collections
that push data to you

• In essence an observable can be created from any synchronous or
asynchronous stream of data.
-

-

PrismTech

you can create an observables that represents the data coming into a DDS
Data Reader, the events from a Button, or even time
you can also create an observable that represent an actual container


asynchronous data

Observable Examples
0

1

2

3

4

5

t

n

val
evens:
Observable[Long]
=
ticks.filter(n
=>
n
%
2
==
0)

x

0

1

x

2

3

4

x
5

t

n

val
bufs:
Observable[Seq[Long]]
=
evens.buffer(2,1)
0
PrismTech

2

4

6

8

n

t


val
ticks:
Observable[Long]
=
Observable.interval(1
seconds)

• The Observer receives

notification concerning new
values, errors or the
completion of the stream —
e.g. no more data

PrismTech

trait
Observer[-‐T]
{

def
onNext(value:
T):
Unit

def
onError(error:
Throwable):
Unit

def
onCompleted():
Unit

}


RxJava Observers

• Beside from the standard RxJava/RxScala Observable, each framework

that wants to integrate with RxJava/RxScala has to provide its own factory
methods to create observables

• For DDS you have to use the DdsObservable defined as follows:
object
DdsObservable
{

def
fromDataReaderData[T](dr:
DataReader[T]):
Observable[T]

def
fromDataReaderEvents[T](dr:
DataReader[T]):
Observable[ReaderEvent[T]]

//
more
methods
which
we’ll
ignore
for
the
time
being

}

PrismTech


DDS Observables

Shapes Application
Application to illustrate the benefits of
using Reactive Programming with DDS

• Three Topics
-

Circle, Square, Triangle

• One Type:
struct ShapeType {
string color;
Pierced shapes represent publications
long x;
long y;
long shapesize;
Spotted shapes represent subscriptions
};
#pragma keylist ShapeType color

PrismTech


• We will use the omnipresent Shapes

• To appreciate the joys of Reactive lets look first at how to implement a
Shapes Logger using traditional “callback” programming

PrismTech


Logging Shapes

Step 1: Implement Listener
class
ShapesListener
extends
DataReaderListener[ShapeType]
{

!

def
onRequestedDeadlineMissed(p1:
RequestedDeadlineMissedEvent[ShapeType]):
Unit

def
onRequestedIncompatibleQos(p1:
RequestedIncompatibleQosEvent[ShapeType]):
Unit

!

def
onSampleRejected(p1:
SampleRejectedEvent[ShapeType]):
Unit

!

def
onLivelinessChanged(p1:
LivelinessChangedEvent[ShapeType]):
Unit

!

def
onDataAvailable(p1:
DataAvailableEvent[ShapeType]):
Unit

!

def
onSubscriptionMatched(p1:
SubscriptionMatchedEvent[ShapeType]):
Unit

!

def
onSampleLost(p1:
SampleLostEvent[ShapeType]):
Unit

}

PrismTech


!

Step 2: Read and Log Data
• From the listener code you’d retrieve the data reader (dr) and do
something like this:

def
onDataAvailable(p1:
DataAvailableEvent[ShapeType]):
Unit
=
{

dr.read.foreach(s
=>
println("("
+
s.color
+
",
"
+
s.x
+
",
"
+
s.y))

}

!

PrismTech


• To keep things simple and more compact we’ll use the Scala binding

Molière Makes it Better
write:

dr
listen
{

case
DataAvailable(_)

=>
{

dr
read()
foreach
(s
=>
println("("
+
s.color
+
",
"
+
s.x
+
",
"
+
s.y))

}

}

PrismTech


• To be fair, if you use Moliere, the Scala API for OpenSplice you could simply

val
shapes
=
DdsObservable.fromDataReaderData(dr)

shapes.subscribe(s
=>

println("("
+
s.color
+
",
"
+
s.x
+
",
"
+
s.y)

)

PrismTech


Reactive Logger

To show a bit more the expressive power of Reactive programming, let’s see
how we can solve the following problems

PrismTech


“Shapes Problems”

• We want to show a triangle that

positioned midway between every
single pair of circle and square of the
same colour

PrismTech


Problem 1: Shape in the Middle

Shapes Marble Diagrams
t

• Conceptually, we need to select on stream, look at it one element at the time
and match it with the corresponding item in the other stream

• Then, we can produce the average
PrismTech


t

x x

t
t

PrismTech


t

x x x x

t
t

P.S.: The dropping may seem a bit counter-intuitive but it is in part due to how the DDS streams work
PrismTech


t

val
circles
=
DdsObservable.fromDataReaderData
{

DataReader[ShapeType](Topic[ShapeType](circle))

}

val
squares
=
DdsObservable.fromDataReaderData
{

DataReader[ShapeType](Topic[ShapeType](square))

}

val
ttopic
=
Topic[ShapeType](triangle)

val
tdw
=
DataWriter[ShapeType](ttopic)

!
!

//
Compute
the
average
between
circle
and
square
of
matching
color
with
flatMap

val
triangles
=
circles.flatMap
{

c
=>
squares.dropWhile(_.color
!=
c.color).take(1).map
{

s
=>
new
ShapeType(s.color,
(s.x
+
c.x)/2,
(s.y
+
c.y)/2,
(s.shapesize
+
c.shapesize)/4)

}

}

!

triangles.subscribe(tdw.write(_))

PrismTech


Midpoint Triangle

//
Compute
the
average
between
circle
and
square
of
matching
colour
with
flatMap

val
triangles
=
circles.flatMap
{

c
=>
!=
c.color).take(1).map
{

s
=>
new
ShapeType(s.color,
(s.x
+
c.x)/2,
(s.y
+
c.y)/2,
(s.shapesize
+
c.shapesize)/4)

}

}

//
Compute
the
average
between
circle
and
square
of
matching
colour
with
for
comprehension

val
triangles
=
for
{

c
<-‐
circles;

s
<-‐
!=
c.color).take(1)

}
yield
new
ShapeType(s.color,
(s.x
+
c.x)/2,
(s.y
+
c.y)/2,
(s.shapesize
+
c.shapesize)/4)

PrismTech


Using for-comprehension

Concluding Remarks
• Many of the requirements of traditional Reactive Systems are applicable to
• Stream Processing Architecture and Reactive Programming can greatly
help in the design and implementation of Reactive Systems that are
efficient, responsive, resilient, scalable and highly available

• As proven through years of deployments DDS provides the ideal
infrastructure technology for Reactive Systems

• The combination of DDS with Reactive Programming makes for a very
powerful platform!

PrismTech


many mainstream application domains

• OpenSplice DDS [http://prismtech.com]
• RxJava [http://bit.ly/rxjava] (OpenSouce Rx Implementation from NetFlix)
• Moliere [http://bit.ly/nuvo-moliere] (This includes Rx Binding for DDS)

PrismTech


References

Building Reactive Applications with DDS

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie Building Reactive Applications with DDS

Ähnlich wie Building Reactive Applications with DDS (20)

Mehr von Angelo Corsaro

Mehr von Angelo Corsaro (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Building Reactive Applications with DDS