Event-Driven Architecture Workshop Agenda and Modern Application Design

Event-Driven Architecture
Workshop
1
A Day In The Streaming Life

AGENDA
2
Discover Event Driven Architecture, Change Data Capture and
Cloud-Native Integration
Introduction
Walkthrough of the lab environment, including a demonstration of
one the labs
Demo
Presenter will distribute the lab URL and credentials
Hands-on Workshop
10:00
11:00

3
How does cloud
change things?

4
Modern application architectures
Ye Olde Architecture Enlightened
postmodernism

5
Modern application architectures
APIs
Events
Data
Enterprise Integration Patterns
Process and Decisions

6
Waterfall
Agile, Incremental, Iterative
Development and delivery model

7
Container
Workload
Key to incremental delivery: Containers

8
What the container is on matters
Platform

9
What’s in the container matters
Runtime
Frameworks
{Your business code}
OS

10
What’s around the container matters
Function-
as-a-service
Messaging
API
management
Rules
Process
automation

11
How it’s all engineered together matters
{Your Code}
Integration
Runtime
Function-
as-a-service
Messaging
API
management
Rules
Process
automation

12
Cloud-Native Community Projects

Red Hat Cloud-Native Dev Platform
13
Our vision is to simplify the creation of cloud-native services and serverless functions with a rich set of
components and tools to match the workloads of modern cloud native apps.
Cloud-native middleware
applications services and
service mesh
Automate Kubernetes
application operations
with DevOps in mind
Tools and standard
processes to increase
developer productivity
on Kubernetes

Supporting Container and Cloud Native workloads
14
PROCESS AUTOMATIONAPPLICATION RUNTIMES
AUTOMATE AND OPTIMIZE
BUSINESS PROCESSES
Develop, deploy, and manage across cloud and on-premise
CORE TOOLS TO BUILD CLOUD
NATIVE & MIGRATE EXISTING APPS
COMPOSE AND INTEGRATE
MICROSERVICES ACROSS AN
ENTERPRISE SERVICE NETWORK
INTEGRATION
Integration with Red Hat Developer, CI/CD tools, & security services
Optimized for Red Hat OpenShift & Kubernetes services
Support organizations desire for choice and process standardization
Emphasis on solution
Simpliﬁed selling motion
Flexible consumption
BROKER

15
Designing
Modern
Applications

Events
Use Cases
● Traditional MOM
● Multicloud
● IoT
● Event streaming
Data
Use Cases
● Data Federation
● Data Virtualization
● Data as APIs
● Change Data Capture
EIPs
Use Cases
● Intelligent Routing
● Service composition
● Connectivity
● iPaaS
● Data Pipelines
APIs
Use Cases
● Microservices
● Mobile Apps
● B2B
● Legacy facade
Elements of Modern Applications
APIs EIPs DataEvents

Introduction:
Events
EVENT-DRIVENARCHITECTURE
18

What is Event Driven Architecture?
19
(EDA) is a way of designing
applications and services to
respond to real-time information
based on the sending and
receiving of information about
individual event notifications

What is an event?
20
Image: https://foto.wuestenigel.com/hand-ready-to-push-domino-pieces/
Event an action or occurrence
recognized by software, often
originating asynchronously
from the external
environment, that may be
handled by the software

What is an event?
21
Source:
Async form of Remote Procedure Call,
contains instructions telling recipient
what to do, may cause a change of state.
QueryCommand
Similar to commands, queries expect a
response returning the results, but does
not cause any change in state.
Event
Immutable state and value of a particular
entity, which occurred during operation
among services.

Types of event consumption patterns
22
Source:
Events stored durably until read by all
registered consumers. Traditional
store-and-forward brokers.
ReplayableConsumable
Events stored durably for specific period
of time or storage capacity. Consumers
can move back and forth of the stream.
Volatile
The event needs to be disseminated to
all consumers online at time of
publication. Not persisted.

Event-driven
Microservices
23

Perfect fit for Microservices
24
● Broadcast
● Parallel Processing
● Modular
● Fault tolerance nature
● real-time responsiveness

25
Event
Sourcing
Aync
Req and Res
Change Data
Capture
Streaming process
Event Driven Architecture
Components
Communication
Query Command Event
Volatile
Consumable
Replayable
Connects
Transformation
Routing
Channel
Cross cluster
On demand
Self-service

Event-driven architecture use cases
26
Streaming between data centers
Reactive notification
Command query
responsibility segregation
(CQRS)
Auditing
Behavior capture
Cache store
Complex event
processing

27
Zero One Two Three Four
There are
absolutely no
asynchronous
communication
in the system.
There are some
adoption of
asynchronized
communication
between
applications.
Proper
streaming and
messaging are
installed with
some level of
high availability
plan.
Ability to
provide limited
self-service
between
departments
Discoverability
of services,
endpoints and
data schemas.
As well as ability
to scale freely
on cloud
platforms.
Provide 100%
scalable self
service for all
developer.
Some
automations are
introduce in to
continuous
delivery
Provide
observability of
the system
environment.
And allow
tracing of
events and
monitoring
performance of
application and
resource usage.
Automations are
built-in to all
continuous
delivery
pipelines
Event Driven Architecture Maturity Level

Event Stream Processing
28

29
A core foundation of modern enterprise IT
Event streaming
Cloud-native Apache Kafka
Topics
Events

30
Educational Slides on Event Streaming
TBD
NOT OURS
CHANGE IT!!!

What is Apache Kafka?
31
Apache Kafka is a distributed system
designed for streams. It is built to be an
horizontally-scalable, fault-tolerant,
commit log, and allows distributed data
streams and stream processing
applications.

Apache Kafka ecosystem
32
● Kafka Core
○ Broker
○ Producer API, Consumer API, Admin API
○ Management tools
● Kafka Connect
● Kafka Streams API
● Mirror Maker / Mirror Maker 2
● REST Proxy for bridging HTTP and Kafka
● Schema Registry
Streams
API
Producer
API
Consumer
API
3rd party
tools
Mirror
Maker
Connect

Running on OpenShift
33
● Red Hat AMQ streams provides:
○ Container images for Apache Kafka, Connect, Zookeeper and
MirrorMaker
○ Kubernetes Operators for managing/configuring Apache Kafka
clusters, topics and users
○ Kafka Consumer, Producer and Admin clients, Kafka Streams
● Upstream Community: Strimzi
○ 100% Open source project licensed under Apache License 2.0
○ Part of the Cloud Native Computing Foundation (CNCF)

Cluster Deployment with AMQ Streams
Topic & User
operators
Zookeeper
cluster
Kafka
cluster
Cluster
operator
Kafka
Custom
Resource

Cluster Update with AMQ Streams
Topic & User
operators
Zookeeper
cluster
Kafka
cluster
Cluster
operator
Kafka
Custom
Resource

Kafka Connect
● Framework for transferring data between Kafka and other data
systems.
● Facilitate data conversion, scaling, load balancing, fault tolerance.
● Connector plugins are deployed into Kafka connect cluster:
○ Well defined API for creating new connectors (with Sink/Source)
○ Apache Kafka itself includes only FileSink and FileSource plugins
(reading records from file and posting them as Kafka messages /
writing Kafka messages to files)
○ Many additional plugins are available outside of Apache Kafka
36

Kafka Streams API
37
● Stream processing framework.
● Streams are Kafka topics (as input and output).
● It’s really just a Java library to include in your application.
● Creates a topology of processing nodes (filter, map, join etc) acting
on a stream:
○ Low level processor API
○ High level DSL
○ Using “internal” topics (when re-partitioning is needed or for
“stateful” transformations)

Kafka Connect + Kafka Streams API
38
Other
System
Other
System
Kafka
Application
Topic
Topic
Processing Processing
Topic
Topic
Topic
Topic
Streams API
Kafka Connect Kafka Connect
Source
Connector
Sink
Connector

Kafka HTTP Proxy Bridge
39
● General Available (GA) since AMQ Streams 1.3
○ Available on RHEL and on OCP
○ On OCP deployed through the Cluster operator
● On RHEL installed and configured manually
○ Can be used to access Kafka using HTTP REST API
○ Tries to maintain compatibility with the other REST Proxies
● But implements only selected features
● AMQP Bridge (part of the same upstream code) is not supported!

Kafka Mirror Maker
40
● Kafka is latency sensitive
○ Zookeeper require low latency to maintain a quorum
○ Kafka is less latency sensitive, but replications and reliable
producers will be slow with high latency networks
● Kafka clusters do not work well when split across multiple
datacenters. It is recommended to setup independent cluster in
each data center and mirror data.
● Usual solution is to create local clusters and mirror data between
them
● Mirroring with Kafka is always asynchronous!

41
Data Center B
Kafka Cluster 1
Topic 1
Partition 0
Data Center A
Kafka Cluster 2
Mirror Maker
Producer
Consumer
Mirror
Maker
Topic 1
Partition 0
Topic 2
Consumer
Producer
Topic 2
Topic 3
Partition 0
Topic 3
Partition 0
Producer
Consumer
Mirror Maker

42
Data access patterns for modern applications
How data patterns evolve in a microservices world
Change Data Capture
Incorporating data in to event-driven architecture
Topics
Data

43
Data Architecture
Traditional n-tier app
DB stays
DB moves
DB embeds

44
Data Access Patterns
DATA VIRTUALIZATION
Federates multiple relational and non-relational sources
Compatible with existing SQL clients
Live CRUD access to back-end data sources
Client-driven interaction
CHANGE DATA CAPTURE
Database changes externalized as events
Process data change events as they happen
Distributed state updates, caching, data lake, etc.
Event-driven interaction

What is Change Data Capture?
45
● Well-established software design pattern
● A system monitors and captures the changes in data so that other
software can respond to those changes
● CDC captures row-level changes to database tables and passes
corresponding change events to a data streaming bus
● Applications can read these change event streams and access the
change events in the order in which they occurred

What is Change Data Capture?
46
Get an event stream with all data and schema changes in your DB
Database
Apache Kafka
?E E E E E E E
Offset 0 Offset 1 Offset 2 Offset 3 Offset 4 Offset 5 Offset 6 Offset 7
Topic

Change Data Capture Connectors
47
● Change Data Capture (CDC) connectors for Kafka Connect
○ Component connects to selected database, reads its
transaction log and publishes it as Kafka messages
○ Supported databases are MySQL, PostgreSQL, MongoDB,
SQL Server
○ The Kafka messages can be send for example in JSON format
● Makes it easy to integrate DB based applications into Kafka
○ No need to write data to DB and send to Kafka
○ Use cases like, for example, microservices integration and data
replication

48
Data Replication
Replicate data to another Database
Feed Analytics system, Data Lake or DWH
Feed data to other teams
Microservices
Excellent fit for microservices architectures:
Propagate data between different services without coupling
Each service keeps optimized views locally
Other Uses
Auditing / Historization
Update / Invalidate caches
Enable full-text search
Update CQRS read models

REST API
ODAT
A
JDBC
ODBC
REST
● Transformation
● Filtering
● Masking
● Authentication
● Authorization
● Query Optimization
● Materialization
● ...
Data Virtualization

50
Cloud-native adoption requires integration
Enterprise integration solves organizational and technical
challenges associated with cloud adoption.
iPaaS
Hype and adoption continue to grow.
Serverless and Knative
Integration concerns begin to surface as first class concepts in
Kubernetes community.
Topics
Enterprise
Integration
Patterns

Organizational Impacts of Cloud Adoption
Central IT as Gatekeeper
Central IT viewed as bottleneck
Developers and LOB taking more ownership
Cloud adoption opens additional paths to production
Best practices and institutional knowledge lost
IT
Central IT as Advisor
Distribute IT knowledge throughout organization
Facilitate self-service and connect participants
Integration Strategy Enablement Team (ISET)
Central IT viewed as catalyst

Technology Impacts of Cloud Adoption
DISTRIBUTED
INTEGRATION
❏ Lightweight
❏ Pattern Based
❏ Event Oriented
❏ Community Sourced
FLEXIBILITY Fuse Standalone Fuse on OpenShift Fuse Online

Apache Camel Going Strong
Ranked #1 ASF
project
considering
commits number
ASF Annual Report 2019
3Camel 3.0 Final - Est. Nov 2019
● 145 releases
● 508 unique contributors
● 2793 stars on Github
● 3639 forks on Github
● 8985 questions on
StackOverFlow
● 68 committers
● 35 PMC members
● 764 Followers on Twitter

54
Cloud-native adoption requires integration
Enterprise integration solves organizational and technical
challenges associated with cloud adoption.
iPaaS
Hype and adoption continue to grow.
Integration concerns begin to surface as ﬁrst class concepts in
Kubernetes community.
Topics
Enterprise
Integration
Patterns

Self-service integration
Self-service
API Design & Implementation
Data Mapping
Connectivity
Intelligent Routing
Extensibility

Self-service messaging
AMQ Console
Create an address
Select type & name it
Choose plan
View summary
Address is created
Namespace view

Kogito
Next-gen Cloud-Native Business Automation
Cloud-Native Business Automation for building intelligent applications,
backed by battle-tested capabilities
A continuation of
Drools, jBPM and
Optaplanner but
completely
redesigned to be
cloud-native!

58
Source:
Quotation: Abraham H. Maslow (1966). The Psychology of Science. p. 15.
Image: https://en.wikipedia.org/wiki/Abraham_Maslow#/media/File:Abraham_Maslow.jpg
I suppose it is tempting, if the
only tool you have is a
hammer, to treat everything
as if it were a nail.
“”
Abraham Maslow, 1966

59
A Complete Foundation for the Event-Driven Enterprise
DESCRIPTION BEST FIT PATTERNS POSSIBLE PATTERNS UNSUITABLE PATTERNS
AMQ BROKER
Traditional messaging broker w/ Queuing & Pub/Sub
Rich feature set inc. JMS 2.0
Best-in-class performance
Based on Apache ActiveMQ Artemis
CONSUMABLE EVENTS
VOLATILE EVENTS
COMMAND
QUERY
REPLAYABLE EVENTS
AMQ INTERCONNECT
Message router
1:1 (anycast) and 1:many (multicast)
Secure messaging backbone for hybrid cloud
Based on Apache Qpid Dispatch Router
COMMAND
QUERY
VOLATILE EVENTS
CONSUMABLE EVENTS
REPLAYABLE EVENTS
AMQ STREAMS
Enterprise distribution of Apache Kafka
Simpliﬁed deployment on OpenShift
Based on Kafka and Strimzi
REPLAYABLE EVENTS
CONSUMABLE EVENTS
COMMAND
QUERY
VOLATILE EVENTS
TRANSACTED/FILTERED EVENTS
AMQ ONLINE
Scalable, “self-service” on OpenShift
Available self-managed and Red Hat-managed
Based on EnMasse, Artemis & Qpid Dispatch Router
COMMAND
QUERY
VOLATILE EVENTS
DURABLE EVENTS
REPLAYABLE EVENTS

Single Message Transformations
60
Modify events before storing in Kafka
Image Source: “Penknife, Swiss Army Knife” by Emilian Robert Vicol , used under CC BY 2.0
● Lightweight single message inline transformation
● Format conversions
○ Time/date fields
○ Extract new row state
● Aggregate sharded tables to single topic
● Keep compatibility with existing consumers
● Transformation does not interact with external systems

61
KAFKA Connects VS Camel route

62
Integration concerns begin to surface as ﬁrst class concepts in
Kubernetes community
Emerging Trends
Enterprise
Integration
Patterns

Apache Camel K
● A platform for directly running integrations on Openshift and Kubernetes
● Based on Operator SDK
● Apache-based, community-driven project
● A subproject of Apache Camel started on August 31st, 2018
https://github.com/apache/camel-k

Integration and Knative
ChannelSource Service
Source
Channel
Service

Integration and Knative
ChannelSource Service

66
Messaging for the hybrid cloud
A uniform messaging overlay for the hybrid-cloud
Emerging Trends
Events

AMQ Interconnect
Event Mesh for Hybrid Cloud
Event Mesh
• Overlay messaging network spanning
private and public cloud for seamless
inter-service communication
• Secure
○ Mutual TLS between routers with
dedicated CA
○ No inbound TCP connections to
private cloud
• Logical addressing, not hosts & ports
Use-cases
• Expose data securely from private to
public cloud (APIs and Events)
• Topology-aware request routing
• Cloud bursting
• Fault-tolerance

71
International, Inc
- Large global company
- Designing its digital transformation strategy
- API-first approach to ease merger & acquisition
- Fundamental pain-point: profits are impaired by
pollution on Earth
Fleur de Lune
- Eco-friendly startup
- Core business focused on space agriculture
- Staggering growth, especially among Millennials
- Features an amazing culture among I.T. workers
- Heavy usage of cloud-native integration technologies
The storyline

Installation
RHPDS
OPENSHIFT
WORKSHOP
4.3
Ansible
CHE
Fuse Online
AMQ Online
AMQ Streams
Infinispan
Camel K
User Projects
Shared Projects
Solution Patterns

Data Engineer vs Scientist Tooling
74

Personas????
Lab one Lab two Lab three Lab four
Developer 100% Developer 100% Developer 100% Developer 100%
Architect 100% Architect 70% Architect 70% Architect 50%
Data Eng 0% Data Eng 100% Data Eng 10% Data Eng 30%

Lab 1
Event Driven Architecture
76

77
● Distributed system
● Multiple machines
● Each service is a process
● Lightweight protocols
Need to talk to each other, but how?
Customers
Billing
Administration
Operations
Customer
Service
Lab One - Event Driven Architecture

invoice
inventory
order
HTTP
HTTP
HTTP

invoice
inventory
order
i-invoice
i-inventory
i-order
AMQONLINE
events
Incomingorders/notifications
HTTP
HTTP
HTTP
SSE
AMQP
AMQP
AMQP
AMQP
HTTP

i-invoice
i-inventory
i-order
AMQONLINE
events
Incomingorders/notifications
HTTP
SSE
AMQP
AMQP
AMQP
AMQP

Lab 2
Change Data Capture &
Apache Kafka
84

85
Lab Two - Change Data Capture (CDC)
Enterprise
| | | | | | |
| | | | | | | |
| | | | | |
Apache Kafka
Earth
Kafka
ConnectMS SQL Server

Lab 3
Cloud-native EIPs
88

Lab Three - Cloud-Native Integration with EIPs
IoT
IoT
IoT
IoT
IoT
IoT
IoT
IoT
Queue
Marshmello..
Marshmello..
Marshmello..
Marshmello..
Marshmello..
Marshmello..
Farm
EdgeSimulator
Config

Queue
streams
streams
Wiretap
Inventory
Config
Premium
Standard
Utility
CBR
Marshmello..
Marshmello..
Marshmello..
Marshmello..
Marshmello..
Marshmello..
Farm
Splitter
Inventory
Store

Shipping
Cargo No
Cart No
mmid
Streams
Premium
Streams
Standard
Premium
Shipping
Assignment
Standard
Shipping
Assignment Shipping
Cargo No
Weight
Data Lake
(Cache)
Data Lake
(Cache)

Shipping
Cargo No
Cart No
mmid
Shipping
Cargo No
Weight
Data Lake
(Cache)
Data Lake
(Cache)
Console Service
API
Inventory
Store
Grafana

Lab 4
Event Sourcing
& CEP
93

Lab Four - Event Sourcing and CEP
Queue
Marshmello..
Marshmello..
Marshmello..
Marshmello..
Marshmello..
Marshmello..
Farm
Streams
Cost Center
Cost Center
Filter
Cost Advisor
Calculate
Cost
Reset offset
Replay
What if cost
increased?

Lab Four - Event Sourcing and CEP
Farm
CEP Kogito
Time
window
Streams
Rules
Determine
Disaster
Events
EventsFarm
Harvest
Events
EventsFarm
Harvest
Every 10 Seconds
Events
EventsFarm
Harvest
Events
EventsFarm
Harvest
Every 10 Seconds
< 150 every 10 Sec means disaster
Streams
Disaster
Streams
Disaster
Farm
Farm
Farm
Farm

Environment URL:
https://tutorial.blah.redhat.com
User: userX
Password: openshift
96

linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHat
Red Hat is the world’s leading provider of enterprise
open source software solutions. Award-winning
support, training, and consulting services make
Red Hat a trusted adviser to the Fortune 500.
Thank you
97

Event-Driven Architecture Workshop Agenda and Modern Application Design

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Event-Driven Architecture Workshop Agenda and Modern Application Design

Ähnlich wie Event-Driven Architecture Workshop Agenda and Modern Application Design (20)

Mehr von Christina Lin

Mehr von Christina Lin (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Event-Driven Architecture Workshop Agenda and Modern Application Design