In software systems, architecture that served the enterprise in the pre-cloud era is no longer sufficient. The need for cohesive customer experience is forcing the modernization of information and data supply chains. To enable next-generation application architecture, organizations are shifting from traditional data silos and batch processing to a streaming-first approach. Event streaming paradigms as seen in Kafka-based architecture are becoming de facto for such solutions. This session will dive deep into the engineering methodology, inspired by domain-driven design, to model business functions in an events-first approach and provide a practice to implement the event streaming architecture. We’ll navigate a journey through application modernization. First, we explore business domain modeling with the swift methodology. Next, we work on a business flow that satisfies the most important functional requirements for the business stakeholder and addresses pain points surfaced from event Storming. Finally, we focus on the technical challenges of the architecture to satisfy the non-functional requirements. An example of a non-functional requirement described in this session is achieving high availability across multiple data centers while maintaining data integrity service level objectives. We accomplished this by moving business-critical workloads from legacy middleware to a Kafka-native solution using the Kafka Streams library.

1. Confidential │ © 2020 VMware, Inc.
Application
Modernization
With Event Streaming
Architecture
David Wadden
Staff Engineer
July 20th, 2020
@ronmervis

2. Confidential │ © 2020 VMware, Inc. 2
David Wadden
Staff Engineer
@ronmervis
picture

3. Confidential │ © 2020 VMware, Inc. 3
➔ Big upfront planning. You’re assessing the entire portfolio at
once before starting the work.
Risk: Fail to get quick wins to build momentum.
➔ Horizontally focused project work. Often we address painful
symptoms in an IT-centric “horizontal” manner.
Risk: You miss root cause and don’t deliver business impact
➔ Your mess for less. Instead of making meaningful changes,
you’re moving apps from one host to the next.
Risk: Won’t get the business-centric benefits you desire.
➔ One-time project versus competency development.
Efforts attempt to “finish” versus build a practice.
Risk: Not set up to continuously keep technical debt low.
Traditional
Modernizatio
n
Approaches

4. Confidential │ © 2020 VMware, Inc.
We Start with Events First

5. Confidential │ © 2020 VMware, Inc.
Service Business capability service
e.g. “Order service”
Queue
Topic
Messaging Q / T
Async communication
Service A
Service B
Service C Service D
Service E
Service F
External Connected external system
e.g. “Weather service”
External 2
External 1
UI
UI
Q Q
T
User
Interface
Entry points
relevant in the narrative
Synchronous
… service calls
Asynchronous
Boris Diagram - Workshop

6. Confidential │ © 2020 VMware, Inc.
Event
Storming
2. Event Storm the system
Thin
Slice
3. Select “Thin Slice(s)” of functionalities
OKRs
1. Define Objectives & Key Results
Tested
Code
6. Produce tested and working code
Boris
4. Drafts the desired notional architecture
Snap-E 5. Fill the backlog
API DATA
PUB / SUB EXT
STORIES UI RISKS
SERVICE A
Deconstructing Monoliths

7. 7Confidential │ © 2020 VMware, Inc.
Business Domain
Decomposition

8. © 2020 VMware, Inc. 8
Get to Function Fast!
Evolve architecture and implementation from simple to production ready
Goals
● Spike risky interactions identified from Boris exercise
● Validate notional architecture with simplest implementation
● Maintain legacy function parity

9. © 2020 VMware, Inc. 9
Get to Function Fast!
Continued...
Anti-goals
● Building for resilience or scale at this stage
● Over-thinking the microservices design
● Implement final state architecture or future proofing
● Use-cases outside of the thin slice

10. © 2020 VMware, Inc. 10
Get to Function Fast!
Continued...
Technical Strategies
● Separate reads and writes
● Spend the time on interface design
● Keep technical complexity minimal
● Implement anti-corruption layer to protect legacy

11. © 2020 VMware, Inc.
● React to domain events published via Kafka,
update model (application state) using
relational or NoSQL store
● Leverage ACLs to integrate with legacy
systems
○ Avoid pollution of new domain with
current technical implementation
● Focus on breaking down monolithic data
structures (i.e., “God objects”: Order,
Product, Trade) across bounded contexts
Initial Technical Design

12. © 2020 VMware, Inc.
Domain Architecture / Boris

13. © 2020 VMware, Inc.
Initial System Architecture

14. 14Confidential │ © 2020 VMware, Inc.
Evolving to
Kafka Streams

15. © 2020 VMware, Inc.Confidential │ ©2020 VMware, Inc.
Non Functional Requirements
● Keep a log of NFRs: availability, recovery, etc.
● Allow the NFRs to organically drive out the need for Kafka
native architecture patterns: active-active and MTTR for
disaster recovery
● Pivot to focusing on the architecture and thinking
through unhappy path cases
Evolve to meet operational goals

16. © 2020 VMware, Inc.
Re-look at example
Initial design shortcomings
● TPS limited by RDBMS table locks
● ACL logic complexity
● Disaster recovery operation
● Cross data center replication challenges

17. © 2020 VMware, Inc.
Leverage same model with Kafka Streams Architecture
● If the domain model is implemented
properly, it can be reused wholesale
within Kafka Streams as an aggregate
rolled up from a given KStreams
processor topology
● The existing model then becomes the
materialized view derived from the event
streams
● The materialized view can then service
any existing queries with minimal
change

18. © 2020 VMware, Inc.
Leverage Kafka Streams

19. © 2020 VMware, Inc.
Evolved System Architecture

20. 20Confidential │ ©2020 VMware, Inc.
Recap

21. Thank You
Confidential │ © 2020 VMware, Inc.
@ronmervis