Serverless Workflow: New approach to Kubernetes service orchestration | DevNation Tech Talk

•

1 like•3,969 views

With the rise of Serverless Architectures, Workflows have gained a renewed interest and usefulness. Typically thought of as centralized and monolithic, they now play a key role in service orchestration and coordination as well as modular processing. With many different architecture approaches already in place, the Cloud Native Computing Foundation (CNCF) has started an initiative to specify serverless workflows to ensure portability and vendor neutrality. In this talk, we introduce the CNCF Serverless Workflow specification and provide examples and demos on top of Kogito, Red Hat's business automation toolkit. You will learn: 1- The what, why, and how of the CNCF Serverless Workflow specification 2- Why using the Serverless Workflow specification and orchestration can improve your serverless architecture 3- When to use CNCF Serverless Workflow and Kogito together and the benefits derived.

Technology

Define a declarative workflow
model that orchestrates event-driven,
serverless applications.
CNCF Serverless WG Sandbox Project
Vendor-neutral
Platform-independent
Apache License 2.0

Provide a common
workflow
model definition.
Common model
allows easier portability
across
platforms/containers.
Common model
can drive common:
*libraries
*tooling
*infrastructure
Describe both stateful
and stateless
orchestration.

TCK
<code>
SDK
<code>
JSON SCHEMA
{ … }
RUNTIME IMPL
conforms to…uses…
EXT
{ … }
KPIs
Sim
Tra
cing
S
p
e
c
R
u
n
t
i
m
e
s

JSON/YAML
Runtime Runtime Runtime
Specification

Trigger 1
Trigger 2
Trigger 3
SinkBinding 1
SinkBinding 2
SinkBinding 3
… …
Service
Discovery
Services /
Deployments

“Container images”“Cloud services”
Define used services

Correlation rulesDeclarative types
cloudevents
Define consumed/produced events

Define services execution
Function reference Event-triggered

Define control-flow logic
A
A B
Sequence
A
A A
Data/Event based
transition
A
Data-based
looping
A B C
Parallel Exec Error handling
(and much more … )

Broker
Knative Resource
Workflow Service
Trigger1
Trigger2
Trigger3
Serving
Service
(http)
Workflow Workflow Workflow
Example - Event-triggered workflow exec

GitHub: https://github.com/serverlessworkflow
Web: http://serverlessworkflow.io
Slack: https://slack.cncf.io/ #serverless-workflow
Meetings: https://www.cncf.io/community/calendar/

Rest
Integration
K
A
F
K
A
Serverless
Workflow
Human Task
Service
Rule
Service
Kogitcloudevents
New Orders Process Orders
Messaging
Custom UI
http://orders-demo-serverless-demos.apps.kogito.automation.rhmw.io/

What's hot

Apache Kafka fundamentally changes how organizations build and deploy a universal data pipeline that is scalable, reliable, and durable enough to meet the needs of digital-first organizations. However, as powerful as Kafka is today, it’s not an event-streaming platform - and getting it there on your own is a long, complicated, and expensive process. Earlier this year Confluent announced Project Metamorphosis - our plan to bring the best characteristics of cloud native systems to Apache Kafka. Since May we’ve begun transforming Confluent Cloud and Confluent Platform to do just that. Join two of our Product Managers, Dan Rosanova and Addison Huddy to: Learn how we’ve evolved Confluent Cloud with the first phase of Project Metamorphosis releases See how Confluent Platform 6.0 brings these transformational, cloud-like qualities to self-managed Kafka Get a sneak peak of our next Metamorphosis theme and how it impacts your Kafka and event-streaming strategy.

How Confluent Completes the Event Streaming Platform (Addison Huddy & Dan Ros...

HostedbyConfluent

Moving existing apps to the cloud

Tiera Fann, MBA

As a Cloud Service Provider, Clever Cloud makes heavy use of messaging in its core architecture. Initially using other messaging solutions, we faced a number of different challenges. Then we discovered Apache Pulsar, which held the promise to solve these challenges. This session will dive into the adoption of Apache Pulsar at Clever Cloud, including how we worked with StreamNative to develop KoP (Kafka on Pulsar), and how we're using Apache Pulsar as the foundation for our cloud backbone.

Apache Pulsar: A Foundation Backbone for Clever Cloud - Pulsar Virtual Summit...

StreamNative

Mark Teehan, Principal Solutions Engineer, Confluent Use the Debezium CDC connector to capture database changes from a Postgres database - or MySQL or Oracle; streaming into Kafka topics and onwards to an external data store. Examine how to setup this pipeline using Docker Compose and Confluent Cloud; and how to use various payload formats, such as avro, protobuf and json-schema. https://www.meetup.com/Singapore-Kafka-Meetup/events/276822852/

From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...

confluent

Cloud migration: it's practically a rite of passage for anyone who's built infrastructure on bare metal. When we migrated our 5-year-old Kafka deployment from the datacenter to GCP, we were faced with the task of making our highly mutable server infrastructure more cloud-friendly. This led to a surprising decision: we chose to run our Kafka cluster on Kubernetes. I'll share war stories from our Kafka migration journey, explain why we chose Kubernetes over arguably simpler options like GCP VMs, and present the lessons we learned while making our way toward a stable and self-healing Kubernetes deployment. I'll also go through some improvements in the more recent Kafka releases that make upgrades crucial for any Kafka deployment on immutable and ephemeral infrastructure. You'll learn what happens when you try to run one complex distributed system on top of another, and come away with some handy tricks for automating cloud cluster management, plus some migration pitfalls to avoid. And if you're not sure whether running Kafka on Kubernetes is right for you, our experiences should provide some extra data points that you can use as you make that decision.

Kafka on Kubernetes: Keeping It Simple (Nikki Thean, Etsy) Kafka Summit SF 2019

confluent

One of the great things about running applications in the cloud is that you only pay for the resources that you use. But that also makes it more important than ever for our applications to be resource-efficient. This becomes even more critical when we use serverless functions. Micronaut is an application framework that provides dependency injection, developer productivity features, and excellent support for Apache Kafka. By performing dependency injection, AOP, and other productivity-enhancing magic at compile time, Micronaut allows us to build smaller, more efficient microservices and serverless functions. In this session, we'll explore the ways that Apache Kafka and Micronaut work together to enable us to build fast, efficient, event-driven applications. Then we'll see it in action, using the AWS Lambda Sink Connector for Confluent Cloud.

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...

HostedbyConfluent

Event streaming allows companies to build more scalable and loosely coupled real-time applications supporting massive concurrency demands and simplifying the construction of services. At the same time, API management provides capabilities to securely control the upstream services consumption, including the event processing infrastructure. This session shows how Kong Konnect Enterprise can complement Kafka Event Streaming, exposing it to new and external consumers while applying specific and critical policies to control its consumption, including API key, OAuth/OIDC and others for authentication, rate limiting, caching, log processing, etc.

Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...

HostedbyConfluent

Robinhood uses Kafka in every line of its business, from stock and crypto trading to clearing and data analytics. One interesting aspect of our architecture is that many of our microservices leveraging Kafka are written in Python. When you combine Python's relatively slow performance coupled, its reliance on process-based parallelism and Robinhood’s scale, the result is a massive fleet of application processes producing to and consuming from our Kafka clusters. This fleet generates an atypical workload on Kafka that warrants a deeper investment in scalability and reliability. This talk discusses our investments in Kafka infrastructure for a large-scale Python-based environment: kafkahood: our librdkafka-based client library wrapper that codifies best practices, sane defaults and deep client-side observability. kafkaproxy: a Rust-based sidecar proxy that reduces connection fan-in from Python gunicorn worker pools to our Kafka clusters. We'll also present challenges we encountered along the way and share our learnings with the audience.

Taming a massive fleet of Python-based Kafka apps at Robinhood | Chandra Kuch...

HostedbyConfluent

Whilst Kafka has the ability to encrypt data in transit, it does not have the functionality out of the box to encrypt data at rest. This places the responsibility of encryption of data placed on message queues on developers. Implementing cryptography correctly in our applications is challenging and time consuming. In this demo-driven talk, I will show you how you can use HashiCorp Vault’s API to implement a simple workflow that offsets the complexity of cryptography to Vault. In just a few lines of code, I will demonstrate how message producers will be able to encrypt its data, whilst message consumers can decrypt message payloads with minimal development effort. I will also show how to troubleshoot common errors from the API. By the end of this talk, you will learn how to implement symmetric and asymmetric encryption of your application data before placing it on Kafka message queues. You will also learn how to implement this workflow using Format Preserving Encryption (FPE).

Encrypting Kafka messages at rest to secure applications | Robert Barnes, Has...

HostedbyConfluent

K8s vs Cloud Foundry

Ivan Borshukov

Annoyed by some bug in your Java application? Need to profile that performance bottleneck in your JVM? And all of this inside a container? Don't worry: we got you covered. Join this session to learn more about using JDK Flight Recorder (JFR) and JDK Mission Control (JMC) to troubleshoot, monitor, and profile Java applications. Also learn how to do all of this for Java applications inside containers, with ContainerJFR.

Profiling Java inside containers with ContainerJFR | DevNation Tech Talk

Serverless Workflow: New approach to Kubernetes service orchestration | DevNation Tech Talk

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Serverless Workflow: New approach to Kubernetes service orchestration | DevNation Tech Talk

Similar to Serverless Workflow: New approach to Kubernetes service orchestration | DevNation Tech Talk (20)

More from Red Hat Developers

More from Red Hat Developers (20)

Recently uploaded

Recently uploaded (20)

Serverless Workflow: New approach to Kubernetes service orchestration | DevNation Tech Talk