Flink Forward Berlin 2018: Viktor Klang - Keynote "The convergence of stream processing and microservice architecture"

•

4 gefällt mir•1,520 views

Two of the main software architectural trends in software development this decade has been the move to streaming data processing, and the move to microservice architecture. Both of these architectures are driven by the needs of managing and mining knowledge from ever increasing volumes of data in a close to real-time fashion—all while being reactive: responsive under failure, and responsive under load. I'm here to tell you that these two trends are converging, and a fusion of the two is both logical and inevitable. In this session we will talk about what a fused approach to stream processing and microservices could look like, what opportunities exist—what software development for business software can look like in the following decade.

Technologie

The convergence of stream processing
& microservice architecture
√Deputy CTO, Lightbend
@viktorklang / v@lightbend.com / viktorklang.com

History of Data Processing
4 Offline Batch processing
4 MapReduce
4 Lambda Architecture

History of Services
4 Client-Server
4 CORBA & DCOM
4 EJB & N-Tier
4 SOA & ESB
4 HTTP-API & REST

Catch of the day
Live1
Stream Processing
&
Microservice Architecture
1
Near real-time

Shared requirements:
4 Available
4 Elastically scalable
4 Resilient & self-healing
4 Loosely coupled & upgradeable

Reactive Systems2
Available (responsive), under failure (resilience) and
under load (elasticity), by being message-driven.
2
https://www.reactivemanifesto.org/

Reactive Streams3
Provides a standard for asynchronous stream
processing with non-blocking back pressure.
3
www.reactive-streams.com & java.util.concurrent.Flow (JDK9+)

Ingredients
4 Stream processing stages: Stateless & Stateful
4 Microservices: RPC-style & Evented
4 Domain Events with Schema/Codec

#define Events4
4 Immutable recordings of observations
4 Can be disregarded but not be retracted once accepted
4 Can be invalidated but (typically) not be deleted
4 Knowledge accrues with every processed event
4
Learn more: https://speakerdeck.com/jboner/designing-events-first-microservices

Realizationevented services are stream processing stages

Is this the solution?
4 Alice: So we we lift evented services into the stream
processing pipeline …
4 Bob: … and then we're done!
4 <record scratch>
4 Narrator: They're forgetting the request-reply use-
case which RPC services excel at.

The improved solution
4 lift the service into a stream stage
4 lift input Events into Commands and submit
4 capture responses & convert into output Events

What if we could
derive microservices
from data streams?

Unification
4 Microservices are a part of a streaming pipeline
4 A pipeline can now be exposed as Microservices
4 We can independently upgrade parts of the pipeline

Developing converged applications
4 Deployment Orchestration
4 Events-First DDD & EventStorming
4 (Meta)Data-driven Continuous Deployment
4 Use of the right tools for each component

Explore what we do at Lightbend:
4 Lightbend Fast Data Platform
4 https://www.lightbend.com/products/fast-data-
platform
4 Lightbend Reactive Platform
4 https://www.lightbend.com/products/reactive-platform
4 Lightbend Enterprise Suite
4 https://www.lightbend.com/products/enterprise-suite

Thank you for attending!
The convergence of stream processing
& microservice architecture
√
Deputy CTO, Lightbend
@viktorklang / v@lightbend.com / viktorklang.com

Weitere ähnliche Inhalte

Was ist angesagt?

Stream Processing in conjunction with a Consistent, Durable, Reliable stream storage is kicking the revolution up a notch in Big Data processing. This modern paradigm is enabling a new generation of data middleware that delivers on the streaming promise of a simplified and unified programming model. From data ingest, transformation, and messaging to search, time series and more, a robust streaming data ecosystem means we’ll all be able to more quickly build applications that solve problems we could not solve before.

Flink Forward San Francisco 2018 keynote: Srikanth Satya - "Stream Processin...

Flink Forward

Nowadays many companies become data rich and intensive. They have millions of users generating billions of interactions and events per day. These massive streams of complex events can be processed and reacted upon to e.g. offer new products, next best actions, communicate to users or detect frauds, and quicker we can do it, the higher value we can generate. Our presentation will be based on our recent experience in building a real-time data analytics platform for telco events. This platform has been jointly built by GetInData and Kcell - the leading telco in Kazakhstan - in just a few months and it currently runs in production at the scale of 10M subscribers and 160K events per second on a still small cluster. It's used as a backbone for personalized marketing campaigns, detecting frauds, cross-sell & up-sell by following the behavior of millions of users in real-time and reacting to it instantly. We will share how we build such platform using current best of breed open-source projects like Flink, Kafka, and Nifi. We won't skimp on the details how we designed and optimized our Flink applications for high load and performance. We will also describe challenges that we faced during development and try to provide some tips what one should pay attention to when developing similar solutions, not only for telco, but also for banks, e-commerce, IoT and other industries.

Flink Forward Berlin 2018: Krzysztof Zarzycki & Alexey Brodovshuk - "Assistin...

Flink Forward

In modern applications of streaming frameworks, stateful streaming is arguably one of the most important usage cases. Flink, as a well-supported streaming framework for stateful streaming, readily helps developers spend less efforts on system deployment and focus more on the business logic. Nevertheless, upgrading from an existing production system to a new one with stateful streaming can still be a challenging task for any development team. In this talk, we will share our experience in migrating an existing system at Appier (an AI-based startup specialized with B2B solutions) to stateful streaming with Flink. We will first discuss how stateful streaming matches our business logic and its potential benefits. Then, we review the obstacles that we have encountered during migration, and present our solutions to conquer them. We hope that our experience and tips shared in this talk hints future users to prepare themselves towards applying Flink in their production systems more painlessly.

Flink Forward Berlin 2018: Wei-Che (Tony) Wei - "Lessons learned from Migrati...

Flink Forward

Stream Processing as helped to turn many monolithic database-centric applications into fast, scalable, and flexible real time applications. However, there are still entire classes of applications that are built against databases, because today's streaming processing model is not yet rich enough to support these applications. We present what we believe is still missing in today's stream processing models, and how we envision to evolve stream processing for the next wave of application to be able to move to a stream processing architecture.

Flink Forward Berlin 2018: Stephan Ewen - Keynote: "Unlocking the next wave o...

Flink Forward

As a distributed stream processing engine, Flink provides users with convenient operators to manipulate data on the fly. Among all these operators, join could be the most complicated one as it requires the capability to cross-analyze various sources simultaneously. In this talk, we aim to give a comprehensive introduction to the stream join in Flink. Specifically, we'll first provide an overview of the different join types and which of them are currently supported by Flink DataStream and Table & SQL APIs. Then we'll discuss some key points when performing distributed stream join. After that, we'd like to focus the rationale and implementation details of the time-windowed join launched in version 1.4. Since there are still a lot of improvements can be made, we'll end our talk by sharing some proposals for the future work.

Flink Forward Berlin 2018: Xingcan Cui - "Stream Join in Flink: from Discrete...

Flink Forward

Extracting insights out of continuously generated data requires a stream processor with powerful data analytics features such as Apache Flink. A stream data pipeline with Flink typically includes a storage component to ingest and serve the data. Pravega is a stream store that ingests and stores stream data permanently, making the data available for tail, catch-up, and historical reads. One important challenge for such stream data pipelines is coping with the variations in the workload. Daily cycles and seasonal spikes might require the provisioning of the application to adapt accordingly. Pravega has a feature called stream scaling, which enables the capacity offered for the ingestion of events of a stream to grow and shrink over time according to workload. Such a feature is useful when the application downstream has the ability of accommodating such changes and also scale its provisioning accordingly. In this presentation, we introduce stream scaling in Pravega and how Flink jobs leverage this feature to rescale stateful jobs according to variations in the workload.

Scaling stream data pipelines with Pravega and Apache Flink

Till Rohrmann

CloudStream service is a Full Management Service in Huawei Cloud. Support several features, such as On-Demand Billing, easy-to-use Stream SQL in online SQL editor, test Stream SQL in real-time style, Multi-tenant, security isolation and so on. We choose Apache Flink as streaming compute platform. Inside of CloudStream Cluster, Flink job can run on Yarn, Mesos, Kubernetes. We also have extended Apache Flink to meet IoT scenario needs. There are specialized tests on Flink reliability with college cooperation. Finally continuously improve the infrastructure around CS including open source projects and cloud services. CloudStream is different with any other real-time analysis cloud service. The development process can also be shared at architecture and principles.

Flink Forward San Francisco 2018: - Jinkui Shi and Radu Tudoran "Flink real-t...

Flink Forward

Video: https://data-artisans.com/flink-forward-berlin/resources/monitoring-flink-with-prometheus Live Demo Code: https://github.com/mbode/flink-prometheus-example Prometheus is a cloud-native monitoring system prioritizing reliability and simplicity – and Flink works really well with it! This session will show you how to leverage the Flink metrics system together with Pronetheus to improve the observability of your jobs. There will be a live demo establishing how everything ties in together. The talk is aimed at people already building and running Flink jobs who would like to gain more insight into them. It is fine if you are not familiar with Prometheus yet as the basic concepts will be introduced. If you have ever wondered how you could use modern monitoring tools to be alerted in the middle of the night in case your Flink job‘s 99th percentile end-to-end latency degraded for some reason, this might just be the talk you are looking for.

Monitoring Flink with Prometheus

Maximilian Bode

Operationalizing Machine Learning models is never easy. Our team at Comcast has been challenged with operationalizing predictive ML models to improve customer care experiences. Using Apache Flink we have been able to apply real-time streaming to all aspects of the Machine Learning lifecycle. This includes data feature exploration and preparation by data scientists, deploying live models to serve near-real-time predictions, and validating results for model retraining and iteration. We will share best practices and lessons learned from Flink’s role in our operationalized lifecycle including: • Executing as the “Prediction Pipeline” – a model container environment for near-real-time streaming and batch predictions • Preparing streaming features and data sets for model training, as input for production model predictions, and for a continually-updated customer context • Using connected streams and savepoints for “Live in the Dark”, multi-variant testing, and validation scenarios • Incorporating Flink’s Queryable State as an approach to the online “Feature Store” – a data catalog for reuse by multiple models and use cases • Enabling versioned models, versioned feature sets, and versioned data through DevOps approaches.

Flink Forward San Francisco 2018: Dave Torok & Sameer Wadkar - "Embedding Fl...

Flink Forward

Apache Flink is a popular stream computing framework for real-time stream computing. Many stream compute algorithms require trailing data in order to compute the intended result. One example is computing the number of user logins in the last 7 days. This creates a dilemma where the results of the stream program are incomplete until the runtime of the program exceeds 7 days. The alternative is to bootstrap the program using historic data to seed the state before shifting to use real-time data. This talk will discuss alternatives to bootstrap programs in Flink. Some alternatives rely on technologies exogenous to the stream program, such as enhancements to the pub/sub layer, that are more generally applicable to other stream compute engines. Other alternatives include enhancements to Flink source implementations. Lyft is exploring another alternative using orchestration of multiple Flink programs. The talk will cover why Lyft pursued this alternative and future directions to further enhance bootstrapping support in Flink.

Flink Forward San Francisco 2018: Gregory Fee - "Bootstrapping State In Apach...

Flink Forward

Flink metrics module allows to use Dropwizard-like metrics and reporters in Flink pipelines. It opens a rich opportunity to not only monitor health of Flink pipelines but also attach real-time business intelligence metrics to run alongside existing Flink data jobs, thus avoiding a need to build a separate BI data-flow infrastructure. However, most of the work still resides with application developers: they have to define how those metrics are registered, calculated and attached to Flink functional operators – and then Flink will do the computation and reporting. The most challenging task is to add BI-like metrics, with key dimensions dynamically extracted from streamed data and metric values likely needed to be aggregated over multiple Flink tasks. We present a toolkit that extends Flink metrics and allows to decorate existing Flink operators with both simple health-check and complex BI-like metrics, updated and observable in real-time through IT monitoring and BI visualization dashboards. This work is based on a real-world production use-case of computing retail merchandise prices in real-time for the Walmart e-commerce catalog using Flink.

Flink Forward San Francisco 2018: Andrew Torson - "Extending Flink metrics: R...

Flink Forward

dA Platform is a production-ready platform for stream processing with Apache Flink®. The Platform includes open source Apache Flink, a stateful stream processing and event-driven application framework, and dA Application Manager, a central deployment and management component. dA Platform schedules clusters on Kubernetes, deploys stateful Flink applications, and controls these applications and their state.

dA Platform Overview

Robert Metzger

“Customer experience is the next big battle ground for telcos,” proclaimed recently Amit Akhelikar, Global Director of Lynx Analytics at TM Forum Live! Asia in Singapore. But, how to fight in this battle? A common approach has been to keep “under control” some well-known network quality indicators, like dropped calls, radio access congestion, availability, and so on; but this has proven not to be enough to keep customers happy, like a siege weapon is not enough to conquer a city. But, what if it were possible to know how customers perceive services, at least most demanded ones, like web browsing or video streaming? That would be like a squad of archers ready to battle. And even having that, how to extract value of it and take actions in no time, giving our skilled archers the right targets? Meet CANVAS (Customer And Network Visualization and AnaltyticS), one of the first LATAM implementations of a Flink-based stream processing use case for a telco, which successfully combines leading and innovative technologies like Apache Hadoop, YARN, Kafka, Nifi, Druid and advanced visualizations with Flink core features like non-trivial stateful stream processing (joins, windows and aggregations on event time) and CEP capabilities for alarm generation, delivering a next-generation tool for SOC (Service Operation Center) teams.

Flink Forward San Francisco 2018: David Reniz & Dahyr Vergara - "Real-time m...

Flink Forward

In Zalando's microservice architecture, each service continuously generates streams of events for the purposes of inter-service communication or data integration. Some of these events describe business processes, e.g. a customer has placed an order or a parcel has been shipped. Out of this, the need to materialize event streams from the central event bus into persistent cloud storage evolved. The temporarily persisted data is then integrated into our relational data warehouse. In this talk we present a materialization engine backed by Apache Flink. We show how we employ Flink’s RESTful API, custom accumulators and stoppable sources to provide another API abstraction layer for deploying, monitoring and controlling our materialization jobs. Our jobs compact event streams depending on event properties and transform their complex JSON structures into flat files for easier integration into the data warehouse.

Flink Forward Berlin 2017: Mihail Vieru - A Materialization Engine for Data I...

Flink Forward

Flink Forward San Francisco 2018: Xu Yang - "Alibaba’s common algorithm platf...

Flink Forward

Stream processing still evolves and changes at a speed that can make it hard to keep up with the developments. Being at the forefront of stream processing technology, the evolution of Apache Flink has mirrored many of these developments and continues to do so. We will take you on a journey through the major milestones of stream processing technology in past years, diving into the latest additions that Apache Flink and other communities introduced to the stream processing landscape, such as Streamng SQL, Time Versioned Tables, cluster-library-duality, language portability, etc. We will take a sneak peek into our crystal ball and present in what the Flink community is working on next.

The Past, Present, and Future of Apache Flink

Aljoscha Krettek

In the last decade, many distributed stream processing engines (SPEs) were developed to perform continuous queries on massive online data. The central design principle of these engines is to handle queries that potentially run forever on data streams with a query-at-a-time model, i.e., each query is optimized and executed separately. In many real applications, streams are not only processed with long-running queries, but also thousands of short-running ad-hoc queries. To support this efficiently, it is essential to share resources and computation for stream ad-hoc queries in a multi-user environment. The goal of this talk is to bridge the gap between stream processing and ad-hoc queries in SPEs by sharing computation and resources. We define three main requirements for ad-hoc shared stream processing: (1) Integration: Ad-hoc query processing should be a composable layer which can extend stream operators, such as join, aggregation, and window operators; (2) Consistency: Ad-hoc query creation and deletion must be performed in a consistent manner and ensure exactly-once semantics and correctness; (3) Performance: In contrast to state-of-the-art SPEs, ad-hoc SPE should not only maximize data throughput but also query throughout via incremental computation and resource sharing. Based on these requirements, we have developed AStream, an ad-hoc, shared computation stream processing framework. To the best of our knowledge, AStream is the first system that supports distributed ad-hoc stream processing. AStream is built on top of Apache Flink. Our experiments show that AStream shows comparable results to Flink for single query deployments and outperforms it in orders of magnitude with multiple queries.

A stream: Ad-hoc Shared Stream Processing - Jeyhun Karimov, DFKI GmbH

Flink Forward

Streaming engines like Apache Flink are redefining ETL and data processing. Data can be extracted, transformed, filtered and written out in real-time with an ease matching that of batch processing. However the real challenge of matching the prowess of batch ETL remains in doing joins, in maintaining state and to have the data be paused or rested dynamically. Netflix has a microservices architecture. Different microservices serve and record different kind of user interactions with the product. Some of these live services generate millions of events per second, all carrying meaningful but often partial information. Things start to get exciting when we want to combine the events coming from one high-traffic microservice to another. Joining these raw events generates rich datasets that are used to train the machine learning models that serve Netflix recommendations. Historically we have done this joining of large volume data-sets in batch. However we asked ourselves if the data is being generated in real-time, why must it not be processed downstream in real time? Why wait a full day to get information from an event that was generated a few mins ago? In this talk, we will share how we solved a complex join of two high-volume event streams using Flink. We will talk about maintaining large state, fault tolerance of a stateful application and strategies for failure recovery.

Flink Forward Berlin 2018: Shriya Arora - "Taming large-state to join dataset...

Flink Forward

The streaming platform team at Lyft has been running Flink jobs in production for more than a year now, powering critical use cases like improving pickup ETA accuracy, dynamic pricing, generating machine learning features for fraud detection, real-time analytics among many others. Broadly, the jobs fall into two abstraction layers: applications (Flink jobs that run on the native platform) and analytics (that leverage Dryft, Lyft’s fully managed data processing engine). This talk will give an overview of the platform architecture, deployment model and user experience. The talk will also dive deeper into some of the challenges and the lessons that were learnt, running Flink jobs at scale, specifically around scaling Flink connectors, dealing with event time skew (source synchronization) and highlight common patterns of problems observed across several Flink jobs. Finally, the talk will give insights into how we are re-architecting the streaming platform @ Lyft using a Kubernetes based deployment.

Running Flink in Production: The good, The bad and The in Between - Lakshmi ...

Flink Forward

As many industries, banking is undergoing a fundamental change because of the software revolution. No longer are banks competing only on interest rates and having the best traders, these days customer experience and having the best engineers are the focus. In this changing world, banks compete with new start-ups, the so-called Fintechs, and with large platform organisations such as Google, Facebook and Apple. At ING, we believe that staying ahead of the game means changing how we interact with our customers, no longer a traditional model of waiting for the customers to come to the bank through our website or apps, but to actively reach out to the customer with information that is relevant to him or her in order to make their financial life frictionless. Many of these changes are driven by reacting to all events that are relevant to the customer, and using streaming analytics to be able to reach out to the customer in milliseconds after the event occurs. Apache Flink is key for ING to achieve this. This presentation addresses how ING approaches the challenge, the role that Apache Flink plays, and the consequences regulations have on how we work with Open Source in general, and with Apache Flink (and data Artisans) in particular. This keynote takes place at Kino 3.

Flink Forward Berlin 2017 Keynote: Ferd Scheepers - Taking away customer fric...

Flink Forward

Was ist angesagt? (20)

Flink Forward San Francisco 2018 keynote: Srikanth Satya - "Stream Processin...

Flink Forward Berlin 2018: Krzysztof Zarzycki & Alexey Brodovshuk - "Assistin...

Flink Forward Berlin 2018: Wei-Che (Tony) Wei - "Lessons learned from Migrati...

Flink Forward Berlin 2018: Stephan Ewen - Keynote: "Unlocking the next wave o...

Flink Forward Berlin 2018: Xingcan Cui - "Stream Join in Flink: from Discrete...

Scaling stream data pipelines with Pravega and Apache Flink

Flink Forward San Francisco 2018: - Jinkui Shi and Radu Tudoran "Flink real-t...

Monitoring Flink with Prometheus

Flink Forward San Francisco 2018: Dave Torok & Sameer Wadkar - "Embedding Fl...

Flink Forward San Francisco 2018: Gregory Fee - "Bootstrapping State In Apach...

Flink Forward San Francisco 2018: Andrew Torson - "Extending Flink metrics: R...

dA Platform Overview

Flink Forward San Francisco 2018: David Reniz & Dahyr Vergara - "Real-time m...

Flink Forward Berlin 2017: Mihail Vieru - A Materialization Engine for Data I...

Flink Forward San Francisco 2018: Xu Yang - "Alibaba’s common algorithm platf...

The Past, Present, and Future of Apache Flink

A stream: Ad-hoc Shared Stream Processing - Jeyhun Karimov, DFKI GmbH

Flink Forward Berlin 2018: Shriya Arora - "Taming large-state to join dataset...

Running Flink in Production: The good, The bad and The in Between - Lakshmi ...

Flink Forward Berlin 2017 Keynote: Ferd Scheepers - Taking away customer fric...

Ähnlich wie Flink Forward Berlin 2018: Viktor Klang - Keynote "The convergence of stream processing and microservice architecture"

DEVNET-1175 OpenDaylight Service Function Chaining

Cisco DevNet

With Viktor Klang, Deputy CTO Lightbend, Inc. As software grows more and more interconnected, and with several generations of software having to interoperate, a new take on the integration of systems is needed—ad hoc, unversioned, and unreplicated scripts just won’t suffice, and the traditional Enterprise Service Bus (ESB) concept has experienced stability, reliability, performance, and scalability problems. In this webinar, Viktor explores a new take on Enterprise Integration Patterns: First, he will explore the Reactive Streams standard, an orchestration layer where transformations are standalone, composable, reusable, and—most importantly—using asynchronous flow-control—back pressure—to maintain predictable, stable, behavior over time. Furthermore, he will go through how one-off workloads relate to continuous, and batch, workloads, and how they can be addressed by that very same orchestration layer. Finally, he will review how this type of design achieves resilience, scalability, and ultimately—responsiveness.

Revitalizing Enterprise Integration with Reactive Streams

Lightbend

Facing Open Banking regulation, rapidly increasing transaction volumes and increasing customer expectations, Nationwide took the decision to take load off their back-end systems through real-time streaming of data changes into Kafka. Hear about how Nationwide started their journey with Kafka, from their initial use case of creating a real-time data cache using Change Data Capture, Kafka and Microservices to how Kafka allowed them to build a stream processing backbone used to reengineer the entire banking experience including online banking, payment processing and mortgage applications. See a working demo of the system and what happens to the system when the underlying infrastructure breaks. Technologies covered include: Change Data Capture, Kafka (Avro, partitioning and replication) and using KSQL and Kafka Streams Framework to join topics and process data.

Introducing Events and Stream Processing into Nationwide Building Society (Ro...

confluent

WSO2 Year End Tech Update Webinar

WSO2

Apache Roadshow Chicago Talk on May 14, 2019 In this talk we’ll look at the ways Apache Thrift can solve performance problems commonly facing next generation applications deployed in performance sensitive capital markets and banking environments. The talk will include practical examples illustrating the construction, performance and resource utilization benefits of Apache Thrift. Apache Thrift is a high-performance cross platform RPC and serialization framework designed to make it possible for organizations to specify interfaces and application wide data structures suitable for serialization and transport over a wide variety of schemes. Due to the unparalleled set of languages supported by Apache Thrift, these interfaces and structs have similar interoperability to REST type services with an order of magnitude improvement in performance. Apache Thrift services are also a perfect fit for container technology, using considerably fewer resources than traditional application server style deployments. Decomposing applications into microservices, packaging them into containers and orchestrating them on systems like Kubernetes can bring great value to an organization; however, it can also take a very fast monolithic application and turn it into a high latency web of slow, resource hungry services. Apache Thrift is a perfect solution to the performance and resource ills of many microservice based endeavors.

Building high performance microservices in finance with Apache Thrift

RX-M Enterprises LLC

Watch this talk here: https://www.confluent.io/online-talks/introducing-events-and-stream-processing-nationwide-building-society Open Banking regulations compel the UK’s largest banks, and building societies to enable their customers to share personal information with other regulated companies securely. As a result companies such as Nationwide Building Society are re-architecting their processes and infrastructure around customer needs to reduce the risk of losing relevance and the ability to innovate. In this online talk, you will learn why, when facing Open Banking regulation and rapidly increasing transaction volumes, Nationwide decided to take load off their back-end systems through real-time streaming of data changes into Apache Kafka®. You will hear how Nationwide started their journey with Apache Kafka®, beginning with the initial use case of creating a real-time data cache using Change Data Capture, Confluent Platform and Microservices. Rob Jackson, Head of Application Architecture, will also cover how Confluent enabled Nationwide to build the stream processing backbone that is being used to re-engineer the entire banking experience including online banking, payment processing and mortgage applications. View now to: -Explore the technologies used by Nationwide to meet the challenges of Open Banking -Understand how Nationwide is using KSQL and Kafka Streams Framework to join topics and process data. -Learn how Confluent Platform can enable enterprises such as Nationwide to embrace the event streaming paradigm -See a working demo of the Nationwide system and what happens when the underlying infrastructure breaks.

Introducing Events and Stream Processing into Nationwide Building Society

confluent

Openshift serverless Solution

Ryan ZhangCheng

These are a series of presentations and knowledge collected from the web to help knowledge sharing at the government of Puerto Rico, created with the hope of helping transform government culture by engaging key personnel in diverse areas of central government IT. We discussed design and development methodologies as well as implementation, network and server technologies that led to the successful launch of the most popular online service in PR.gov, in the hope that the knowledge is retained and used to prevent problems that have plagued digital services of the past. How did Puerto Rico build the New Good standing Certificate Online Service? How did it scale to handle millions of visitors while having 0 licensing costs? This is the technical overview of the design, philosophy and implementation. - Good standing certificate knowledge transfer presentation by Andrés Colón Note on attribution: some content such as logos and designs were used from the web. Rights remain with their original authors. Thanks for sharing with the world.

Building Modern Digital Services on Scalable Private Government Infrastructur...

Andrés Colón Pérez

Microservices, events, containers, and orchestrators are dominating our vernacular today. As operations teams adapt to support these technologies in production, cloud-native platforms like Cloud Foundry and Kubernetes have quickly risen to serve as force multipliers of automation, productivity and value. Kafka is providing developers a critically important component as they build and modernize applications to cloud-native architecture. This talk will explore: • Why cloud-native platforms and why run Kafka on Kubernetes? • What kind of workloads are best suited for this combination? • Tips to determine the path forward for legacy monoliths in your application portfolio • Running Kafka as a Streaming Platform on Container Orchestration

Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...

confluent

Full lifecycle of a microservice

Luigi Bennardis

Day in the life event-driven workshop

Christina Lin

Microservices have emerged as a widely discussed and adopted way to build modern and scalable applications. They are easier to build, manage and maintain than monoliths due to smaller code bases; they isolate complexity, allowing for smaller, more agile teams to create services, and they are flexible—allowing the use of various platforms, programming languages, and tools since these choices affect only an individual service and a small team at a time.

Application Modernisation through Event-Driven Microservices

confluent

Move fast and make things with microservices

Mithun Arunan

Databus - LinkedIn's Change Data Capture Pipeline

Sunil Nagaraj

LDP4j: A framework for the development of interoperable read-write Linked Da...

Nandana Mihindukulasooriya

PortoTechHub - Hail Hydrate! From Stream to Lake with Apache Pulsar and Friends https://portotechhub.com/conference-2021/ Timothy Spann Developer Advocate StreamNative A cloud data lake that is empty is not useful to anyone. How can you quickly, scalably and reliably fill your cloud data lake with diverse sources of data you already have and new ones you never imagined you needed. Utilizing open source tools from Apache, the FLiP stack enables any data engineer, programmer or analyst to build reusable modules with low or no code. FLiP utilizes Apache NiFi, Apache Pulsar, Apache Flink and MiNiFi agents to load CDC, Logs, REST, XML, Images, PDFs, Documents, Text, semistructured data, unstructured data, structured data and a hundred data sources you could never dream of streaming before. I will teach you how to fish in the deep end of the lake and return a data engineering hero. Let's hope everyone is ready to go from 0 to Petabyte hero. TRACK RIBEIRA Fri 07:00 — 50 min 19-Nov-2021

PortoTechHub - Hail Hydrate! From Stream to Lake with Apache Pulsar and Friends

Timothy Spann

Meetup Openshift Geneva 03/10

MagaliDavidCruz

Microservices, events, containers, and orchestrators are dominating our vernacular today. As operations teams adapt to support these technologies in production, cloud-native platforms like Pivotal Cloud Foundry and Kubernetes have quickly risen to serve as force multipliers of automation, productivity and value. Apache Kafka® is providing developers a critically important component as they build and modernize applications to cloud-native architecture. This talk will explore: • Why cloud-native platforms and why run Apache Kafka on Kubernetes? • What kind of workloads are best suited for this combination? • Tips to determine the path forward for legacy monoliths in your application portfolio • Demo: Running Apache Kafka as a Streaming Platform on Kubernetes

Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...

confluent

Have you ever wondered how companies keep their databases online through unplanned disasters? Have you wondered which high availability features might be best for our shop? In this session we will go over a quick look at log shipping, database mirroring, transactional replication, failover cluster instances and availability groups. John will identify pros and cons for each features and share some tips from the field.

Introduction to High Availability with SQL Server

John Sterrett

This presentation will describe how to go beyond a "Hello world" stream application and build a real-time data-driven product. We will present architectural patterns, go through tradeoffs and considerations when deciding on technology and implementation strategy, and describe how to put the pieces together. We will also cover necessary practical pieces for building real products: testing streaming applications, and how to evolve products over time. Presented at highloadstrategy.com 2016 by Øyvind Løkling (Schibsted Products & Technology), joint work with Lars Albertsson (independent, www.mapflat.com).

Building real time data-driven products

Lars Albertsson

Ähnlich wie Flink Forward Berlin 2018: Viktor Klang - Keynote "The convergence of stream processing and microservice architecture" (20)

DEVNET-1175 OpenDaylight Service Function Chaining

Revitalizing Enterprise Integration with Reactive Streams

Introducing Events and Stream Processing into Nationwide Building Society (Ro...

WSO2 Year End Tech Update Webinar

Building high performance microservices in finance with Apache Thrift

Introducing Events and Stream Processing into Nationwide Building Society

Openshift serverless Solution

Building Modern Digital Services on Scalable Private Government Infrastructur...

Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...

Full lifecycle of a microservice

Day in the life event-driven workshop

Application Modernisation through Event-Driven Microservices

Move fast and make things with microservices

Databus - LinkedIn's Change Data Capture Pipeline

LDP4j: A framework for the development of interoperable read-write Linked Da...

PortoTechHub - Hail Hydrate! From Stream to Lake with Apache Pulsar and Friends

Meetup Openshift Geneva 03/10

Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...

Introduction to High Availability with SQL Server

Building real time data-driven products

Mehr von Flink Forward

Apache Flink is a distributed stream processing framework that allows users to process and analyze data in real-time. At LinkedIn, we developed a fully managed stream processing platform on Flink running on K8s to power hundreds of stream processing pipelines in production. This platform is the backbone for other infra systems like Search, Espresso (internal document store) and feature management etc. We provide a rich authoring and testing environment which allows users to create, test, and deploy their streaming jobs in a self-serve fashion within minutes. Users can focus on their business logic, leaving the Flink platform to take care of management aspects such as split deployment, resource provisioning, auto-scaling, job monitoring, alerting, failure recovery and much more. In this talk, we will introduce the overall platform architecture, highlight the unique value propositions that it brings to stream processing at LinkedIn and share the experiences and lessons we have learned.

Building a fully managed stream processing platform on Flink at scale for Lin...

Flink Forward

Flink Forward San Francisco 2022. When running Flink jobs, skew is a common problem that results in wasted resources and limited scalability. In the past years, we have helped our customers and users solve various skew-related issues in their Flink jobs or clusters. In this talk, we will present the different types of skew that users often run into: data skew, key skew, event time skew, state skew, and scheduling skew, and discuss solutions for each of them. We hope this will serve as a guideline to help you reduce skew in your Flink environment. by Jun Qin & Karl Friedrich

Evening out the uneven: dealing with skew in Flink

Flink Forward

Flink Forward San Francisco 2022. To improve Amazon Alexa experiences and support machine learning inference at scale, we built an automated end-to-end solution for incremental model building or fine-tuning machine learning models through continuous learning, continual learning, and/or semi-supervised active learning. Customer privacy is our top concern at Alexa, and as we build solutions, we face unique challenges when operating at scale such as supporting multiple applications with tens of thousands of transactions per second with several dependencies including near-real time inference endpoints at low latencies. Apache Flink helps us transform and discover metrics in near-real time in our solution. In this talk, we will cover the challenges that we faced, how we scale the infrastructure to meet the needs of ML teams across Alexa, and go into how we enable specific use cases that use Apache Flink on Amazon Kinesis Data Analytics to improve Alexa experiences to delight our customers while preserving their privacy. by Aansh Shah

“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...

Flink Forward

Flink Forward San Francisco 2022. Probably everyone who has written stateful Apache Flink applications has used one of the fault-tolerant keyed state primitives ValueState, ListState, and MapState. With RocksDB, however, retrieving and updating items comes at an increased cost that you should be aware of. Sometimes, these may not be avoidable with the current API, e.g., for efficient event-time stream-sorting or streaming joins where you need to iterate one or two buffered streams in the right order. With FLIP-220, we are introducing a new state primitive: BinarySortedMultiMapState. This new form of state offers you to (a) efficiently store lists of values for a user-provided key, and (b) iterate keyed state in a well-defined sort order. Both features can be backed efficiently by RocksDB with a 2x performance improvement over the current workarounds. This talk will go into the details of the new API and its implementation, present how to use it in your application, and talk about the process of getting it into Flink. by Nico Kruber

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...

Flink Forward

Flink Forward San Francisco 2022. The Apache Flink Kubernetes Operator provides a consistent approach to manage Flink applications automatically, without any human interaction, by extending the Kubernetes API. Given the increasing adoption of Kubernetes based Flink deployments the community has been working on a Kubernetes native solution as part of Flink that can benefit from the rich experience of community members and ultimately make Flink easier to adopt. In this talk we give a technical introduction to the Flink Kubernetes Operator and demonstrate the core features and use-cases through in-depth examples." by Thomas Weise

Introducing the Apache Flink Kubernetes Operator

Flink Forward

Flink Forward San Francisco 2022. Resource Elasticity is a frequently requested feature in Apache Flink: Users want to be able to easily adjust their clusters to changing workloads for resource efficiency and cost saving reasons. In Flink 1.13, the initial implementation of Reactive Mode was introduced, later releases added more improvements to make the feature production ready. In this talk, we’ll explain scenarios to deploy Reactive Mode to various environments to achieve autoscaling and resource elasticity. We’ll discuss the constraints to consider when planning to use this feature, and also potential improvements from the Flink roadmap. For those interested in the internals of Flink, we’ll also briefly explain how the feature is implemented, and if time permits, conclude with a short demo. by Robert Metzger

Autoscaling Flink with Reactive Mode

Flink Forward

Flink Forward San Francisco 2022. Flink consumers read from Kafka as a scalable, high throughput, and low latency data source. However, there are challenges in scaling out data streams where migration and multiple Kafka clusters are required. Thus, we introduced a new Kafka source to read sharded data across multiple Kafka clusters in a way that conforms well with elastic, dynamic, and reliable infrastructure. In this presentation, we will present the source design and how the solution increases application availability while reducing maintenance toil. Furthermore, we will describe how we extended the existing KafkaSource to provide mechanisms to read logical streams located on multiple clusters, to dynamically adapt to infrastructure changes, and to perform transparent cluster migrations and failover. by Mason Chen

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Flink Forward

Flink Forward San Francisco 2022. Next time you want to integrate with a new destination for a demo, concept or production application, the Async Sink framework will bootstrap development, allowing you to move quickly without compromise. In Flink 1.15 we introduced the Async Sink base (FLIP-171), with the goal to encapsulate common logic and allow developers to focus on the key integration code. The new framework handles things like request batching, buffering records, applying backpressure, retry strategies, and at least once semantics. It allows you to focus on your business logic, rather than spending time integrating with your downstream consumers. During the session we will dive deep into the internals to uncover how it works, why it was designed this way, and how to use it. We will code up a new sink from scratch and demonstrate how to quickly push data to a destination. At the end of this talk you will be ready to start implementing your own Flink sink using the new Async Sink framework. by Steffen Hausmann & Danny Cranmer

One sink to rule them all: Introducing the new Async Sink

Flink Forward

Flink Forward San Francisco 2022. In normal situations, the default Kafka consumer and producer configuration options work well. But we all know life is not all roses and rainbows and in this session we’ll explore a few knobs that can save the day in atypical scenarios. First, we'll take a detailed look at the parameters available when reading from Kafka. We’ll inspect the params helping us to spot quickly an application lock or crash, the ones that can significantly improve the performance and the ones to touch with gloves since they could cause more harm than benefit. Moreover we’ll explore the partitioning options and discuss when diverging from the default strategy is needed. Next, we’ll discuss the Kafka Sink. After browsing the available options we'll then dive deep into understanding how to approach use cases like sinking enormous records, managing spikes, and handling small but frequent updates.. If you want to understand how to make your application survive when the sky is dark, this session is for you! by Olena Babenko

Tuning Apache Kafka Connectors for Flink.pptx

Flink Forward

Flink Forward San Francisco 2022. Pinterest is a visual discovery engine that serves over 433MM users. Stream processing allows us to unlock value from realtime data for pinners. At Pinterest, we adopt Flink as the unified streaming processing engine. In this talk, we will share our journey in building a stream processing platform with Flink and how we onboarding critical use cases to the platform. Pinterest has supported 90+near realtime streaming applications. We will cover the problem statement, how we evaluate potential solutions and our decision to build the framework. by Rainie Li & Kanchi Masalia

Flink powered stream processing platform at Pinterest

Flink Forward

Flink Forward San Francisco 2022. This talk will take you on the long journey of Apache Flink into the cloud-native era. It started all the way from where Hadoop and YARN were the standard way of deploying and operating data applications. We're going to deep dive into the cloud-native set of principles and how they map to the Apache Flink internals and recent improvements. We'll cover fast checkpointing, fault tolerance, resource elasticity, minimal infrastructure dependencies, industry-standard tooling, ease of deployment and declarative APIs. After this talk you'll get a broader understanding of the operational requirements for a modern streaming application and where the current limits are. by David Moravek

Apache Flink in the Cloud-Native Era

Flink Forward

Flinkn Forward San Francisco 2022. In this talk, we will cover various topics around performance issues that can arise when running a Flink job and how to troubleshoot them. We’ll start with the basics, like understanding what the job is doing and what backpressure is. Next, we will see how to identify bottlenecks and which tools or metrics can be helpful in the process. Finally, we will also discuss potential performance issues during the checkpointing or recovery process, as well as and some tips and Flink features that can speed up checkpointing and recovery times. by Piotr Nowojski

Where is my bottleneck? Performance troubleshooting in Flink

Flink Forward

Flink Forward San Francisco 2022. Running natively on Kubernetes, using the new Apache Flink Kubernetes Operator is a great way to deploy and manage Flink application and session deployments. In this presentation, we provide: - A brief overview of Kubernetes operators and their benefits. - Introduce the five levels of the operator maturity model. - Introduce the newly released Apache Flink Kubernetes Operator and FlinkDeployment CRs - Dockerfile modifications you can make to swap out UBI images and Java of the underlying Flink Operator container - Enhancements we're making in: - Versioning/Upgradeability/Stability - Security - Demo of the Apache Flink Operator in-action, with a technical preview of an upcoming product using the Flink Kubernetes Operator. - Lessons learned - Q&A by James Busche & Ted Chang

Using the New Apache Flink Kubernetes Operator in a Production Deployment

Flink Forward

Flink Forward San Francisco 2022. The Table API is one of the most actively developed components of Flink in recent time. Inspired by databases and SQL, it encapsulates concepts many developers are familiar with. It can be used with both bounded and unbounded streams in a unified way. But from afar it can be difficult to keep track of what this API is capable of and how it relates to Flink's other APIs. In this talk, we will explore the current state of Table API. We will show how it can be used as a batch processor, a changelog processor, or a streaming ETL tool with many built-in functions and operators for deduplicating, joining, and aggregating data. By comparing it to the DataStream API we will highlight differences and elaborate on when to use which API. We will demonstrate hybrid pipelines in which both APIs interact with one another and contribute their unique strengths. Finally, we will take a look at some of the most recent additions as a first step to stateful upgrades. by David Andreson

The Current State of Table API in 2022

Flink Forward

Flink Forward San Francisco 2022. Based on the new Flink-Pulsar connector, we implemented Flink's TableAPI and Catalog to help users to interact with the Pulsar cluster via Flink SQL easily. We would like to go through the design and implementation of the SQL connector in the following aspects: 1. Two different modes of use Pulsar as a metadata store 2. Data format transformation and management 3. SQL semantics support within Pulsar context by Sijie Guo & Neng Lu

Flink SQL on Pulsar made easy

Flink Forward

Flink Forward San Francisco 2022. At Bloomberg, we deal with high volumes of real-time market data. Our clients expect to be notified of any anomalies in this market data, which may indicate volatile movements in the markets, notable trades, forthcoming events, or system failures. The parameters for these alerts are always evolving and our clients can update them dynamically. In this talk, we'll cover how we utilized the open source Apache Flink and Siddhi SQL projects to build a distributed, scalable, low-latency and dynamic rule-based, real-time alerting system to solve our clients' needs. We'll also cover the lessons we learned along our journey. by Ajay Vyasapeetam & Madhuri Jain

Dynamic Rule-based Real-time Market Data Alerts

Flink Forward

Flink Forward San Francisco 2022. At Stripe we have created a complete end to end exactly-once processing pipeline to process financial data at scale, by combining the exactly-once power from Flink, Kafka, and Pinot together. The pipeline provides exactly-once guarantee, end-to-end latency within a minute, deduplication against hundreds of billions of keys, and sub-second query latency against the whole dataset with trillion level rows. In this session we will discuss the technical challenges of designing, optimizing, and operating the whole pipeline, including Flink, Kafka, and Pinot. We will also share our lessons learned and the benefits gained from exactly-once processing. by Xiang Zhang & Pratyush Sharma & Xiaoman Dong

Exactly-Once Financial Data Processing at Scale with Flink and Pinot

Flink Forward

Flink Forward San Francisco 2022. What if my data is already in order? Stream Processing has given us an elegant and powerful solution for running analytic queries and logic over high volumes of continuously arriving data. However, in both Apache Flink and Apache Beam, the notion of time-ordering is baked in at a very low level, making it difficult to express computations that are interested in a semantic-, rather than time-ordering of the data. In financial services, what often matters the most about the data moving between systems is not when the data was created, but in what order, to the extent that many institutions engineer a global sequencing over all data entering and produced by their systems to achieve complete determinism. How, then, can financial institutions and others best employ Stream Processing on streams of data that are already ordered? I will cover various techniques that can make this work, as well as seek input from the community on how Flink might be improved to better support these use-cases. by Patrick Lucas

Processing Semantically-Ordered Streams in Financial Services

Flink Forward

Flink Forward San Francisco 2022. In modern data platform architectures, stream processing engines such as Apache Flink are used to ingest continuous streams of data into data lakes such as Apache Iceberg. Streaming ingestion to iceberg tables can suffer by two problems (1) small files problem that can hurt read performance (2) poor data clustering that can make file pruning less effective. To address those two problems, we propose adding a shuffling stage to the Flink Iceberg streaming writer. The shuffling stage can intelligently group data via bin packing or range partition. This can reduce the number of concurrent files that every task writes. It can also improve data clustering. In this talk, we will explain the motivations in details and dive into the design of the shuffling stage. We will also share the evaluation results that demonstrate the effectiveness of smart shuffling. by Gang Ye & Steven Wu

Tame the small files problem and optimize data layout for streaming ingestion...

Flink Forward

Flink Forward San Francisco 2022. Goldman Sachs's Data Lake platform serves as the firm's centralized data platform, ingesting 140K (and growing!) batches per day of Datasets of varying shape and size. Powered by Flink and using metadata configured by platform users, ingestion applications are generated dynamically at runtime to extract, transform, and load data into centralized storage where it is then exported to warehousing solutions such as Sybase IQ, Snowflake, and Amazon Redshift. Data Latency is one of many key considerations as producers and consumers have their own commitments to satisfy. Consumers range from people/systems issuing queries, to applications using engines like Spark, Hive, and Presto to transform data into refined Datasets. Apache Iceberg allows our applications to not only benefit from consistency guarantees important when running on eventually consistent storage like S3, but also allows us the opportunity to improve our batch processing patterns with its scalability-focused features. by Andreas Hailu

Batch Processing at Scale with Flink & Iceberg

Flink Forward

Mehr von Flink Forward (20)

Building a fully managed stream processing platform on Flink at scale for Lin...

Evening out the uneven: dealing with skew in Flink

“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...

Introducing the Apache Flink Kubernetes Operator

Autoscaling Flink with Reactive Mode

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

One sink to rule them all: Introducing the new Async Sink

Tuning Apache Kafka Connectors for Flink.pptx

Flink powered stream processing platform at Pinterest

Apache Flink in the Cloud-Native Era

Where is my bottleneck? Performance troubleshooting in Flink

Using the New Apache Flink Kubernetes Operator in a Production Deployment

The Current State of Table API in 2022

Flink SQL on Pulsar made easy

Dynamic Rule-based Real-time Market Data Alerts

Exactly-Once Financial Data Processing at Scale with Flink and Pinot

Processing Semantically-Ordered Streams in Financial Services

Tame the small files problem and optimize data layout for streaming ingestion...

Batch Processing at Scale with Flink & Iceberg

Kürzlich hochgeladen

ICT role in 21st century education and its challenges

rafiqahmad00786416

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

MINDCTI Revenue Release Quarter One 2024

MIND CTI

Dubai, often portrayed as a shimmering oasis in the desert, faces its own set of challenges, including the occasional threat of flooding. Despite its reputation for opulence and modernity, the emirate is not immune to the forces of nature. In recent years, Dubai has experienced sporadic but significant floods, testing the resilience of its infrastructure and communities. Among the critical lifelines in this bustling metropolis is the Dubai International Airport, a bustling hub that connects the city to the world. This article explores the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Orbitshub

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

Passkeys: Developing APIs to enable passwordless authentication Cody Salas, Sr Developer Advocate | Solutions Architect - Yubico Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

apidays

The microservices honeymoon is over. When starting a new project or revamping a legacy monolith, teams started looking for alternatives to microservices. The Modular Monolith, or 'Modulith', is an architecture that reaps the benefits of (vertical) functional decoupling without the high costs associated with separate deployments. This talk will delve into the advantages and challenges of this progressive architecture, beginning with exploring the concept of a 'module', its internal structure, public API, and inter-module communication patterns. Supported by spring-modulith, the talk provides practical guidance on addressing the main challenges of a Modultith Architecture: finding and guarding module boundaries, data decoupling, and integration module-testing. You should not miss this talk if you are a software architect or tech lead seeking practical, scalable solutions. About the author With two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Victor Rentea

Accelerating FinTech Innovation: Unleashing API Economy and GenAI Vasa Krishnan, Chief Technology Officer - FinResults Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

apidays

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

DBX First Quarter 2024 Investor Presentation

Dropbox

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Six Myths about Ontologies: The Basics of Formal Ontology

johnbeverley2021

Understanding the FAA Part 107 License ..

Christopher Logan Kennedy

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Kürzlich hochgeladen (20)

ICT role in 21st century education and its challenges

AWS Community Day CPH - Three problems of Terraform

MINDCTI Revenue Release Quarter One 2024

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Why Teams call analytics are critical to your entire business

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

DBX First Quarter 2024 Investor Presentation

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

Boost Fertility New Invention Ups Success Rates.pdf

Six Myths about Ontologies: The Basics of Formal Ontology

Understanding the FAA Part 107 License ..

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Flink Forward Berlin 2018: Viktor Klang - Keynote "The convergence of stream processing and microservice architecture"

1. The convergence of stream processing & microservice architecture √Deputy CTO, Lightbend @viktorklang / v@lightbend.com / viktorklang.com

3. Overview

4. History of Data Processing 4 Offline Batch processing 4 MapReduce 4 Lambda Architecture

5. History of Services 4 Client-Server 4 CORBA & DCOM 4 EJB & N-Tier 4 SOA & ESB 4 HTTP-API & REST

6. Today

7. Catch of the day Live1 Stream Processing & Microservice Architecture 1 Near real-time

8. Shared requirements: 4 Available 4 Elastically scalable 4 Resilient & self-healing 4 Loosely coupled & upgradeable

9. Reactive Systems2 Available (responsive), under failure (resilience) and under load (elasticity), by being message-driven. 2 https://www.reactivemanifesto.org/

10. Reactive Streams3 Provides a standard for asynchronous stream processing with non-blocking back pressure. 3 www.reactive-streams.com & java.util.concurrent.Flow (JDK9+)

11. Ingredients 4 Stream processing stages: Stateless & Stateful 4 Microservices: RPC-style & Evented 4 Domain Events with Schema/Codec

12. #define Events4 4 Immutable recordings of observations 4 Can be disregarded but not be retracted once accepted 4 Can be invalidated but (typically) not be deleted 4 Knowledge accrues with every processed event 4 Learn more: https://speakerdeck.com/jboner/designing-events-first-microservices

13. Services (evented)

14. Realizationevented services are stream processing stages

15. The near future: Convergence

16. Is this the solution? 4 Alice: So we we lift evented services into the stream processing pipeline … 4 Bob: … and then we're done! 4 <record scratch> 4 Narrator: They're forgetting the request-reply use- case which RPC services excel at.

17. Services (rpc)

18. The improved solution 4 lift the service into a stream stage 4 lift input Events into Commands and submit 4 capture responses & convert into output Events

19. What if we could derive microservices from data streams?

20. Ingestion endpoints

21. Simple Projection endpoints

22. Complex Projection endpoints

23. Unification 4 Microservices are a part of a streaming pipeline 4 A pipeline can now be exposed as Microservices 4 We can independently upgrade parts of the pipeline

24. Developing converged applications 4 Deployment Orchestration 4 Events-First DDD & EventStorming 4 (Meta)Data-driven Continuous Deployment 4 Use of the right tools for each component

25. Beyond

26. Extend to the Edges

27. End of stream

28. Explore what we do at Lightbend: 4 Lightbend Fast Data Platform 4 https://www.lightbend.com/products/fast-data- platform 4 Lightbend Reactive Platform 4 https://www.lightbend.com/products/reactive-platform 4 Lightbend Enterprise Suite 4 https://www.lightbend.com/products/enterprise-suite

29. Thank you for attending! The convergence of stream processing & microservice architecture √ Deputy CTO, Lightbend @viktorklang / v@lightbend.com / viktorklang.com

Flink Forward Berlin 2018: Viktor Klang - Keynote "The convergence of stream processing and microservice architecture"

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Flink Forward Berlin 2018: Viktor Klang - Keynote "The convergence of stream processing and microservice architecture"

Ähnlich wie Flink Forward Berlin 2018: Viktor Klang - Keynote "The convergence of stream processing and microservice architecture" (20)

Mehr von Flink Forward

Mehr von Flink Forward (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Flink Forward Berlin 2018: Viktor Klang - Keynote "The convergence of stream processing and microservice architecture"