Migrating applications to serverless Apache Kafka + KSQL

•Als PPTX, PDF herunterladen•

2 gefällt mir•150 views

The document describes the steps to migrate a legacy movie ratings application to a serverless architecture using Apache Kafka and KSQL. The legacy application uses a monolithic, database-centric design to calculate average movie ratings. The migration plan involves: 1) Using Kafka Connect to extract rating data to Kafka topics, 2) Setting up a Confluent Cloud Kafka cluster, 3) Replicating data to the cloud cluster, 4) Processing data with KSQL queries, 5) Building microservices powered by KSQL output, and 6) Decommissioning the monolith once migration is complete.

Technologie

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migrating Applications to Serverless
Apache Kafka and KSQL
Tim Berglund
S e s s i o n I D
Sr. Director, Developer Relations
Confluent

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Our Movie Rating Service, Legacy Edition
• Users rate movies, ratings go into Kafka
• Monolithic, database-centric application calculates averages
• Serves them to users through a web UI and API
Movie
Ratings
Users
Movies
Top Rated Movies
My Favorite Movie
Moviegoers

Kafka served well
• Decouples event input from processing
• Easily understood abstraction for event processing
• Not exactly a pleasure to operate, but we can’t complain
Movie
Ratings
Users
Movies
Top Rated Movies
My Favorite Movie
Moviegoers

Our Monolith’s Problems
• It will do a bad job managing complexity as our service grows
• The Kafka Consumer code is bespoke
• It is a textbook pre-cloud architecture
• We cannot trivially scale to larger message volumes
Movie
Ratings
Users
Movies
Top Rated Movies
My Favorite Movie
Moviegoers

Our Refactoring Plan
• Capture Users and Movies as Kafka topics
• Migrate all topics to Confluent Cloud using Confluent Replicator
• Refactor monolith to microservices
• Keep web UI nearly untouched
• Never touch the on-prem system until the migration is complete
Movie
Ratings
Users
Movies
Top Rated Movies
My Favorite Movie
Moviegoers

Step One: Fewer Databases
• Use Kafka Connect to extract Users and Movies tables to Kafka topics
Movie
Ratings
Moviegoers
Movies
Users
Kafka
Connect

Step Two: Spin up a Confluent Cloud cluster
• We want to get out of the business of managing Kafka ourselves
Movie
Ratings
Movies
Users

Step Three: Deploy Confluent Replicator
• Use Kafka Connect to extract Users and Movies tables to Kafka topics
Movie
Ratings
Movies
Users
Movie
Ratings
Movies
Users
Replicator
Replicator
Replicator

Step Four: Convert to KSQL
• Bespoke Consumer code implements non-differentiated functionality
Movie
Ratings
Movies
Users
CREATE TABLE movie_ratings AS
SELECT title,
SUM(rating)/COUNT(rating) AS avg_rating,
COUNT(rating) AS num_ratings
FROM ratings
LEFT OUTER JOIN movies
ON ratings.movie_id = movies.movie_id
GROUP BY title;

Step Four: Convert to KSQL
• The rating averaging query
Movie
Ratings
Movies
Users
Rated
Movies
KSQL
magic goes here

Step Four: Convert to KSQL
• The user favorite query
Movie
Ratings
Movies
Users
Rated
Movies
KSQL
magic goes here
more
KSQL
magic goes here
User
Favorites

Step Five: Extract the rating average service
• Now serve rating averages from KSQL output
• Monolith no longer serves these results
Rating Averages
Rated
Movies

Step Six: Extract the user favorite service
• Now serve rating averages from KSQL output
• Monolith no longer serves these results
User Favorites User
Favorites

Step Seven: Stand down the monolith
• Now serve rating averages from KSQL output
• Monolith no longer serves these results
User Favorites
Rated Movies
User
Favorites
Movie
Ratings
Moviegoers
Rated
Movies
so much
KSQL
magic
Movies
Users

Step Eight: Stand down Replicator
• All data is in Confluent Cloud now
• For hybrid on-prem/cloud deployment

Thank you!
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tim Berglund
@tlberglund
https://kafka-tutorials.confluent.io
https://slackpass.io/confluentcommunity

Weitere ähnliche Inhalte

Was ist angesagt?

Input features are the building blocks for machine learning models. You cannot have a great model without great features. By building on top of Apache Pulsar's infinite retention of events, we built infrastructure to serve features in production and to generate training datasets. It allowed our machine learning teams to change, test, and deploy personalization features at an extraordinary rate to 10s of millions of end-users. This talk will discuss: - What event-sourcing is and why it's so powerful for machine learning infrastructure. - How we built the StreamSQL feature store on top of Pulsar, Flink, and Cassandra. - How a feature store accelerates ML development.

StreamSQL Feature Store (Apache Pulsar Summit)

Simba Khadder

At Hootsuite, we've been transitioning from a single monolithic PHP application to a set of scalable Scala-based microservices. To avoid excessive coupling between services, we've implemented an event system using Apache Kafka that allows events to be reliably produced + consumed asynchronously from services as well as data stores. In this presentation, I talk about: - Why we chose Kafka - How we set up our Kafka clusters to be scalable, highly available, and multi-data-center aware. - How we produce + consume events - How we ensure that events can be understood by all parts of our system (Some that are implemented in other programming languages like PHP and Python) and how we handle evolving event payload data.

Building an Event Bus at Scale

jimriecken

Whilst Kafka has the ability to encrypt data in transit, it does not have the functionality out of the box to encrypt data at rest. This places the responsibility of encryption of data placed on message queues on developers. Implementing cryptography correctly in our applications is challenging and time consuming. In this demo-driven talk, I will show you how you can use HashiCorp Vault’s API to implement a simple workflow that offsets the complexity of cryptography to Vault. In just a few lines of code, I will demonstrate how message producers will be able to encrypt its data, whilst message consumers can decrypt message payloads with minimal development effort. I will also show how to troubleshoot common errors from the API. By the end of this talk, you will learn how to implement symmetric and asymmetric encryption of your application data before placing it on Kafka message queues. You will also learn how to implement this workflow using Format Preserving Encryption (FPE).

Encrypting Kafka messages at rest to secure applications | Robert Barnes, Has...

HostedbyConfluent

Building a derived data store using Kafka

Venu Ryali

Introduction to Kafka

Akash Vacher

Kafka aws

Ariel Moskovich

Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...

Nitin Kumar

How to Lock Down Apache Kafka and Keep Your Streams Safe

confluent

Fundamentals and Architecture of Apache Kafka

Angelo Cesaro

Confluent building a real-time streaming platform using kafka streams and k...

Thomas Alex

Apache kafka

Long Nguyen

Even though Kafka is scalable by design, proper handling of over one petabyte of data a day requires much more than Kafka’s scalability. Several challenges present themselves in a data centric business at this scale. These challenges include capacity planning, provisioning, message auditing, monitoring and alerting, rebalancing workloads with changes in traffic patterns, data lineage, handling service degradation and system outages, optimizing cost, upgrades, etc. In this talk we describe how at Pinterest we tackle some of these challenges and share some of the key lessons that we learned in the process. Specifically we will share how we: * Automate Kafka cluster maintenance * Manage over 150K partitions * Manage upgrade lifecycle * Track / troubleshoot thousands of data pipelines

Organic Growth and A Good Night Sleep: Effective Kafka Operations at Pinteres...

confluent

What happened when our biggest and most important Kafka cluster went rogue all of a sudden, and while trying to recover it, a single, crucial misconfiguration made things even worse? At a company like Taboola, where service availability and latency are our top priority, this was a disaster. With 300K messages/sec and 250TB of messages produced each day to our on-premise Kafka clusters, and mirrored to our central Kafka cluster, we always try to ensure Kafka behaves well under high loads of traffic and unexpected cluster failures. So when our main Kafka cluster went crazy we had a serious issue on our hands. This session is the story of how we learned the hard way about mitigating cluster failures with the proper configurations in place.

Oops! I started a broker | Yinon Kahta, Taboola

HostedbyConfluent

___________________________________________ Meetup#7 | Session 2 | 21/03/2018 | Taboola _____________________________________________ In this talk, we will present our multi-DC Kafka architecture, and discuss how we tackle sending and handling 10B+ messages per day, with maximum availability and no tolerance for data loss. Our architecture includes technologies such as Cassandra, Spark, HDFS, and Vertica - with Kafka as the backbone that feeds them all.

Distributed Kafka Architecture Taboola Scale

Apache Kafka TLV

Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...

LINE Corporation

Have you ever migrated Kafka clusters from one data center to another being completely transparent to client applications? At PayPal, as part of a massive datacenter migration initiative, Kafka team successfully moved all PayPal Kafka traffic across data centers. This initiative involved migrating 20+ Kafka clusters (1000+ broker and zookeeper nodes), as well as 60+ mirrormaker groups which seamlessly handle Kafka traffic volumes as high as 1 trillion messages per day. Throughout the course of this migration, applications required no modification, encountered 0% service outage, 0% message loss and duplicated messages. The whole migration process was fully transparent to Kafka applications. In this session, you will learn the strategies, techniques and tools the PayPal Kafka team has utilized for managing the migration process. You will also learn the lessons and pitfalls they experienced during this exercise, as well as the secret sauce of making the migration successful.

How did we move the mountain? - Migrating 1 trillion+ messages per day across...

HostedbyConfluent

Introduction to apache kafka

Dimitris Kontokostas

As architectures have become more complex, no single protocol or service can fulfill every use case. As a result, teams have turned to leveraging a multitude of services, like Kafka, REST, GraphQL, gRPC, SOAP and MQTT depending on the use case. However, this introduces a multitude of problems. The design experience is inconsistent, as each service has its own specification or schema, from OpenAPI to Avro, Protobuf to AsyncAPI. Each API or service requires different utilities, products or SDKs to simply send and receive data. Creating tests or mocks requires imperfect and fragile scripts. Join this talk to learn how SmartBear is building the world's first universal and protocol-agnostic API platform and the lessons learned along the way.

Becoming Protocol-Agnostic with Kafka, REST, GraphQL & gRPC | Tyler Mills, Sm...

HostedbyConfluent

Capture the Streams of Database Changes

confluent

Organizations have a need to protect Personally Identifiable Information (PII). As Event Streaming Architecture (ESA) becomes ubiquitous in the enterprise, the prevalence of PII within data streams will only increase. Data architects must be cognizant of how their data pipelines can allow for potential leaks. In highly distributed systems, zero-trust networking has become an industry best practice. We can do the same with Kafka by introducing message-level security. A DevSecOps Engineer with some Kafka experience can leverage Kafka Streams to protect PII by enforcing role-based access control using Open Policy Agent. Rather than implementing a REST API to handle message-level security, Kafka Streams can filter, or even transform outgoing messages in order to redact PII data while leveraging the native capabilities of Kafka. In our proposed presentation, we will provide a live demonstration that consists of two consumers subscribing to the same Kafka topic, but receiving different messages based on the rules specified in Open Policy Agent. At the conclusion of the presentation, we will provide attendees with a GitHub repository, so that they can enjoy a sandbox environment for hands-on experimentation with message-level security.

Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...

HostedbyConfluent

Was ist angesagt? (20)

StreamSQL Feature Store (Apache Pulsar Summit)

Building an Event Bus at Scale

Encrypting Kafka messages at rest to secure applications | Robert Barnes, Has...

Building a derived data store using Kafka

Introduction to Kafka

Kafka aws

Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...

How to Lock Down Apache Kafka and Keep Your Streams Safe

Fundamentals and Architecture of Apache Kafka

Confluent building a real-time streaming platform using kafka streams and k...

Apache kafka

Organic Growth and A Good Night Sleep: Effective Kafka Operations at Pinteres...

Oops! I started a broker | Yinon Kahta, Taboola

Distributed Kafka Architecture Taboola Scale

Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...

How did we move the mountain? - Migrating 1 trillion+ messages per day across...

Introduction to apache kafka

Becoming Protocol-Agnostic with Kafka, REST, GraphQL & gRPC | Tyler Mills, Sm...

Capture the Streams of Database Changes

Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...

Ähnlich wie Migrating applications to serverless Apache Kafka + KSQL

Going Cloud Native with IBM Cloud and NetflixOSS for Dev@Pulse

aspyker

Immutable infrastructure tsap_v2

Volodymyr Tsap

Serverless architectures let you build and deploy applications and services with infrastructure resources that require zero administration. In the past, you had to provision and scale servers to run your application code, install and operate distributed databases, and build and run custom software to handle API requests. Now, AWS provides a stack of scalable, fully-managed services that eliminates these operational complexities. In this session, you learn about the concepts and benefits of serverless architectures and the basics of the serverless stack AWS provides (e.g., AWS Lambda and Amazon API Gateway). We discuss use cases such as data processing, website backends, serverless applications and "operational glue". After that, you get practical tips and tricks, best practices, and architecture patterns that you can take back and implement immediately.

AWS re:Invent 2016: Getting Started with Serverless Architectures (CMP211)

Amazon Web Services

Gcp intro-20160721

Haeseung Lee

In this workshop, we build out an end-to-end Amazon AppStream 2.0 environment for your organization. We create a master image containing desktop application and configure a streaming fleet and streaming stack. We walk through network configuration options, and we show you how to connect to resources in your VPC. Finally, we show you how to create streaming URLs that users need to access their applications. To complete this workshop, you must bring your laptop, have an individual AWS account that has already been provisioned, and have working knowledge of AWS concepts. Also, it is beneficial to attend the session, "Securely Deliver Desktop Applications with Amazon AppStream 2.0.”

Build an AppStream 2.0 Environment to Deliver Desktop Applications to Any Com...

Amazon Web Services

Active web page chapter for reading purpose

SambalSwetank

Yelowsoft delivers super app to bbr one of the biggest ride hailing companies...

Yelowsoft

Yelowsoft delivers super app to bbr one of the biggest ride hailing companies...

Yelowsoft

Oscon2014 Netflix API - Top 10 Lessons Learned

Sangeeta Narayanan

Can We Make Maps from Videos? ~From AI Algorithm to Engineering for Continuou...

DeNA

Scalability and flexibility can make or break your online video success. When putting an online video plan in place for your organization, you want to be certain that you will be able to accommodate adoption and usage, increasing content quantity and quality, and any new features you may need. In Kaltura Inspire this webinar we discussed the fundamentals of API-driven online video solutions and how you can leverage this to gain scalability, flexibility, and control of end-user experience. How to avoid potentially slow, risky, and error prone manual deployments by using the Kaltura API to fully automate your cluster deployments and upgrades. See how you could plan and execute a video platform that better fits your needs and have more influence on how your users experience online video.

Kaltura Inspire Webinar: API Driven Video Platform - The Key to Scalability a...

Zohar Babin

About captionator english 20150622

Will Kim

Canada DevOps Summit 2020 Presentation Nov_03_2020

Varun Manik

Spring Cloud Pipelines is an opinionated framework that automates the creation of structured continuous deployment pipelines. In this presentation we’ll go through the contents of the Spring Cloud Pipelines project. We’ll start a new project for which we’ll have a deployment pipeline set up in no time. We’ll deploy to Cloud Foundry and check if our application is backwards compatible so that we can roll it back on production.

Continuous Deployment of your Application @SpringOne

ciberkleid

This session provides an overview of the AWS Digital Media Partner ecosystem and explains how APN partners, including ISVs, SIs and solution providers, leverage the AWS Cloud to enable media workloads to be executed in the cloud. Included is an overview of the AWS media workflow categories and a discussion of the alignment of these partners to these categories as well as to the sub-vertical industries that comprise Media and Entertainment.

[AWS LA Media & Entertainment Event 2015]: M&E Ecosystem Update Q4 2015

Amazon Web Services

Geeeez, after demanding you unit test, system test, black box test, white box test, test-test-test everything, your manager is now demanding you load test your brand spanking new Oracle web application. How on earth can you do this? This technical presentation will explain the concepts behind preparing for load testing, the Http protocol's request/response model, and live demonstrations using Oracle's Http Analyzer and Apache's JMeter to stress test your Oracle web application. The presentation is suitable for anybody, be it DBAs or developers, who are concerned about the performance of any web based application, possibly an Apex or JDeveloper or 3rd party web application. Knowledge of Apex or JDeveloper is not mandatory for this presentation and they will not be covered in any depth.

Take a load off! Load testing your Oracle APEX or JDeveloper web applications

Sage Computing Services

Top 10 Lessons Learned from the Netflix API - OSCON 2014

Daniel Jacobson

Continuous Deployment of your Application - SpringOne Tour Dallas

VMware Tanzu

IBM Connect 2016: Speaker Session with Teresa Deane, Senior Developer, BCC

BCC - Solutions for IBM Collaboration Software

Presented by Yaron Inger, CTO @Lightricks in the TLV iOS Developers Meetup 23/12/15 http://www.meetup.com/Tel-Aviv-iOS-Developers-Meetup Enlight is an iOS app designed to be a one-stop-shop for all your photo editing needs, that's been recently selected by Apple as the *2015 App of the Year* in Canada, UK, Germany, China and more, and as runner-up in the US. In this talk I will reveal some of the concepts and infrastructures that enabled us to create this extremely complex app with a relatively small team of designers and developers.

Yaron Inger - Enlight - Inside the app of the year

tlv-ios-dev

Ähnlich wie Migrating applications to serverless Apache Kafka + KSQL (20)

Going Cloud Native with IBM Cloud and NetflixOSS for Dev@Pulse

Immutable infrastructure tsap_v2

AWS re:Invent 2016: Getting Started with Serverless Architectures (CMP211)

Gcp intro-20160721

Build an AppStream 2.0 Environment to Deliver Desktop Applications to Any Com...

Active web page chapter for reading purpose

Yelowsoft delivers super app to bbr one of the biggest ride hailing companies...

Oscon2014 Netflix API - Top 10 Lessons Learned

Can We Make Maps from Videos? ~From AI Algorithm to Engineering for Continuou...

Kaltura Inspire Webinar: API Driven Video Platform - The Key to Scalability a...

About captionator english 20150622

Canada DevOps Summit 2020 Presentation Nov_03_2020

Continuous Deployment of your Application @SpringOne

[AWS LA Media & Entertainment Event 2015]: M&E Ecosystem Update Q4 2015

Take a load off! Load testing your Oracle APEX or JDeveloper web applications

Top 10 Lessons Learned from the Netflix API - OSCON 2014

Continuous Deployment of your Application - SpringOne Tour Dallas

IBM Connect 2016: Speaker Session with Teresa Deane, Senior Developer, BCC

Yaron Inger - Enlight - Inside the app of the year

Mehr von confluent

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

confluent

Santander Stream Processing with Apache Flink

confluent

Unlocking the Power of IoT: A comprehensive approach to real-time insights

confluent

El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real. Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real. En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.

Workshop híbrido: Stream Processing con Flink

confluent

Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace. In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms. You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes. Don't miss out on this opportunity to learn from industry experts and take your business to the next level.

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

confluent

La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.

AWS Immersion Day Mapfre - Confluent

confluent

Eventos y Microservicios - Santander TechTalk

confluent

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

confluent

Citi TechTalk Session 2: Kafka Deep Dive

confluent

Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.

Build real-time streaming data pipelines to AWS with Confluent

confluent

Q&A with Confluent Professional Services: Confluent Service Mesh

confluent

Citi Tech Talk: Event Driven Kafka Microservices

confluent

An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities. It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks. This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.

Confluent & GSI Webinars series - Session 3

confluent

Transforming applications built with traditional messaging solutions such as TIBCO, MQ and Solace to be scalable, reliable and ready for the move to cloud How can applications built with traditional messaging technologies like TIBCO, Solace and IBM MQ be modernised and be made cloud ready? What are the advantages to Event Streaming approaches to pub/sub vs traditional message queues? What are the strengeths and weaknesses of both approaches, and what use cases and requirements are actually a better fit for messaging than Kafka?

Citi Tech Talk: Messaging Modernization

confluent

Citi Tech Talk: Data Governance for streaming and real time data

confluent

Confluent & GSI Webinars series: Session 2

confluent

Vous apprendrez également à : • Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données • Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience • Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés

Data In Motion Paris 2023

confluent

Confluent Partner Tech Talk with Synthesis

confluent

The Future of Application Development - API Days - Melbourne 2023

confluent

The Playful Bond Between REST And Data Streams

confluent

Mehr von confluent (20)

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

Santander Stream Processing with Apache Flink

Unlocking the Power of IoT: A comprehensive approach to real-time insights

Workshop híbrido: Stream Processing con Flink

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

AWS Immersion Day Mapfre - Confluent

Eventos y Microservicios - Santander TechTalk

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

Citi TechTalk Session 2: Kafka Deep Dive

Build real-time streaming data pipelines to AWS with Confluent

Q&A with Confluent Professional Services: Confluent Service Mesh

Citi Tech Talk: Event Driven Kafka Microservices

Confluent & GSI Webinars series - Session 3

Citi Tech Talk: Messaging Modernization

Citi Tech Talk: Data Governance for streaming and real time data

Confluent & GSI Webinars series: Session 2

Data In Motion Paris 2023

Confluent Partner Tech Talk with Synthesis

The Future of Application Development - API Days - Melbourne 2023

The Playful Bond Between REST And Data Streams

Kürzlich hochgeladen

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Angeliki Cooney has spent over twenty years at the forefront of the life sciences industry, working out of Wynantskill, NY. She is highly regarded for her dedication to advancing the development and accessibility of innovative treatments for chronic diseases, rare disorders, and cancer. Her professional journey has centered on strategic consulting for biopharmaceutical companies, facilitating digital transformation, enhancing omnichannel engagement, and refining strategic commercial practices. Angeliki's innovative contributions include pioneering several software-as-a-service (SaaS) products for the life sciences sector, earning her three patents. As the Senior Vice President of Life Sciences at Avenga, Angeliki orchestrated the firm's strategic entry into the U.S. market. Avenga, a renowned digital engineering and consulting firm, partners with significant entities in the pharmaceutical and biotechnology fields. Her leadership was instrumental in expanding Avenga's client base and establishing its presence in the competitive U.S. market.

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

Angeliki Cooney

Keynote 2: APIs in 2030: The Risk of Technological Sleepwalk Paolo Malinverno, Growth Advisor - The Business of Technology Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

apidays

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

The microservices honeymoon is over. When starting a new project or revamping a legacy monolith, teams started looking for alternatives to microservices. The Modular Monolith, or 'Modulith', is an architecture that reaps the benefits of (vertical) functional decoupling without the high costs associated with separate deployments. This talk will delve into the advantages and challenges of this progressive architecture, beginning with exploring the concept of a 'module', its internal structure, public API, and inter-module communication patterns. Supported by spring-modulith, the talk provides practical guidance on addressing the main challenges of a Modultith Architecture: finding and guarding module boundaries, data decoupling, and integration module-testing. You should not miss this talk if you are a software architect or tech lead seeking practical, scalable solutions. About the author With two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Victor Rentea

Accelerating FinTech Innovation: Unleashing API Economy and GenAI Vasa Krishnan, Chief Technology Officer - FinResults Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

apidays

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Exploring Multimodal Embeddings with Milvus

Zilliz

💥 You’re lucky! We’ve found two different (lead) developers that are willing to share their valuable lessons learned about using UiPath Document Understanding! Based on recent implementations in appealing use cases at Partou and SPIE. Don’t expect fancy videos or slide decks, but real and practical experiences that will help you with your own implementations. 📕 Topics that will be addressed: • Training the ML-model by humans: do or don't? • Rule-based versus AI extractors • Tips for finding use cases • How to start 👨‍🏫👨‍💻 Speakers: o Dion Morskieft, RPA Product Owner @Partou o Jack Klein-Schiphorst, Automation Developer @Tacstone Technology

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

UiPathCommunity

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

Retrieval augmented generation (RAG) is the most popular style of large language model application to emerge from 2023. The most basic style of RAG works by vectorizing your data and injecting it into a vector database like Milvus for retrieval to augment the text output generated by an LLM. This is just the beginning. One of the ways that we can extend RAG, and extend AI, is through multilingual use cases. Typical RAG is done in English using embedding models that are trained in English. In this talk, we’ll explore how RAG could work in languages other than English. We’ll explore French, Chinese, and Polish.

Introduction to Multilingual Retrieval Augmented Generation (RAG)

Zilliz

Six Myths about Ontologies: The Basics of Formal Ontology

johnbeverley2021

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

Discover the innovative features and strategic vision that keep WSO2 an industry leader. Explore the exciting 2024 roadmap of WSO2 API management, showcasing innovations, unified APIM/APK control plane, natural language API interaction, and cloud native agility. Discover how open source solutions, microservices architecture, and cloud native technologies unlock seamless API management in today's dynamic landscapes. Leave with a clear blueprint to revolutionize your API journey and achieve industry success!

WSO2's API Vision: Unifying Control, Empowering Developers

WSO2

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

MadyBayot

Dubai, often portrayed as a shimmering oasis in the desert, faces its own set of challenges, including the occasional threat of flooding. Despite its reputation for opulence and modernity, the emirate is not immune to the forces of nature. In recent years, Dubai has experienced sporadic but significant floods, testing the resilience of its infrastructure and communities. Among the critical lifelines in this bustling metropolis is the Dubai International Airport, a bustling hub that connects the city to the world. This article explores the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Orbitshub

Vector Search -An Introduction in Oracle Database 23ai.pptx

Remote DBA Services

Kürzlich hochgeladen (20)

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

AWS Community Day CPH - Three problems of Terraform

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

presentation ICT roal in 21st century education

Exploring Multimodal Embeddings with Milvus

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

Apidays New York 2024 - The value of a flexible API Management solution for O...

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Introduction to Multilingual Retrieval Augmented Generation (RAG)

Six Myths about Ontologies: The Basics of Formal Ontology

Corporate and higher education May webinar.pptx

WSO2's API Vision: Unifying Control, Empowering Developers

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Vector Search -An Introduction in Oracle Database 23ai.pptx

Migrating applications to serverless Apache Kafka + KSQL

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Migrating Applications to Serverless Apache Kafka and KSQL Tim Berglund S e s s i o n I D Sr. Director, Developer Relations Confluent

3. Our Movie Rating Service, Legacy Edition • Users rate movies, ratings go into Kafka • Monolithic, database-centric application calculates averages • Serves them to users through a web UI and API Movie Ratings Users Movies Top Rated Movies My Favorite Movie Moviegoers

4. Kafka served well • Decouples event input from processing • Easily understood abstraction for event processing • Not exactly a pleasure to operate, but we can’t complain Movie Ratings Users Movies Top Rated Movies My Favorite Movie Moviegoers

5. Or can we?

6. Our Monolith’s Problems • It will do a bad job managing complexity as our service grows • The Kafka Consumer code is bespoke • It is a textbook pre-cloud architecture • We cannot trivially scale to larger message volumes Movie Ratings Users Movies Top Rated Movies My Favorite Movie Moviegoers

8. Our Refactoring Plan • Capture Users and Movies as Kafka topics • Migrate all topics to Confluent Cloud using Confluent Replicator • Refactor monolith to microservices • Keep web UI nearly untouched • Never touch the on-prem system until the migration is complete Movie Ratings Users Movies Top Rated Movies My Favorite Movie Moviegoers

9. Step One: Fewer Databases • Use Kafka Connect to extract Users and Movies tables to Kafka topics Movie Ratings Moviegoers Movies Users Kafka Connect

10. Step Two: Spin up a Confluent Cloud cluster • We want to get out of the business of managing Kafka ourselves Movie Ratings Movies Users

11. Step Three: Deploy Confluent Replicator • Use Kafka Connect to extract Users and Movies tables to Kafka topics Movie Ratings Movies Users Movie Ratings Movies Users Replicator Replicator Replicator

12. Step Four: Convert to KSQL • Bespoke Consumer code implements non-differentiated functionality Movie Ratings Movies Users CREATE TABLE movie_ratings AS SELECT title, SUM(rating)/COUNT(rating) AS avg_rating, COUNT(rating) AS num_ratings FROM ratings LEFT OUTER JOIN movies ON ratings.movie_id = movies.movie_id GROUP BY title;

13. Step Four: Convert to KSQL • The rating averaging query Movie Ratings Movies Users Rated Movies KSQL magic goes here

14. Step Four: Convert to KSQL • The user favorite query Movie Ratings Movies Users Rated Movies KSQL magic goes here more KSQL magic goes here User Favorites

15. Step Five: Extract the rating average service • Now serve rating averages from KSQL output • Monolith no longer serves these results Rating Averages Rated Movies

16. Step Six: Extract the user favorite service • Now serve rating averages from KSQL output • Monolith no longer serves these results User Favorites User Favorites

17. Step Seven: Stand down the monolith • Now serve rating averages from KSQL output • Monolith no longer serves these results User Favorites Rated Movies User Favorites Movie Ratings Moviegoers Rated Movies so much KSQL magic Movies Users

18. Step Eight: Stand down Replicator • All data is in Confluent Cloud now • For hybrid on-prem/cloud deployment

Migrating applications to serverless Apache Kafka + KSQL

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Migrating applications to serverless Apache Kafka + KSQL

Ähnlich wie Migrating applications to serverless Apache Kafka + KSQL (20)

Mehr von confluent

Mehr von confluent (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Migrating applications to serverless Apache Kafka + KSQL