One sink to rule them all: Introducing the new Async Sink

•Als PPTX, PDF herunterladen•

0 gefällt mir•366 views

Flink Forward San Francisco 2022. Next time you want to integrate with a new destination for a demo, concept or production application, the Async Sink framework will bootstrap development, allowing you to move quickly without compromise. In Flink 1.15 we introduced the Async Sink base (FLIP-171), with the goal to encapsulate common logic and allow developers to focus on the key integration code. The new framework handles things like request batching, buffering records, applying backpressure, retry strategies, and at least once semantics. It allows you to focus on your business logic, rather than spending time integrating with your downstream consumers. During the session we will dive deep into the internals to uncover how it works, why it was designed this way, and how to use it. We will code up a new sink from scratch and demonstrate how to quickly push data to a destination. At the end of this talk you will be ready to start implementing your own Flink sink using the new Async Sink framework. by Steffen Hausmann & Danny Cranmer

Technologie

ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
© 2022, Amazon Web Services, Inc. or its affiliates.
One sink to rule them all
Introducing the new Async Sink
Danny Cranmer
Sr. Software Development Engineer, AWS
Steffen Hausmann
Principal Streaming Architect, AWS

ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
Apache Flink’s connector ecosystem
2
Amazon Kinesis
Data Streams
Elasticsearch
RabbitMQ
Google PubSub

ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
How to build a high quality connector
Buffering Send batch
requests
Retry failed
requests
Checkpointing Rate limiting and
Backpressure

ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
Async Sink internals
internal buffer
endpoint
message request entries
batch request
response
requeue
failed requests
checkpoint
sink to
AsyncSinkWriter
request entries -> batch request
ElementConverter
message -> request entry
AsyncSinkWriter
response -> failed requests
AsyncSink
WriterStateSerializer
4

ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
How the Async Sink manages throughput
6
endpoint
sink subtask3
sink subtask2
sink subtask1
1. Limit the number of in-flight request entries
2. If a batch request was successful, increase
the limit by a constant factor
3. If a batch request failed, cut the limit in half
Additive Increase/Multiplicative Decrease

ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
Async Sink configuration
endpoint
message request entries
batch request
sink to
maxRecordSizeInBytes
8
maxInFlightRequests
maxBatchSize
maxBatchSizeInBytes
maxBufferedRequests
maxTimeInBufferMs
internal buffer

ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
Current limitations
At least
once
Ordering Thread pool
management

ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
What’s Next?
FLIP-242
Configurable
Rate Limiting
FLIP-252
Amazon DynamoDB
Sink
Wider
Community
Adoption

ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
Thank you!
© 2022, Amazon Web Services, Inc. or its affiliates.
Danny Cranmer Steffen Hausmann

Weitere ähnliche Inhalte

Was ist angesagt?

Flink Forward San Francisco 2022. At Stripe we have created a complete end to end exactly-once processing pipeline to process financial data at scale, by combining the exactly-once power from Flink, Kafka, and Pinot together. The pipeline provides exactly-once guarantee, end-to-end latency within a minute, deduplication against hundreds of billions of keys, and sub-second query latency against the whole dataset with trillion level rows. In this session we will discuss the technical challenges of designing, optimizing, and operating the whole pipeline, including Flink, Kafka, and Pinot. We will also share our lessons learned and the benefits gained from exactly-once processing. by Xiang Zhang & Pratyush Sharma & Xiaoman Dong

Exactly-Once Financial Data Processing at Scale with Flink and Pinot

Flink Forward

Flink Forward San Francisco 2022. In normal situations, the default Kafka consumer and producer configuration options work well. But we all know life is not all roses and rainbows and in this session we’ll explore a few knobs that can save the day in atypical scenarios. First, we'll take a detailed look at the parameters available when reading from Kafka. We’ll inspect the params helping us to spot quickly an application lock or crash, the ones that can significantly improve the performance and the ones to touch with gloves since they could cause more harm than benefit. Moreover we’ll explore the partitioning options and discuss when diverging from the default strategy is needed. Next, we’ll discuss the Kafka Sink. After browsing the available options we'll then dive deep into understanding how to approach use cases like sinking enormous records, managing spikes, and handling small but frequent updates.. If you want to understand how to make your application survive when the sky is dark, this session is for you! by Olena Babenko

Tuning Apache Kafka Connectors for Flink.pptx

Flink Forward

Flink Forward San Francisco 2022. Pinterest is a visual discovery engine that serves over 433MM users. Stream processing allows us to unlock value from realtime data for pinners. At Pinterest, we adopt Flink as the unified streaming processing engine. In this talk, we will share our journey in building a stream processing platform with Flink and how we onboarding critical use cases to the platform. Pinterest has supported 90+near realtime streaming applications. We will cover the problem statement, how we evaluate potential solutions and our decision to build the framework. by Rainie Li & Kanchi Masalia

Flink powered stream processing platform at Pinterest

Flink Forward

Flink Forward San Francisco 2022. Task Managers constantly running out of memory? Flink job keeps restarting from cryptic Akka exceptions? Flink job running but doesn’t seem to be processing any records? We share practical learnings from running thousands of Flink Jobs for different use-cases and take a look at common challenges they have experienced such as out-of-memory errors, timeouts and job stability. We will cover memory tuning, S3 and Akka configurations to address common pitfalls and the approaches that we take on automating health monitoring and management of Flink jobs at scale. by Hong Teoh & Usamah Jassat

Practical learnings from running thousands of Flink jobs

Flink Forward

Apache Flink is a popular stream computing framework for real-time stream computing. Many stream compute algorithms require trailing data in order to compute the intended result. One example is computing the number of user logins in the last 7 days. This creates a dilemma where the results of the stream program are incomplete until the runtime of the program exceeds 7 days. The alternative is to bootstrap the program using historic data to seed the state before shifting to use real-time data. This talk will discuss alternatives to bootstrap programs in Flink. Some alternatives rely on technologies exogenous to the stream program, such as enhancements to the pub/sub layer, that are more generally applicable to other stream compute engines. Other alternatives include enhancements to Flink source implementations. Lyft is exploring another alternative using orchestration of multiple Flink programs. The talk will cover why Lyft pursued this alternative and future directions to further enhance bootstrapping support in Flink. Speaker Gregory Fee, Principal Engineer, Lyft

Bootstrapping state in Apache Flink

DataWorks Summit

Deploying Flink on Kubernetes - David Anderson

Ververica

Flink Forward San Francisco 2022. Resource Elasticity is a frequently requested feature in Apache Flink: Users want to be able to easily adjust their clusters to changing workloads for resource efficiency and cost saving reasons. In Flink 1.13, the initial implementation of Reactive Mode was introduced, later releases added more improvements to make the feature production ready. In this talk, we’ll explain scenarios to deploy Reactive Mode to various environments to achieve autoscaling and resource elasticity. We’ll discuss the constraints to consider when planning to use this feature, and also potential improvements from the Flink roadmap. For those interested in the internals of Flink, we’ll also briefly explain how the feature is implemented, and if time permits, conclude with a short demo. by Robert Metzger

Autoscaling Flink with Reactive Mode

Flink Forward

Apache Flink is the foundation for Decodable's real-time SaaS data platform. Flink runs critical data processing jobs with strong security requirements. In addition, Decodable has to scale to thousands of tenants, power various use cases, provide an intuitive user experience and maintain cost-efficiency. We've learned a lot of lessons while building and maintaining the platform. In this talk, I'll share the top 3 toughest challenges building and operating this platform with Flink, and how we solved them.

The top 3 challenges running multi-tenant Flink at scale

Flink Forward

Flink Forward San Francisco 2022. The Apache Flink Kubernetes Operator provides a consistent approach to manage Flink applications automatically, without any human interaction, by extending the Kubernetes API. Given the increasing adoption of Kubernetes based Flink deployments the community has been working on a Kubernetes native solution as part of Flink that can benefit from the rich experience of community members and ultimately make Flink easier to adopt. In this talk we give a technical introduction to the Flink Kubernetes Operator and demonstrate the core features and use-cases through in-depth examples." by Thomas Weise

Introducing the Apache Flink Kubernetes Operator

Flink Forward

Apache Flink Deep Dive

DataWorks Summit

Flink Forward San Francisco 2022. Goldman Sachs's Data Lake platform serves as the firm's centralized data platform, ingesting 140K (and growing!) batches per day of Datasets of varying shape and size. Powered by Flink and using metadata configured by platform users, ingestion applications are generated dynamically at runtime to extract, transform, and load data into centralized storage where it is then exported to warehousing solutions such as Sybase IQ, Snowflake, and Amazon Redshift. Data Latency is one of many key considerations as producers and consumers have their own commitments to satisfy. Consumers range from people/systems issuing queries, to applications using engines like Spark, Hive, and Presto to transform data into refined Datasets. Apache Iceberg allows our applications to not only benefit from consistency guarantees important when running on eventually consistent storage like S3, but also allows us the opportunity to improve our batch processing patterns with its scalability-focused features. by Andreas Hailu

Batch Processing at Scale with Flink & Iceberg

Flink Forward

Flink Forward San Francisco 2022. At Bloomberg, we deal with high volumes of real-time market data. Our clients expect to be notified of any anomalies in this market data, which may indicate volatile movements in the markets, notable trades, forthcoming events, or system failures. The parameters for these alerts are always evolving and our clients can update them dynamically. In this talk, we'll cover how we utilized the open source Apache Flink and Siddhi SQL projects to build a distributed, scalable, low-latency and dynamic rule-based, real-time alerting system to solve our clients' needs. We'll also cover the lessons we learned along our journey. by Ajay Vyasapeetam & Madhuri Jain

Dynamic Rule-based Real-time Market Data Alerts

Flink Forward

Flink Forward San Francisco 2022. A particular feature in our system relies on a streaming 90-minute trailing window of 1-minute samples - implemented as a lookaside cache - to speed up a particular query, allowing our customers to rapidly see an overview of their estate. Across our entire customer base, there is a substantial amount of data flowing into this cache - ~1,000,000 entries/second, with the entire cache requiring ~600GB of RAM. The current implementation is simplistic but expensive. In this talk I describe a replacement implementation as a stateful streaming Flink application leveraging Queryable State. This Flink application reduces the net cost by ~90%. In this session, the implementation is described in detail, including windowing considerations, a sliding-window state buffer that avoids the sliding window replication penalty, and a comparison of queryable state and Redis queries. The talk concludes with a frank discussion of when this distinctive approach is, and is not, appropriate. by Ron Crocker

Using Queryable State for Fun and Profit

Flink Forward

Ever tried to get get clarity on what kinds of memory there are and how to tune each of them ? If not, very likely your jobs are configured incorrectly. As we found out, its is not straightforward and it is not well documented either. This session will provide information on the types of memory to be aware of, the calculations involved in determining how much is allocated to each type of memory and how to tune it depending on the use case.

Demystifying flink memory allocation and tuning - Roshan Naik, Uber

Flink Forward

Flink Forward San Francisco 2022. The world is constantly changing. Data is continuously produced and thus should be consumed in a similar fashion by enterprise systems. Only this enables real-time decisions at scale. Message logs such as Apache Kafka can be found in almost every architecture, while databases and other batch systems still provide the foundation. Change Data Capture (CDC) propagates changes downstream. In this talk, we will highlight what it means to be a general data processor and how Flink can act as an integration hub. We present the current state of Flink and how it can power various use cases on both finite and infinite streams. We demonstrate Flink's SQL engine as a changelog processor that is shipped with an ecosystem tailored to process CDC data and maintain materialized views. We will use Kafka as an upsert log, Debezium for connecting to databases, and enrich streams of various sources. Finally, we will combine Flink's Table API with DataStream API for event-driven applications beyond SQL. by Timo Walther

Changelog Stream Processing with Apache Flink

Flink Forward

Click-Through Example for Flink’s KafkaConsumer Checkpointing

Robert Metzger

Flink Forward San Francisco 2022. The Table API is one of the most actively developed components of Flink in recent time. Inspired by databases and SQL, it encapsulates concepts many developers are familiar with. It can be used with both bounded and unbounded streams in a unified way. But from afar it can be difficult to keep track of what this API is capable of and how it relates to Flink's other APIs. In this talk, we will explore the current state of Table API. We will show how it can be used as a batch processor, a changelog processor, or a streaming ETL tool with many built-in functions and operators for deduplicating, joining, and aggregating data. By comparing it to the DataStream API we will highlight differences and elaborate on when to use which API. We will demonstrate hybrid pipelines in which both APIs interact with one another and contribute their unique strengths. Finally, we will take a look at some of the most recent additions as a first step to stateful upgrades. by David Andreson

The Current State of Table API in 2022

Flink Forward

Getting Started with Confluent Schema Registry

confluent

Flink Forward San Francisco 2022. Running natively on Kubernetes, using the new Apache Flink Kubernetes Operator is a great way to deploy and manage Flink application and session deployments. In this presentation, we provide: - A brief overview of Kubernetes operators and their benefits. - Introduce the five levels of the operator maturity model. - Introduce the newly released Apache Flink Kubernetes Operator and FlinkDeployment CRs - Dockerfile modifications you can make to swap out UBI images and Java of the underlying Flink Operator container - Enhancements we're making in: - Versioning/Upgradeability/Stability - Security - Demo of the Apache Flink Operator in-action, with a technical preview of an upcoming product using the Flink Kubernetes Operator. - Lessons learned - Q&A by James Busche & Ted Chang

Using the New Apache Flink Kubernetes Operator in a Production Deployment

Flink Forward

Apache Flink is a distributed stream processing framework that allows users to process and analyze data in real-time. At LinkedIn, we developed a fully managed stream processing platform on Flink running on K8s to power hundreds of stream processing pipelines in production. This platform is the backbone for other infra systems like Search, Espresso (internal document store) and feature management etc. We provide a rich authoring and testing environment which allows users to create, test, and deploy their streaming jobs in a self-serve fashion within minutes. Users can focus on their business logic, leaving the Flink platform to take care of management aspects such as split deployment, resource provisioning, auto-scaling, job monitoring, alerting, failure recovery and much more. In this talk, we will introduce the overall platform architecture, highlight the unique value propositions that it brings to stream processing at LinkedIn and share the experiences and lessons we have learned.

Building a fully managed stream processing platform on Flink at scale for Lin...

Flink Forward

Was ist angesagt? (20)

Exactly-Once Financial Data Processing at Scale with Flink and Pinot

Tuning Apache Kafka Connectors for Flink.pptx

Flink powered stream processing platform at Pinterest

Practical learnings from running thousands of Flink jobs

Bootstrapping state in Apache Flink

Deploying Flink on Kubernetes - David Anderson

Autoscaling Flink with Reactive Mode

The top 3 challenges running multi-tenant Flink at scale

Introducing the Apache Flink Kubernetes Operator

Apache Flink Deep Dive

Batch Processing at Scale with Flink & Iceberg

Dynamic Rule-based Real-time Market Data Alerts

Using Queryable State for Fun and Profit

Demystifying flink memory allocation and tuning - Roshan Naik, Uber

Changelog Stream Processing with Apache Flink

Click-Through Example for Flink’s KafkaConsumer Checkpointing

The Current State of Table API in 2022

Getting Started with Confluent Schema Registry

Using the New Apache Flink Kubernetes Operator in a Production Deployment

Building a fully managed stream processing platform on Flink at scale for Lin...

Ähnlich wie One sink to rule them all: Introducing the new Async Sink

Use Amazon EC2 Systems Manager to Perform Automated Resilience Testing in You...

Amazon Web Services

"Catsndogs.lol is a (fictional) company that needs help deploying and scaling its container-based application. During this workshop, attendees will join the new DevOps team at CatsnDogs.lol, and help the company to manage their applications using Amazon EC2 Container Service (ECS), and help release new features to make our customers happier than ever. Attendees will get hands-on with service and container-instance auto-scaling, spot-fleet integration, container placement strategies, service discovery, secrets management with AWS Systems Manager Parameter Store, time-based and event-based scheduling, and automated deployment pipelines. If you are a developer interested in learning more about how Amazon ECS can accelerate your application development and deployment workflows, or if you are a systems administrator or devops person interested in understanding how Amazon ECS can simplify the operational model associated with running containers at scale, then this workshop is for you. You should have basic familiarity with ECS, EC2, and IAM. The workshop requires: A laptop (Windows, OSX, or Linux) with the AWSCLI or AWS Powershell installed An AWS account with administrative permissions (including the ability to create IAM roles and policies) created at least 24 hours in advance."

CON317_Advanced container management at catsndogs.lol

Amazon Web Services

Most organizations today run their production workloads inside Amazon Virtual Private Cloud (Amazon VPC). This software-defined network structure provides the boundaries that are needed for the security that an organization and its customers require. For most organizations, the natural evolution in their architecture, security, and environment involves migrating from a single VPC to multiple VPCs in the same AWS Region and across many other AWS Regions. The question of how to enforce security policies while simplifying the flow of traffic between multiple VPCs, the data center, and remote offices while adhering to AWS best practices becomes an intricate one to answer. In this chalk talk, we provide solutions to scenarios like these and more. Topics include Amazon security groups, NACLs, static and dynamic VPNs, AWS Direct Connect, IPS and IDS, transit VPC architectures, designing for security, and more.

From One to Many: Diving Deeper into Evolving VPC Design (ARC310-R2) - AWS re...

Amazon Web Services

Event-driven architectures can help decouple services and simplify dependencies as your application grows. There are now new options for developers looking to benefit from this approach with the release of AWS Lambda controls for asynchronous and stream processing events, Lambda event destinations, and Amazon EventBridge, a serverless event bus service. In this tech talk, we'll demonstrate strategies for managing events to help you build more resilient event-driven applications. You will learn how to customize responses to Lambda function results with new service features to handle retries, failures, and prevent “poison pill” events. We will also demonstrate how to configure Lambda event destinations to route execution results to other AWS services, and how to set up routing rules in Amazon EventBridge that react to all of your data sources.

AWS re:Invent serverless recap day: Controls for-managing-events-in-serverles...

⛷️ Ben Smith

by Itzik Paz, Solutions Architect & Rich Cowper, Solutions Architect Manager, AWS This presentation compares three modern architecture patterns that startups are building their businesses around. It includes a realistic analysis of cost, team management, and security implications of each approach. It covers AWS Elastic Beanstalk, Amazon ECS, Docker, Amazon API Gateway, AWS Lambda, Amazon DynamoDB, and Amazon CloudFront.

Born in the Cloud, Built like a Startup

Amazon Web Services

How do you ensure that a containerized system can handle the needs of your application? Designing and testing for performance is a critical aspect of operating containerized architectures at scale. In this session, we cover best practices for designing perfomant containerized applications on AWS using Kubernetes. We also show you how State Street deployed a high-performance database at scale using Amazon Elastic Container Service for Kubernetes (Amazon EKS).

Running a High-Performance Kubernetes Cluster with Amazon EKS (CON318-R1) - A...

Amazon Web Services

AWS DevDay Berlin - Automating building blocks choices you will face with con...

Cobus Bernard

Scaling a microservice-based infrastructure can be challenging in terms of both technical implementation and developer workflow. In this talk, AWS Solutions Architect Pierre Steckmeyer will be joined by Will McCutchen, Architect at BuzzFeed, to discuss Amazon ECS as a platform for building a robust infrastructure for microservices. We will look at the key attributes of microservice architectures and how Amazon ECS supports these requirements in production, from configuration to sophisticated workload scheduling to networking capabilities to resource optimization. We will also examine what it takes to build an end-to-end platform on top of the wider AWS ecosystem, and what it's like to migrate a large engineering organization from a monolithic approach to microservices.

Advanced Patterns in Microservices Implementation with Amazon ECS - CON402 - ...

Amazon Web Services

Interstella 8888 is an intergalactic trading company that deals in rare resources, but their antiquated monolithic logistics systems are causing the business to lose money. Join this workshop to learn how to set up a CI/CD pipeline for containerized microservices. You'll get hands-on experience deploying Docker container images using Amazon ECS AWS CloudFormation, AWS CodeBuild, and AWS CodePipline, automating everything from code check-in to production. AWS credits are provided. Bring your laptop, and have an active AWS account.

Interstella 8888: CICD for Containers on AWS - CON319 - re:Invent 2017

Amazon Web Services

"Interstella 8888 is an intergalactic trading company that deals in rare resources, but their antiquated monolithic logistics systems are causing the business to lose money. Join this workshop to learn how to set up a CI/CD pipeline for containerized microservices. You'll get hands-on experience deploying Docker container images using Amazon ECS AWS CloudFormation, AWS CodeBuild, and AWS CodePipline, automating everything from code check-in to production. AWS credits are provided. Bring your laptop, and have an active AWS account."

CON319_Interstella GTC CICD for Containers on AWS

Amazon Web Services

Introduction to AWS Fargate & Amazon Elastic Container Service for Kubernetes

Amazon Web Services

AWS DevDay Vienna - Automating building blocks choices you will face with con...

Cobus Bernard

CI/CD for a Docker Node.JS application using Code* services. This session will walkthrough what a solution like this would look like, what Code* services are used, how your build will work, and how deploys will work. The purpose of this session is to allow customers to see how to deploy their containerized applications in Amazon Elastic Container Service (ECS) Fargate using our CI/CD solutions. Come with your questions and pain points. We will also talk about how to use Bitbucket as your source control rather than Code Commit for the many customers already using BitBucket and Jenkins.

CI/CD with AWS Developer Tools and Fargate

Amazon Web Services

Aws container services overview

Patricio Vazquez

Containers on AWS: What You Missed at re:Invent 2017 - AWS Online Tech Talks

Amazon Web Services

Cisco’s video solutions were historically designed for on-premises dedicated hardware deployments. Typically, major releases occurred annually or bi-annually. The release process lacked the ability to absorb frequent changes and adapt to rapid market trends. This session looks into how Cisco’s IVP Solution team evolved a production system from its monolithic design into a microservices platform, leveraging cloud services, automated deployments, and delivery pipelines. Through this transition the team adopted a biweekly deployment cadence. This ultimately enabled a fast-paced migration to an AWS environment, using AWS services such as Amazon EC2, Amazon RDS, and Amazon Elasticsearch Service.

DEV329_Cisco’s Journey from Monolith to Microservices

Amazon Web Services

VMware cloud on AWS

Amazon Web Services

DevSecOps is the premise that everyone in the software development lifecycle is responsible for security. DevSecOps aims to embed security in every part of the development process. In this workshop, participants explore taking a standard CI/CD pipeline and adding security stages to improve security posture. Learn how to use AWS CodeCommit and AWS CodePipeline to build and publish golden AMI images. Also, learn how to modify pipeline flow to add security test cases. You also have to opportunity to perform CVE analysis and code analysis using Amazon Inspector and perform observational container analysis using Amazon GuardDuty.

Adding the Sec to Your DevOps Pipelines (SEC332-R1) - AWS re:Invent 2018

Amazon Web Services

Building an Immersive VR Streaming Solution on AWS - AWS Online Tech Talks

Amazon Web Services

Running and managing a Kubernetes environment at scale can be easier than you think! In this session we will share some expert tips that can help you run production grade Kubernetes deployments on AWS. We will review native AWS services you can use to improve security, deployments, networking, logging and monitoring, and CI/CD. We will also highlight third party services that are popular across a multitude of customers. Speaker: Mitch Beaumont, Solutions Architect, AWS

Expert Tips for Successful Kubernetes Deployments on AWS

Amazon Web Services

Ähnlich wie One sink to rule them all: Introducing the new Async Sink (20)

Use Amazon EC2 Systems Manager to Perform Automated Resilience Testing in You...

CON317_Advanced container management at catsndogs.lol

From One to Many: Diving Deeper into Evolving VPC Design (ARC310-R2) - AWS re...

AWS re:Invent serverless recap day: Controls for-managing-events-in-serverles...

Born in the Cloud, Built like a Startup

Running a High-Performance Kubernetes Cluster with Amazon EKS (CON318-R1) - A...

AWS DevDay Berlin - Automating building blocks choices you will face with con...

Advanced Patterns in Microservices Implementation with Amazon ECS - CON402 - ...

Interstella 8888: CICD for Containers on AWS - CON319 - re:Invent 2017

CON319_Interstella GTC CICD for Containers on AWS

Introduction to AWS Fargate & Amazon Elastic Container Service for Kubernetes

AWS DevDay Vienna - Automating building blocks choices you will face with con...

CI/CD with AWS Developer Tools and Fargate

Aws container services overview

Containers on AWS: What You Missed at re:Invent 2017 - AWS Online Tech Talks

DEV329_Cisco’s Journey from Monolith to Microservices

VMware cloud on AWS

Adding the Sec to Your DevOps Pipelines (SEC332-R1) - AWS re:Invent 2018

Building an Immersive VR Streaming Solution on AWS - AWS Online Tech Talks

Expert Tips for Successful Kubernetes Deployments on AWS

Mehr von Flink Forward

Flink Forward San Francisco 2022. Based on the new Flink-Pulsar connector, we implemented Flink's TableAPI and Catalog to help users to interact with the Pulsar cluster via Flink SQL easily. We would like to go through the design and implementation of the SQL connector in the following aspects: 1. Two different modes of use Pulsar as a metadata store 2. Data format transformation and management 3. SQL semantics support within Pulsar context by Sijie Guo & Neng Lu

Flink SQL on Pulsar made easy

Flink Forward

Flink Forward San Francisco 2022. What if my data is already in order? Stream Processing has given us an elegant and powerful solution for running analytic queries and logic over high volumes of continuously arriving data. However, in both Apache Flink and Apache Beam, the notion of time-ordering is baked in at a very low level, making it difficult to express computations that are interested in a semantic-, rather than time-ordering of the data. In financial services, what often matters the most about the data moving between systems is not when the data was created, but in what order, to the extent that many institutions engineer a global sequencing over all data entering and produced by their systems to achieve complete determinism. How, then, can financial institutions and others best employ Stream Processing on streams of data that are already ordered? I will cover various techniques that can make this work, as well as seek input from the community on how Flink might be improved to better support these use-cases. by Patrick Lucas

Processing Semantically-Ordered Streams in Financial Services

Flink Forward

Flink Forward San Francisco 2022. At Flink Forward, we get to hear creative, unique use cases, often on the bleeding edge of some of the most exciting current technologies. This talk will give you a chance to get to open up the hood on our driven and innovative Open Source community. I will cover what our community has been working on this past year, and how this work relates to our (Ververica's) exciting new Flink engineering roadmap! I will also go through some best practices and upcoming opportunities for getting involved in this community! by Caito Scherr

Welcome to the Flink Community!

Flink Forward

Flink Forward San Francisco 2022. Apache Flink is a powerful stream processing platform that enables users to build complex real time applications. Flink SQL provides a SQL interface that implements standard SQL. While the standard SQL provides a perfect interface for batch processing, in stream processing context, it can result is ambiguity and complex syntax. As an example, consider these three types of streams: Append-only stream, Retract stream and Upsert stream. Using standard SQL, we would represent all of these streams as Table along with the Table concept in batch processing. Such overloading of concepts can result in ambiguity in SQL statements in streaming context. In this talk, we will present extensions to the Flink SQL that simplify SQL statements in the context of stream processing. We will show how such extensions work in the context of a Flink application using different use cases. These extensions are only sugar syntax and users should be able to use Flink SQL as is if they desire. by Hojjat Jafarpour

Extending Flink SQL for stream processing use cases

Flink Forward

Flink Forward San Francisco 2022. Neuro-ID analyzes web behavior at a large scale to determine visitors' intent on web pages, specifically in the online lending industry. When users interact with an online loan application, our software analyzes their behavior to determine if the applicant may be potentially fraudulent. Lenders can then request various scores describing the applicant's intentions in real-time to use to make decisions during the application flow. Flink gives our product the ability to observe behavior in a stateful manner. As an applicant interacts with an online loan application, a Flink application is used to compare earlier actions to later actions. This processing in Flink can determine the applicant's intent throughout the process of the application. by Jeff Niemann & Randy Hanak

Large Scale Real Time Fraudulent Web Behavior Detection

Flink Forward

Flink Forward San Francisco 2022. Being in the payments space, Stripe requires strict correctness and freshness guarantees. We rely on Flink as the natural solution for delivering on this in support of our Change Data Capture (CDC) infrastructure. We heavily rely on CDC as a tool for capturing data change streams from our databases without critically impacting database reliability, scalability, and maintainability. Data derived from these streams is used broadly across the business and powers many of our critical financial reporting systems totalling over $640 Billion in payment volume annually. We use many components of Flink’s flexible DataStream API to perform aggregations and abstract away the complexities of stream processing from our downstreams. In this talk, we’ll walk through our experience from the very beginning to what we have in production today. We’ll share stories around the technical details and trade-offs we encountered along the way. by Jeff Chao

Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...

Flink Forward

Flink Forward San Francisco 2022. Apache Flink and Delta Lake together allow you to build the foundation for your data lakehouses by ensuring the reliability of your concurrent streams from processing to the underlying cloud object-store. Together, the Flink/Delta Connector enables you to store data in Delta tables such that you harness Delta’s reliability by providing ACID transactions and scalability while maintaining Flink’s end-to-end exactly-once processing. This ensures that the data from Flink is written to Delta Tables in an idempotent manner such that even if the Flink pipeline is restarted from its checkpoint information, the pipeline will guarantee no data is lost or duplicated thus preserving the exactly-once semantics of Flink. by Scott Sandre & Denny Lee

Building Reliable Lakehouses with Apache Flink and Delta Lake

Flink Forward

Flink Forward San Francisco 2022. With a real-time processing engine like Flink and a transactional storage layer like Hudi, it has never been easier to build end-to-end low-latency data platforms connecting sources like Kafka to data lake storage. Come learn how to blend Lakehouse architectural patterns with real-time processing pipelines with Flink and Hudi. We will dive deep on how Flink can leverage the newest features of Hudi like multi-modal indexing that dramatically improves query and write performance, data skipping that reduces the query latency by 10x for large datasets, and many more innovations unique to Flink and Hudi. by Ethan Guo & Kyle Weller

How to build a streaming Lakehouse with Flink, Kafka, and Hudi

Flink Forward

Mehr von Flink Forward (8)

Flink SQL on Pulsar made easy

Processing Semantically-Ordered Streams in Financial Services

Welcome to the Flink Community!

Extending Flink SQL for stream processing use cases

Large Scale Real Time Fraudulent Web Behavior Detection

Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...

Building Reliable Lakehouses with Apache Flink and Delta Lake

How to build a streaming Lakehouse with Flink, Kafka, and Hudi

Kürzlich hochgeladen

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

MINDCTI Revenue Release Quarter One 2024

MIND CTI

A Principled Technologies deployment guide Conclusion Deploying VMware Cloud Foundation 5.1 on next gen Dell PowerEdge servers brings together critical virtualization capabilities and high-performing hardware infrastructure. Relying on our hands-on experience, this deployment guide offers a comprehensive roadmap that can guide your organization through the seamless integration of advanced VMware cloud solutions with the performance and reliability of Dell PowerEdge servers. In addition to the deployment efficiency, the Cloud Foundation 5.1 and PowerEdge solution delivered strong performance while running a MySQL database workload. By leveraging VMware Cloud Foundation 5.1 and PowerEdge servers, you could help your organization embrace cloud computing with confidence, potentially unlocking a new level of agility, scalability, and efficiency in your data center operations.

Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...

Principled Technologies

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Miguel Araújo

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

Manulife - Insurer Innovation Award 2024

The Digital Insurer

🐬 The future of MySQL is Postgres 🐘

RTylerCroy

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

This presentation explores the impact of HTML injection attacks on web applications, detailing how attackers exploit vulnerabilities to inject malicious code into web pages. Learn about the potential consequences of such attacks and discover effective mitigation strategies to protect your web applications from HTML injection vulnerabilities. for more information visit https://bostoninstituteofanalytics.org/category/cyber-security-ethical-hacking/

HTML Injection Attacks: Impact and Mitigation Strategies

Boston Institute of Analytics

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Kürzlich hochgeladen (20)

Data Cloud, More than a CDP by Matt Robison

MINDCTI Revenue Release Quarter One 2024

Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

AWS Community Day CPH - Three problems of Terraform

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Artificial Intelligence: Facts and Myths

Automating Google Workspace (GWS) & more with Apps Script

Apidays New York 2024 - The value of a flexible API Management solution for O...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

A Domino Admins Adventures (Engage 2024)

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Manulife - Insurer Innovation Award 2024

🐬 The future of MySQL is Postgres 🐘

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

HTML Injection Attacks: Impact and Mitigation Strategies

Boost Fertility New Invention Ups Success Rates.pdf

One sink to rule them all: Introducing the new Async Sink

1. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. One sink to rule them all Introducing the new Async Sink Danny Cranmer Sr. Software Development Engineer, AWS Steffen Hausmann Principal Streaming Architect, AWS

2. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. Apache Flink’s connector ecosystem 2 Amazon Kinesis Data Streams Elasticsearch RabbitMQ Google PubSub

3. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. How to build a high quality connector Buffering Send batch requests Retry failed requests Checkpointing Rate limiting and Backpressure

4. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. Async Sink internals internal buffer endpoint message request entries batch request response requeue failed requests checkpoint sink to AsyncSinkWriter request entries -> batch request ElementConverter message -> request entry AsyncSinkWriter response -> failed requests AsyncSink WriterStateSerializer 4

5. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. How the Async Sink manages throughput 6 endpoint sink subtask3 sink subtask2 sink subtask1 1. Limit the number of in-flight request entries 2. If a batch request was successful, increase the limit by a constant factor 3. If a batch request failed, cut the limit in half Additive Increase/Multiplicative Decrease

7. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. Async Sink configuration endpoint message request entries batch request sink to maxRecordSizeInBytes 8 maxInFlightRequests maxBatchSize maxBatchSizeInBytes maxBufferedRequests maxTimeInBufferMs internal buffer

9. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. What’s Next? FLIP-242 Configurable Rate Limiting FLIP-252 Amazon DynamoDB Sink Wider Community Adoption

10. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. Thank you! © 2022, Amazon Web Services, Inc. or its affiliates. Danny Cranmer Steffen Hausmann

One sink to rule them all: Introducing the new Async Sink

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie One sink to rule them all: Introducing the new Async Sink

Ähnlich wie One sink to rule them all: Introducing the new Async Sink (20)

Mehr von Flink Forward

Mehr von Flink Forward (8)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

One sink to rule them all: Introducing the new Async Sink