Flink Forward Berlin 2018: Tobias Lindener - "Approximate standing queries on Stream Processing"

•

3 gefällt mir•875 views

Data analytics in its infancy has taken off with the development of SQL. Yet, at web-scale, even simple analytics queries can prove challenging within (Distributed-) Stream Processing environments. Two such examples are Count and Count Distinct. Because of the key-oriented nature of these queries, traditionally such queries would result in ever increasing memory demand. Through approximation techniques with fixed-size memory consumption, said tasks are feasible and potentially more resource efficient within streaming systems. This is demonstrated by integrating Yahoo Data Sketches on Apache Flink. The evaluation highlights the resource efficiency as well as the challenges of approximation techniques (e.g. varying accuracy) and potential for tuning depending on the dataset. Furthermore, challenges in integrating the components within the existing Streaming interfaces(e.g. Table API) and stateful processing are presented.

Technologie

This project has received funding from the European Union’s Horizon 2020
research and innovation program under grant agreement No. 688191.
@tobiaslindener
lindener@kth.se
Approximate Standing
Queries on Apache Flink

Overview
1. Introduction
2. Background
3. Design & Implementation
4. Results
04.09.2018 2

It is better to use a crude approximation and know the truth, plus or minus
10 percent, than demand an exact solution and know nothing at all
In Arthur Bloch, The Complete Murphy's Law: A Definitive Collection (1991), 126
04.09.2018 3

Unbounded Stream
04.09.2018 4
ID_A
P_A
Key Count
A 1
Time
Infinite
Growth

Unbounded Stream
04.09.2018 5
ID_A
P_A
ID_B
P_A
Key Count
A 1
B 1
Time
Infinite
Growth

Unbounded Stream
04.09.2018 6
ID_A
P_A
ID_C
P_A
ID_B
P_A
Key Count
A 1
B 1
C 1
Time
Infinite
Growth

Unbounded Stream
04.09.2018 7
ID_A
P_A
ID_C
P_A
ID_D
P_A
ID_B
P_A
Key Count
A 1
B 1
C 1
D 1
Time
Infinite
Growth

Unbounded Stream
04.09.2018 8
ID_A
P_A
ID_C
P_A
ID_D
P_A
ID_C
P_A
ID_B
P_A
Key Count
A 1
B 1
C 2
D 1
Time
Infinite
Growth

Unbounded Stream
04.09.2018 9
ID_A
P_A
ID_C
P_A
ID_D
P_A
ID_C
P_A
ID_B
P_A
Key Count
A 1
B 1
C 2
D 1
Time
Infinite
Growth

Approximation Algorithms
Use-cases
• Identify heavy hitters (Count)
• Cardinality Estimation (Count Distinct)
Algorithms
• Frequent Item Estimation
• HyperLogLog
• Quantile Estimation
04.09.2018 10

Processing
04.09.2018 11
Flink Architecture (Apache Software Foundation, 2018)
Approximate Queries

Processing
04.09.2018 12
Flink Architecture (Apache Software Foundation, 2018)
Approximate
Queries

Estimate Deviation - Frequent Items
WIKITRACE DATASET TOP 6000 URL AMAZON DATASET TOP 1000 REVIEWER
04.09.2018 16

Estimate Deviation - HyperLogLog
04.09.2018 17

Native
Implementation
• Potentially increased efficiency
• Reduced overhead
• Stateful processing
Integration with
Table API
• CQL with Approximate Queries
• Support for Count Distinct
Queryable State • Reduced data handling
Future Work
04.09.2018 18

Queryable State
04.09.2018 19
Stream Elements Query Results
Sketch
Function
Time

Queryable State
04.09.2018 20
Stream Elements Query Results
Sketch
Function
Time
Query Results
REST API

Conclusion
CHALLENGES
• Efficient Grouping (HLL)
• Stateful Native Implementation
• Skewed Datasets
LEARNINGS
• Importance of Data
Distribution
• Performance Advantages
• Algorithm Parameters
04.09.2018 21

Team
04.09.2018 22
Tobias Lindener
Consultant @ Netlight
Theodore Vasiloudis
Researcher @ RISE
Paris Carbone
PhD candidate @ KTH
Slides
bit.ly/2LULyoZ

References & Links
• https://github.com/tlindener/ApproximateQueries
• https://datasketches.github.io/
• Daniel Anderson, Pryce Bevan, Kevin Lang, Edo Liberty, Lee
Rhodes, Justin Thaler. A High-Performance Algorithm for
Identifying Frequent Items in Data Streams.
• Kevin Lang, Back to the Future: an Even More Nearly Optimal
Cardinality Estimation Algorithm.
04.09.2018 23

Evaluation Environment
▪ WikiTrace Dataset (9 GB)
▪ Address (6,708,723 distinct urls)
▪ Amazon Rating Dataset (3 GB)
▪ ProductId (9,874,210 distinct
items)
▪ ReviewerId (21,176,521 distinct
users)
▪ Ryzen 1600 (6C/12T)
▪ 16GB RAM
▪ Ubuntu 18.04
▪ OpenJDK 8
▪ JVM tuned for max 10 GB heap
▪ Flink 1.4.2
▪ Standalone mode
▪ Evaluation through python scripts
04.09.2018 24

Weitere ähnliche Inhalte

Was ist angesagt?

Just add Imagination

KNIMESlides

KNIME Data Science Learnathon: From Raw Data To Deployment - Paris - November...

KNIMESlides

In this section, you will program Amazon FreeRTOS to use the sensors on the CC3220SF-LAUNCHXL board. First, you will be able to get the temperature sensor readings from the board and send to AWS IoT Core. Next, you will get the accelerometer sensor readings and send to AWS IoT Core. You will then learn how to use Rules Engine to send an alert to your mobile phone when the board tilts beyond a 45-degree angle.

AWS IoT Day - Using AWS IoT Core

Amazon Web Services

Beginners guide to weather and climate data

Margriet Groenendijk

Open Source Story and what’s new in KNIME Software

KNIMESlides

You’re currently using the open source KNIME Analytics Platform, but looking for more functionality - especially for working across teams and business units? KNIME Server is the enterprise software for team based collaboration, automation, management, and deployment of data science workflows, data, and guided analytics. Non experts are given access to data science via KNIME Server WebPortal or can use REST APIs to integrate workflows as analytic services to applications, IoT, and systems. These slides are from a webinar where we introduce all KNIME Server features. It covers everything you need to manage your analytics at scale - deploying your workflows for sharing and collaboration, scheduling and automating tasks, templating and version control, as well as enterprise integration. it highlights the power of the REST API of KNIME Server, and the KNIME WebPortal - the ideal way for bringing data analytics to your non experts. View the webinar at: https://www.youtube.com/watch?v=qFV4P9-enZk

Sharing and Deploying Data Science with KNIME Server

KNIMESlides

From Raw Data to Deployment

KNIMESlides

Get Your Aircraft Spare Parts Inventory Management Off the Ground

PTC

Llorenc Muntaner, Senior Frontend Engineer at Onedot, shared what challenges AI startups face and what technical solutions they adopt to overcome them during Meetup in Barcelona. About Onedot Onedot's artificial intelligence (AI)-driven software helps businesses reduce manual work in product data management by 8x, speed up time-to-market by 90% and increase revenue up to 10%. Onedot makes this possible by radically improving data quality–automating the data integration, cleaning and categorisation process. Onedot features a user-friendly web interface designed for non-technical users. Unlike traditional rule-based ETL or scripting tools, Onedot is a ‘plug-and-play' system that continuously learns from business expert feedback and adapts to constantly changing data formats, structures, and nomenclatures. To learn more about Onedot, visit: https://www.onedot.com.

Stack for high performance in data-intensive operations

Onedot

REF compliance - what Jisc is doing

Repository Fringe

Weather is part of our everyday lives. Who doesn’t check the rain radar before heading out, or the weather forecast when planning a weekend away? But where does this data come from, and what is it made of? The answer is a mix of measurements, models and statistics, meaning that the use of weather and climate data can get complex very quickly. These slides provide a brief overview of the science behind weather and climate forecasts and provides you with the tools to get started with weather data - even if you aren't a meteorologist. Learn how to connect weather data to other data sources, how to visualize weather and climate data in an interactive weather dashboard embedded in a Python notebook, and other ways you can use weather data for yourself, from examples using weather APIs, maps, PixieDust and Machine Learning.

ODSC Europe: Weather and Climate Data: Not Just for Meteorologists

Margriet Groenendijk

Efficiency analyzer for wallblower iiot_quest

Uma Santharam

Introduction to the IBM Watson Data Platform

Margriet Groenendijk

Digitalt Partnerskap - Hur företag kan stärka varandra IBM och Tyréns har tillsammans paketerat en Akustiskt IoT lösning för monitorering av tågräls där vi kombinerat Tyréns djupa ingenjörsexpertis med IBM's Cloud och Analyticskompetens samt globala räckvidd för att erbjuda en first-of-a-kind lösning för att övervaka skicket på tågräls. Talare: Fredrik Alpen, Digital transformation facilitator, IBM Presentationen hölls på Watson Kista Summit 2018

Acoustic io t rail monitoring.pptx

IBM Sverige

GI2016 ppt charvat senslog api as tools for collection of big vgi data

IGN Vorstand

DSD-INT 2018 Impact of flooding on critical infrastructures - Mulder

Deltares

GeoApps is Capgemini’s Earth Observation (EO) application portal allowing researchers and industrial partners to request and visualise EO products generated from the latest satellite image data. COPA is an orchestration framework for containerised applications that enables the design and run of complex workflows. This session will explore how GeoApps will implement COPA for the orchestration of EO data workflows.

Smart orchestrator for pipeline processing chain applied to space data cwin18...

Capgemini

Ground Penetrating Radar (GPR) | Military | UXO

Sensors & Software Inc.

re:Inventから見えたDeepRacer Leagueで勝つための心構え

Takatoshi Ohmura

Integrating vert.x v2

Francisco Alves

Was ist angesagt? (20)

Just add Imagination

KNIME Data Science Learnathon: From Raw Data To Deployment - Paris - November...

AWS IoT Day - Using AWS IoT Core

Beginners guide to weather and climate data

Open Source Story and what’s new in KNIME Software

Sharing and Deploying Data Science with KNIME Server

From Raw Data to Deployment

Get Your Aircraft Spare Parts Inventory Management Off the Ground

Stack for high performance in data-intensive operations

REF compliance - what Jisc is doing

ODSC Europe: Weather and Climate Data: Not Just for Meteorologists

Efficiency analyzer for wallblower iiot_quest

Introduction to the IBM Watson Data Platform

Acoustic io t rail monitoring.pptx

GI2016 ppt charvat senslog api as tools for collection of big vgi data

DSD-INT 2018 Impact of flooding on critical infrastructures - Mulder

Smart orchestrator for pipeline processing chain applied to space data cwin18...

Ground Penetrating Radar (GPR) | Military | UXO

re:Inventから見えたDeepRacer Leagueで勝つための心構え

Integrating vert.x v2

Ähnlich wie Flink Forward Berlin 2018: Tobias Lindener - "Approximate standing queries on Stream Processing"

Network of Networks Share Group Spring Update

Lora Cecere

Integrating, exposing and managing distributed data with RESTful APIs and op...

3scale

State of enterprise data science

Yan Xu

Cips Meetup Auckland-Mirko-Kleiner-lean-agile-procurement-201904-1.0

Mirko Kleiner

K8s & cloud native past, present and future

Cheryl Hung

digitalization in oil & gas at an inflection point

Joseph Elkhoury

K8s & cloud native past, present and future

Cheryl Hung

TechEvent DWH Modernization

Trivadis

Testing microservices is challenging. Dividing a system into components naturally creates inter-service dependencies, and each service has its own performance and fault-tolerance characteristics that need to be validated during development, the QA process, and continually in production. Join Daniel Bryant to learn about the theory, techniques and practices needed to overcome this challenge. – Introduction to the challenges of testing distributed microservice systems – Learn how to isolate tests within a complex microservice ecosystem – Introduction to consumer-driven contract testing – Explore how API simulation can be used for testing work undertaken during DevOps, legacy system and high-volume load testing – Implementing fault-injection testing to validate nonfunctional requirements in development and QA – An introduction and discussion of the need for continually validating microservice systems running in production, both through observability and chaos engineering

Jax London 2018: "Testing Microservices from Development to Production"

Daniel Bryant

IoT Market in Canada

Nikhil Mittal

Artificial Intelligence and the Cognitive Revolution – the next frontier?

Level

Early adopter group and closing of webinar - João Fernandes (CERN)

Helix Nebula The Science Cloud

Produktdatenmanagement mit Neo4j

Neo4j

Neo4j and Technip Energies Italy executed an Innovation Lab Sprint. The goal of the laboratory has been to frame, design and prototype the use case identified by their colleagues of Planning, Equipment and Construction disciplines, by applying Knowledge Graph technology, as the way to connect the data to gain information and insights as an immediate value, that is: – capturing engineering deliverable milestone chain by gaining insights into a schedule – performing reasoning on information, evidence and data – extracting insights from data

Technip Energies Italy: Planning is a graph matter

Neo4j

Enabling Edge Analytics of IoT Data: The Case of LoRaWAN

Hong-Linh Truong

INTERFACE, by apidays - APIs: the next 10 years June 8, 9 & 10 2022 The Evolution of Data Movement Michel Tricot, Co-founder and CEO at Airbyte ------------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/ Deep dive into the API industry with our reports: https://www.apidays.global/industry-reports/ Subscribe to our global newsletter: https://apidays.typeform.com/to/i1MPEW

INTERFACE, by apidays - The Evolution of Data Movement.pdf

apidays

Repository Power: How Repositories can support Open Access Mandates (OR2015 O...

OpenAIRE

Easy SPARQLing for the Building Performance Professional

Martin Kaltenböck

Flink Meetup Septmeber 2017 2018

Christos Hadjinikolis

ODSC18, London, How to build high performing weighted XGBoost ML Model for Re...

Alok Singh

Ähnlich wie Flink Forward Berlin 2018: Tobias Lindener - "Approximate standing queries on Stream Processing" (20)

Network of Networks Share Group Spring Update

Integrating, exposing and managing distributed data with RESTful APIs and op...

State of enterprise data science

Cips Meetup Auckland-Mirko-Kleiner-lean-agile-procurement-201904-1.0

K8s & cloud native past, present and future

digitalization in oil & gas at an inflection point

K8s & cloud native past, present and future

TechEvent DWH Modernization

Jax London 2018: "Testing Microservices from Development to Production"

IoT Market in Canada

Artificial Intelligence and the Cognitive Revolution – the next frontier?

Early adopter group and closing of webinar - João Fernandes (CERN)

Produktdatenmanagement mit Neo4j

Technip Energies Italy: Planning is a graph matter

Enabling Edge Analytics of IoT Data: The Case of LoRaWAN

INTERFACE, by apidays - The Evolution of Data Movement.pdf

Repository Power: How Repositories can support Open Access Mandates (OR2015 O...

Easy SPARQLing for the Building Performance Professional

Flink Meetup Septmeber 2017 2018

ODSC18, London, How to build high performing weighted XGBoost ML Model for Re...

Mehr von Flink Forward

Apache Flink is a distributed stream processing framework that allows users to process and analyze data in real-time. At LinkedIn, we developed a fully managed stream processing platform on Flink running on K8s to power hundreds of stream processing pipelines in production. This platform is the backbone for other infra systems like Search, Espresso (internal document store) and feature management etc. We provide a rich authoring and testing environment which allows users to create, test, and deploy their streaming jobs in a self-serve fashion within minutes. Users can focus on their business logic, leaving the Flink platform to take care of management aspects such as split deployment, resource provisioning, auto-scaling, job monitoring, alerting, failure recovery and much more. In this talk, we will introduce the overall platform architecture, highlight the unique value propositions that it brings to stream processing at LinkedIn and share the experiences and lessons we have learned.

Building a fully managed stream processing platform on Flink at scale for Lin...

Flink Forward

Flink Forward San Francisco 2022. When running Flink jobs, skew is a common problem that results in wasted resources and limited scalability. In the past years, we have helped our customers and users solve various skew-related issues in their Flink jobs or clusters. In this talk, we will present the different types of skew that users often run into: data skew, key skew, event time skew, state skew, and scheduling skew, and discuss solutions for each of them. We hope this will serve as a guideline to help you reduce skew in your Flink environment. by Jun Qin & Karl Friedrich

Evening out the uneven: dealing with skew in Flink

Flink Forward

Flink Forward San Francisco 2022. To improve Amazon Alexa experiences and support machine learning inference at scale, we built an automated end-to-end solution for incremental model building or fine-tuning machine learning models through continuous learning, continual learning, and/or semi-supervised active learning. Customer privacy is our top concern at Alexa, and as we build solutions, we face unique challenges when operating at scale such as supporting multiple applications with tens of thousands of transactions per second with several dependencies including near-real time inference endpoints at low latencies. Apache Flink helps us transform and discover metrics in near-real time in our solution. In this talk, we will cover the challenges that we faced, how we scale the infrastructure to meet the needs of ML teams across Alexa, and go into how we enable specific use cases that use Apache Flink on Amazon Kinesis Data Analytics to improve Alexa experiences to delight our customers while preserving their privacy. by Aansh Shah

“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...

Flink Forward

Flink Forward San Francisco 2022. Probably everyone who has written stateful Apache Flink applications has used one of the fault-tolerant keyed state primitives ValueState, ListState, and MapState. With RocksDB, however, retrieving and updating items comes at an increased cost that you should be aware of. Sometimes, these may not be avoidable with the current API, e.g., for efficient event-time stream-sorting or streaming joins where you need to iterate one or two buffered streams in the right order. With FLIP-220, we are introducing a new state primitive: BinarySortedMultiMapState. This new form of state offers you to (a) efficiently store lists of values for a user-provided key, and (b) iterate keyed state in a well-defined sort order. Both features can be backed efficiently by RocksDB with a 2x performance improvement over the current workarounds. This talk will go into the details of the new API and its implementation, present how to use it in your application, and talk about the process of getting it into Flink. by Nico Kruber

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...

Flink Forward

Flink Forward San Francisco 2022. The Apache Flink Kubernetes Operator provides a consistent approach to manage Flink applications automatically, without any human interaction, by extending the Kubernetes API. Given the increasing adoption of Kubernetes based Flink deployments the community has been working on a Kubernetes native solution as part of Flink that can benefit from the rich experience of community members and ultimately make Flink easier to adopt. In this talk we give a technical introduction to the Flink Kubernetes Operator and demonstrate the core features and use-cases through in-depth examples." by Thomas Weise

Introducing the Apache Flink Kubernetes Operator

Flink Forward

Flink Forward San Francisco 2022. Resource Elasticity is a frequently requested feature in Apache Flink: Users want to be able to easily adjust their clusters to changing workloads for resource efficiency and cost saving reasons. In Flink 1.13, the initial implementation of Reactive Mode was introduced, later releases added more improvements to make the feature production ready. In this talk, we’ll explain scenarios to deploy Reactive Mode to various environments to achieve autoscaling and resource elasticity. We’ll discuss the constraints to consider when planning to use this feature, and also potential improvements from the Flink roadmap. For those interested in the internals of Flink, we’ll also briefly explain how the feature is implemented, and if time permits, conclude with a short demo. by Robert Metzger

Autoscaling Flink with Reactive Mode

Flink Forward

Flink Forward San Francisco 2022. Flink consumers read from Kafka as a scalable, high throughput, and low latency data source. However, there are challenges in scaling out data streams where migration and multiple Kafka clusters are required. Thus, we introduced a new Kafka source to read sharded data across multiple Kafka clusters in a way that conforms well with elastic, dynamic, and reliable infrastructure. In this presentation, we will present the source design and how the solution increases application availability while reducing maintenance toil. Furthermore, we will describe how we extended the existing KafkaSource to provide mechanisms to read logical streams located on multiple clusters, to dynamically adapt to infrastructure changes, and to perform transparent cluster migrations and failover. by Mason Chen

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Flink Forward

Flink Forward San Francisco 2022. Next time you want to integrate with a new destination for a demo, concept or production application, the Async Sink framework will bootstrap development, allowing you to move quickly without compromise. In Flink 1.15 we introduced the Async Sink base (FLIP-171), with the goal to encapsulate common logic and allow developers to focus on the key integration code. The new framework handles things like request batching, buffering records, applying backpressure, retry strategies, and at least once semantics. It allows you to focus on your business logic, rather than spending time integrating with your downstream consumers. During the session we will dive deep into the internals to uncover how it works, why it was designed this way, and how to use it. We will code up a new sink from scratch and demonstrate how to quickly push data to a destination. At the end of this talk you will be ready to start implementing your own Flink sink using the new Async Sink framework. by Steffen Hausmann & Danny Cranmer

One sink to rule them all: Introducing the new Async Sink

Flink Forward

Flink Forward San Francisco 2022. In normal situations, the default Kafka consumer and producer configuration options work well. But we all know life is not all roses and rainbows and in this session we’ll explore a few knobs that can save the day in atypical scenarios. First, we'll take a detailed look at the parameters available when reading from Kafka. We’ll inspect the params helping us to spot quickly an application lock or crash, the ones that can significantly improve the performance and the ones to touch with gloves since they could cause more harm than benefit. Moreover we’ll explore the partitioning options and discuss when diverging from the default strategy is needed. Next, we’ll discuss the Kafka Sink. After browsing the available options we'll then dive deep into understanding how to approach use cases like sinking enormous records, managing spikes, and handling small but frequent updates.. If you want to understand how to make your application survive when the sky is dark, this session is for you! by Olena Babenko

Tuning Apache Kafka Connectors for Flink.pptx

Flink Forward

Flink Forward San Francisco 2022. Pinterest is a visual discovery engine that serves over 433MM users. Stream processing allows us to unlock value from realtime data for pinners. At Pinterest, we adopt Flink as the unified streaming processing engine. In this talk, we will share our journey in building a stream processing platform with Flink and how we onboarding critical use cases to the platform. Pinterest has supported 90+near realtime streaming applications. We will cover the problem statement, how we evaluate potential solutions and our decision to build the framework. by Rainie Li & Kanchi Masalia

Flink powered stream processing platform at Pinterest

Flink Forward

Flink Forward San Francisco 2022. This talk will take you on the long journey of Apache Flink into the cloud-native era. It started all the way from where Hadoop and YARN were the standard way of deploying and operating data applications. We're going to deep dive into the cloud-native set of principles and how they map to the Apache Flink internals and recent improvements. We'll cover fast checkpointing, fault tolerance, resource elasticity, minimal infrastructure dependencies, industry-standard tooling, ease of deployment and declarative APIs. After this talk you'll get a broader understanding of the operational requirements for a modern streaming application and where the current limits are. by David Moravek

Apache Flink in the Cloud-Native Era

Flink Forward

Flinkn Forward San Francisco 2022. In this talk, we will cover various topics around performance issues that can arise when running a Flink job and how to troubleshoot them. We’ll start with the basics, like understanding what the job is doing and what backpressure is. Next, we will see how to identify bottlenecks and which tools or metrics can be helpful in the process. Finally, we will also discuss potential performance issues during the checkpointing or recovery process, as well as and some tips and Flink features that can speed up checkpointing and recovery times. by Piotr Nowojski

Where is my bottleneck? Performance troubleshooting in Flink

Flink Forward

Flink Forward San Francisco 2022. Running natively on Kubernetes, using the new Apache Flink Kubernetes Operator is a great way to deploy and manage Flink application and session deployments. In this presentation, we provide: - A brief overview of Kubernetes operators and their benefits. - Introduce the five levels of the operator maturity model. - Introduce the newly released Apache Flink Kubernetes Operator and FlinkDeployment CRs - Dockerfile modifications you can make to swap out UBI images and Java of the underlying Flink Operator container - Enhancements we're making in: - Versioning/Upgradeability/Stability - Security - Demo of the Apache Flink Operator in-action, with a technical preview of an upcoming product using the Flink Kubernetes Operator. - Lessons learned - Q&A by James Busche & Ted Chang

Using the New Apache Flink Kubernetes Operator in a Production Deployment

Flink Forward

Flink Forward San Francisco 2022. The Table API is one of the most actively developed components of Flink in recent time. Inspired by databases and SQL, it encapsulates concepts many developers are familiar with. It can be used with both bounded and unbounded streams in a unified way. But from afar it can be difficult to keep track of what this API is capable of and how it relates to Flink's other APIs. In this talk, we will explore the current state of Table API. We will show how it can be used as a batch processor, a changelog processor, or a streaming ETL tool with many built-in functions and operators for deduplicating, joining, and aggregating data. By comparing it to the DataStream API we will highlight differences and elaborate on when to use which API. We will demonstrate hybrid pipelines in which both APIs interact with one another and contribute their unique strengths. Finally, we will take a look at some of the most recent additions as a first step to stateful upgrades. by David Andreson

The Current State of Table API in 2022

Flink Forward

Flink Forward San Francisco 2022. Based on the new Flink-Pulsar connector, we implemented Flink's TableAPI and Catalog to help users to interact with the Pulsar cluster via Flink SQL easily. We would like to go through the design and implementation of the SQL connector in the following aspects: 1. Two different modes of use Pulsar as a metadata store 2. Data format transformation and management 3. SQL semantics support within Pulsar context by Sijie Guo & Neng Lu

Flink SQL on Pulsar made easy

Flink Forward

Flink Forward San Francisco 2022. At Bloomberg, we deal with high volumes of real-time market data. Our clients expect to be notified of any anomalies in this market data, which may indicate volatile movements in the markets, notable trades, forthcoming events, or system failures. The parameters for these alerts are always evolving and our clients can update them dynamically. In this talk, we'll cover how we utilized the open source Apache Flink and Siddhi SQL projects to build a distributed, scalable, low-latency and dynamic rule-based, real-time alerting system to solve our clients' needs. We'll also cover the lessons we learned along our journey. by Ajay Vyasapeetam & Madhuri Jain

Dynamic Rule-based Real-time Market Data Alerts

Flink Forward

Flink Forward San Francisco 2022. At Stripe we have created a complete end to end exactly-once processing pipeline to process financial data at scale, by combining the exactly-once power from Flink, Kafka, and Pinot together. The pipeline provides exactly-once guarantee, end-to-end latency within a minute, deduplication against hundreds of billions of keys, and sub-second query latency against the whole dataset with trillion level rows. In this session we will discuss the technical challenges of designing, optimizing, and operating the whole pipeline, including Flink, Kafka, and Pinot. We will also share our lessons learned and the benefits gained from exactly-once processing. by Xiang Zhang & Pratyush Sharma & Xiaoman Dong

Exactly-Once Financial Data Processing at Scale with Flink and Pinot

Flink Forward

Flink Forward San Francisco 2022. What if my data is already in order? Stream Processing has given us an elegant and powerful solution for running analytic queries and logic over high volumes of continuously arriving data. However, in both Apache Flink and Apache Beam, the notion of time-ordering is baked in at a very low level, making it difficult to express computations that are interested in a semantic-, rather than time-ordering of the data. In financial services, what often matters the most about the data moving between systems is not when the data was created, but in what order, to the extent that many institutions engineer a global sequencing over all data entering and produced by their systems to achieve complete determinism. How, then, can financial institutions and others best employ Stream Processing on streams of data that are already ordered? I will cover various techniques that can make this work, as well as seek input from the community on how Flink might be improved to better support these use-cases. by Patrick Lucas

Processing Semantically-Ordered Streams in Financial Services

Flink Forward

Flink Forward San Francisco 2022. In modern data platform architectures, stream processing engines such as Apache Flink are used to ingest continuous streams of data into data lakes such as Apache Iceberg. Streaming ingestion to iceberg tables can suffer by two problems (1) small files problem that can hurt read performance (2) poor data clustering that can make file pruning less effective. To address those two problems, we propose adding a shuffling stage to the Flink Iceberg streaming writer. The shuffling stage can intelligently group data via bin packing or range partition. This can reduce the number of concurrent files that every task writes. It can also improve data clustering. In this talk, we will explain the motivations in details and dive into the design of the shuffling stage. We will also share the evaluation results that demonstrate the effectiveness of smart shuffling. by Gang Ye & Steven Wu

Tame the small files problem and optimize data layout for streaming ingestion...

Flink Forward

Flink Forward San Francisco 2022. Goldman Sachs's Data Lake platform serves as the firm's centralized data platform, ingesting 140K (and growing!) batches per day of Datasets of varying shape and size. Powered by Flink and using metadata configured by platform users, ingestion applications are generated dynamically at runtime to extract, transform, and load data into centralized storage where it is then exported to warehousing solutions such as Sybase IQ, Snowflake, and Amazon Redshift. Data Latency is one of many key considerations as producers and consumers have their own commitments to satisfy. Consumers range from people/systems issuing queries, to applications using engines like Spark, Hive, and Presto to transform data into refined Datasets. Apache Iceberg allows our applications to not only benefit from consistency guarantees important when running on eventually consistent storage like S3, but also allows us the opportunity to improve our batch processing patterns with its scalability-focused features. by Andreas Hailu

Batch Processing at Scale with Flink & Iceberg

Flink Forward

Mehr von Flink Forward (20)

Building a fully managed stream processing platform on Flink at scale for Lin...

Evening out the uneven: dealing with skew in Flink

“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...

Introducing the Apache Flink Kubernetes Operator

Autoscaling Flink with Reactive Mode

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

One sink to rule them all: Introducing the new Async Sink

Tuning Apache Kafka Connectors for Flink.pptx

Flink powered stream processing platform at Pinterest

Apache Flink in the Cloud-Native Era

Where is my bottleneck? Performance troubleshooting in Flink

Using the New Apache Flink Kubernetes Operator in a Production Deployment

The Current State of Table API in 2022

Flink SQL on Pulsar made easy

Dynamic Rule-based Real-time Market Data Alerts

Exactly-Once Financial Data Processing at Scale with Flink and Pinot

Processing Semantically-Ordered Streams in Financial Services

Tame the small files problem and optimize data layout for streaming ingestion...

Batch Processing at Scale with Flink & Iceberg

Kürzlich hochgeladen

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

Advantages of Hiring UIUX Design Service Providers for Your Business

Pixlogix Infotech

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Miguel Araújo

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

🐬 The future of MySQL is Postgres 🐘

RTylerCroy

Tata AIG General Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

Scaling API-first – The story of a global engineering organization

Radu Cotescu

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

[2024]Digital Global Overview Report 2024 Meltwater.pdf

hans926745

What are drone anti-jamming systems? The drone anti-jamming systems and anti-spoof technology protect against interference, jamming, and spoofing of the UAVs. To protect their security, countries are beginning to research drone anti-jamming systems, also known as drone strike weapons. The anti-jam and anti-spoof technology protects against interference, jamming and spoofing. A drone strike weapon is a drone attack weapon that can attack and destroy enemy drones. So what is so unique about this amazing system?

What Are The Drone Anti-jamming Systems Technology?

Antenna Manufacturer Coco

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

Kürzlich hochgeladen (20)

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Strategies for Landing an Oracle DBA Job as a Fresher

Partners Life - Insurer Innovation Award 2024

Advantages of Hiring UIUX Design Service Providers for Your Business

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Powerful Google developer tools for immediate impact! (2023-24 C)

🐬 The future of MySQL is Postgres 🐘

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Apidays New York 2024 - The value of a flexible API Management solution for O...

Scaling API-first – The story of a global engineering organization

Automating Google Workspace (GWS) & more with Apps Script

Boost PC performance: How more available memory can improve productivity

Boost Fertility New Invention Ups Success Rates.pdf

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

[2024]Digital Global Overview Report 2024 Meltwater.pdf

What Are The Drone Anti-jamming Systems Technology?

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Data Cloud, More than a CDP by Matt Robison

Flink Forward Berlin 2018: Tobias Lindener - "Approximate standing queries on Stream Processing"

1. This project has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No. 688191. @tobiaslindener lindener@kth.se Approximate Standing Queries on Apache Flink

2. Overview 1. Introduction 2. Background 3. Design & Implementation 4. Results 04.09.2018 2

3. It is better to use a crude approximation and know the truth, plus or minus 10 percent, than demand an exact solution and know nothing at all In Arthur Bloch, The Complete Murphy's Law: A Definitive Collection (1991), 126 04.09.2018 3

4. Unbounded Stream 04.09.2018 4 ID_A P_A Key Count A 1 Time Infinite Growth

5. Unbounded Stream 04.09.2018 5 ID_A P_A ID_B P_A Key Count A 1 B 1 Time Infinite Growth

6. Unbounded Stream 04.09.2018 6 ID_A P_A ID_C P_A ID_B P_A Key Count A 1 B 1 C 1 Time Infinite Growth

7. Unbounded Stream 04.09.2018 7 ID_A P_A ID_C P_A ID_D P_A ID_B P_A Key Count A 1 B 1 C 1 D 1 Time Infinite Growth

8. Unbounded Stream 04.09.2018 8 ID_A P_A ID_C P_A ID_D P_A ID_C P_A ID_B P_A Key Count A 1 B 1 C 2 D 1 Time Infinite Growth

9. Unbounded Stream 04.09.2018 9 ID_A P_A ID_C P_A ID_D P_A ID_C P_A ID_B P_A Key Count A 1 B 1 C 2 D 1 Time Infinite Growth

10. Approximation Algorithms Use-cases • Identify heavy hitters (Count) • Cardinality Estimation (Count Distinct) Algorithms • Frequent Item Estimation • HyperLogLog • Quantile Estimation 04.09.2018 10

11. Processing 04.09.2018 11 Flink Architecture (Apache Software Foundation, 2018) Approximate Queries

12. Processing 04.09.2018 12 Flink Architecture (Apache Software Foundation, 2018) Approximate Queries

13. Approximate Query API (1) 04.09.2018 13

14. Approximate Query API (2) 04.09.2018 14

15. Sketch Capacity 04.09.2018 15

16. Estimate Deviation - Frequent Items WIKITRACE DATASET TOP 6000 URL AMAZON DATASET TOP 1000 REVIEWER 04.09.2018 16

17. Estimate Deviation - HyperLogLog 04.09.2018 17

18. Native Implementation • Potentially increased efficiency • Reduced overhead • Stateful processing Integration with Table API • CQL with Approximate Queries • Support for Count Distinct Queryable State • Reduced data handling Future Work 04.09.2018 18

19. Queryable State 04.09.2018 19 Stream Elements Query Results Sketch Function Time

20. Queryable State 04.09.2018 20 Stream Elements Query Results Sketch Function Time Query Results REST API

21. Conclusion CHALLENGES • Efficient Grouping (HLL) • Stateful Native Implementation • Skewed Datasets LEARNINGS • Importance of Data Distribution • Performance Advantages • Algorithm Parameters 04.09.2018 21

22. Team 04.09.2018 22 Tobias Lindener Consultant @ Netlight Theodore Vasiloudis Researcher @ RISE Paris Carbone PhD candidate @ KTH Slides bit.ly/2LULyoZ

23. References & Links • https://github.com/tlindener/ApproximateQueries • https://datasketches.github.io/ • Daniel Anderson, Pryce Bevan, Kevin Lang, Edo Liberty, Lee Rhodes, Justin Thaler. A High-Performance Algorithm for Identifying Frequent Items in Data Streams. • Kevin Lang, Back to the Future: an Even More Nearly Optimal Cardinality Estimation Algorithm. 04.09.2018 23

24. Evaluation Environment ▪ WikiTrace Dataset (9 GB) ▪ Address (6,708,723 distinct urls) ▪ Amazon Rating Dataset (3 GB) ▪ ProductId (9,874,210 distinct items) ▪ ReviewerId (21,176,521 distinct users) ▪ Ryzen 1600 (6C/12T) ▪ 16GB RAM ▪ Ubuntu 18.04 ▪ OpenJDK 8 ▪ JVM tuned for max 10 GB heap ▪ Flink 1.4.2 ▪ Standalone mode ▪ Evaluation through python scripts 04.09.2018 24

Flink Forward Berlin 2018: Tobias Lindener - "Approximate standing queries on Stream Processing"

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Flink Forward Berlin 2018: Tobias Lindener - "Approximate standing queries on Stream Processing"

Ähnlich wie Flink Forward Berlin 2018: Tobias Lindener - "Approximate standing queries on Stream Processing" (20)

Mehr von Flink Forward

Mehr von Flink Forward (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Flink Forward Berlin 2018: Tobias Lindener - "Approximate standing queries on Stream Processing"