Real-Time Attribution with Structured Streaming and Databricks Delta with Caryl Yuhas

•

2 gefällt mir•584 views

Companies doing any kind of advertising typically have an attribution process that joins users’ conversions with the impressions that they were served or that they clicked on. The standard workflow is typically a batch job that runs every few hours or once a day. However, as technology gets more sophisticated, advertisers are looking for more real-time reporting and results. This talk presents an example of a foundational architecture for near real-time attribution and advanced analytics against real-time impression and conversion data using Structured Streaming and Databricks Delta.

Daten & Analysen

Caryl Yuhas, Databricks
Real-Time Attribution with
Structured Streaming and
Databricks Delta
#ExpSAIS13

Introduction
2#ExpSAIS13
• Goal:
Provide tools and information
that can help you build more
real-time / lower latency
attribution pipelines
• Crawl, Walk, Run: Pull Model
Carylpreviously MediaMath / SE / PM
for Attribution, SA for Databricks

Getting Started
3#ExpSAIS13
• What is Attribution?
Image Source: www.mediamath.com

Introduction
What is Databricks Delta?
Delta is a data management capability that
brings data reliability and performance
optimizations to the cloud data lake.
4#ExpSAIS13

Stream-to-Sink BEFORE
5#ExpSAIS13
Reprocessing
Data Lake
λ-arch
λ-arch
Streaming
Analytics
Reporting
Events
Validation
λ-arch
Validation
Reprocessing
Compaction
Partitioned
Compact
Small Files
Scheduled to
Avoid Compaction
1
2
3
1
1
3
4
4
4
2

Stream-to-Sink AFTER
6#ExpSAIS13
Reprocessing
λ-arch
Streaming
Analytics
Reporting
Events
Validation
λ-arch
Validation
Reprocessing
Compaction
Partitioned
1
2
Compact
Small Files
3
1
4
4
2
DELTA
Optimize
3
ZOrder
3

Attribution in Practice
7
impressions conversionsJOIN
#ExpSAIS13
attributed impressions

Attribution Challenges
Scale
• Often dealing with millions to billions of data
points per attribution window
Complexity
• Simple, last-click model is still common
• MTA and more sophisticated attribution on rise
8#ExpSAIS13

High Level Attribution Pipeline
9#ExpSAIS13

Attribution in Practice
10
impressions conversionsJOIN
#ExpSAIS13
attributed impressions

Data Architecture
11#ExpSAIS13
impression stream
conversion stream conversions table
impressions table
attributed table
last touch
attributed table
weighted
attribution views
(filters, logic, etc.)

System Architecture
12#ExpSAIS13
STRUCTURED
STREAMING
Amazon
Kinesis

Unification of Streaming + Batch
DEMO
13#ExpSAIS13

• How can we optimize performance?
• Levers:
– Delta Tools
• Optimize
• ZOrder
• Caching
• Data Skipping
– Join on Stream
– Cluster Size
Managing Performance
14#ExpSAIS13

Handling Complexity
• Flexibility with Complex Logic
– Forking streams
– Logic on query vs. in-stream
• Late or Corrected Data
– Upserts
– Views automatically update when raw data changed
15#ExpSAIS13

Conclusion
• Unification of Batch & Streaming
• Easy APIs for Managing Performance
• Flexible and Scalable Analytics on Near
Real-Time Data
16#ExpSAIS13

Weitere ähnliche Inhalte

Was ist angesagt?

Omid: A Transactional Framework for HBase

DataWorks Summit/Hadoop Summit

At Strava we have extensively leveraged Apache Spark to explore our data of over a billion activities, from tens of millions of athletes. This talk will be a survey of the more unique and exciting applications: A Global Heatmap gives a ~2 meter resolution density map of one billion runs, rides, and other activities consisting of three trillion GPS points from 17 billion miles of exercise data. The heatmap was rewritten from a non-scalable system into a highly scalable Spark job enabling great gains in speed, cost, and quality. Locally sensitive hashing for GPS traces was used to efficiently cluster 1 billion activities. Additional processes categorize and extract data from each cluster, such as names and statistics. Clustering gives an automated process to extract worldwide geographical patterns of athletes. Applications include route discovery, recommendation systems, and detection of events and races. A coarse spatiotemporal index of all activity data is stored in Apache Cassandra. Spark streaming jobs maintain this index and compute all space-time intersections (“flybys”) of activities in this index. Intersecting activity pairs are then checked for spatiotemporal correlation, indicated by connected components in the graph of highly correlated pairs form “Group Activities”, creating a social graph of shared activities and workout partners. Data from several hundred thousand runners was used to build an improved model of the relationship between running difficulty and elevation gradient (Grade Adjusted Pace).

Strava Labs: Exploring a Billion Activity Dataset from Athletes with Apache S...

Databricks

Stateful processing is one of the most challenging aspects of distributed, fault-tolerant stream processing. The DataFrame APIs in Structured Streaming make it easy for the developer to express their stateful logic, either implicitly (streaming aggregations) or explicitly (mapGroupsWithState). However, there are a number of moving parts under the hood which makes all the magic possible. In this talk, I will dive deep into different stateful operations (streaming aggregations, deduplication and joins) and how they work under the hood in the Structured Streaming engine.

Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...

Databricks

Leveraging Kafka for Big Data in Real Time Bidding, Analytics, ML & Campaign ...

Helena Edelson

The RAPIDS Accelerator for Apache Spark is a plugin that enables the power of GPUs to be leveraged in Spark DataFrame and SQL queries, improving the performance of ETL pipelines. User-defined functions (UDFs) in the query appear as opaque transforms and can prevent the RAPIDS Accelerator from processing some query operations on the GPU. This presentation discusses how users can leverage the RAPIDS Accelerator UDF Compiler to automatically translate some simple UDFs to equivalent Catalyst operations that are processed on the GPU. The presentation also covers how users can provide a GPU version of Scala, Java, or Hive UDFs for maximum control and performance. Sample UDFs for each case will be shown along with how the query plans are impacted when the UDFs are processed on the GPU.

Speed up UDFs with GPUs using the RAPIDS Accelerator

Databricks

Parallelization of Structured Streaming Jobs Using Delta Lake

Databricks

My new industry acronym: PSTL the Parallelized Streaming Transformation Loader (Pron. PiSToL) is an architecture for highly scalable and reliable, data ingestion pipelines While there is guidance on using; Apache Kafka™ for Streaming (or non-Streaming), and Apache Spark™ for Transformations, and Loading data (e.g., COPY) into an HP-Vertica™ columnar Data Warehouse, there is very little prescriptive guidance on how to truly parallelize a unified data pipeline - until now.

HPBigData2015 PSTL kafka spark vertica

Jack Gudenkauf

As Real Time Analytics built on Spark Streaming framework gains increasing adoption and deployment in the data center, the need for hardware accelerators to scale for performance and offer lower latency for certain applications is becoming increasingly apparent. Recently reconfigurable accelerators based on FPGAs (Field Programmable Gate Arrays) have been proposed to accelerate the analytics workloads with support for low-latency streaming. This is becoming a feasible and attractive option now as major CSPs (Cloud Service Providers) announce deployment of FPGAs- as-a-Service (FPGAaaS). However there are certain challenges in using FPGA accelerators in the cloud that first must be overcome: (1) Development of FPGA accelerators is challenging because of the lack of mature high level language support and standard accelerator interfaces; (2) Using hardware accelerators from high level application frameworks like Spark require special consideration for the efficient transfer of data to and from the application; and (3) Managing the FPGA accelerators for sharing across different applications is limited by the lack of a common resource sharing paradigm. In this session, we’ll present a runtime framework that addresses these challenges. The framework will present highly optimized FPGA based accelerated functions as a service (AFaaS) to the application. The framework will demonstrate seamless integration of accelerators into existing Big Data Frameworks using Java, Scala, Pyton or other high level language APIs. The framework will also support the transfer of streaming data to/from the FPGA to the application and the sharing of accelerator libraries across different applications. Key takeaways: This paper demonstrates the feasibility of accelerating Real Time Analytics applications using Spark Streaming with accelerated functions based on FPGAaaS to deliver higher performance efficiencies at lower latencies. The paper will provide performance data for Spark Streaming applications using our solution compared to current native software implementations.

Accelerating Real Time Analytics with Spark Streaming and FPGAaaS with Prabha...

Databricks

Building the Petcare Data Platform using Delta Lake and 'Kyte': Our Spark ETL...

Databricks

Stateful processing is one of the most challenging aspects of distributed, fault-tolerant stream processing. The DataFrame APIs in Structured Streaming make it very easy for the developer to express their stateful logic, either implicitly (streaming aggregations) or explicitly (mapGroupsWithState). However, there are a number of moving parts under the hood which makes all the magic possible. In this talk, I am going to dive deeper into how stateful processing works in Structured Streaming. In particular, I am going to discuss the following. – Different stateful operations in Structured Streaming – How state data is stored in a distributed, fault-tolerant manner using State Stores – How you can write custom State Stores for saving state to external storage systems.

Deep dive into stateful stream processing in structured streaming by Tathaga...

Databricks

Building a Streaming Microservice Architecture: with Apache Spark Structured ...

Databricks

Big Data Tools in AWS

Shu-Jeng Hsieh

Streaming Big Data & Analytics For Scale

Helena Edelson

Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)

Spark Summit

Change Data Capture CDC is a typical use case in Real-Time Data Warehousing. It tracks the data change log -binlog- of a relational database [OLTP], and replay these change log timely to an external storage to do Real-Time OLAP, such as delta/kudu. To implement a robust CDC streaming pipeline, lots of factors should be concerned, such as how to ensure data accuracy , how to process OLTP source schema changed, whether it is easy to build for variety databases with less code.

Simplify CDC Pipeline with Spark Streaming SQL and Delta Lake

Databricks

Continuing with the objectives to make Spark faster, easier, and smarter, Apache Spark 3.1 extends its scope with more than 1500 resolved JIRAs. We will talk about the exciting new developments in the Apache Spark 3.1 as well as some other major initiatives that are coming in the future. In this talk, we want to share with the community many of the more important changes with the examples and demos. The following features are covered: the SQL features for ANSI SQL compliance, new streaming features, and Python usability improvements, the performance enhancements and new tuning tricks in query compiler.

Deep Dive into the New Features of Apache Spark 3.1

Databricks

When you run an Apache Spark application on a large cluster, you want to make sure you’re getting the most from that cluster. Any CPU or memory left on the table represents either a waste of money or a lost opportunity to speed up your Spark jobs. What many people don’t realize is how sensitive Spark cluster utilization is to the resource manager. Resource managers decide how to allocate cluster resources among the many users and applications contending for them. In this deep dive session, we will discuss how Spark integrates with two common open source resource managers, YARN and Mesos, as well as a new commercial product called IBM Spectrum Conductor with Spark. You will learn how resource managers arbitrate resources in multi-user/multi-tenant Spark clusters, and how this affects application performance. You will come away with new techniques for tuning Spark resource management to optimize goals like speed and fairness. The session will include a demo of a new open source benchmark designed to help analyse Spark multi-user/multi-tenant performance. The benchmark uses Spark SQL and machine learning jobs to load the cluster, and can be used during a pre-production cycle to tune Spark and resource manager configurations.

Deep Dive Into Apache Spark Multi-User Performance Michael Feiman, Mikhail Ge...

Databricks

Lambda architecture

Szilveszter Molnár

Google cloud Dataflow & Apache Flink

Iván Fernández Perea

La collecte de données au sein d'un DataLake sans impacter les systèmes opérationnels est un challenge pour de nombreuses entreprises. Lors du meetup Paris Data Engineers du 26 mars 2019, Dimitri Capitaine nous a présenté Data Collector qui est un outil de Change Data Capture (CDC) développé en interne chez OVH. Data Collector est capable d'assurer une réplication fiable et performante des bases de données jusqu'au DataLake. Hugo Larcher nous a alors présenté un cas d'utilisation autour de l'exploitation de données aéronautiques avec une touche d'IoT et de DataViz.

Change Data Capture with Data Collector @OVH

Paris Data Engineers !

Was ist angesagt? (20)

Omid: A Transactional Framework for HBase

Strava Labs: Exploring a Billion Activity Dataset from Athletes with Apache S...

Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...

Leveraging Kafka for Big Data in Real Time Bidding, Analytics, ML & Campaign ...

Speed up UDFs with GPUs using the RAPIDS Accelerator

Parallelization of Structured Streaming Jobs Using Delta Lake

HPBigData2015 PSTL kafka spark vertica

Accelerating Real Time Analytics with Spark Streaming and FPGAaaS with Prabha...

Building the Petcare Data Platform using Delta Lake and 'Kyte': Our Spark ETL...

Deep dive into stateful stream processing in structured streaming by Tathaga...

Building a Streaming Microservice Architecture: with Apache Spark Structured ...

Big Data Tools in AWS

Streaming Big Data & Analytics For Scale

Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)

Simplify CDC Pipeline with Spark Streaming SQL and Delta Lake

Deep Dive into the New Features of Apache Spark 3.1

Deep Dive Into Apache Spark Multi-User Performance Michael Feiman, Mikhail Ge...

Lambda architecture

Google cloud Dataflow & Apache Flink

Change Data Capture with Data Collector @OVH

Ähnlich wie Real-Time Attribution with Structured Streaming and Databricks Delta with Caryl Yuhas

Most data practitioners grapple with data reliability issues—it’s the bane of their existence. Data engineers, in particular, strive to design, deploy, and serve reliable data in a performant manner so that their organizations can make the most of their valuable corporate data assets. Delta Lake is an open-source storage layer that brings ACID transactions to Apache Spark™ and big data workloads. Built on open standards, Delta Lake employs co-designed compute and storage and is compatible with Spark API’s. It powers high data reliability and query performance to support big data use cases, from batch and streaming ingests, fast interactive queries to machine learning. In this tutorial we will discuss the requirements of modern data engineering, the challenges data engineers face when it comes to data reliability and performance and how Delta Lake can help. Through presentation, code examples and notebooks, we will explain these challenges and the use of Delta Lake to address them. You will walk away with an understanding of how you can apply this innovation to your data architecture and the benefits you can gain. This tutorial will be both instructor-led and hands-on interactive session. Instructions on how to get tutorial materials will be covered in class. What you’ll learn: Understand the key data reliability challenges How Delta Lake brings reliability to data lakes at scale Understand how Delta Lake fits within an Apache Spark™ environment How to use Delta Lake to realize data reliability improvements Prerequisites A fully-charged laptop (8-16GB memory) with Chrome or Firefox Pre-register for Databricks Community Edition

Building Reliable Data Lakes at Scale with Delta Lake

Databricks

The Briefing Room with Dr. Robin Bloor and Think Big, a Teradata Company Live Webcast April 7, 2015 Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=4114b87441ab7b2b4c52f6b24776e5a1 The more things change in Big Data, the more they stay the same. Indeed, there are many similarities between a Hadoop-based Data Lake and today’s modern Data Warehouse. Regardless of platform, information workers must still be able to turn their assets into action quickly, without taking a hit on governance or downstream performance. Register for this episode of The Briefing Room to hear veteran Analyst Dr. Robin Bloor as he explains the challenges facing organizations who endeavor on Big Data projects. He’ll be briefed by Rick Stellwagen of Think Big, a Teradata Company, who will outline his company’s approach to handling Big Data implementations. Rick will discuss the role of the data lake, and how timely response of queries is critical for reporting and analysis. Visit InsideAnalysis.com for more information.

The Great Lakes: How to Approach a Big Data Implementation

Inside Analysis

The Briefing Room with Dr. Robin Bloor and Teradata RainStor Live Webcast October 13, 2015 Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=012bb2c290097165911872b1f241531d Hadoop data lakes are emerging as peers to corporate data warehouses. However, successful data management solutions require a fusion of all relevant data, new and old, which has proven challenging for many companies. With a data lake that’s been optimized for fast queries, solid governance and lifecycle management, users can take data management to a whole new level. Register for this episode of The Briefing Room to learn from veteran Analyst Dr. Robin Bloor as he discusses the relevance of data lakes in today’s information landscape. He’ll be briefed by Mark Cusack of Teradata, who will explain how his company’s archiving solution has developed into a storage point for raw data. He’ll show how the proven compression, scalability and governance of Teradata RainStor combined with Hadoop can enable an optimized data lake that serves as both reservoir for historical data and as a "system of record” for the enterprise. Visit InsideAnalysis.com for more information.

First in Class: Optimizing the Data Lake for Tighter Integration

Inside Analysis

Comcast is one of the leading providers of communications, entertainment, and cable products and services. At the heart of it is Comcast RDK providing the backbone of telemetry to the industry. RDK (Reference Design Kit) is pre-bundled opensource firmware for a complete home platform covering video, broadband and IoT devices. RDK team at Comcast analyzes petabytes of data, collected every 15 minutes from 70 million devices (video and broadband and IoT devices) installed in customer homes. They run ETL and aggregation pipelines and publish analytical dashboards on a daily basis to reduce customer calls and firmware rollout. The analysis is also used to calculate WIFI happiness index which is a critical KPI for Comcast customer experience. In addition to this, RDK team also does release tracking by analyzing the RDK firmware quality. SQL Analytics allows customers to operate a lakehouse architecture that provides data warehousing performance at data lake economics for up to 4x better price/performance for SQL workloads than traditional cloud data warehouses. We present the results of the “Test and Learn” with SQL Analytics and the delta engine that we worked in partnership with the Databricks team. We present a quick demo introducing the SQL native interface, the challenges we faced with migration, The results of the execution and our journey of productionizing this at scale.

SQL Analytics Powering Telemetry Analysis at Comcast

Databricks

C19013010 the tutorial to build shared ai services session 2

Bill Liu

Meetup: Streaming Data Pipeline Development In this interactive session, Tim will lead participants through how to best build streaming data pipelines. He will cover how to build applications from some common use cases and highlight tips, tricks, best practices and patterns. He will show how to build the easy way and then dive deep into the underlying open source technologies including Apache NiFi, Apache Flink, Apache Kafka and Apache Iceberg. If you wish to follow along, please download open source projects beforehand. You can also download this helpful streaming platform: https://docs.cloudera.com/csp-ce/latest/installation/topics/csp-ce-installing-ce.html All source code and slides will be shared for those interested in building their own FLaNK Apps. https://www.flankstack.dev/ You can join the meeting virtually here: https://cloudera.zoom.us/j/91603330726 Speaker - Tim Spann Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Pulsar, Apache Kafka, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science.

Meetup: Streaming Data Pipeline Development

Timothy Spann

Whether to take data ingestion cycles off the ETL tool and the data warehouse or to facilitate competitive Data Science and building algorithms in the organization, the data lake – a place for unmodeled and vast data – will be provisioned widely in 2020. Though it doesn’t have to be complicated, the data lake has a few key design points that are critical, and it does need to follow some principles for success. Avoid building the data swamp, but not the data lake! The tool ecosystem is building up around the data lake and soon many will have a robust lake and data warehouse. We will discuss policy to keep them straight, send data to its best platform, and keep users’ confidence up in their data platforms. Data lakes will be built in cloud object storage. We’ll discuss the options there as well. Get this data point for your data lake journey.

ADV Slides: When and How Data Lakes Fit into a Modern Data Architecture

DATAVERSITY

2018 02-08-what's-new-in-apache-spark-2.3

Chester Chen

Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard

Paris Data Engineers !

As organizations modernize their data and analytics platforms, the data lake concept has gained momentum as a shared enterprise resource for supporting insights across multiple lines of business. The perception is that data lakes are vast, slow-moving bodies of data, but innovations like Apache Kafka for streaming-first architectures put real-time data flows at the forefront. Combining real-time alerts and fast-moving data with rich historical analysis lets you respond quickly to changing business conditions with powerful data lake analytics to make smarter decisions. Join this complimentary webinar with industry experts from 451 Research and Arcadia Data who will discuss: - Business requirements for combining real-time streaming and ad hoc visual analytics. - Innovations in real-time analytics using tools like Confluent’s KSQL. - Machine-assisted visualization to guide business analysts to faster insights. - Elevating user concurrency and analytic performance on data lakes. - Applications in cybersecurity, regulatory compliance, and predictive maintenance on manufacturing equipment all benefit from streaming visualizations.

Accelerating Data Lakes and Streams with Real-time Analytics

Arcadia Data

Today’s data-driven companies have a choice to make – where do we store our data? As the move to the cloud continues to be a driving factor, the choice becomes either the data warehouse (Snowflake et al) or the data lake (AWS S3 et al). There are pro’s and con’s for each approach. While the data warehouse will give you strong data management with analytics, they don’t do well with semi-structured and unstructured data with tightly coupled storage and compute, not to mention expensive vendor lock-in. On the other hand, data lakes allow you to store all kinds of data and are extremely affordable, but they’re only meant for storage and by themselves provide no direct value to an organization. Enter the Open Data Lakehouse, the next evolution of the data stack that gives you the openness and flexibility of the data lake with the key aspects of the data warehouse like management and transaction support. In this webinar, you’ll hear from Ali LeClerc who will discuss the data landscape and why many companies are moving to an open data lakehouse. Ali will share more perspective on how you should think about what fits best based on your use case and workloads, and how some real world customers are using Presto, a SQL query engine, to bring analytics to the data lakehouse.

Data Warehouse or Data Lake, Which Do I Choose?

DATAVERSITY

"Data mesh is a relatively recent architectural innovation, espoused as one of the best ways to fix analytic data. We renegotiate aged social conventions by focusing on treating data as a product, with a clearly defined data product owner, akin to that of any other product. In addition, we focus on building out a self-service platform with integrated governance, letting consumers safely access and use the data they need to solve their business problems. Data mesh is prescribed as a solution for _analytical data_, so that conventionally analytical results (think weekly sales or monthly revenue reports) can be more accurately and predictably computed. But what about non-analytical business operations? Would they not also benefit from data products backed by self-service capabilities and dedicated owners? If you've ever provided a customer with an analytical report that differed from their operational conclusions, then this talk is for you. Adam discusses the resounding successes he has seen from applying data mesh _off-label_ to both analytical and operational domains. The key? Event streams. Well-defined, incrementally updating data products that can power both real-time and batch-based applications, providing a single source of data for a wide variety of application and analytical use cases. Adam digs into the common areas of success seen across numerous clients and customers and provides you with a set of practical guidelines for implementing your own minimally viable data mesh. Finally, Adam covers the main social and technical hurdles that you'll encounter as you implement your own data mesh. Learn about important data use cases, data domain modeling techniques, self-service platforms, and building an iteratively successful data mesh."

Off-Label Data Mesh: A Prescription for Healthier Data

HostedbyConfluent

Part 2: Apache Kudu: Extending the Capabilities of Operational and Analytic D...

Cloudera, Inc.

This presentation is geared toward enterprise architects and senior IT leaders looking to drive more value from their data by learning about cloud data lake management. As businesses focus on leveraging big data to drive digital transformation, technology leaders are struggling to keep pace with the high volume of data coming in at high speed and rapidly evolving technologies. What's needed is an approach that helps you turn petabytes into profit. Cloud data lakes and cloud data warehouses have emerged as a popular architectural pattern to support next-generation analytics. Informatica's comprehensive AI-driven cloud data lake management solution natively ingests, streams, integrates, cleanses, governs, protects and processes big data workloads in multi-cloud environments. Please leave any questions or comments below.

How to Architect a Serverless Cloud Data Lake for Enhanced Data Analytics

Informatica

Data lakes often fail because they are only accessible by highly-skilled data scientists and not by business users. But BI tools have been able to access data warehouses for years, so what gives? In this talk, we’ll discuss: - Why existing BI tools are architected well for data warehouses, but not data lakes. - The pros and cons of each architecture. - Why every organization should have two BI standards: one for data warehouses and one for data lakes.

A Tale of Two BI Standards

Arcadia Data

Azure data analytics platform - A reference architecture

Rajesh Kumar

Apache Spark 2.0 set the architectural foundations of Structure in Spark, Unified high-level APIs, Structured Streaming, and the underlying performant components like Catalyst Optimizer and Tungsten Engine. Since then the Spark community has continued to build new features and fix numerous issues in releases Spark 2.1 and 2.2. Continuing forward in that spirit, the upcoming release of Apache Spark 2.3 has made similar strides too, introducing new features and resolving over 1300 JIRA issues. In this talk, we want to share with the community some salient aspects of soon to be released Spark 2.3 features: • Kubernetes Scheduler Backend • PySpark Performance and Enhancements • Continuous Structured Streaming Processing • DataSource v2 APIs • Structured Streaming v2 APIs

What's New in Upcoming Apache Spark 2.3

Databricks

Unconference Round Table Notes

Timothy Spann

BIG DATA ANALYTICS MEANS “IN-DATABASE” ANALYTICS

TIBCO Spotfire

발표영상 다시보기: https://youtu.be/iosyR9DKVSU AWS는 데이터 수집, 저장, 분석 및 처리에 대한 고객 요구에 따른 다양한 클라우드 서비스를 제공하고 있습니다. 본 세션에서는 고객의 데이터 활용 요구 사항에 따라 선택할 수 있는 AWS 서비스 소개와 함께 데이터의 수집과 분석, 기계 학습을 활용한 비지니스 통찰력을 얻음으로서 데이터의 가치를 최대로 높일 수 있는 혁신적인 방법을 알려드립니다.

AWS를 통한 데이터 분석 및 처리의 새로운 혁신 기법 - 김윤건, AWS사업개발 담당:: AWS Summit Online Korea 2020

Amazon Web Services Korea

Ähnlich wie Real-Time Attribution with Structured Streaming and Databricks Delta with Caryl Yuhas (20)

Building Reliable Data Lakes at Scale with Delta Lake

The Great Lakes: How to Approach a Big Data Implementation

First in Class: Optimizing the Data Lake for Tighter Integration

SQL Analytics Powering Telemetry Analysis at Comcast

C19013010 the tutorial to build shared ai services session 2

Meetup: Streaming Data Pipeline Development

ADV Slides: When and How Data Lakes Fit into a Modern Data Architecture

2018 02-08-what's-new-in-apache-spark-2.3

Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard

Accelerating Data Lakes and Streams with Real-time Analytics

Data Warehouse or Data Lake, Which Do I Choose?

Off-Label Data Mesh: A Prescription for Healthier Data

Part 2: Apache Kudu: Extending the Capabilities of Operational and Analytic D...

How to Architect a Serverless Cloud Data Lake for Enhanced Data Analytics

A Tale of Two BI Standards

Azure data analytics platform - A reference architecture

What's New in Upcoming Apache Spark 2.3

Unconference Round Table Notes

BIG DATA ANALYTICS MEANS “IN-DATABASE” ANALYTICS

AWS를 통한 데이터 분석 및 처리의 새로운 혁신 기법 - 김윤건, AWS사업개발 담당:: AWS Summit Online Korea 2020

Mehr von Databricks

DW Migration Webinar-March 2022.pptx

Databricks

The world of data architecture began with applications. Next came data warehouses. Then text was organized into a data warehouse. Then one day the world discovered a whole new kind of data that was being generated by organizations. The world found that machines generated data that could be transformed into valuable insights. This was the origin of what is today called the data lakehouse. The evolution of data architecture continues today. Come listen to industry experts describe this transformation of ordinary data into a data architecture that is invaluable to business. Simply put, organizations that take data architecture seriously are going to be at the forefront of business tomorrow. This is an educational event. Several of the authors of the book Building the Data Lakehouse will be presenting at this symposium.

Data Lakehouse Symposium | Day 1 | Part 1

Databricks

Data Lakehouse Symposium | Day 1 | Part 2

Databricks

Data Lakehouse Symposium | Day 2

Databricks

Data Lakehouse Symposium | Day 4

Databricks

In this session, learn how to quickly supplement your on-premises Hadoop environment with a simple, open, and collaborative cloud architecture that enables you to generate greater value with scaled application of analytics and AI on all your data. You will also learn five critical steps for a successful migration to the Databricks Lakehouse Platform along with the resources available to help you begin to re-skill your data teams.

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

Databricks

Bad data leads to bad decisions and broken customer experiences. Organizations depend on complete and accurate data to power their business, maintain efficiency, and uphold customer trust. With thousands of datasets and pipelines running, how do we ensure that all data meets quality standards, and that expectations are clear between producers and consumers? Investing in shared, flexible components and practices for monitoring data health is crucial for a complex data organization to rapidly and effectively scale. At Zillow, we built a centralized platform to meet our data quality needs across stakeholders. The platform is accessible to engineers, scientists, and analysts, and seamlessly integrates with existing data pipelines and data discovery tools. In this presentation, we will provide an overview of our platform’s capabilities, including: Giving producers and consumers the ability to define and view data quality expectations using a self-service onboarding portal Performing data quality validations using libraries built to work with spark Dynamically generating pipelines that can be abstracted away from users Flagging data that doesn’t meet quality standards at the earliest stage and giving producers the opportunity to resolve issues before use by downstream consumers Exposing data quality metrics alongside each dataset to provide producers and consumers with a comprehensive picture of health over time

Democratizing Data Quality Through a Centralized Platform

Databricks

Data scientists face numerous challenges throughout the data science workflow that hinder productivity. As organizations continue to become more data-driven, a collaborative environment is more critical than ever — one that provides easier access and visibility into the data, reports and dashboards built against the data, reproducibility, and insights uncovered within the data.. Join us to hear how Databricks’ open and collaborative platform simplifies data science by enabling you to run all types of analytics workloads, from data preparation to exploratory analysis and predictive analytics, at scale — all on one unified platform.

Learn to Use Databricks for Data Science

Databricks

Application performance monitoring (APM) has become the cornerstone of software engineering allowing engineering teams to quickly identify and remedy production issues. However, as the world moves to intelligent software applications that are built using machine learning, traditional APM quickly becomes insufficient to identify and remedy production issues encountered in these modern software applications. As a lead software engineer at NewRelic, my team built high-performance monitoring systems including Insights, Mobile, and SixthSense. As I transitioned to building ML Monitoring software, I found the architectural principles and design choices underlying APM to not be a good fit for this brand new world. In fact, blindly following APM designs led us down paths that would have been better left unexplored. In this talk, I draw upon my (and my team’s) experience building an ML Monitoring system from the ground up and deploying it on customer workloads running large-scale ML training with Spark as well as real-time inference systems. I will highlight how the key principles and architectural choices of APM don’t apply to ML monitoring. You’ll learn why, understand what ML Monitoring can successfully borrow from APM, and hear what is required to build a scalable, robust ML Monitoring architecture.

Why APM Is Not the Same As ML Monitoring

Databricks

Autonomy and ownership are core to working at Stitch Fix, particularly on the Algorithms team. We enable data scientists to deploy and operate their models independently, with minimal need for handoffs or gatekeeping. By writing a simple function and calling out to an intuitive API, data scientists can harness a suite of platform-provided tooling meant to make ML operations easy. In this talk, we will dive into the abstractions the Data Platform team has built to enable this. We will go over the interface data scientists use to specify a model and what that hooks into, including online deployment, batch execution on Spark, and metrics tracking and visualization.

The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix

Databricks

In this talk, I will dive into the stage level scheduling feature added to Apache Spark 3.1. Stage level scheduling extends upon Project Hydrogen by improving big data ETL and AI integration and also enables multiple other use cases. It is beneficial any time the user wants to change container resources between stages in a single Apache Spark application, whether those resources are CPU, Memory or GPUs. One of the most popular use cases is enabling end-to-end scalable Deep Learning and AI to efficiently use GPU resources. In this type of use case, users read from a distributed file system, do data manipulation and filtering to get the data into a format that the Deep Learning algorithm needs for training or inference and then sends the data into a Deep Learning algorithm. Using stage level scheduling combined with accelerator aware scheduling enables users to seamlessly go from ETL to Deep Learning running on the GPU by adjusting the container requirements for different stages in Spark within the same application. This makes writing these applications easier and can help with hardware utilization and costs. There are other ETL use cases where users want to change CPU and memory resources between stages, for instance there is data skew or perhaps the data size is much larger in certain stages of the application. In this talk, I will go over the feature details, cluster requirements, the API and use cases. I will demo how the stage level scheduling API can be used by Horovod to seamlessly go from data preparation to training using the Tensorflow Keras API using GPUs. The talk will also touch on other new Apache Spark 3.1 functionality, such as pluggable caching, which can be used to enable faster dataframe access when operating from GPUs.

Stage Level Scheduling Improving Big Data and AI Integration

Databricks

In this talk, I would like to introduce an open-source tool built by our team that simplifies the data conversion from Apache Spark to deep learning frameworks. Imagine you have a large dataset, say 20 GBs, and you want to use it to train a TensorFlow model. Before feeding the data to the model, you need to clean and preprocess your data using Spark. Now you have your dataset in a Spark DataFrame. When it comes to the training part, you may have the problem: How can I convert my Spark DataFrame to some format recognized by my TensorFlow model? The existing data conversion process can be tedious. For example, to convert an Apache Spark DataFrame to a TensorFlow Dataset file format, you need to either save the Apache Spark DataFrame on a distributed filesystem in parquet format and load the converted data with third-party tools such as Petastorm, or save it directly in TFRecord files with spark-tensorflow-connector and load it back using TFRecordDataset. Both approaches take more than 20 lines of code to manage the intermediate data files, rely on different parsing syntax, and require extra attention for handling vector columns in the Spark DataFrames. In short, all these engineering frictions greatly reduced the data scientists’ productivity. The Databricks Machine Learning team contributed a new Spark Dataset Converter API to Petastorm to simplify these tedious data conversion process steps. With the new API, it takes a few lines of code to convert a Spark DataFrame to a TensorFlow Dataset or a PyTorch DataLoader with default parameters. In the talk, I will use an example to show how to use the Spark Dataset Converter to train a Tensorflow model and how simple it is to go from single-node training to distributed training on Databricks.

Simplify Data Conversion from Spark to TensorFlow and PyTorch

Databricks

There is no doubt Kubernetes has emerged as the next generation of cloud native infrastructure to support a wide variety of distributed workloads. Apache Spark has evolved to run both Machine Learning and large scale analytics workloads. There is growing interest in running Apache Spark natively on Kubernetes. By combining the flexibility of Kubernetes and scalable data processing with Apache Spark, you can run any data and machine pipelines on this infrastructure while effectively utilizing resources at disposal. In this talk, Rajesh Thallam and Sougata Biswas will share how to effectively run your Apache Spark applications on Google Kubernetes Engine (GKE) and Google Cloud Dataproc, orchestrate the data and machine learning pipelines with managed Apache Airflow on GKE (Google Cloud Composer). Following topics will be covered: – Understanding key traits of Apache Spark on Kubernetes- Things to know when running Apache Spark on Kubernetes such as autoscaling- Demonstrate running analytics pipelines on Apache Spark orchestrated with Apache Airflow on Kubernetes cluster.

Scaling your Data Pipelines with Apache Spark on Kubernetes

Databricks

Pipelines have become ubiquitous, as the need for stringing multiple functions to compose applications has gained adoption and popularity. Common pipeline abstractions such as “fit” and “transform” are even shared across divergent platforms such as Python Scikit-Learn and Apache Spark. Scaling pipelines at the level of simple functions is desirable for many AI applications, however is not directly supported by Ray’s parallelism primitives. In this talk, Raghu will describe a pipeline abstraction that takes advantage of Ray’s compute model to efficiently scale arbitrarily complex pipeline workflows. He will demonstrate how this abstraction cleanly unifies pipeline workflows across multiple platforms such as Scikit-Learn and Spark, and achieves nearly optimal scale-out parallelism on pipelined computations. Attendees will learn how pipelined workflows can be mapped to Ray’s compute model and how they can both unify and accelerate their pipelines with Ray.

Scaling and Unifying SciKit Learn and Apache Spark Pipelines

Databricks

In this talk about zipline, we will introduce a new type of windowing construct called a sawtooth window. We will describe various properties about sawtooth windows that we utilize to achieve online-offline consistency, while still maintaining high-throughput, low-read latency and tunable write latency for serving machine learning features.We will also talk about a simple deployment strategy for correcting feature drift – due operations that are not “abelian groups”, that operate over change data.

Sawtooth Windows for Feature Aggregations

Databricks

We want to present multiple anti patterns utilizing Redis in unconventional ways to get the maximum out of Apache Spark.All examples presented are tried and tested in production at Scale at Adobe. The most common integration is spark-redis which interfaces with Redis as a Dataframe backing Store or as an upstream for Structured Streaming. We deviate from the common use cases to explore where Redis can plug gaps while scaling out high throughput applications in Spark. Niche 1 : Long Running Spark Batch Job – Dispatch New Jobs by polling a Redis Queue · Why? o Custom queries on top a table; We load the data once and query N times · Why not Structured Streaming · Working Solution using Redis Niche 2 : Distributed Counters · Problems with Spark Accumulators · Utilize Redis Hashes as distributed counters · Precautions for retries and speculative execution · Pipelining to improve performance

Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

Databricks

In the era of microservices, decentralized ML architectures and complex data pipelines, data quality has become a bigger challenge than ever. When data is involved in complex business processes and decisions, bad data can, and will, affect the bottom line. As a result, ensuring data quality across the entire ML pipeline is both costly, and cumbersome while data monitoring is often fragmented and performed ad hoc. To address these challenges, we built whylogs, an open source standard for data logging. It is a lightweight data profiling library that enables end-to-end data profiling across the entire software stack. The library implements a language and platform agnostic approach to data quality and data monitoring. It can work with different modes of data operations, including streaming, batch and IoT data. In this talk, we will provide an overview of the whylogs architecture, including its lightweight statistical data collection approach and various integrations. We will demonstrate how the whylogs integration with Apache Spark achieves large scale data profiling, and we will show how users can apply this integration into existing data and ML pipelines.

Re-imagine Data Monitoring with whylogs and Spark

Databricks

Machine learning (ML) models are typically part of prediction queries that consist of a data processing part (e.g., for joining, filtering, cleaning, featurization) and an ML part invoking one or more trained models. In this presentation, we identify significant and unexplored opportunities for optimization. To the best of our knowledge, this is the first effort to look at prediction queries holistically, optimizing across both the ML and SQL components. We will present Raven, an end-to-end optimizer for prediction queries. Raven relies on a unified intermediate representation that captures both data processing and ML operators in a single graph structure. This allows us to introduce optimization rules that (i) reduce unnecessary computations by passing information between the data processing and ML operators (ii) leverage operator transformations (e.g., turning a decision tree to a SQL expression or an equivalent neural network) to map operators to the right execution engine, and (iii) integrate compiler techniques to take advantage of the most efficient hardware backend (e.g., CPU, GPU) for each operator. We have implemented Raven as an extension to Spark’s Catalyst optimizer to enable the optimization of SparkSQL prediction queries. Our implementation also allows the optimization of prediction queries in SQL Server. As we will show, Raven is capable of improving prediction query performance on Apache Spark and SQL Server by up to 13.1x and 330x, respectively. For complex models, where GPU acceleration is beneficial, Raven provides up to 8x speedup compared to state-of-the-art systems. As part of the presentation, we will also give a demo showcasing Raven in action.

Raven: End-to-end Optimization of ML Prediction Queries

Databricks

Semantic segmentation is the classification of every pixel in an image/video. The segmentation partitions a digital image into multiple objects to simplify/change the representation of the image into something that is more meaningful and easier to analyze [1][2]. The technique has a wide variety of applications ranging from perception in autonomous driving scenarios to cancer cell segmentation for medical diagnosis. Exponential growth in the datasets that require such segmentation is driven by improvements in the accuracy and quality of the sensors generating the data extending to 3D point cloud data. This growth is further compounded by exponential advances in cloud technologies enabling the storage and compute available for such applications. The need for semantically segmented datasets is a key requirement to improve the accuracy of inference engines that are built upon them. Streamlining the accuracy and efficiency of these systems directly affects the value of the business outcome for organizations that are developing such functionalities as a part of their AI strategy. This presentation details workflows for labeling, preprocessing, modeling, and evaluating performance/accuracy. Scientists and engineers leverage domain-specific features/tools that support the entire workflow from labeling the ground truth, handling data from a wide variety of sources/formats, developing models and finally deploying these models. Users can scale their deployments optimally on GPU-based cloud infrastructure to build accelerated training and inference pipelines while working with big datasets. These environments are optimized for engineers to develop such functionality with ease and then scale against large datasets with Spark-based clusters on the cloud.

Processing Large Datasets for ADAS Applications using Apache Spark

Databricks

At Adobe Experience Platform, we ingest TBs of data every day and manage PBs of data for our customers as part of the Unified Profile Offering. At the heart of this is a bunch of complex ingestion of a mix of normalized and denormalized data with various linkage scenarios power by a central Identity Linking Graph. This helps power various marketing scenarios that are activated in multiple platforms and channels like email, advertisements etc. We will go over how we built a cost effective and scalable data pipeline using Apache Spark and Delta Lake and share our experiences. What are we storing? Multi Source – Multi Channel Problem Data Representation and Nested Schema Evolution Performance Trade Offs with Various formats Go over anti-patterns used (String FTW) Data Manipulation using UDFs Writer Worries and How to Wipe them Away Staging Tables FTW Datalake Replication Lag Tracking Performance Time!

Massive Data Processing in Adobe Using Delta Lake

Databricks

Mehr von Databricks (20)

DW Migration Webinar-March 2022.pptx

Data Lakehouse Symposium | Day 1 | Part 1

Data Lakehouse Symposium | Day 1 | Part 2

Data Lakehouse Symposium | Day 2

Data Lakehouse Symposium | Day 4

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

Democratizing Data Quality Through a Centralized Platform

Learn to Use Databricks for Data Science

Why APM Is Not the Same As ML Monitoring

The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix

Stage Level Scheduling Improving Big Data and AI Integration

Simplify Data Conversion from Spark to TensorFlow and PyTorch

Scaling your Data Pipelines with Apache Spark on Kubernetes

Scaling and Unifying SciKit Learn and Apache Spark Pipelines

Sawtooth Windows for Feature Aggregations

Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

Re-imagine Data Monitoring with whylogs and Spark

Raven: End-to-end Optimization of ML Prediction Queries

Processing Large Datasets for ADAS Applications using Apache Spark

Massive Data Processing in Adobe Using Delta Lake

Kürzlich hochgeladen

Vadodara 💋 Call Girl 7737669865 Call Girls in Vadodara Escort service book now Booking Contact Details :- WhatsApp Chat :- +91-7737669865 We offer all types of girls of your choice with space. Our escorts are fully cooperative and understand your needs. All types of call girls like Housewives, College girls,#K09 Russian girls, Muslim girls, Afghani girls, Bengali girls, Working girls, south Indian girls, Punjabi girls, etc. In-Call: — You Can Reach At Our Place in Bangalore Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: — Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Hygienic: — Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Bangalore Our Services and Rates: – One Shot — 2500/in call (time ½ hour), 5000/out call Two shot with one girl — 5000/in call (time 1 hour), 6000/out call Body to body massage with sex- 3000/in call (time 1 hour) full night for one person– 8000/in call, 10000/out call (shot limit 4 shot) We are available 24*7 all days of the year

Vadodara 💋 Call Girl 7737669865 Call Girls in Vadodara Escort service book now

gargpaaro

Digital Transformation Playbook by Graham Ware

Graham Ware

Top profile Call Girls In Vadodara [ 7014168258 ] Call Me For Genuine Models We are available 24*7 Booking Contact Details :- WhatsApp Chat :- +91-7014168258 If you're looking for India Call girls you've come to the right place. You'll find some of the most beautiful call girls in our location with. These ladies have pleasing personalities, hot figures, and a passion for physical pleasure. Call girls in India Lucknow Many men have booked them for their erotic and soul-mixing performances, which are sure to leave you with unforgettable memories. #K09 Escort Service India is available in the city for men and women of all ages. They can satisfy your sexual needs and will make your experience even more enjoyable and memorable. Whether you're looking for a blow-job, stripping, lovemaking, or other dirty acts, you'll be able to find a match for your tastes and budget. These highly trained professionals will help you have an unforgettable night. One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7014168258 We are available 24*7 all days of the year. Call us — 7014168258 Thank you for Visiting.

Top profile Call Girls In Vadodara [ 7014168258 ] Call Me For Genuine Models ...

gajnagarg

Discover Why Less is More in B2B Research

michael115558

Gomti Nagar & best call girls in Lucknow | 9548273370 Independent Escorts & Dating Escorts Service CALL GIRL IN Lucknow 9548273370 ❤CALL GIRLS IN ESCORT SERVICE❤CALL GIRL IN #j11 We are Providing :- ● – Private independent collage Going girls . ● – independent Models . ● – House Wife’s . ● – Private Independent House Wife’s ● – Corporate M.N.C Working Profiles . ● – Call Center Girls . ● – Live Band Girls . ●- Foreigners & Many More . Service type: 1.In call 2.out call 3. full Lip to Lip kiss 4.69 5.b-job without Condom 6. Hard Core sex & Much More. 7 Body to Body Touch 8 Kissing 9 Sucking Boobs and More 10 Enjoy by Hand 11 Relax By Oral 12 Sex with Happy Ending • In Call and Out Call Service • 3* 5* 7* Hotels Service • 24 Hours Available • Indian, Russian, Punjabi, Kashmiri Escorts • Real Models, College Girls, House Wife, Also Available • Short Time and Full Time Service Available • Hygienic Full AC Neat and Clean Rooms Avail. In Hotel 24 hours • Daily Escorts Staff Available • Minimum to Maximu m Range Available.c

Gomti Nagar & best call girls in Lucknow | 9548273370 Independent Escorts & D...

HyderabadDolls

Aspirational Block Program Block Syaldey District - Almora

GovindSinghDasila

Top profile Call Girls In Satna [ 7014168258 ] Call Me For Genuine Models We are available 24*7 Booking Contact Details :- WhatsApp Chat :- +91-7014168258 If you're looking for India Call girls you've come to the right place. You'll find some of the most beautiful call girls in our location with. These ladies have pleasing personalities, hot figures, and a passion for physical pleasure. Call girls in India Lucknow Many men have booked them for their erotic and soul-mixing performances, which are sure to leave you with unforgettable memories. #K09 Escort Service India is available in the city for men and women of all ages. They can satisfy your sexual needs and will make your experience even more enjoyable and memorable. Whether you're looking for a blow-job, stripping, lovemaking, or other dirty acts, you'll be able to find a match for your tastes and budget. These highly trained professionals will help you have an unforgettable night. One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7014168258 We are available 24*7 all days of the year. Call us — 7014168258 Thank you for Visiting.

Top profile Call Girls In Satna [ 7014168258 ] Call Me For Genuine Models We ...

nirzagarg

Jodhpur Park | Call Girls in Kolkata Phone No 8005736733 Elite Escort Service Available 24/7 Hire Our agency presents a selection of young, charming call girls #J11 available for bookings at Oyo Hotels. Experience high-class escort services at pocket-friendly rates, with our female escorts exuding both beauty and a delightful personality, ready to meet your desires. Whether it's Housewives,#J11 College girls, Russian girls, Muslim girls, or any other preference, we offer a diverse range of options to cater to your tastes. We provide both in-call and out-call services for your convenience. Our in-call location in Kolkata ensures cleanliness, hygiene, and 100% safety, while our out-call services offer doorstep delivery for added ease. We value your time and money, hence we kindly request pic collectors, time-passers, and bargain hunters to refrain from contacting us. Our services feature various packages at competitive rates: One shot: ₹2000/in-call, ₹5000/out-call Two shots with one girl: ₹3500/in-call, ₹6000/out-call Body to body massage with sex: ₹3000/in-call Full night for one person: ₹7000/in-call, ₹10000/out-call Full night for more than 1 person: Contact us at 🔝 8005736733 🔝. for details Operating 24/7, we serve various locations in Kolkata, including Green Park, near metro stations. For premium call girl services in Kolkata 🔝 8005736733 🔝. Thank you for considering us!

Jodhpur Park | Call Girls in Kolkata Phone No 8005736733 Elite Escort Service...

HyderabadDolls

Gartner's Data Analytics Maturity Model.pptx

chadhar227

RESEARCH-FINAL-DEFENSE-PPT-TEMPLATE.pptx

ronsairoathenadugay

Top profile Call Girls In Begusarai [ 7014168258 ] Call Me For Genuine Models We are available 24*7 Booking Contact Details :- WhatsApp Chat :- +91-7014168258 If you're looking for India Call girls you've come to the right place. You'll find some of the most beautiful call girls in our location with. These ladies have pleasing personalities, hot figures, and a passion for physical pleasure. Call girls in India Lucknow Many men have booked them for their erotic and soul-mixing performances, which are sure to leave you with unforgettable memories. #K09 Escort Service India is available in the city for men and women of all ages. They can satisfy your sexual needs and will make your experience even more enjoyable and memorable. Whether you're looking for a blow-job, stripping, lovemaking, or other dirty acts, you'll be able to find a match for your tastes and budget. These highly trained professionals will help you have an unforgettable night. One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7014168258 We are available 24*7 all days of the year. Call us — 7014168258 Thank you for Visiting.

Top profile Call Girls In Begusarai [ 7014168258 ] Call Me For Genuine Models...

nirzagarg

Gulbai Tekra * Cheap Call Girls In Ahmedabad Phone No 8005736733 Elite Escort Service Available 24/7 Hire Booking Contact Details :- WhatsApp Chat :- +91-6378878445 We offer all types of girls of your choice with space. Our escorts are fully cooperative and understand your needs.#K09 All types of call girls like Housewives, College girls,#K09 Russian girls, Muslim girls, Afghani girls, Bengali girls, Working girls, south Indian girls, Punjabi girls, etc. In-Call: — You Can Reach At Our Place in Bangalore Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: — Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Hygienic: — Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Bangalore Our Services and Rates: – One Shot — 2500/in call (time ½ hour), 5000/out call Two shot with one girl — 5000/in call (time 1 hour), 6000/out call Body to body massage with sex- 3000/in call (time 1 hour) full night for one person– 8000/in call, 10000/out call (shot limit 4 shot) We are available 24*7 all days of the year

Gulbai Tekra * Cheap Call Girls In Ahmedabad Phone No 8005736733 Elite Escort...

gragchanchal546

Discover how TrafficWave Generator can drive targeted, engaging visitors to your online assets and propel your business to new heights. Unlock the power of free, set-and-forget traffic today. Featuring a user-friendly interface, set-and-forget functionality, and the ability to monetize the traffic with your affiliate links or product offers, TrafficWave Generator is a game-changer for online entrepreneurs, bloggers, and e-commerce store owners alike. Backed by a 30-day money-back guarantee and comprehensive bonus offerings, this software has the potential to transform your traffic-generation and content-creation efforts, unlocking new levels of success in the digital marketing landscape.

TrafficWave Generator Will Instantly drive targeted and engaging traffic back...

SOFTTECHHUB

Abortion pills in Kuwait city !🌆+918761049707^) Where to get cytotec pills in salmiyah, WhatsApp '''(+918761049707)''' Explore Discreet and Safe Abortion Pill Options in Dubai, Empowering Women With Confidential and Medically Supervised Choices For Reproductive Health" Abortion Pills In Dubai, Abu Dhabi/Alain/Sharjah/RAK City Satwa-Mifepristone and Misoprostol Available in Dubai/Abu Dhabi - i Pills in Dubai and Cost Of Cytotec in WhatsApp (+918761049707) Abortion Pills in Dubai/Abu Dubai/ Abu Dhabi. Are you stranded with unwanted pregnancy in Dubai, Abu Dhabi , the United Arab Emirates (U.A.E), Qatar, Oman, Saudi Arabia or Kuwait? you can now contact us now on Whatsapp Dr AJ:+918761049707to buy safe abortion pills In Dubai, Abu Dhabi, Sharjah, Al Ain, Ajman, RAK City, Ras Al Khaimah and Fujairah to terminate an unwanted pregnancy in Dubai and the United Arab Emirates. Get your Discreet 100% Safe (+918761049707 )*Effective Abortion Pills For Sale in Dubai, Abu Dhabi, KUWAIT, QATAR, BAHRAIN, DOHA, SALMIYA, Sharjah, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE. BUY Mifepristone and Misoprostol (Cytotec), Mtp Kit In UAE. Abortion pills available in UAE (United Arab Emirates), Saudi Arabia, Kuwait, Oman, Bahrain and Qatar. Contact us today. +918761049707 -The UAE’s leading abortion care service in Dubai. Abortion Treatment. Medical Abortion. Surgical Abortion. Find A Clinic like Dr Maria Abortion clinic in Dubai We have Abortion Pills / Cytotec Tablets Available in Dubai, Abu Dhabi, KUWAIT, QATAR, BAHRAIN, DOHA, SALMIYA, Sharjah, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE., buy cytotec in Dubai abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain We sell original abortion medicine which includes: Cytotec 200mcg (Misoprostol), Mifepristone, Mifegest-kit, Misoclear, Emergency contraceptive pills, Morning after sex pills, ipills, pills to prevent pregnancy 72 hours after sex. All our pills are manufactured by reputable medical manufacturing companies like PFIZER. Medical abortion is easy and effective for everyone to perform in their own privacy. There are very few complications that may arise from medical abortion if one follows the right guidelines as instructed by the obstetrician. Abortion Pills in Dubai Can Now Be Offered at Dr Maria Abortion clinic in Dubai, F.D.A. Mifepristone and Misoprostol, the first of two drugs in medication abortions, previously had to be dispensed only by clinics, doctors or a few mail-order pharmacies like Dr Maria Abortion clinic in Dubai . Now, We can provide it. For the first time, retail pharmacies, like Dr Maria Abortion clinic in Dubai, will be Able to offer abortion pills in Dubai under a regulatory change made Tuesday by the Food and Drug Administration. The action could significantly expand access to abortion through medication. Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg

In Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi Arabia

ahmedjiabur940

Top Call Girls in Balaghat 9332606886Call Girls Advance Cash On Delivery Service Booking Now open +91- 9332606886 Pune VIP Call Girls is highly qualified professionals. #S10 They will offer only top quality services and never try to scam or drain you of money; in fact, they strive to give complete satisfaction so that each cent spent was worth its worth. Furthermore, these professionals pride themselves in keeping customer details confidential; any information gleaned will never be shared with any third party.$V15 Two shots with one girl: ₹4500/in-call, ₹7500/out-call Body to body massage with : ₹5500/in-call Full night for one person: ₹7000/in-call, ₹12000/out-call •𝐂𝐨𝐥𝐥𝐞𝐠𝐞 𝐆𝐢𝐫𝐥𝐬 •𝐇𝐨𝐮𝐬𝐞𝐰𝐢𝐯𝐞𝐬 •𝐇𝐢𝐠𝐡 𝐏𝐫𝐨𝐟𝐢𝐥𝐞 𝐆𝐢𝐫𝐥𝐬 •𝐑𝐚𝐦𝐩 𝐌𝐨𝐝𝐞𝐥𝐬 •𝐅𝐨𝐫𝐞𝐢𝐠𝐧𝐞𝐫 𝐆𝐢𝐫𝐥𝐬 𝐎𝐮𝐫 𝐞𝐬𝐜𝐨𝐫𝐭 𝐬𝐞𝐫𝐯𝐢𝐜𝐞𝐬 •𝐇𝐉 (𝐇𝐚𝐧𝐝 𝐉𝐨𝐛) •𝐁𝐉 (𝐁𝐥𝐨𝐰𝐣𝐨𝐛) # Completion # (Oral To Completion) # Special Massage # O-Level (Oral ) # Oral With A Noncondom) # COB (Come On Body) # Extraball (Have Many Times) # All Meetings 100%Safe

Top Call Girls in Balaghat 9332606886Call Girls Advance Cash On Delivery Ser...

kumargunjan9515

Building Real-Time Pipelines With FLaNK Timothy Spann, Principal Developer Advocate, Streaming - Cloudera Future of Data meetup, startup grind, AI Camp The combination of Apache Flink, Apache NiFi, and Apache Kafka for building real-time data processing pipelines is extremely powerful, as demonstrated by this case study using the FLaNK-MTA project. The project leverages these technologies to process and analyze real-time data from the New York City Metropolitan Transportation Authority (MTA). FLaNK-MTA demonstrates how to efficiently collect, transform, and analyze high-volume data streams, enabling timely insights and decision-making. Apache NiFi Apache Kafka Apache Flink Apache Iceberg LLM Generative AI Slack Postgresql

DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK

Timothy Spann

5CL-ADBA,5cladba, Chinese supplier, safety is guaranteed

amy56318795

Top profile Call Girls In bhavnagar [ 7014168258 ] Call Me For Genuine Models We are available 24*7 Booking Contact Details :- WhatsApp Chat :- +91-7014168258 If you're looking for India Call girls you've come to the right place. You'll find some of the most beautiful call girls in our location with. These ladies have pleasing personalities, hot figures, and a passion for physical pleasure. Call girls in India Lucknow Many men have booked them for their erotic and soul-mixing performances, which are sure to leave you with unforgettable memories. #K09 Escort Service India is available in the city for men and women of all ages. They can satisfy your sexual needs and will make your experience even more enjoyable and memorable. Whether you're looking for a blow-job, stripping, lovemaking, or other dirty acts, you'll be able to find a match for your tastes and budget. These highly trained professionals will help you have an unforgettable night. One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7014168258 We are available 24*7 all days of the year. Call us — 7014168258 Thank you for Visiting.

Top profile Call Girls In bhavnagar [ 7014168258 ] Call Me For Genuine Models...

gajnagarg

Klinik_ Apotek Onlin 085657271886 Solusi Menggugurkan Masalah Kehamilan Anda Jual Obat Aborsi Asli KLINIK ABORSI TERPEECAYA _ Jual Obat Aborsi Cytotec Misoprostol Asli 100% Ampuh Hanya 3 Jam Langsung Gugur || OBAT PENGGUGUR KANDUNGAN AMPUH MANJUR OBAT ABORSI OLINE" APOTIK Jual Obat Cytotec, Gastrul, Gynecoside Asli Ampuh. JUAL ” Obat Aborsi Tuntas | Obat Aborsi Manjur | Obat Aborsi Ampuh | Obat Penggugur Janin | Obat Pencegah Kehamilan | Obat Pelancar Haid | Obat terlambat Bulan | Ciri Obat Aborsi Asli | Obat Telat Bulan | Pil Aborsi Asli | Cara Menggugurkan Konten | Cara Aborsi Tuntas | Harga Obat Aborsi Asli | Pil Aborsi | Jual Obat Aborsi Cytotec | Cara Aborsi Sendiri | Cara Aborsi Usia 1 Bulan | Cara Aborsi Usia 2 Tahun | Cara Aborsi Usia 3 Bulan | Obat Aborsi Usia 4 Bulan | Cara Abrasi Usia 5 Bulan | Cara Menggugurkan Konten | Kandungan Obat Penggugur | Cara Menghitung Usia Konten | Cara Mengatasi Terlambat Bulan | Penjual Obat Aborsi Asli | Obat Aborsi Garansi | Kandungan Obat Peluntur | Obat Telat Datang Bulan | Obat Telat Haid | Obat Aborsi Paling Murah | Klinik Jual Obat Aborsi | Jual Pil Cytotec | Apotik Jual Obat Aborsi | Kandungan Dokter Abrasi | Cara Aborsi Cepat | Jual Obat Aborsi Bergaransi | Jual Obat Cytotec Asli | Obat Aborsi Aman Manjur | Obat Misoprostol Cytotec Asli. "APA ITU ABORSI" “Aborsi Adalah dengan membendung hormon yang di perlukan untuk mempertahankan kehamilan yaitu hormon progesteron, karena hormon ini dibendung, maka jalur kehamilan mulai membuka dan leher rahim menjadi melunak,sehingga mengeluarkan darah yang merupakan tanda bahwa obat telah bekerja || maksimal 1 jam obat diminum || PENJELASAN OBAT ABORSI USIA 1 _7 BULAN Pada usia kandungan ini, pasien akan merasakan sakit yang sedikit tidak berlebihan || sekitar 1 jam ||. namun hanya akan terjadi pada saatdarah keluar merupakan pertanda menstruasi. Hal ini dikarenakan pada usiakandungan 3 bulan,janin sudah terbentuk sebesar kepalan tangan orang dewasa. Cara kerja obat aborsi : JUAL OBAT ABORSI AMPUH dosis 3 bulan secara umum sama dengan cara kerja || DOSIS OBAT ABORSI 2 bulan”, hanya berbedanya selain mengisolasijanin juga menghancurkan janin dengan formula methotrexate dikandungdidalamnya. Formula methotrexate ini sangat ampuh untuk menghancurkan janinmenjadi serpihan-serpihan kecil akan sangat berguna pada saat dikeluarkan nanti. APA ALASAN WANITA MELAKUKAN ABORSI? Aborsi di lakukan wanita hamil baik yang sudah menikah maupun belum menikah dengan berbagai alasan , akan tetapi alasan yang utama adalah alasan-alasan non medis (termasuk aborsi sendiri / di sengaja/ buatan] MELAYANI PEMESANAN OBAT ABORSI SETIAP HARI, SIAP KIRIM KESELURUH KOTA BESAR DI INDONESIA DAN LUAR NEGERI. HUBUNGI PEMESANAN LEBIH NYAMAN VIA WA/: 085657271886

Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...

ZurliaSoop

Charbagh + Female Escorts Service in Lucknow | Starting ₹,5K To @25k with A/C 9548273370 Escort Service CALL GIRL IN Lucknow 9548273370 ❤CALL GIRLS IN ESCORT SERVICE❤CALL GIRL IN #j11 We are Providing :- ● – Private independent collage Going girls . ● – independent Models . ● – House Wife’s . ● – Private Independent House Wife’s ● – Corporate M.N.C Working Profiles . ● – Call Center Girls . ● – Live Band Girls . ●- Foreigners & Many More . Service type: 1.In call 2.out call 3. full Lip to Lip kiss 4.69 5.b-job without Condom 6. Hard Core sex & Much More. 7 Body to Body Touch 8 Kissing 9 Sucking Boobs and More 10 Enjoy by Hand 11 Relax By Oral 12 Sex with Happy Ending • In Call and Out Call Service • 3* 5* 7* Hotels Service • 24 Hours Available • Indian, Russian, Punjabi, Kashmiri Escorts • Real Models, College Girls, House Wife, Also Available • Short Time and Full Time Service Available • Hygienic Full AC Neat and Clean Rooms Avail. In Hotel 24 hours • Daily Escorts Staff Available • Minimum to Maximu m Range Available.c

Charbagh + Female Escorts Service in Lucknow | Starting ₹,5K To @25k with A/C...

HyderabadDolls

Kürzlich hochgeladen (20)

Vadodara 💋 Call Girl 7737669865 Call Girls in Vadodara Escort service book now

Digital Transformation Playbook by Graham Ware

Top profile Call Girls In Vadodara [ 7014168258 ] Call Me For Genuine Models ...

Discover Why Less is More in B2B Research

Gomti Nagar & best call girls in Lucknow | 9548273370 Independent Escorts & D...

Aspirational Block Program Block Syaldey District - Almora

Top profile Call Girls In Satna [ 7014168258 ] Call Me For Genuine Models We ...

Jodhpur Park | Call Girls in Kolkata Phone No 8005736733 Elite Escort Service...

Gartner's Data Analytics Maturity Model.pptx

RESEARCH-FINAL-DEFENSE-PPT-TEMPLATE.pptx

Top profile Call Girls In Begusarai [ 7014168258 ] Call Me For Genuine Models...

Gulbai Tekra * Cheap Call Girls In Ahmedabad Phone No 8005736733 Elite Escort...

TrafficWave Generator Will Instantly drive targeted and engaging traffic back...

In Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi Arabia

Top Call Girls in Balaghat 9332606886Call Girls Advance Cash On Delivery Ser...

DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK

5CL-ADBA,5cladba, Chinese supplier, safety is guaranteed

Top profile Call Girls In bhavnagar [ 7014168258 ] Call Me For Genuine Models...

Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...

Charbagh + Female Escorts Service in Lucknow | Starting ₹,5K To @25k with A/C...

Real-Time Attribution with Structured Streaming and Databricks Delta with Caryl Yuhas

1. Caryl Yuhas, Databricks Real-Time Attribution with Structured Streaming and Databricks Delta #ExpSAIS13

2. Introduction 2#ExpSAIS13 • Goal: Provide tools and information that can help you build more real-time / lower latency attribution pipelines • Crawl, Walk, Run: Pull Model Carylpreviously MediaMath / SE / PM for Attribution, SA for Databricks

3. Getting Started 3#ExpSAIS13 • What is Attribution? Image Source: www.mediamath.com

4. Introduction What is Databricks Delta? Delta is a data management capability that brings data reliability and performance optimizations to the cloud data lake. 4#ExpSAIS13

5. Stream-to-Sink BEFORE 5#ExpSAIS13 Reprocessing Data Lake λ-arch λ-arch Streaming Analytics Reporting Events Validation λ-arch Validation Reprocessing Compaction Partitioned Compact Small Files Scheduled to Avoid Compaction 1 2 3 1 1 3 4 4 4 2

6. Stream-to-Sink AFTER 6#ExpSAIS13 Reprocessing λ-arch Streaming Analytics Reporting Events Validation λ-arch Validation Reprocessing Compaction Partitioned 1 2 Compact Small Files 3 1 4 4 2 DELTA Optimize 3 ZOrder 3

7. Attribution in Practice 7 impressions conversionsJOIN #ExpSAIS13 attributed impressions

8. Attribution Challenges Scale • Often dealing with millions to billions of data points per attribution window Complexity • Simple, last-click model is still common • MTA and more sophisticated attribution on rise 8#ExpSAIS13

9. High Level Attribution Pipeline 9#ExpSAIS13

10. Attribution in Practice 10 impressions conversionsJOIN #ExpSAIS13 attributed impressions

11. Data Architecture 11#ExpSAIS13 impression stream conversion stream conversions table impressions table attributed table last touch attributed table weighted attribution views (filters, logic, etc.)

12. System Architecture 12#ExpSAIS13 STRUCTURED STREAMING Amazon Kinesis

13. Unification of Streaming + Batch DEMO 13#ExpSAIS13

14. • How can we optimize performance? • Levers: – Delta Tools • Optimize • ZOrder • Caching • Data Skipping – Join on Stream – Cluster Size Managing Performance 14#ExpSAIS13

15. Handling Complexity • Flexibility with Complex Logic – Forking streams – Logic on query vs. in-stream • Late or Corrected Data – Upserts – Views automatically update when raw data changed 15#ExpSAIS13

16. Conclusion • Unification of Batch & Streaming • Easy APIs for Managing Performance • Flexible and Scalable Analytics on Near Real-Time Data 16#ExpSAIS13

Real-Time Attribution with Structured Streaming and Databricks Delta with Caryl Yuhas

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Real-Time Attribution with Structured Streaming and Databricks Delta with Caryl Yuhas

Ähnlich wie Real-Time Attribution with Structured Streaming and Databricks Delta with Caryl Yuhas (20)

Mehr von Databricks

Mehr von Databricks (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Real-Time Attribution with Structured Streaming and Databricks Delta with Caryl Yuhas