Apache Beam @ GCPUG.TW Flink.TW 20161006

•

2 gefällt mir•366 views

Introduce to Apache Beam Dive in to Beam's architecture and live demo running data pipeline on different runners such as Google Dataflow, Flink and Spark

Software

Apache Beam in
Data Pipeline
Randy Huang 
2016/10/06

Who am I
• Data Architect @ VMFive
• Fluentd/Embulk fans

Overview
• Deﬁne Data Pipeline
• Architecture
• How to write Beam
• Demo

Goal
• Provide an abstraction layer between data
processing’s code and the execution runtime.
• Batch processing and Streaming Jobs in one
world.
• Beam SDK open the door to write once, run
anywhere.*
on-premise and non-Google cloud

Supported Runners
• Google Cloud Dataﬂow (Block/Non-Blocking)
• Apache Flink 1.1.2
• Apache Spark 1.6.2 Hadoop 2.2.0 Kafka 0.8.2.1

Architecture
• Pipelines
• Translators
• Runners

programming tips/ Flink
• Use the Flink DataStream API in Java and Scala
• Use the Beam API directly in Java (and soon
Python) with the Flink runner

SDK
• Four Parts :
• Pipeline : Streaming & Batch Processing
• PCollection
• Transform
• I/O : Source & Sink

for Flink user
• we encourage users to use either of the Beam or Flink
APIs to implement their Flink jobs for stream data
processing.
• But Native Flink API -
• backwards-compatible API
• built-in libraries (e.g., CEP and upcoming SQL)
• key-value state (with the ability to query that state in
the future)
http://data-artisans.com/why-apache-beam/

Demo
• GDELT project
• EventCount by Location
Pileline

Recap
• Write the general data pipeline, and choose your
runner

Next…
• New Runners, SDK (python still dev)
• DSL

Another things
• BigQuery have DML support!!! https://goo.gl/
lcZQVZ
• DataStudio Beta in Taiwan is available
• Embulk
• Fluentd v0.14.6 - 2016/09/07

Weitere ähnliche Inhalte

Was ist angesagt?

cLoki: Like Loki but for ClickHouse

Altinity Ltd

Apache Beam (unified Batch and strEAM processing!) is a new Apache incubator project. Originally based on years of experience developing Big Data infrastructure within Google (such as MapReduce, FlumeJava, and MillWheel), it has now been donated to the OSS community at large. Come learn about the fundamentals of out-of-order stream processing, and how Beam’s powerful tools for reasoning about time greatly simplify this complex task. Beam provides a model that allows developers to focus on the four important questions that must be answered by any stream processing pipeline: What results are being calculated? Where in event time are they calculated? When in processing time are they materialized? How do refinements of results relate? Furthermore, by cleanly separating these questions from runtime characteristics, Beam programs become portable across multiple runtime environments, both proprietary (e.g., Google Cloud Dataflow) and open-source (e.g., Flink, Spark, et al).

Fundamentals of Stream Processing with Apache Beam, Tyler Akidau, Frances Perry

confluent

Building the next generation Spark SQL engine at speed poses new challenges to both automation and testing. At Databricks, we are implementing a new testing framework for assessing the quality and performance of new developments as they produced. Having more than 1,200 worldwide contributors, Apache Spark follows a rapid pace of development. At this scale, new testing tooling such as random query and data generation, fault injection, longevity stress, and scalability tests are essential to guarantee a reliable and performance Spark later in production. By applying such techniques, we will demonstrate the effectiveness of our testing infrastructure by drilling-down into cases where correctness and performance regressions have been found early. In addition, showing how they have been root-caused and fixed to prevent regressions in production and boosting the continuous delivery of new features.

Fast and Reliable Apache Spark SQL Engine

Databricks

Apache Flink @ Alibaba - Seattle Apache Flink Meetup

Bowen Li

Is it possible to build an efficient, focused web crawler using Flink? That was the question that led to the creation of the flink-crawler open source project. In this talk I’ll discuss how we use Flink’s support for AsyncFunctions and iterations to create a scalable web crawler that continuously and efficiently performs a focused web crawl with no additional infrastructure. I’ll also discuss some of the testing and debugging challenges encountered when using features such as AsyncFunctions and iterations.

Flink Forward San Francisco 2018: Ken Krugler - "Building a scalable focused ...

Flink Forward

Apache Airflow (incubating) NL HUG Meetup 2016-07-19

Bolke de Bruin

Mark Teehan, Principal Solutions Engineer, Confluent Use the Debezium CDC connector to capture database changes from a Postgres database - or MySQL or Oracle; streaming into Kafka topics and onwards to an external data store. Examine how to setup this pipeline using Docker Compose and Confluent Cloud; and how to use various payload formats, such as avro, protobuf and json-schema. https://www.meetup.com/Singapore-Kafka-Meetup/events/276822852/

From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...

confluent

http://flink-forward.org/kb_sessions/flink-in-zalandos-world-of-microservices/ In this talk we present Zalando’s microservices architecture, introduce Saiki – our next generation data integration and distribution platform on AWS and show how we employ stream processing with Apache Flink for near-real time business intelligence. Zalando is one of the largest online fashion retailers in Europe. In order to secure our future growth and remain competitive in this dynamic market, we are transitioning from a monolithic to a microservices architecture and from a hierarchical to an agile organization. We first have a look at how business intelligence processes have been working inside Zalando for the last years and present our current approach – Saiki. It is a scalable, cloud-based data integration and distribution infrastructure that makes data from our many microservices readily available for analytical teams. We no longer live in a world of static data sets, but are instead confronted with endless streams of events that constantly inform us about relevant happenings from all over the enterprise. The processing of these event streams enables us to do near-real time business intelligence. In this context we have evaluated Apache Flink vs. Apache Spark in order to choose the right stream processing framework. Given our requirements, we decided to use Flink as part of our technology stack, alongside with Kafka and Elasticsearch. With these technologies we are currently working on two use cases: a near real-time business process monitoring solution and streaming ETL. Monitoring our business processes enables us to check if technically the Zalando platform works. It also helps us analyze data streams on the fly, e.g. order velocities, delivery velocities and to control service level agreements. On the other hand, streaming ETL is used to relinquish resources from our relational data warehouse, as it struggles with increasingly high loads. In addition to that, it also reduces the latency and facilitates the platform scalability. Finally, we have an outlook on our future use cases, e.g. near-real time sales and price monitoring. Another aspect to be addressed is to lower the entry barrier of stream processing for our colleagues coming from a relational database background.

Javier Lopez_Mihail Vieru - Flink in Zalando's World of Microservices - Flink...

Flink Forward

Understanding and Improving Code Generation

Databricks

For a long time we discuss how much data we can keep in Kafka. Can we store data forever or do we remove data after a while and maybe having the history in a data lake on Object Storage or HDFS? With the advent of Tiered Storage in Confluent Enterprise Platform, storing data much longer in Kafka is much very feasible. So can we replace a traditional data lake with just Kafka? Maybe at least for the raw data? But what about accessing the data, for example using SQL? KSQL allows for processing data in a streaming fashion using an SQL like dialect. But what about reading all data of a topic? You can reset the offset and still use KSQL. But there is another family of products, so-called query engines for Big Data. They originate from the idea of reading Big Data sources such as HDFS, object storage or HBase, using the SQL language. Presto, Apache Drill and Dremio are the most popular solutions in that space. Lately these query engines also added support for Kafka topics as a source of data. With that you can read a topic as a table and join it with information available in other data sources. The idea of course is not real-time streaming analytics but batch analytics directly on the Kafka topic, without having to store it in a big data storage. This talk answers, how well these tools support Kafka as a data source. What serialization formats do they support? Is there some form of predicate push-down supported or do we have to always read the complete topic? How performant is a query against a topic, compared to a query against the same data sitting in HDFS or an object store? And finally, will this allow us to replace our data lake or at least part of it by Apache Kafka?

Kafka as your Data Lake - is it Feasible? (Guido Schmutz, Trivadis) Kafka Sum...

HostedbyConfluent

Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder. Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Samza) and in-house technologies have helped Uber scale.

Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day

Ankur Bansal

IoT and Event Streaming at Scale with Apache Kafka

confluent

Whether you are deploying a new application in Microservices or transitioning from a monolithic database application to a cloud-ready architecture, you will inevitably face the decision of either creating a service mesh of API’s – or – using an event bus for better durability, reliability and extensibility of your application. If you choose to go the event bus route, Kafka is an excellent choice for several reasons. One key technology not to overlook is Avro Schemas. They provide a definition for your event payload, just like an API, to ensure all of the event consumers can reliably consume the events. They also handle schema evolution as requirements change and much, much more. In this talk we will discuss all the nuances and considerations around using Avro Schemas for your JSON event payloads. From developer tools, to DevOps approaches, versioning, governance and some “gotchas” we found when working with Avro Schemas and the Confluent Schema Registry.

Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...

HostedbyConfluent

Flink has supported Apache Mesos officially since the 1.2 release and many users have been using them together even before that. The latest releases 1.4 and 1.5 (not released at the time of writing) add a deeper integration for resource schedulers, such as Mesos, which also resulted in many new features around this integration. But what does that mean in practice for operating large cluster? In this talk, we will discuss operational best practices-alongside with some pitfalls- for operating large Flink cluster on top of Apache Mesos, including topics such as: * Deployments, * Monitoring, * Scaling, * Upgrades, * Debugging.

Flink Forward San Francisco 2018: Jörg Schad and Biswajit Das - "Operating Fl...

Flink Forward

Machine Learning (ML) is separated into model training and model inference. ML frameworks typically use a data lake like HDFS or S3 to process historical data and train analytic models. But it’s possible to completely avoid such a data store, using a modern streaming architecture. This talk compares a modern streaming architecture to traditional batch and big data alternatives and explains benefits like the simplified architecture, the ability of reprocessing events in the same order for training different models, and the possibility to build a scalable, mission-critical ML architecture for real time predictions with muss less headaches and problems. The talk explains how this can be achieved leveraging Apache Kafka, Tiered Storage and TensorFlow.

Apache Kafka, Tiered Storage and TensorFlow for Streaming Machine Learning wi...

confluent

Bitsy graph database

LambdaZen LLC

Elastic Data Processing with Apache Flink and Apache Pulsar More and more applications are using Flink for low-latency data processing. Flink unifies batch and stream processing using one computation engine. However in reality, in order to really unify batch and stream processing, it requires a data system offers one unified data representation for both batch and streaming data. Nowadays, streaming data is typically stored in a log storage or messaging system, while batch data is stored in distributed filesystem and object stores. That means that data scientists still need write two different computing jobs to access same data stored in different data systems. Apache Pulsar is the next generation messaging and streaming data system. It was originally built at Yahoo, and has graduated from Apache Incubator and become a Top-Level-Project. Pulsar separates messaging serving and data storage into two layers. Such layered architecture provides high throughput and low-latency while ensuring high availability and scalability. Pulsar’s segment centric storage design along with layered architecture makes Pulsar a perfect unbounded streaming data system, which can well fit into Flink’s computation model. In this talk, Sijie Guo from Apache Pulsar PMC, will introduce Pulsar and its layered architecture and segment-centric storage, detailing how this architecture can well integrate with Flink to provide elastic unified batch and stream processing.

Flink Forward San Francisco 2019: Elastic Data Processing with Apache Flink a...

Flink Forward

Airflow presentation

Ilias Okacha

In 2019 we presented “Secure Kafka at scale in true multi-tenant environment” at SFO Kafka summit. Back then, kafka was mainly used for event driven architectures, high-throughput pub/sub use cases and as a data-plane for log aggregation and for transporting metadata & metrics. A lot has changed since then - Kafka plant has grown to handle 400B incoming events in a day just in production, introduced stretch cluster pattern in addition to Active-Active cluster replication pattern. Moreover, new use cases have emerged in using Kafka for near real time stream processing and data moving pipelines across the cloud environments. Moving data in near-real time across the system is a hard problem to solve. Kafka Connect is a framework to stream data in/out of kafka reliably and can be used to achieve near-real time data moving pipelines. In this talk, we will present how kafka adoption has evolved over the last couple of years in our space and deep dive into how we approached in providing Managed Kafka Connect, a newest addition to our service portfolio.

Changing landscapes in data integration - Kafka Connect for near real-time da...

HostedbyConfluent

What do you do when you've two different technologies on the upstream and the downstream that are both rapidly being adopted industrywide? How do you bridge them scalably and robustly? At Wework, the upstream data was being brokered by Kafka and the downstream consumers were highly scalable gRPC services. While Kafka was capable of efficiently channeling incoming events in near real-time from a variety of sensors that were used in select Wework spaces, the downstream gRPC services that were user-facing were exceptionally good at serving requests in a concurrent and robust manner. This was a formidable combination, if only there was a way to effectively bridge these two in an optimized way. Luckily, sink Connectors came to the rescue. However, there weren't any for gRPC sinks! So we wrote one. In this talk, we will briefly focus on the advantages of using Connectors, creating new Connectors, and specifically spend time on gRPC sink Connector and its impact on Wework's data pipeline.

Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...

confluent

Was ist angesagt? (20)

cLoki: Like Loki but for ClickHouse

Fundamentals of Stream Processing with Apache Beam, Tyler Akidau, Frances Perry

Fast and Reliable Apache Spark SQL Engine

Apache Flink @ Alibaba - Seattle Apache Flink Meetup

Flink Forward San Francisco 2018: Ken Krugler - "Building a scalable focused ...

Apache Airflow (incubating) NL HUG Meetup 2016-07-19

From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...

Javier Lopez_Mihail Vieru - Flink in Zalando's World of Microservices - Flink...

Understanding and Improving Code Generation

Kafka as your Data Lake - is it Feasible? (Guido Schmutz, Trivadis) Kafka Sum...

Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day

IoT and Event Streaming at Scale with Apache Kafka

Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...

Flink Forward San Francisco 2018: Jörg Schad and Biswajit Das - "Operating Fl...

Apache Kafka, Tiered Storage and TensorFlow for Streaming Machine Learning wi...

Bitsy graph database

Flink Forward San Francisco 2019: Elastic Data Processing with Apache Flink a...

Airflow presentation

Changing landscapes in data integration - Kafka Connect for near real-time da...

Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...

Andere mochten auch

FEA Final Report - Jawanza Bassue

Jawanza Bassue

Presenter: Kenn Knowles, Software Engineer, Google & Apache Beam (incubating) PPMC member Apache Beam (incubating) is a programming model and library for unified batch & streaming big data processing. This talk will cover the Beam programming model broadly, including its origin story and vision for the future. We will dig into how Beam separates concerns for authors of streaming data processing pipelines, isolating what you want to compute from where your data is distributed in time and when you want to produce output. Time permitting, we might dive deeper into what goes into building a Beam runner, for example atop Apache Apex.

Apache Beam (incubating)

Apache Apex

http://flink-forward.org/kb_sessions/apache-beam-a-unified-model-for-batch-and-streaming-data-processing/ Unbounded, unordered, global-scale datasets are increasingly common in day-to-day business, and consumers of these datasets have detailed requirements for latency, cost, and completeness. Apache Beam (incubating) defines a new data processing programming model that evolved from more than a decade of experience within Google, including MapReduce, FlumeJava, MillWheel, and Cloud Dataflow. Beam handles both batch and streaming use cases and neatly separates properties of the data from runtime characteristics, allowing pipelines to be portable across multiple runtimes, both open-source (e.g., Apache Flink, Apache Spark, et al.) and proprietary (e.g., Google Cloud Dataflow). This talk will cover the basics of Apache Beam, touch on its evolution, describe main concepts in the programming model, and compare with similar systems. We’ll go from a simple scenario to a relatively complex data processing pipeline, and finally demonstrate execution of that pipeline on multiple runtimes.

Kenneth Knowles - Apache Beam - A Unified Model for Batch and Streaming Data...

Flink Forward

Presentación1

pipe-alejo

Shizuoka pref public_library(20101020)

真岡本

159267237 extractos-alquimicos

geogomes the best

06 asp.net session08

Niit Care

Otsuma(2010727)

真岡本

Mantenimiento de computo

sebassevitas

CacharreAndo

YcRF

03 asp.net session04

Niit Care

Presentación del sistema genesis y portal institucional j a

Idocaedro aplicacion

Presentación junio

simple present tense

07 asp.net session10

Mi materia preferida

Jadwal kegiatan ujian smk muhammadiyah 1

heri baskoro

02 asp.net session02

Niit Care

Introduction to Drools

giurca

Andere mochten auch (20)

FEA Final Report - Jawanza Bassue

Apache Beam (incubating)

Kenneth Knowles - Apache Beam - A Unified Model for Batch and Streaming Data...

Presentación1

Shizuoka pref public_library(20101020)

159267237 extractos-alquimicos

06 asp.net session08

Otsuma(2010727)

Mantenimiento de computo

CacharreAndo

03 asp.net session04

Presentación del sistema genesis y portal institucional j a

Idocaedro aplicacion

Presentación junio

simple present tense

07 asp.net session10

Mi materia preferida

Jadwal kegiatan ujian smk muhammadiyah 1

02 asp.net session02

Introduction to Drools

Ähnlich wie Apache Beam @ GCPUG.TW Flink.TW 20161006

Introduction to Apache Beam

Knoldus Inc.

The world of big data involves an ever-changing field of players. Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms. In a way, Apache Beam is a glue that can connect the big data ecosystem together; it enables users to "run any data processing pipeline anywhere." This talk will briefly cover the capabilities of the Beam model for data processing and discuss its architecture, including the portability model. We’ll focus on the present state of the community and the current status of the Beam ecosystem. We’ll cover the state of the art in data processing and discuss where Beam is going next, including completion of the portability framework and the Streaming SQL. Finally, we’ll discuss areas of improvement and how anybody can join us on the path of creating the glue that interconnects the big data ecosystem. Speaker Davor Bonaci, Apache Software Foundation; Simbly, V.P. of Apache Beam; Founder/CEO at Operiant

Present and future of unified, portable, and efficient data processing with A...

DataWorks Summit

http://flink-forward.org/kb_sessions/flink-and-beam-current-state-roadmap/ It is no secret that the Dataflow model, which evolved from Google’s MapReduce, Flume, and MillWheel, has been a major influence to Apache Flink’s streaming API. The essentials of this model are captured in Apache Beam. Beam provides the Dataflow API with the option to deploy to various backends (e.g. Flink, Spark). In this talk we will examine the current state of the Flink Runner. Beam’s Runners manage the translation of the Beam API into the backend API. The Beam project itself has made an effort to summarize the capabilities of each Runner to provide an overview of the supported API concepts. From all open sources backends, Flink is currently the Runner which supports the most features. We will look at the supported Beam features and their counterpart in Flink. Further, we will look at potential improvements and upcoming features of the Flink Runner.

Maximilian Michels - Flink and Beam

Flink Forward

Reactive Streams 1.0.0 is now live, and so are our implementations in Akka Streams 1.0 and Slick 3.0. Reactive Streams is an engineering collaboration between heavy hitters in the area of streaming data on the JVM. With the Reactive Streams Special Interest Group, we set out to standardize a common ground for achieving statically-typed, high-performance, low latency, asynchronous streams of data with built-in non-blocking back pressure—with the goal of creating a vibrant ecosystem of interoperating implementations, and with a vision of one day making it into a future version of Java. Akka (recent winner of “Most Innovative Open Source Tech in 2015”) is a toolkit for building message-driven applications. With Akka Streams 1.0, Akka has incorporated a graphical DSL for composing data streams, an execution model that decouples the stream’s staged computation—it’s “blueprint”—from its execution (allowing for actor-based, single-threaded and fully distributed and clustered execution), type safe stream composition, an implementation of the Reactive Streaming specification that enables back-pressure, and more than 20 predefined stream “processing stages” that provide common streaming transformations that developers can tap into (for splitting streams, transforming streams, merging streams, and more). Slick is a relational database query and access library for Scala that enables loose-coupling, minimal configuration requirements and abstraction of the complexities of connecting with relational databases. With Slick 3.0, Slick now supports the Reactive Streams API for providing asynchronous stream processing with non-blocking back-pressure. Slick 3.0 also allows elegant mapping across multiple data types, static verification and type inference for embedded SQL statements, compile-time error discovery, and JDBC support for interoperability with all existing drivers.

A Deeper Look Into Reactive Streams with Akka Streams 1.0 and Slick 3.0

Legacy Typesafe (now Lightbend)

Madrid Meetup

Sri Ambati

"Introduction to Sparkling Water" — Jakub Hava, Senior Software Engineer at H2O.ai. Jakub Hava enjoys dealing with problems and learning new programming languages. At H2O.ai, Kuba works on Sparkling Water project and its integration with the rest of the H2O.ai ecosystem. The speech includes a live demo showing how to create a Sparkling Water pipeline with H2O.ai's XGBoost model — no terminal needed, all we need is Jupyter!

"Introduction to Sparkling Water" — Jakub Hava, Senior Software Engineer, at ...

Provectus

DBCC 2021 - FLiP Stack for Cloud Data Lakes

Timothy Spann

In this paper we present an extensible software workbench for supporting the effective and dynamic prototyping of multimodal interactive systems. We hypothesize the construction of such applications to be based on the assembly of several components, namely various and sometimes interchangeable modalities at the input, fusion-fission components, and also several modalities at the output. Successful realization of advanced interactions can benefit from early prototyping and the iterative implementation of design requires the easy integration, combination, replacement, or upgrade of components. We have designed and implemented a thin integration platform able to manage these key elements, and thus provide the research community a tool to bridge the gap of the current support for multimodal applications implementation. The platform is included within a workbench offering visual editors, non-intrusive tools, components and techniques to assemble various modalities provided in different implementation technologies, while keeping a high level of performance of the integrated system.

An Open Source Workbench for Prototyping Multimodal Interactions Based on Off...

Jean Vanderdonckt

A Short Presentation on Kafka

Mostafa Jubayer Khan

開放原始碼 Ch1.2 intro - oss - apahce foundry (ver 2.0)

My own sweet home!

Apache flink

Janu Jahnavi

Apache Arrow at DataEngConf Barcelona 2018

Wes McKinney

Stream processing applications built on Apache Apex run on Hadoop clusters and typically power analytics use cases where availability, flexible scaling, high throughput, low latency and correctness are essential. These applications consume data from a variety of sources, including streaming sources like Apache Kafka, Kinesis or JMS, file based sources or databases. Processing results often need to be stored in external systems (sinks) for downstream consumers (pub-sub messaging, real-time visualization, Hive and other SQL databases etc.). Apex has the Malhar library with a wide range of connectors and other operators that are readily available to build applications. We will cover key characteristics like partitioning and processing guarantees, generic building blocks for new operators (write-ahead-log, incremental state saving, windowing etc.) and APIs for application specification.

BigDataSpain 2016: Stream Processing Applications with Apache Apex

Thomas Weise

Apache flink

Janu Jahnavi

Apache flink

Janu Jahnavi

http://www.learntek.org/product/apache-flink/ Apache Flink is an open source stream processing framework developed by the Apache Software Foundation. The core of Apache Flink is a distributed streaming dataflow engine written in Java and Scala. Apache Flink’s dataflow programming model provides event-at-a-time processing on both finite and infinite datasets. At a basic level, Flink programs consist of streams and transformations. Conceptually, a stream is a (potentially never-ending) flow of data records, and a transformation is an operation that takes one or more streams as input, and produces one or more output streams as a result. Programs can be written in Java, Scala, Python, and SQL and are automatically compiled and optimized into dataflow programs that are executed in a cluster or cloud environment. http://www.learntek.org Learntek is global online training provider on Big Data Analytics, Hadoop, Machine Learning, Deep Learning, IOT, AI, Cloud Technology, DEVOPS, Digital Marketing and other IT and Management courses. We are dedicated to designing, developing and implementing training programs for students, corporate employees and business professional.

Apache Flink Online Training

Learntek1

Cloud lunch and learn real-time streaming in azure

Timothy Spann

Seattle Spark Meetup Mobius CSharp API

shareddatamsft

E2E Data Pipeline - Apache Spark/Airflow/Livy

Rikin Tanna

Current & Future Use-Cases of OpenDaylight

abhijit2511

Ähnlich wie Apache Beam @ GCPUG.TW Flink.TW 20161006 (20)

Introduction to Apache Beam

Present and future of unified, portable, and efficient data processing with A...

Maximilian Michels - Flink and Beam

A Deeper Look Into Reactive Streams with Akka Streams 1.0 and Slick 3.0

Madrid Meetup

"Introduction to Sparkling Water" — Jakub Hava, Senior Software Engineer, at ...

DBCC 2021 - FLiP Stack for Cloud Data Lakes

An Open Source Workbench for Prototyping Multimodal Interactions Based on Off...

A Short Presentation on Kafka

開放原始碼 Ch1.2 intro - oss - apahce foundry (ver 2.0)

Apache flink

Apache Arrow at DataEngConf Barcelona 2018

BigDataSpain 2016: Stream Processing Applications with Apache Apex

Apache flink

Apache Flink Online Training

Cloud lunch and learn real-time streaming in azure

Seattle Spark Meetup Mobius CSharp API

E2E Data Pipeline - Apache Spark/Airflow/Livy

Current & Future Use-Cases of OpenDaylight

Kürzlich hochgeladen

Diamond Application Development Crafting Solutions with Precision

SolGuruz

Introducing MyIntelliAccount™ Cloud Accounting Software as a Service (SaaS) , the complete system for simplifying business accounting needs. MyIntelliAccount Cloud Accounting SaaS is an easy to understand and easy to use application and system designed for the Web with supporting applications for iOS and Android devices. Designed to work like a natural extension of your web browser, the user interface for MyIntelliAccount Cloud Accounting SaaS is intuitive and thoughtfully organized to help you easily navigate and access your business accounting information. Because our company takes the time to research, study, and understand the applications we develop, we are confident that once you use MyIntelliAccount Cloud Accounting SaaS , you will be asking yourself how you ever got along without it.

Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...

MyIntelliSource, Inc.

Conference: Engage2024 in Antwerp Type: Workshop Speakers: Florian Vogler, Henning Kunz, Christoph Adler Title: Navigating the Future with The Hitchhiker's Guide to Notes and Domino 14 Abstract: Embark on an exhilarating journey with industry trailblazers Florian Vogler, Henning Kunz, and Christoph Adler in this not-to-be-missed workshop at the forefront of the tech universe. Get ready for a thrilling kick-off as we navigate the current state of the HCL universe, setting the stage for an exploration of the groundbreaking Notes and Domino 14. Discover the latest enhancements and revolutionary features that will redefine your experience. In this interactive session, unlock a treasure trove of tips and tricks to elevate your utilization of version 14, both with and without the game-changing panagenda MarvelClient. Brace yourself for also diving into Nomad, Nomad Web, and VoltMX, expanding your horizons in the expansive HCL landscape. Be a part of this exclusive opportunity to stay ahead in the ever-evolving world of HCL technologies. Your journey to mastering Notes and Domino 14 begins here. And remember, in the spirit of intergalactic exploration, don't forget to bring your towel!

W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...

panagenda

Test automation is a cornerstone of software development and quality assurance in today's rapidly evolving digital landscape. Its significance cannot be overstated. Businesses can enhance efficiency, productivity, and accelerate software delivery to market through automation, streamlining testing processes effectively. This comprehensive guide addresses the best practices for test automation in 2024. It offers a detailed checklist to empower you to optimize your automation efforts and maintain a competitive edge.

The Ultimate Test Automation Guide_ Best Practices and Tips.pdf

kalichargn70th171

Software Quality Assurance Interview Questions

Arshad QA

5 Signs You Need a Fashion PLM Software.pdf

Wave PLM

How To Troubleshoot Collaboration Apps for the Modern Connected Worker

ThousandEyes

Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live Booking Contact Details :- WhatsApp Chat :- [+91-9999965857 ] The Best Call Girls Delhi At Your Service Russian Call Girls Delhi Doing anything intimate with can be a wonderful way to unwind from life's stresses, while having some fun. These girls specialize in providing sexual pleasure that will satisfy your fetishes; from tease and seduce their clients to keeping it all confidential - these services are also available both install and outcall, making them great additions for parties or business events alike. Their expert sex skills include deep penetration, oral sex, cum eating and cum eating - always respecting your wishes as part of the experience (29-April-2024(PSS)

Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live

Call Girls In Delhi Whatsup 9873940964 Enjoy Unlimited Pleasure

👉 𝑾𝑰𝑳𝑳 𝑷𝑹𝑶𝑽𝑰𝑫𝑬 𝒀𝑶𝑼 𝑾𝑰𝑻𝑯 𝑺𝑬𝑿𝒀 𝑴𝑶𝑫𝑬𝑳𝑺 𝑾𝑯𝑶 𝑾𝑰𝑳𝑳 𝑫𝑨𝑵𝑪𝑬 & 𝑫𝑹𝑰𝑵𝑲 𝑾𝑰𝑻𝑯 𝒀𝑶𝑼 𝑨𝑵𝑫 𝑨𝑳𝑺𝑶 𝑷𝑹𝑶𝑽𝑰𝑫𝑬 𝒀𝑶𝑼 𝑺𝑬𝑿𝑼𝑨𝑳 𝑩𝑶𝑫𝒀 𝑻𝑶 𝑩𝑶𝑫𝒀 𝑴𝑨𝑺𝑺𝑨𝑮𝑬 𝑾𝑰𝑻𝑯 𝑺𝑬𝑿. 👉𝒀𝑶𝑼 𝑴𝑨𝒀 𝑻𝑨𝑲𝑬 𝑻𝑯𝑬𝑴 𝑶𝑼𝑻 𝑭𝑶𝑹 𝑨 𝑷𝑨𝑹𝑻𝒀 𝑶𝑹 𝑨𝑳𝑺𝑶 𝑭𝑶𝑹 𝑨𝑵𝒀 𝑷𝑹𝑰𝑽𝑨𝑻𝑬 𝑷𝑨𝑹𝑻𝑰𝑬𝑺. 👉𝑻𝑯𝑬𝑺𝑬 𝑮𝑰𝑹𝑳𝑺 𝑨𝑹𝑬 𝑰𝑵𝑻𝑬𝑹𝑬𝑺𝑻𝑬𝑫 𝑰𝑵 𝑯𝑨𝑽𝑰𝑵𝑮 𝑺𝑶𝑴𝑬 𝑭𝑼𝑵 𝑾𝑰𝑻𝑯 𝒀𝑶𝑼 𝑨𝑵𝑫 𝑾𝑰𝑳𝑳 𝑬𝑵𝑺𝑼𝑹𝑬 𝑻𝑯𝑨𝑻 𝒀𝑶𝑼 𝑯𝑨𝑽𝑬 𝑪𝑶𝑴𝑷𝑳𝑬𝑻𝑬 𝑭𝑼𝑵. 28/April/2024 {{🎗️SCH🎗️}} 8923113531 How to book call girls The Booking process is particularly time-saving and more comfortable for us. Below mentioned are the steps you need to follow to hire our call girl: Step 1 - Visit our Call Girls Service website Step 2 - check for the portfolios of our sexy and hot call girls Step 3 - check on the services provided by the specific call girls Step 4 - Once you have selected the call girls and respective services, go to the contact us page. Step 5 - on the contact page, you will get our phone number, WhatsApp number, and email address. You can choose any one of them to connect with us. Call Us – 8923113531 ✣ ✤ ✥ ✦ 𝑻𝑰𝑴𝑬 𝑾𝑨𝑺𝑻𝑬𝑹𝑺 𝑨𝑵𝑫 𝑩𝑨𝑹𝑮𝑨𝑰𝑵𝑬𝑹𝑺 𝑨𝑹𝑬 𝑷𝑳𝑬𝑨𝑺𝑬 𝑬𝑿𝑪𝑼𝑺𝑬, 𝑾𝑬 𝑹𝑬𝑺𝑷𝑬𝑪𝑻 𝒀𝑶𝑼𝑹 𝑺𝑨𝑭𝑬𝑻𝒀 𝑨𝑵𝑫 𝑷𝑹𝑰𝑽𝑨𝑪𝒀 𝑨𝑵𝑫 𝑬𝑿𝑷𝑬𝑪𝑻 𝑻𝑯𝑬 𝑺𝑨𝑴𝑬 𝑭𝑹𝑶𝑴 𝒀𝑶𝑼.✣ ✤ ✥ ✦ Duration Normal Girls Housewives High-Profile Models (Celebrities) 1 Shot ₹ 8,000 ₹ 10,000 ₹ 15,000 2 Shots ₹ 15,000 ₹ 15,000 ₹ 25,000 Full Day ₹ 35,000 ₹ 35,000 ₹ 40,000 Full Night ₹ 40,000 ₹ 40,000 ₹ 50,000 Weekend ₹ 45,000 ₹ 45,000 ₹ 50,000 Party Nights ₹ 50,000 ₹ 50,000 ₹ 60,000

CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️

anilsa9823

(Vivek)Call Us, 8448380779,Call girls in Delhi NCr – We Offer best in class call girls. escort Service At Affordable Price At low Rate with Space Night 8000 We Are One Of The Oldest Escort and Call girls Agencies in Delhi. You Will Find That Our Female Escorts Are Full Of Fun, Sexy And They Would Love Enjoy Your Company. We Have A Fantastic Selection Of Escort Ladies Available For In-Calls As Well As Out-Calls. Our Escorts Are Not Only Beautiful But All Have Great Personalities Making Them The Perfect Companion For Any Occasion. In-Call:- You Can Come At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call:- You have To Come Pick The Girl From My Place We Are Also Provide Door Step Services (Delhi Ncr, Noida, Gurgaon, Faridabad, Ghaziabad Note:- Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic:- Full Ac room And Clean Rooms Available In Hotel 24 * 7 Hourly In Delhi NCR More Details, With WhatsApp Number, +91-8448380779

call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️

Delhi Call girls

In an era where security concerns are paramount, the integration of artificial intelligence (AI) into CCTV cameras has revolutionized surveillance capabilities. One of the most significant advancements is the ability to achieve real-time threat detection, enabling immediate responses to potential security breaches. This blog explores how AI is reshaping surveillance through real-time threat detection and the implications of this technology.

Optimizing AI for immediate response in Smart CCTV

shikhaohhpro

+971565801893 Mtp-Kit (500MG) Prices » Dubai [(+971565801893**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Leen Whatsapp +971565801893 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971565801893''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971565801893' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Clinic in Abu Dhabi, United Arab Emirates.+971565801893

+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...

Health

At TECUNIQUE, we're a stable and steadily growing Indian software services company with over 14 years of industry experience. Specializing in offshore software development and quality assurance services, we've built a reputation for delivering unique and effective solutions to start-ups, software development companies, enterprises, and digital agencies. We pride ourselves on our commitment to excellence and innovation. By blending insightful business domain knowledge with exceptional technical prowess, we craft tailor-made solutions that meet the unique needs of our clients. Our dedicated teams are adept in specific technologies, ensuring seamless integration of skills and delivering reliable, scalable, and high-quality software solutions aligned with our clients' preferences. Bespoke Dedicated Teams: Crafted to meet your specific needs and technology preferences, our dedicated teams are committed to delivering top-notch software solutions. Offshore Software Development: Accelerate your software development and scale up quickly with our 12+ years of expertise in offshore development. Quality Assurance Services: Ensure the quality of your software products with our dedicated teams of experienced QA professionals. IT Staff Augmentation: Overcome skill gaps with our client-centric software team, offering staff augmentation services. Expert Software Services: Unlock our capabilities in custom software development, product development, and quality assurance. Mission and Vision: Our mission at TECUNIQUE is to be the catalyst for our clients' success in the dynamic domain of software development. Rooted in our core values of respect, authenticity, and responsibility, we strive to ease the software outsourcing experience, reducing both time and cost to market for our clients. We envision ourselves as the leading Indian software services company, renowned for our unwavering commitment to excellence and innovation. www.tecunique.com

TECUNIQUE: Success Stories: IT Service provider

mohitmore19

A Secure and Reliable Document Management System is Essential.docx

ComplianceQuest1

Looking for an efficient way to manage your finances? Look no further than our money management app. With easy-to-use features, you can track your expenses, create budgets, and monitor your savings goals all in one place. Our app provides real-time updates on your spending habits and helps you make smarter financial decisions. Take control of your finances today with our user-friendly money management app.

Right Money Management App For Your Financial Goals

Jhone kinadey

Data spaces in distributed environments should be allowed to evolve in agile ways providing data space owners with large flexibility about which data they store. Agility and heterogeneity, however, jeopardize data exchanges because representations may build on varying ontologies and data consumers may not rely on the semantic correctness of their queries in the context of semantically heterogeneous, evolving data spaces. Graph data spaces are one example of a powerful model for representing and querying data whose semantics may change over time. To assert and enforce conditions on individual graph data spaces, shape languages (e.g SHACL) have been developed. We investigate the question of how querying and programming can be guarded by reasoning over SHACL constraints in a distributed setting and we sketch a picture of how a future landscape based on semantically heterogeneous data spaces might look like.

Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...

Steffen Staab

Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...

kellynguyen01

Investing in AI transformation today The modern business advantage: Uncovering deep insights with AI Organizations around the world have come to recognize AI as the transformative technology that enables them to gain real business advantage. AI’s ability to organize vast quantities of data allows those who implement it to uncover deep business insights, augment human expertise, drive operational efficiency, transform their products, and better serve their customers

Microsoft AI Transformation Partner Playbook.pdf

Willy Marroquin (WillyDevNET)

How To Use Server-Side Rendering with Nuxt.js

Andolasoft Inc

Document contains steps to getting ups and running quickly with MyTimeClock Employee Scheduling and Time Keeping Cloud Software as a Service Solution, Web version. Try MyTimeClock or any of our other software packages risk-free by registering for a FREE ACCOUNT at https://register.myintellisource.com/. If you would like more information about our company or its software, follow us on Facebook, Instagram, LinkedIn, Twitter, or YouTube, visit our home page at https://www.myintellisource.com/, or send us an email at cs@myintellisource.com. Take care and have a great day.

Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...

MyIntelliSource, Inc.

Kürzlich hochgeladen (20)

Diamond Application Development Crafting Solutions with Precision

Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...

W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...

The Ultimate Test Automation Guide_ Best Practices and Tips.pdf

Software Quality Assurance Interview Questions

5 Signs You Need a Fashion PLM Software.pdf

How To Troubleshoot Collaboration Apps for the Modern Connected Worker

Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live

CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️

call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️

Optimizing AI for immediate response in Smart CCTV

+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...

TECUNIQUE: Success Stories: IT Service provider

A Secure and Reliable Document Management System is Essential.docx

Right Money Management App For Your Financial Goals

Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...

Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...

Microsoft AI Transformation Partner Playbook.pdf

How To Use Server-Side Rendering with Nuxt.js

Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...

Apache Beam @ GCPUG.TW Flink.TW 20161006

1. Apache Beam in Data Pipeline Randy Huang  2016/10/06

2. Who am I • Data Architect @ VMFive • Fluentd/Embulk fans

3. Overview • Deﬁne Data Pipeline • Architecture • How to write Beam • Demo

4. Data Pipeline Input Algorithm Output

5. Why Apache Beam?

6. Data Pipeline’s world is chaos

7. Goal • Provide an abstraction layer between data processing’s code and the execution runtime. • Batch processing and Streaming Jobs in one world. • Beam SDK open the door to write once, run anywhere.* on-premise and non-Google cloud

8. Supported Runners • Google Cloud Dataﬂow (Block/Non-Blocking) • Apache Flink 1.1.2 • Apache Spark 1.6.2 Hadoop 2.2.0 Kafka 0.8.2.1

9. API, model, and engine

10. Architecture • Pipelines • Translators • Runners

11. programming tips/ Flink • Use the Flink DataStream API in Java and Scala • Use the Beam API directly in Java (and soon Python) with the Flink runner

12. SDK • Four Parts : • Pipeline : Streaming & Batch Processing • PCollection • Transform • I/O : Source & Sink

13. for Flink user • we encourage users to use either of the Beam or Flink APIs to implement their Flink jobs for stream data processing. • But Native Flink API - • backwards-compatible API • built-in libraries (e.g., CEP and upcoming SQL) • key-value state (with the ability to query that state in the future) http://data-artisans.com/why-apache-beam/

14. Demo • GDELT project • EventCount by Location Pileline

15. Recap • Write the general data pipeline, and choose your runner

16. Next… • New Runners, SDK (python still dev) • DSL

17. Another things • BigQuery have DML support!!! https://goo.gl/ lcZQVZ • DataStudio Beta in Taiwan is available • Embulk • Fluentd v0.14.6 - 2016/09/07

18. forward secure

19. remember to setup nginx

Apache Beam @ GCPUG.TW Flink.TW 20161006

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie Apache Beam @ GCPUG.TW Flink.TW 20161006

Ähnlich wie Apache Beam @ GCPUG.TW Flink.TW 20161006 (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Apache Beam @ GCPUG.TW Flink.TW 20161006