I²: Interactive Real-Time Visualization for Streaming Data with Apache Flink and Apache Zeppelin - Flink Forward Berlin 2017

•

2 gefällt mir•811 views

We present I², an interactive development environment for real-time analysis pipelines, which is based on Apache Flink and Apache Zeppelin. The sheer amount of available streaming data frequently makes it impossible to visualize all data points at the same time. I² coordinates running Flink jobs and corresponding visualizations such that only the currently depicted data points are processed in Flink and transferred towards the front end. We show how Flink jobs can adapt to changed visualization properties at runtime to allow interactive data exploration on high bandwidth data streams. Moreover, we present a data reduction technique which minimizes data transfer while providing loss free time-series plots. We show I² in a live demonstration in which we replay recorded sensor data from a football match (ca. 12k event/s). I² was first presented at EDBT'17 where it was awarded as best demonstration. The demonstration is available as open source at github.com/TU-Berlin-DIMA/i2.

Software

Berlin, Sep 11-13, 2017
I²
INTERACTIVE REAL-TIME VISUALIZATION FOR STREAMING
DATA WITH APACHE FLINK AND APACHE ZEPPELIN

It’s worth it to postpone checking your mails!
We connect Flink and Zeppelin to visualize data-streams in real-time.
We make front-end settings available to running Flink jobs.
We adapt running Flink jobs to visualization requirements.
We reduce the amount of processed and transferred data while providing loss-free plots.
i.e., we visualize 12.000 events per second without crashing the front end.
We enable visualization-driven development of Flink jobs.
Jonas Traub & Philipp Grulich | I²: Interactive real-time visualization for streaming data with Apache Flink and Apache Zeppelin | 2

I²
two types of interactivity
(i) through code changes (ii) through an interactive visualization GUI
change and deploy the code of analysis
pipelines and corresponding result
visualizations in a one-click fashion
change visualization properties
(e.g. the zoom level in a map) while the
underlying Flink job adapts at runtime
Rapid data-driven development
of data analysis pipelines
Reduces processed and transferred data
while still providing loss-free visualizations
Jonas Traub & Philipp Grulich | I²: Interactive real-time visualization for streaming data with Apache Flink and Apache Zeppelin | 3

Architecture Overview
Apache Zeppelin
Jonas Traub & Philipp Grulich | I²: Interactive real-time visualization for streaming data with Apache Flink and Apache Zeppelin | 4

I² - Live Demonstration
The DEBS 2013 Grand Challenge
Christopher Mutschler, Holger Ziekow, Zbigniew Jerzak; DEBS’13
Data:
● Sensor data from a football match
(speed, acceleration, and position of the ball and players)
● Up to 2000 Hz frequency
● roughly 12.000 data points per second
Jonas Traub & Philipp Grulich | I²: Interactive real-time visualization for streaming data with Apache Flink and Apache Zeppelin | 5

Demo
Jonas Traub & Philipp Grulich | I²: Interactive real-time visualization for streaming data with Apache Flink and Apache Zeppelin | 6
I²
Development
Environment

Visualization Front End
Jonas Traub & Philipp Grulich | I²: Interactive real-time visualization for streaming data with Apache Flink and Apache Zeppelin | 8
UI components push visualization
properties to the flink cluster
Stream only visible data points
towards the front end

Adaptive Operator Example: Adapt to Visualization Settings
Control Messages:
Updates to a filter threshold
Data Processing:
Filter according to
current threshold
Jonas Traub & Philipp Grulich | I²: Interactive real-time visualization for streaming data with Apache Flink and Apache Zeppelin | 10

Loss-free Visualization with Reduced Costs
VLDB
2014
Big data time series often contain more data per pixel than we can display.
reduce data to
4 values per pixel column
still provide a loss-free plot
Jonas Traub & Philipp Grulich | I²: Interactive real-time visualization for streaming data with Apache Flink and Apache Zeppelin | 12

Jonas Traub & Philipp Grulich | I²: Interactive real-time visualization for streaming data with Apache Flink and Apache Zeppelin | 13
I² introduces a streaming ready version of M4

The Impact of I² on the Visualization Front End
Jonas Traub & Philipp Grulich | I²: Interactive real-time visualization for streaming data with Apache Flink and Apache Zeppelin | 14

Thank you for joining our Session!
Jonas Traub & Philipp Grulich | I²: Interactive real-time visualization for streaming data with Apache Flink and Apache Zeppelin | 15
Summary
- Live visualization with Flink and Zeppelin
- Adapt jobs to changed setting at runtime
- Reduce data transfer and processing effort w/o quality-loss
- Support the development with live-visualization
Open Source This talk and I² are supported by the EU Horizon 2020 Projects
Proteus (687691) and Streamline (688191) and by the German
Ministry for Education and Research as Berlin Big Data Center
(01IS14013A) and Software Campus (01IS12056)

Weitere ähnliche Inhalte

Was ist angesagt?

Watch the webinar video on demand at: www.safe.com/webinars Find out how to quickly prepare LiDAR data to meet your requirements with FME, the leading technology for spatial data transformation. Through demonstrations, you'll see how you can easily perform coordinate system re-projection, format translation, and integration with GIS, CAD and raster data on millions or billions of points in seconds. We'll also share how the enhancements in FME 2012 make it even easier to get the most out of LiDAR data.

Prepare LiDAR Data To Meet Your Requirements

Safe Software

FUTURESTACK13: Software analytics with Project Rubicon from Alex Kroman Engin...

New Relic

工研院OMEGA ZONE自造者空間邀請在地各產業領域的你，透過空間的開源學習、場域實作、跨界分享，激盪出創新、創意、創業的無限可能。這次空間邀請到Maker PRO 哈爸來進行這次LASS（Location Aware Sensing System）「環境感測器網路系統」開源計畫的緣起與執行經驗，聽聽哈爸如何結合自造者運動，讓每個關心相同議題與擁有共同問題的人，透過各自專業與資源一起解決問題與共享成果。主講：哈爸/MakerPRO社群顧問

LASS 與網路公民科學-1

ITRI-OMEGA ZONE

Getting Started with FME 2017

Sterling Geo

1Spatial Australia: Introduction and getting started with fme 2017

1Spatial

Building a real time Tweet map with Flink in six weeks

Matthias Kricke

Jeremy Foran [BAI Communications] | Detecting Subway Overcrowding in Real Tim...

InfluxData

Seattle kafka meetup nov 2015 published siphon

Nitin Kumar

Boost dataviz with Python, OW2online, June 2020

OW2

Life in operations department can be chaotic - performance issues, outages, errors, requests from multiple sources, ongoing projects. It may be overwhelming, especially when you decide to migrate your core products to another provider in the meantime - AWS in our case. During this presentation, I’ll share with you our experience - how we went full scrum in ops team but I will not avoid topics related to technology behind this migration.

Agile sre fly beyond the clouds olx core went aws- Wojciech Krysmann

PROIDEA

Jovian Data Amazon Final Version

Satya Ramachandran

WHODIS_kearns_presentation.v0a

Edward Kearns

In this workshop we will set up a streaming framework which will process realtime data of traffic sensors installed within the Belgian road system. Starting with the intake of the data, you will learn best practices and the recommended approach to split the information into events in a way that won’t come back to haunt you. With some basic stream operations (count, filter, … ) you will get to know the data and experience how easy it is to get things done with Spring Boot & Spring Cloud Stream. But since simple data processing is not enough to fulfill all your streaming needs, we will also let you experience the power of windows. After this workshop, tumbling, sliding and session windows hold no more mysteries and you will be a true streaming wizard.

Stream Processing Live Traffic Data with Kafka Streams

Tim Ysewyn

Geo-Trending Example

David E Drummond

Hyperloglog Lightning Talk

Simon Prickett

Howtomakeyourown gi sdashboard

GeoMedeelel

Continuously Updating Query Results over Real-Time Linked Data

Ruben Taelman

PowerStream: Propelling Energy Innovation with Predictive Analytics

SingleStore

Pattern matching over event streams is increasingly being employed in many areas including financial services and click stream analysis. Flink, as a true stream processing engine, emerges as a natural candidate for these usecases. In this talk, we will present FlinkCEP, a library for Complex Event Processing (CEP) based on Flink. At the conceptual level, we will see the different patterns the library can support, we will present the main building blocks we implemented to support them, and we will discuss possible future additions that will further enhance the coverage of the library. At the practical level, we will show how the integration of FlinkCEP with Flink allows the former to take advantage of Flink's rich ecosystem (e.g. connectors) and its stream processing capabilities, such as support for event-time processing, exactly-once state semantics, fault-tolerance, savepoints and high throughput.

Flink Forward Berlin 2017: Kostas Kloudas - Complex Event Processing with Fli...

Flink Forward

Stream processing still evolves and changes at a speed that can make it hard to keep up with the developments. Being at the forefront of stream processing technology, the evolution of Apache Flink has mirrored many of these developments and continues to do so. We will take you on a journey through the major milestones of stream processing technology in past years, diving into the latest additions that Apache Flink and other communities introduced to the stream processing landscape, such as Streamng SQL, Time Versioned Tables, cluster-library-duality, language portability, etc. We will take a sneak peek into our crystal ball and present in what the Flink community is working on next.

The Past, Present, and Future of Apache Flink®

Aljoscha Krettek

Was ist angesagt? (20)

Prepare LiDAR Data To Meet Your Requirements

FUTURESTACK13: Software analytics with Project Rubicon from Alex Kroman Engin...

LASS 與網路公民科學-1

Getting Started with FME 2017

1Spatial Australia: Introduction and getting started with fme 2017

Building a real time Tweet map with Flink in six weeks

Jeremy Foran [BAI Communications] | Detecting Subway Overcrowding in Real Tim...

Seattle kafka meetup nov 2015 published siphon

Boost dataviz with Python, OW2online, June 2020

Agile sre fly beyond the clouds olx core went aws- Wojciech Krysmann

Jovian Data Amazon Final Version

WHODIS_kearns_presentation.v0a

Stream Processing Live Traffic Data with Kafka Streams

Geo-Trending Example

Hyperloglog Lightning Talk

Howtomakeyourown gi sdashboard

Continuously Updating Query Results over Real-Time Linked Data

PowerStream: Propelling Energy Innovation with Predictive Analytics

Flink Forward Berlin 2017: Kostas Kloudas - Complex Event Processing with Fli...

The Past, Present, and Future of Apache Flink®

Ähnlich wie I²: Interactive Real-Time Visualization for Streaming Data with Apache Flink and Apache Zeppelin - Flink Forward Berlin 2017

I²: Interactive Real-Time Visualization for Streaming Data

Jonas Traub

Measure your app internals with InfluxDB and Symfony2

Corley S.r.l.

Aljoscha Krettek offers a very short introduction to stream processing before diving into writing code and demonstrating the features in Apache Flink that make truly robust stream processing possible, with a focus on correctness and robustness in stream processing. All of this will be done in the context of a real-time analytics application that we’ll be modifying on the fly based on the topics we’re working though, as Aljoscha exercises Flink’s unique features, demonstrates fault recovery, clearly explains why event time is such an important concept in robust, stateful stream processing, and covers the features you need in a stream processor to do robust, stateful stream processing in production. We’ll also use a real-time analytics dashboard to visualize the results we’re computing in real time, allowing us to easily see the effects of the code we’re developing as we go along. Topics include: * Apache Flink * Stateful stream processing * Event time versus processing time * Fault tolerance * State management in the face of faults * Savepoints * Data reprocessing

Robust stream processing with Apache Flink

Aljoscha Krettek

Alan Pope [InfluxData] | Data Collectors | InfluxDays 2022

InfluxData

Flink Community Update 2015 June

Márton Balassi

Netflow Protocol

Wajid Hassan

Real time stock processing with apache nifi, apache flink and apache kafka

Timothy Spann

LWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis

Jonas Traub

This presentation is given by Brunon Dillenseger, Orange. Abstract: At the middle of this year, Orange switched its internal software forge (so-called Orange Forge) to a completely new hardware infrastructure and a new version of the Tuleap forge software. With more than 1000 connected users daily, among 16000 registered users worldwide, working on more than 5000 active projects, this switch was definitely a critical operation. Could this new brand new Orange Forge platform cope with such a traffic? If so, what about users' quality of experience? This is what we tried to know using OW2's load testing project CLIF. Feedback says it was a good idea to do so.

Benmarking Orange Forge with CLIF, OW2con'15, November 17, Paris

OW2

These slides contain the opening remarks and talk #2 from the first Seattle Apache Flink meetup which had the following talks. Date: Jan 17th, 2018, Wednesday Location: Bellevue, WA OPENING REMARKS (~5min) TALK #1 (~45min) Haitao Wang, Senior Staff Engineer at Alibaba, will give a presentation on large-scale streaming processing with Flink and Flink SQL at Alibaba and several internal use cases. See separate Slideshare: https://www.slideshare.net/dataArtisans/201801-seattle-apache-flink-meetup-talk-1-apache-flink-at-alibaba/edit TALK #2 (~30min) Bowen Li will talk about details of future meetup planning and logistics. He will also present how OfferUp, the largest mobile marketplace in the U.S., does large-scale stream processing with Flink to better serve local buyers and sellers, and what they have contributed to Flink's DataStream APIs, state backends, metrics system, and connectors. We may also talk about what's new in Flink 1.4 and how users can leverage these new features, and what Flink 1.5 would look like and what's users vision on Flink. SPONSOR: OfferUp Attendees included: Alibaba Group, OfferUp, Uber, Amazon Web Services, Google, Microsoft, Zions Bank, Gridpoint, Dell/EMC, NeoPrime, Nordstrom, Snowflake, Tableau, Oracle, Expedia, Grab, Snapchat, and many others.

2018-01 Seattle Apache Flink Meetup at OfferUp, Opening Remarks and Talk 2

Ververica

RIPE NCC Tools and Services - An Update

RIPE NCC

Apache flink

Ahmed Nader

For many industries the need to group together related events based on a period of activity or inactivity is key. Advertising businesses, content producers are just a few examples of where session windows can be used to better understand user behavior. While such sessionization has been possible in Apache Kafka up to this point, implementing it has been rather complex and required leveraging low-level APIs. In the most recent release of Kafka, however, new capabilities have been added making session windows much easier to implement. In this online talk, we’ll introduce the concept of a session window, talk about common use cases, and walk through how Apache Kafka can be used for session-oriented use cases.

user Behavior Analysis with Session Windows and Apache Kafka's Streams API

confluent

Experiences from a Field Test using ICN for Live Video Streaming

Ericsson

What does it take to go from no flow support, to handling huge volumes of heterogeneous flow data in a 100% open-source monitoring stack, in a real-world environment? Expect a brief refresher on flows, an overview of the customer environment, and discussion of the engineering challenges faced. A medium dive follows into the movement of flow data from ingest to query and display, the solution architecture as it exists today, and lessons learned and their application to the project roadmap.

OSMC 2021 | Handling 250K flows per second with OpenNMS: a case study

NETWAYS

Apache flink

Janu Jahnavi

Apache NiFi Crash Course - San Jose Hadoop Summit

Aldrin Piri

Using Apache Kafka to Analyze Session Windows

confluent

Stream processing with Apache Flink @ OfferUp

Bowen Li

As a data scientist I frequently need to create web apps to provide interactive functionality, deliver data APIs or simply publish results. It is now easier than ever to deploy your data driven web app by using cloud based application platforms to do the heavy lifting. Cloud Foundry (http://cloudfoundry.org) is an open source public and private cloud platform that enables simple app deployment, scaling and connectivity to data services like PostgreSQL, MongoDB, Redis and Cassandra. Resources: http://www.ianhuston.net/2015/01/cloud-foundry-for-data-science-talk/

Cloud Foundry for Data Science

Ian Huston

Ähnlich wie I²: Interactive Real-Time Visualization for Streaming Data with Apache Flink and Apache Zeppelin - Flink Forward Berlin 2017 (20)

I²: Interactive Real-Time Visualization for Streaming Data

Measure your app internals with InfluxDB and Symfony2

Robust stream processing with Apache Flink

Alan Pope [InfluxData] | Data Collectors | InfluxDays 2022

Flink Community Update 2015 June

Netflow Protocol

Real time stock processing with apache nifi, apache flink and apache kafka

LWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis

Benmarking Orange Forge with CLIF, OW2con'15, November 17, Paris

2018-01 Seattle Apache Flink Meetup at OfferUp, Opening Remarks and Talk 2

RIPE NCC Tools and Services - An Update

Apache flink

user Behavior Analysis with Session Windows and Apache Kafka's Streams API

Experiences from a Field Test using ICN for Live Video Streaming

OSMC 2021 | Handling 250K flows per second with OpenNMS: a case study

Apache flink

Apache NiFi Crash Course - San Jose Hadoop Summit

Using Apache Kafka to Analyze Session Windows

Stream processing with Apache Flink @ OfferUp

Cloud Foundry for Data Science

Mehr von Jonas Traub

This talk was presented at code.talks 2022 in Hamburg. Learn more about code.talks at: https://codetalks.de/ Download Slides at: https://codetalks.de/speakers#talk-1128?event=7 Video recording on Youtube: https://www.youtube.com/watch?v=GC-hKOvQBus In this session, we will provide an introduction to REASON, also known as ReasonML. At SAP, we use REASON to build our Master Data Integration (MDI) Service as part of SAPs Business Technology Platform. MDI is a highly scalable, distributed, multi-tenant system powered by functional programming. REASON, is our language of choice for MDI: It lets you write simple, fast, and type save code. With REASON, you rarely have to annotate types, but everything gets checked for you through REASON’s type inference system, reducing bugs and increasing code maintainability. REASON translates to OCaml or JavaScript, providing access to both worlds and enabling it to run on NodeJS. During the session, we will highlight the advantages of the syntax compared to Java and introduce you to the beauty of immutability, tail recursion and other relevant concepts. Our session targets programmers who are not yet familiar with functional programming, as well as functional programmers who want to learn more about REASON.

Definitely not Java! A Hands-on Introduction to Efficient Functional Programm...

Jonas Traub

This talk was presented for the SoftwareCampus Alumni e.V. on 07.12.2020. For more Information about the program check https://softwarecampus-alumni.de/ and https://softwarecampus.de/ Abstract: The Internet of Things (IoT) consists of billions of devices which form a cloud of network-connected sensor nodes. These sensor nodes supply a vast number of data streams with massive amounts of sensor data. Real-time sensor data enables diverse applications including traffic-aware navigation, machine monitoring, and home automation. In this talk, we will dive into recent research which optimizes real-time data gathering and data analysis in the IoT. The talk will provide an overview of available techniques which can be deployed on sensor nodes, intermediate network nodes, and central analysis systems. We will look into the state-of-the-art in practice and research and make you aware of important tradeoffs in real-time IoT data analysis. CV: Jonas Traub is a postdoctoral researcher at the Database Systems and Information Management group at TU Berlin. His main research interests include stream processing, sensor data analysis, and data acquisition techniques. In his PhD, he studied efficient data gathering, processing, and transmission in the IoT. His research shows that one can save up to 87% in sensor reads and data transfers by applying smart data reduction techniques on sensor nodes. He further introduced a demand-based control layer which optimizes the data acquisition from thousands of sensors. With his Scotty-framework, he contributed a general aggregation technique for streaming systems which outperforms alternative solutions by an order of magnitude in throughput. His work received a Best Paper Award at the 22nd International Conference on Extending Database Technology (EDBT). Prior to his work at TU Berlin, he studied at KTH Stockholm and DHBW Stuttgart and worked several years at IBM in Germany and the USA. Jonas is an alumnus of Software Campus where he worked with SAP as industry partner.

Efficient Data Stream Processing in the Internet of Things - SoftwareCampus A...

Jonas Traub

This presentation was held at Code.Talks 2019 in Hamburg. A video is available at: https://www.youtube.com/watch?v=K1y5dJvP1jM Window aggregation is a core operation in data stream processing. Stream Processing Systems, like Flink or Storm, implement general aggregation techniques which perform poorly under specific workloads (e.g. Sliding Windows). To this end, we present Scotty, a new highly-efficient window operator. Scotty exploits specific workload properties such as the type of aggregation functions (e.g., invertible, associative), window types (e.g., sliding, sessions), windowing measures (e.g., time- or countbased), and stream (dis)order. This allows Scotty to outperform systems like Flink by up to one order of magnitude. The structure of this talk is threefold: First, we give an introduction to the semantics and implementations of window aggregations in modern Stream Processing Systems. Second, we discuss the design of Scotty and show why Scotty is able to outperform the default window operators of many stream processing systems. Third, we give a hands-on introduction to Scotty and demonstrate how it can be integrated into standard Flink, Storm, or Beam stream processing pipelines. Scotty and its connectors are available as open-source (https://github.com/TU-Berlin-DIMA/scotty-window-processor) and contributions are highly welcome.

code.talks 2019 - Scotty: Efficient Window Aggregation for your Stream Proces...

Jonas Traub

Window aggregation is a core operation in data stream processing. Existing aggregation techniques focus on reducing latency, eliminating redundant computations, and minimizing memory usage. However, each technique operates under different assumptions with respect to workload characteristics such as properties of aggregation functions (e.g., invertible, associative), window types (e.g., sliding, sessions), windowing measures (e.g., time- or countbased), and stream (dis)order. Violating the assumptions of a technique can deem it unusable or drastically reduce its performance. In this talk, we present Scotty an implementation of a general stream slicing technique for window aggregation. This technique automatically adapts to workload characteristics to improve performance without sacrificing its general applicability. Our experiments show that Scotty outperforms alternative implementations, like the default window operator in Flink, by up to one order of magnitude. Furthermore, we present how to use Scotty as a library in Flink, Storm, or Beam without changing the underlying Stream Processing System. General stream slicing was first published at EDBT 2019 (http://www.user.tu-berlin.de/powibol/assets/publications/traub-efficient-window-aggregation-with-general-stream-slicing-edbt-2019.pdf) where it received the Best Paper Award. The Scotty library and its connectors are available as open-source (https://github.com/TU-Berlin-DIMA/scotty-window-processor) and contributions are highly welcome.

FlinkForward Berlin 2019 - Scotty: Efficient Window Aggregation with General ...

Jonas Traub

This talk by Steffen Zeuch presents our VLDB 2019 paper about "Analyzing Efficient Stream Processing on Modern Hardware". This paper is about showing the potential of hardware-tailored code compilation and data ingestion at memory speed for a scale-up SPE. Analyze state-of-the-art streaming systems and identify sources of inefficiency. We investigate the data-related and processing-related design space and derive design changes for streaming systems to exploit modern hardware more efficiently. In order to efficiently scale up, one should avoid managed runtimes, use a compilation-based approach to produce hardware-tailored code, avoid queues and use operator fusion, and use late merge instead of partitioning. The full paper with all our findings is available online: http://www.vldb.org/pvldb/vol12/p516-zeuch.pdf

Analyzing Efficient Stream Processing on Modern Hardware (VLDB 2019 Presentat...

Jonas Traub

In April 2019, we did an USA excursion and presented selected publications of the TU Berlin DIMA and the DFKI IAM research groups. This slide set contains the four teaser talks which we presented on the tour: 1) Jonas Traub: Optimized On-Demand Data Streaming from Sensor Nodes 2) Sebastian Breß: Generating Custom Code for Efficient Query Execution on Heterogeneous Processors 3) Martin Kiefer: Estimating Join Selectivities using Bandwidth Optimized Kernel Density Models 4) Andreas Kunft: BlockJoin: Efficient Matrix Partitioning through Joins

Database Research at TU Berlin DIMA and DFKI IAM - USA Excursion Slides 2019

Jonas Traub

The Paper "Efficient Window Aggregation with General Stream Slicing" by Jonas Traub, Philipp M. Grulich, Alejandro Rodriguez Cuellar, Sebastian Breß, Tilmann Rabl, and Volker Markl was selected as best paper of the International Conference on Extending Database Technology (EDBT) 2019. Abstract: Window aggregation is a core operation in data stream processing. Existing aggregation techniques focus on reducing latency, eliminating redundant computations, and minimizing memory usage. However, each technique operates under different assumptions with respect to workload characteristics such as properties of aggregation functions (e.g., invertible, associative), window types (e.g., sliding, sessions), windowing measures (e.g., time- or countbased), and stream (dis)order. Violating the assumptions of a technique can deem it unusable or drastically reduce its performance. In this paper, we present the first general stream slicing technique for window aggregation. General stream slicing automatically adapts to workload characteristics to improve performance without sacrificing its general applicability. As a prerequisite, we identify workload characteristics which affect the performance and applicability of aggregation techniques. Our experiments show that general stream slicing outperforms alternative concepts by up to one order of magnitude.

Efficient Window Aggregation with General Stream Slicing (EDBT 2019, Best Paper)

Jonas Traub

Resense: Transparent Record and Replay of Sensor Data in the Internet of Things Resense was presented as Demonstration at the 22nd International Conference on Extending Database Technology (EDBT) in Lisbon, Portugal and received the Best Demonstration Award (http://edbticdt2019.inesc-id.pt/?awards_demo_edbt). AUTHORS: Dimitrios Giouroukis, Julius Hülsmann, Janis von Bleichert, Morgan Geldenhuys, Tim Stullich, Felipe Oliveira Gutierrez, Jonas Traub, Kaustubh Beedkar, Volker Markl ABSTRACT As the scientific interest in the Internet of Things (IoT) continues to grow, emulating IoT infrastructure involving a large number of heterogeneous sensors plays a crucial role. Existing research on emulating sensors is often tailored to specific hardware and/or software, which makes it difficult to reproduce and extend. In this paper we show how to emulate different kinds of sensors in a unified way that makes the downstream application agnostic as to whether the sensor data is acquired from real sensors or is read from memory using emulated sensors. We propose the Resense framework that allows for replaying sensor data using emulated sensors and provides an easy-to-use software for setting up and executing IoT experiments involving a large number of heterogeneous sensors. We demonstrate various aspects of Resense in the context of a sports analytics application using real-world sensor data and a set of Raspberry Pis.

Resense: Transparent Record and Replay of Sensor Data in the Internet of Thin...

Jonas Traub

Computing aggregates over windows is at the core of virtually every stream processing job. Typical stream processing applications involve overlapping windows and, therefore, cause redundant computations. Several techniques prevent this redundancy by sharing partial aggregates among windows. However, these techniques do not support out-of-order processing and session windows. Out-of-order processing is a key requirement to deal with delayed tuples in case of source failures such as temporary sensor outages. Session windows are widely used to separate different periods of user activity from each other. Current versions of Apache Flink use Window Buckets to process stream aggregations with session windows and out-of-order tuples. This Approach does not share partial aggregates among overlapping windows. In our talk, we present Scotty, a high throughput operator for window discretization and aggregation in Apache Flink. Scotty splits streams into non-overlapping slices and computes partial aggregates per slice. These partial aggregates are shared among all overlapping windows including session windows. Scotty introduces the first slicing technique which (1) enables stream slicing for session windows in addition to tumbling and sliding windows and (2) processes out-of-order tuples efficiently. Scotty was first published at ICDE 2018 (http://www.user.tu-berlin.de/powibol/assets/publications/traub-scotty-icde-2018.pdf).

Flink Forward 2018: Efficient Window Aggregation with Stream Slicing

Jonas Traub

This poster was presented at ICDE 2018. Abstract: Computing aggregates over windows is at the core of virtually every stream processing job. Typical stream processing applications involve overlapping windows and, therefore, cause redundant computations. Several techniques prevent this redundancy by sharing partial aggregates among windows. However, these techniques do not support out-of-order processing and session windows. Out-of-order processing is a key requirement to deal with delayed tuples in case of source failures such as temporary sensor outages. Session windows are widely used to separate different periods of user activity from each other. In this paper, we present Scotty, a high throughput operator for window discretization and aggregation. Scotty splits streams into non-overlapping slices and computes partial aggregates per slice. These partial aggregates are shared among all concurrent queries with arbitrary combinations of tumbling, sliding, and session windows. Scotty introduces the first slicing technique which (1) enables stream slicing for session windows in addition to tumbling and sliding windows and (2) processes out-of-order tuples efficiently. Our technique is generally applicable to a broad group of dataflow systems which use a unified batch and stream processing model. Our experiments show that we achieve a throughput an order of magnitude higher than alternative state-of-the-art solutions.

Scotty: Efficient Window Aggregation for Out-of-Order Stream Processing

Jonas Traub

Paper: Scalable Detection of Concept Drifts on Data Streams with Parallel Adaptive Windowing Abstract: Machine learning techniques for data stream analysis suffer from concept drifts such as changed user preferences, varying weather conditions, or economic changes. These concept drifts cause wrong predictions and lead to incorrect business decisions. Concept drift detection methods such as adaptive windowing (Adwin) allow for adapting to concept drifts on the fly. In this paper, we examine Adwin in detail and point out its throughput bottlenecks. We then introduce several parallelization alternatives to address these bottlenecks. Our optimizations lead to a speedup of two orders of magnitude over the original Adwin implementation. Thus, we explore parallel adaptive windowing to provide scalable concept detection for high-velocity data streams with millions of tuples per second.

Scalable Detection of Concept Drifts on Data Streams with Parallel Adaptive W...

Jonas Traub

This poster was presented at the 21st International Conference on Extending Database Technology (EDBT), March 26-29, 2018. Paper: Efficient SIMD Vectorization for Hashing in OpenCL Abstract: Hashing is at the core of many efficient database operators such as hash-based joins and aggregations. Vectorization is a technique that uses Single Instruction Multiple Data (SIMD) instructions to process multiple data elements at once. Applying vectorization to hash tables results in promising speedups for build and probe operations. However, vectorization typically requires intrinsics – low-level APIs in which functions map to processorspecific SIMD instructions. Intrinsics are specific to a processor architecture and result in complex and difficult to maintain code. OpenCL is a parallel programming framework which provides a higher abstraction level than intrinsics and is portable to different processors. Thus, OpenCL avoids processor dependencies, which results in improved code maintainability. In this paper, we add efficient, vectorized hashing primitives to OpenCL. Our results show that OpenCL-based vectorization is competitive to intrinsics on CPUs but not on Xeon Phi coprocessors.

Efficient SIMD Vectorization for Hashing in OpenCL

Jonas Traub

About the Workshop: The Stream Reasoning Workshop took place from January 16th to 17th, 2018. Processing, querying and reasoning over streaming data is studied in different communities such as KR&R, Semantic Web, Databases, Stream Processing, Complex Event Processing, etc., where researchers have different perspectives and face different challenges. This workshop aims at advancing Stream Reasoning as research theme by bringing together these different views and goals. In addition to invited talks, the workshop will provide opportunities for all participants to engage in discussions on open problems and future directions. (http://www.ifi.uzh.ch/en/ddis/events/streamreasoning2018.html) About the Talk: Real-time sensor data enables diverse applications such as smart metering, traffic monitoring, and sport analysis. In the Internet of Things, billions of sensor nodes form a sensor cloud and offer data streams to analysis systems. However, it is impossible to transfer all available data with maximal frequencies to all applications. Therefore, we need to tailor data streams to the demand of applications. We contribute a technique that optimizes communication costs while maintaining the desired accuracy. Our technique schedules reads across huge amounts of sensors based on the data-demands of a huge amount of concurrent queries. We introduce user-defined sampling functions that define the data-demand of queries and facilitate various adaptive sampling techniques, which decrease the amount of transferred data. Moreover, we share sensor reads and data transfers among queries. Our experiments with real-world data show that our approach saves up to 87% in data transmissions.

UZH Stream Reasoning Workshop 2018: Optimized On-Demand Data Streaming from S...

Jonas Traub

JT@UCSB - On-Demand Data Streaming from Sensor Nodes and A quick overview of ...

Jonas Traub

LWA 2015: The Apache Flink Platform (Poster)

Jonas Traub

Mehr von Jonas Traub (15)