I²: Interactive Real-Time Visualization for Streaming Data

•

0 gefällt mir•216 views

This is our poster for the demonstration "I²: Interactive Real-Time Visualization for Streaming Data" which was awarded as best demonstration at EDBT 2017. The paper and the source code are available on GitHub: https://github.com/TU-Berlin-DIMA/i2

Software

github.com/TU-Berlin-DIMA/i2
hub.docker.com/r/tuberlindima/i2I²
I²: Interactive Real-Time Visualization for Streaming Data
Technische Universität Berlin (dima.tu-berlin.de)
Jonas Traub
jonas.traub@tu-berlin.de
Nikolaas Steenbergen
nikolaas.steenbergen@dfki.de
Philipp Grulich
philipp.grulich@dfki.de
Tilmann Rabl
rabl@tu-berlin.de
Volker Markl
volker.markl@tu-berlin.de
Try it! - It’s all open source!
Interactive Development Interactive Visualization
German Research Center for Artificial Intelligence (dfki.de)
Apache Flink – flink.apache.org
Apache Zeppelin – zeppelin.apache.org
I² - Two Types of Interactivity
Change your program and deploy your updates with just one click.
Develop real-time data visualizations while operating on live data.
Explore live data in visualizations. The underlying cluster job adapts at
runtime to your settings and sends the required data to the dashboard.
Adaptive Data
Stream Processing
Pipelines in
Apache Flink .
Development and
Live-Visualization
in Apache Zeppelin
Notebooks.
Example Dashboard:
– Sensor data from a
football match.
– Adaptive Flink job.
– Interactivity:
– player selection.
– different metrics.
– range of the
depicted history.
Architecture Overview
Efficient Real-Time Visualization of Time Series Data Adaptive Flink Operators
CPU UtilizationFrame Rate
Performance Evaluation
I² seamlessly connects live data visualization with the development
of analysis pipelines for streaming data.
1. Develop stream analysis pipelines and visualizations.
2. Deploy your code with just one click.
3. Discover the incoming live data.
I² observes visualization properties and adapts the Flink job at runtime.
The visualization no longer suffers from massive ingestion rates.
We provide runtime adaptive operators.
Example: A runtime adaptive filter
operator for variable thresholds.
1. There is a trade off
between the length of
the depicted history and
visualization precision
(pixel columns per time).
2. We need exactly four
data points per pixel column
to provide a loss-free plot of
time series data.
[M4, Jugel et al., VLDB’14]
– Transfer four values per pixel column.
– Constant workload at the front end.
– The front end is independent from the
ingestion rate at the Flink cluster.
Without I²:
Unresponsive dashboard shortly
after start-up (CPU overload).
With I²:
Constant 60Hz frame rate.
Without I²:
CPU cannot keep up with the
massive ingestion rates.
With I²:
Reduced and constant CPU load.
This project has received funding from the European Union’s
Horizon 2020 research and innovation program under grant
agreement No 688191.
Software Campus
(01IS12056) & BBDC
(01IS14013A)

Weitere ähnliche Inhalte

Was ist angesagt?

Using Trimble TX9 terrestrial laser scanner my surveying team scanned an active runway in Western Australia to a tolerance spec of 3mm. The teams were working a live site providing aircraft right of way meant the teams had to setup and takedown scanner and targets to yield to any aircraft movements around the site and airspace. Surveyors took a ground based approach over drone UAS to maintain tighter vertical control than can be achieved using drone capture. We were tasked with looking for deviations, rutting and areas to derive Pavement Condition Index (PCI) criteria for their asset. Once the data was captured surveying teams utilized TopoDot to assemble the raw scans into a consolidated model. They then attempted to use the pavement roughness algorithms in the software against the close to 3.4b points of classified data but had to split the datasets into halves and quads in order for the processing runs to complete. The Bentley product has an inbuilt “Road condition tool” which reports on pavement roughness characteristics but has preset expected pavement widths, roads not runway widths, set in the software. We explained to our surveyors that the algorithms might run faster in another product. It allowed us to explore FME as a point cloud processing workflow using feature tables functionality to quickly generate the statistics required for reporting deliverables using the entire dataset in one process.

How Rough Is Your Runway?

Safe Software

Real time-collaborative-editor-presentation

bflueras

Get productive with python Visual Studio 2019

Taswar Bhatti

Managing Floorplans with AutoCAD and ArcGIS Using FME

Lecture15 fsm i_ic

Project_abstract

ToDoList

Ict 2019 v2

Dev Games!

WTF is Modeling, Anyway!?

Neil Gunther

21st International Workshop on“Single Molecule Spectroscopy and Super-resolu...

Dirk Hähnel

RAW GNSS in Android Nugat

Lukasz Kosma Bonenberg

(slides 4) Visual Computing: Geometry, Graphics, and Vision

Frank Nielsen

Denis Reznik Data driven future

Аліна Шепшелей

Was ist angesagt? (14)

How Rough Is Your Runway?

Real time-collaborative-editor-presentation

Get productive with python Visual Studio 2019

Managing Floorplans with AutoCAD and ArcGIS Using FME

Lecture15 fsm i_ic

Project_abstract

ToDoList

Ict 2019 v2

Dev Games!

WTF is Modeling, Anyway!?

21st International Workshop on“Single Molecule Spectroscopy and Super-resolu...

RAW GNSS in Android Nugat

(slides 4) Visual Computing: Geometry, Graphics, and Vision

Denis Reznik Data driven future

Ähnlich wie I²: Interactive Real-Time Visualization for Streaming Data

Aoyagi Lab Colloquium - 2015-06-01

Michele Bianchi

Inria Tech Talk : Améliorez vos applications de robotique & réalité augmentée

Stéphanie Roger

We present I², an interactive development environment for real-time analysis pipelines, which is based on Apache Flink and Apache Zeppelin. The sheer amount of available streaming data frequently makes it impossible to visualize all data points at the same time. I² coordinates running Flink jobs and corresponding visualizations such that only the currently depicted data points are processed in Flink and transferred towards the front end. We show how Flink jobs can adapt to changed visualization properties at runtime to allow interactive data exploration on high bandwidth data streams. Moreover, we present a data reduction technique which minimizes data transfer while providing loss free time-series plots. We show I² in a live demonstration in which we replay recorded sensor data from a football match (ca. 12k event/s). I² was first presented at EDBT'17 where it was awarded as best demonstration. The demonstration is available as open source at github.com/TU-Berlin-DIMA/i2.

Flink Forward Berlin 2017: Jonas Traub, Philipp Grulich - I²: Interactive rea...

Flink Forward

I²: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...

Jonas Traub

This presentation is given by Brunon Dillenseger, Orange. Abstract: At the middle of this year, Orange switched its internal software forge (so-called Orange Forge) to a completely new hardware infrastructure and a new version of the Tuleap forge software. With more than 1000 connected users daily, among 16000 registered users worldwide, working on more than 5000 active projects, this switch was definitely a critical operation. Could this new brand new Orange Forge platform cope with such a traffic? If so, what about users' quality of experience? This is what we tried to know using OW2's load testing project CLIF. Feedback says it was a good idea to do so.

Benmarking Orange Forge with CLIF, OW2con'15, November 17, Paris

OW2

Advanced view of projects raspberry pi list raspberry pi projects

WiseNaeem

Using OSGi for the Realization of Complex Building Management Systems - Peter...

mfrancis

Presentation of glpi project, OW2con'19, June 12-13, Paris.

OW2

This issue’s feature article, Tuning Autonomous Driving Using Intel® System Studio, illustrates how the tools in Intel System Studio give embedded systems and connected device developers an integrated development environment to build, debug, and tune performance and power usage. Continuing the theme of tuning edge applications, Building Fast Data Compression Code for Cloud and Edge Applications shows how to use the Intel® Integrated Performance Primitives to speed data compression.

Parallel universe-issue-29

DESMOND YUEN

Continuous Delivery to the Cloud: Automate Thru Production with CI + Spinnaker

VMware Tanzu

Fuzzing malware for fun & profit. Applying Coverage-Guided Fuzzing to Find Bu...

Maksim Shudrak

The Italian Institute for Nuclear Physics (INFN) has a long experience in the field of distributed scientific computing, mainly in the framework of GRID computing. In the last years an interest towards the cloud computing paradigm has arisen inside the INFN scientific and technological communities, leading to the growth of new activities aiming at creating a new distributed computing environment that takes advantage of the flexibility offered by cloud technologies. In this contribution we will give an overview of the activities carried out by the INFN IT community in this direction leveraging OpenStack as the main Cloud Management Framework (provider). We will present the two aspects of the INFN - OpenStack venture: - setup and operation of OpenStack based infrastructures: the challenging set-ups and peculiar characteristics of local and geographically distributed cloud infrastructures present in various INFN sites, as well as the prototype of the multi-site INFN Corporate Cloud infrastructure - a geographically distributed cloud environment, fully redundant and highly available, hosted in a limited number of INFN sites; - development of new and/or improvements of already existing OpenStack components in order to: support federated identities and provide privacy and distributed authorization move beyond static allocation and partitioning of both storage and computing resources in data centers distribute and deploy applications in a flexible way exploit distributed computing and storage resources through transparent network interconnections The presentation will describe the national and international projects in which INFN is involved, highlighting their objectives and the solutions adopted, the work done and achieved results as well as future steps and related cloud activities involving the OpenStack community.

Infn os summit_2015_v1.0

caifti

Preserving CAD: A briefing - Emerging trends - Architecture

Ruggero Lancia

IoT and Practical Laboratories

Universidad Nacional de Educación a Distancia

I/O Challenges in Brain Tissue Simulation

inside-BigData.com

Generally a network simulator is used to analyse the performance and behaviour of a network. The Simulation software plays a vital role in real world implementation. The hardware setup of network topologies are very costly and strenuous to modify often. The simulators act as the protocols for a system. The simulators in network such as Ns 2, Ns 3, OMNeT , NetSim, J SIM, REAL, OPNET, OMNEST, QualNetTraNS, NTCUns etc. It is quite a tedious process to select network simulator that is based up on the requirement for a users specified job. This paper gives a comparison and a general analysis of various network simulators. J. Sumitha | S. Swathi | Ellakkiya. M "Simulations on Computer Network: An Improved Study in the Simulator Methodologies and Their Applications" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-5 | Issue-4 , June 2021, URL: https://www.ijtsrd.compapers/ijtsrd41176.pdf Paper URL: https://www.ijtsrd.comcomputer-science/simulation/41176/simulations-on-computer-network-an-improved-study-in-the-simulator-methodologies-and-their-applications/j-sumitha

Simulations on Computer Network An Improved Study in the Simulator Methodolog...

YogeshIJTSRD

Measure your app internals with InfluxDB and Symfony2

Corley S.r.l.

Nt1320 Unit 6

Ashley Jean

apidays LIVE Helsinki & North 2022_Apps without APIs

apidays

Presentation1.2.pptx

pranaykusuma

Ähnlich wie I²: Interactive Real-Time Visualization for Streaming Data (20)

Aoyagi Lab Colloquium - 2015-06-01

Inria Tech Talk : Améliorez vos applications de robotique & réalité augmentée

Flink Forward Berlin 2017: Jonas Traub, Philipp Grulich - I²: Interactive rea...

I²: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...

Benmarking Orange Forge with CLIF, OW2con'15, November 17, Paris

Advanced view of projects raspberry pi list raspberry pi projects

Using OSGi for the Realization of Complex Building Management Systems - Peter...

Presentation of glpi project, OW2con'19, June 12-13, Paris.

Parallel universe-issue-29

Continuous Delivery to the Cloud: Automate Thru Production with CI + Spinnaker

Fuzzing malware for fun & profit. Applying Coverage-Guided Fuzzing to Find Bu...

Infn os summit_2015_v1.0

Preserving CAD: A briefing - Emerging trends - Architecture

IoT and Practical Laboratories

I/O Challenges in Brain Tissue Simulation

Simulations on Computer Network An Improved Study in the Simulator Methodolog...

Measure your app internals with InfluxDB and Symfony2

Nt1320 Unit 6

apidays LIVE Helsinki & North 2022_Apps without APIs

Presentation1.2.pptx

Mehr von Jonas Traub

This talk was presented at code.talks 2022 in Hamburg. Learn more about code.talks at: https://codetalks.de/ Download Slides at: https://codetalks.de/speakers#talk-1128?event=7 Video recording on Youtube: https://www.youtube.com/watch?v=GC-hKOvQBus In this session, we will provide an introduction to REASON, also known as ReasonML. At SAP, we use REASON to build our Master Data Integration (MDI) Service as part of SAPs Business Technology Platform. MDI is a highly scalable, distributed, multi-tenant system powered by functional programming. REASON, is our language of choice for MDI: It lets you write simple, fast, and type save code. With REASON, you rarely have to annotate types, but everything gets checked for you through REASON’s type inference system, reducing bugs and increasing code maintainability. REASON translates to OCaml or JavaScript, providing access to both worlds and enabling it to run on NodeJS. During the session, we will highlight the advantages of the syntax compared to Java and introduce you to the beauty of immutability, tail recursion and other relevant concepts. Our session targets programmers who are not yet familiar with functional programming, as well as functional programmers who want to learn more about REASON.

Definitely not Java! A Hands-on Introduction to Efficient Functional Programm...

Jonas Traub

This talk was presented for the SoftwareCampus Alumni e.V. on 07.12.2020. For more Information about the program check https://softwarecampus-alumni.de/ and https://softwarecampus.de/ Abstract: The Internet of Things (IoT) consists of billions of devices which form a cloud of network-connected sensor nodes. These sensor nodes supply a vast number of data streams with massive amounts of sensor data. Real-time sensor data enables diverse applications including traffic-aware navigation, machine monitoring, and home automation. In this talk, we will dive into recent research which optimizes real-time data gathering and data analysis in the IoT. The talk will provide an overview of available techniques which can be deployed on sensor nodes, intermediate network nodes, and central analysis systems. We will look into the state-of-the-art in practice and research and make you aware of important tradeoffs in real-time IoT data analysis. CV: Jonas Traub is a postdoctoral researcher at the Database Systems and Information Management group at TU Berlin. His main research interests include stream processing, sensor data analysis, and data acquisition techniques. In his PhD, he studied efficient data gathering, processing, and transmission in the IoT. His research shows that one can save up to 87% in sensor reads and data transfers by applying smart data reduction techniques on sensor nodes. He further introduced a demand-based control layer which optimizes the data acquisition from thousands of sensors. With his Scotty-framework, he contributed a general aggregation technique for streaming systems which outperforms alternative solutions by an order of magnitude in throughput. His work received a Best Paper Award at the 22nd International Conference on Extending Database Technology (EDBT). Prior to his work at TU Berlin, he studied at KTH Stockholm and DHBW Stuttgart and worked several years at IBM in Germany and the USA. Jonas is an alumnus of Software Campus where he worked with SAP as industry partner.

Efficient Data Stream Processing in the Internet of Things - SoftwareCampus A...

Jonas Traub

This presentation was held at Code.Talks 2019 in Hamburg. A video is available at: https://www.youtube.com/watch?v=K1y5dJvP1jM Window aggregation is a core operation in data stream processing. Stream Processing Systems, like Flink or Storm, implement general aggregation techniques which perform poorly under specific workloads (e.g. Sliding Windows). To this end, we present Scotty, a new highly-efficient window operator. Scotty exploits specific workload properties such as the type of aggregation functions (e.g., invertible, associative), window types (e.g., sliding, sessions), windowing measures (e.g., time- or countbased), and stream (dis)order. This allows Scotty to outperform systems like Flink by up to one order of magnitude. The structure of this talk is threefold: First, we give an introduction to the semantics and implementations of window aggregations in modern Stream Processing Systems. Second, we discuss the design of Scotty and show why Scotty is able to outperform the default window operators of many stream processing systems. Third, we give a hands-on introduction to Scotty and demonstrate how it can be integrated into standard Flink, Storm, or Beam stream processing pipelines. Scotty and its connectors are available as open-source (https://github.com/TU-Berlin-DIMA/scotty-window-processor) and contributions are highly welcome.

code.talks 2019 - Scotty: Efficient Window Aggregation for your Stream Proces...

Jonas Traub

Window aggregation is a core operation in data stream processing. Existing aggregation techniques focus on reducing latency, eliminating redundant computations, and minimizing memory usage. However, each technique operates under different assumptions with respect to workload characteristics such as properties of aggregation functions (e.g., invertible, associative), window types (e.g., sliding, sessions), windowing measures (e.g., time- or countbased), and stream (dis)order. Violating the assumptions of a technique can deem it unusable or drastically reduce its performance. In this talk, we present Scotty an implementation of a general stream slicing technique for window aggregation. This technique automatically adapts to workload characteristics to improve performance without sacrificing its general applicability. Our experiments show that Scotty outperforms alternative implementations, like the default window operator in Flink, by up to one order of magnitude. Furthermore, we present how to use Scotty as a library in Flink, Storm, or Beam without changing the underlying Stream Processing System. General stream slicing was first published at EDBT 2019 (http://www.user.tu-berlin.de/powibol/assets/publications/traub-efficient-window-aggregation-with-general-stream-slicing-edbt-2019.pdf) where it received the Best Paper Award. The Scotty library and its connectors are available as open-source (https://github.com/TU-Berlin-DIMA/scotty-window-processor) and contributions are highly welcome.

FlinkForward Berlin 2019 - Scotty: Efficient Window Aggregation with General ...

Jonas Traub

This talk by Steffen Zeuch presents our VLDB 2019 paper about "Analyzing Efficient Stream Processing on Modern Hardware". This paper is about showing the potential of hardware-tailored code compilation and data ingestion at memory speed for a scale-up SPE. Analyze state-of-the-art streaming systems and identify sources of inefficiency. We investigate the data-related and processing-related design space and derive design changes for streaming systems to exploit modern hardware more efficiently. In order to efficiently scale up, one should avoid managed runtimes, use a compilation-based approach to produce hardware-tailored code, avoid queues and use operator fusion, and use late merge instead of partitioning. The full paper with all our findings is available online: http://www.vldb.org/pvldb/vol12/p516-zeuch.pdf

Analyzing Efficient Stream Processing on Modern Hardware (VLDB 2019 Presentat...

Jonas Traub

In April 2019, we did an USA excursion and presented selected publications of the TU Berlin DIMA and the DFKI IAM research groups. This slide set contains the four teaser talks which we presented on the tour: 1) Jonas Traub: Optimized On-Demand Data Streaming from Sensor Nodes 2) Sebastian Breß: Generating Custom Code for Efficient Query Execution on Heterogeneous Processors 3) Martin Kiefer: Estimating Join Selectivities using Bandwidth Optimized Kernel Density Models 4) Andreas Kunft: BlockJoin: Efficient Matrix Partitioning through Joins

Database Research at TU Berlin DIMA and DFKI IAM - USA Excursion Slides 2019

Jonas Traub

The Paper "Efficient Window Aggregation with General Stream Slicing" by Jonas Traub, Philipp M. Grulich, Alejandro Rodriguez Cuellar, Sebastian Breß, Tilmann Rabl, and Volker Markl was selected as best paper of the International Conference on Extending Database Technology (EDBT) 2019. Abstract: Window aggregation is a core operation in data stream processing. Existing aggregation techniques focus on reducing latency, eliminating redundant computations, and minimizing memory usage. However, each technique operates under different assumptions with respect to workload characteristics such as properties of aggregation functions (e.g., invertible, associative), window types (e.g., sliding, sessions), windowing measures (e.g., time- or countbased), and stream (dis)order. Violating the assumptions of a technique can deem it unusable or drastically reduce its performance. In this paper, we present the first general stream slicing technique for window aggregation. General stream slicing automatically adapts to workload characteristics to improve performance without sacrificing its general applicability. As a prerequisite, we identify workload characteristics which affect the performance and applicability of aggregation techniques. Our experiments show that general stream slicing outperforms alternative concepts by up to one order of magnitude.

Efficient Window Aggregation with General Stream Slicing (EDBT 2019, Best Paper)

Jonas Traub

Resense: Transparent Record and Replay of Sensor Data in the Internet of Things Resense was presented as Demonstration at the 22nd International Conference on Extending Database Technology (EDBT) in Lisbon, Portugal and received the Best Demonstration Award (http://edbticdt2019.inesc-id.pt/?awards_demo_edbt). AUTHORS: Dimitrios Giouroukis, Julius Hülsmann, Janis von Bleichert, Morgan Geldenhuys, Tim Stullich, Felipe Oliveira Gutierrez, Jonas Traub, Kaustubh Beedkar, Volker Markl ABSTRACT As the scientific interest in the Internet of Things (IoT) continues to grow, emulating IoT infrastructure involving a large number of heterogeneous sensors plays a crucial role. Existing research on emulating sensors is often tailored to specific hardware and/or software, which makes it difficult to reproduce and extend. In this paper we show how to emulate different kinds of sensors in a unified way that makes the downstream application agnostic as to whether the sensor data is acquired from real sensors or is read from memory using emulated sensors. We propose the Resense framework that allows for replaying sensor data using emulated sensors and provides an easy-to-use software for setting up and executing IoT experiments involving a large number of heterogeneous sensors. We demonstrate various aspects of Resense in the context of a sports analytics application using real-world sensor data and a set of Raspberry Pis.

Resense: Transparent Record and Replay of Sensor Data in the Internet of Thin...

Jonas Traub

Computing aggregates over windows is at the core of virtually every stream processing job. Typical stream processing applications involve overlapping windows and, therefore, cause redundant computations. Several techniques prevent this redundancy by sharing partial aggregates among windows. However, these techniques do not support out-of-order processing and session windows. Out-of-order processing is a key requirement to deal with delayed tuples in case of source failures such as temporary sensor outages. Session windows are widely used to separate different periods of user activity from each other. Current versions of Apache Flink use Window Buckets to process stream aggregations with session windows and out-of-order tuples. This Approach does not share partial aggregates among overlapping windows. In our talk, we present Scotty, a high throughput operator for window discretization and aggregation in Apache Flink. Scotty splits streams into non-overlapping slices and computes partial aggregates per slice. These partial aggregates are shared among all overlapping windows including session windows. Scotty introduces the first slicing technique which (1) enables stream slicing for session windows in addition to tumbling and sliding windows and (2) processes out-of-order tuples efficiently. Scotty was first published at ICDE 2018 (http://www.user.tu-berlin.de/powibol/assets/publications/traub-scotty-icde-2018.pdf).

Flink Forward 2018: Efficient Window Aggregation with Stream Slicing

Jonas Traub

This poster was presented at ICDE 2018. Abstract: Computing aggregates over windows is at the core of virtually every stream processing job. Typical stream processing applications involve overlapping windows and, therefore, cause redundant computations. Several techniques prevent this redundancy by sharing partial aggregates among windows. However, these techniques do not support out-of-order processing and session windows. Out-of-order processing is a key requirement to deal with delayed tuples in case of source failures such as temporary sensor outages. Session windows are widely used to separate different periods of user activity from each other. In this paper, we present Scotty, a high throughput operator for window discretization and aggregation. Scotty splits streams into non-overlapping slices and computes partial aggregates per slice. These partial aggregates are shared among all concurrent queries with arbitrary combinations of tumbling, sliding, and session windows. Scotty introduces the first slicing technique which (1) enables stream slicing for session windows in addition to tumbling and sliding windows and (2) processes out-of-order tuples efficiently. Our technique is generally applicable to a broad group of dataflow systems which use a unified batch and stream processing model. Our experiments show that we achieve a throughput an order of magnitude higher than alternative state-of-the-art solutions.

Scotty: Efficient Window Aggregation for Out-of-Order Stream Processing

Jonas Traub

Paper: Scalable Detection of Concept Drifts on Data Streams with Parallel Adaptive Windowing Abstract: Machine learning techniques for data stream analysis suffer from concept drifts such as changed user preferences, varying weather conditions, or economic changes. These concept drifts cause wrong predictions and lead to incorrect business decisions. Concept drift detection methods such as adaptive windowing (Adwin) allow for adapting to concept drifts on the fly. In this paper, we examine Adwin in detail and point out its throughput bottlenecks. We then introduce several parallelization alternatives to address these bottlenecks. Our optimizations lead to a speedup of two orders of magnitude over the original Adwin implementation. Thus, we explore parallel adaptive windowing to provide scalable concept detection for high-velocity data streams with millions of tuples per second.

Scalable Detection of Concept Drifts on Data Streams with Parallel Adaptive W...

Jonas Traub

This poster was presented at the 21st International Conference on Extending Database Technology (EDBT), March 26-29, 2018. Paper: Efficient SIMD Vectorization for Hashing in OpenCL Abstract: Hashing is at the core of many efficient database operators such as hash-based joins and aggregations. Vectorization is a technique that uses Single Instruction Multiple Data (SIMD) instructions to process multiple data elements at once. Applying vectorization to hash tables results in promising speedups for build and probe operations. However, vectorization typically requires intrinsics – low-level APIs in which functions map to processorspecific SIMD instructions. Intrinsics are specific to a processor architecture and result in complex and difficult to maintain code. OpenCL is a parallel programming framework which provides a higher abstraction level than intrinsics and is portable to different processors. Thus, OpenCL avoids processor dependencies, which results in improved code maintainability. In this paper, we add efficient, vectorized hashing primitives to OpenCL. Our results show that OpenCL-based vectorization is competitive to intrinsics on CPUs but not on Xeon Phi coprocessors.

Efficient SIMD Vectorization for Hashing in OpenCL

Jonas Traub

About the Workshop: The Stream Reasoning Workshop took place from January 16th to 17th, 2018. Processing, querying and reasoning over streaming data is studied in different communities such as KR&R, Semantic Web, Databases, Stream Processing, Complex Event Processing, etc., where researchers have different perspectives and face different challenges. This workshop aims at advancing Stream Reasoning as research theme by bringing together these different views and goals. In addition to invited talks, the workshop will provide opportunities for all participants to engage in discussions on open problems and future directions. (http://www.ifi.uzh.ch/en/ddis/events/streamreasoning2018.html) About the Talk: Real-time sensor data enables diverse applications such as smart metering, traffic monitoring, and sport analysis. In the Internet of Things, billions of sensor nodes form a sensor cloud and offer data streams to analysis systems. However, it is impossible to transfer all available data with maximal frequencies to all applications. Therefore, we need to tailor data streams to the demand of applications. We contribute a technique that optimizes communication costs while maintaining the desired accuracy. Our technique schedules reads across huge amounts of sensors based on the data-demands of a huge amount of concurrent queries. We introduce user-defined sampling functions that define the data-demand of queries and facilitate various adaptive sampling techniques, which decrease the amount of transferred data. Moreover, we share sensor reads and data transfers among queries. Our experiments with real-world data show that our approach saves up to 87% in data transmissions.

UZH Stream Reasoning Workshop 2018: Optimized On-Demand Data Streaming from S...

Jonas Traub

JT@UCSB - On-Demand Data Streaming from Sensor Nodes and A quick overview of ...

Jonas Traub

LWA 2015: The Apache Flink Platform (Poster)

Jonas Traub

LWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis

Jonas Traub

Mehr von Jonas Traub (16)