Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Analyzing Efficient Stream Processing on Modern Hardware (VLDB 2019 Presentation)

68 Aufrufe

Veröffentlicht am

This talk by Steffen Zeuch presents our VLDB 2019 paper about "Analyzing Efficient Stream Processing on Modern Hardware".

This paper is about showing the potential of hardware-tailored code compilation and data ingestion at memory speed for a scale-up SPE. Analyze state-of-the-art streaming systems and identify sources of inefficiency. We investigate the data-related and processing-related design space and derive design changes for streaming systems to exploit modern hardware more efficiently. In order to efficiently scale up, one should avoid managed runtimes, use a compilation-based approach to produce hardware-tailored code, avoid queues and use operator fusion, and use late merge instead of partitioning.

The full paper with all our findings is available online:
http://www.vldb.org/pvldb/vol12/p516-zeuch.pdf

Veröffentlicht in: Wissenschaft
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Analyzing Efficient Stream Processing on Modern Hardware (VLDB 2019 Presentation)

  1. 1. Steffen Zeuch / DFKI GmbH / 08/27/2019 1/14 Analyzing Efficient Stream Processing on Modern Hardware Steffen Zeuch, Bonaventura Del Monte, Jeyhun Karimov, Clemens Lutz, Manuel Renz, Jonas Traub, Sebastian Breß, Tilmann Rabl, Volker Markl
  2. 2. Steffen Zeuch / DFKI GmbH / 08/27/2019 2/14 What is this paper about? This paper is about showing the potential of hardware-tailored code compilation and data ingestion at memory speed for a scale-up SPE. This paper is not about benchmarking existing SPEs.
  3. 3. Steffen Zeuch / DFKI GmbH / 08/27/2019 3/14 Is ingestion at memory-speed possible? Network is not the bottleneck in the future.
  4. 4. Steffen Zeuch / DFKI GmbH / 08/27/2019 4/14 What is possible? No SPE is yet ready for processing at memory speed.
  5. 5. Steffen Zeuch / DFKI GmbH / 08/27/2019 5/14 What did we do? • Analyze state-of-the-art streaming systems and identify sources of inefficiency. • Investigate data-related and processing-related design space. • Derive design changes for streaming systems to exploit modern hardware more efficiently.
  6. 6. Steffen Zeuch / DFKI GmbH / 08/27/2019 6/14 How do SPEs transfer data? Queues are the major bottleneck for scale-up processing.
  7. 7. Steffen Zeuch / DFKI GmbH / 08/27/2019 7/14 How do SPEs parallelize a query? For scale-up, there are alternatives to partitioning.
  8. 8. Steffen Zeuch / DFKI GmbH / 08/27/2019 8/14 How do SPEs execute a query? All systems use an interpretation based approach. All systems, except streambox, use a managed runtime.
  9. 9. Steffen Zeuch / DFKI GmbH / 08/27/2019 9/14 What’s the scale-up performance? Yahoo Streaming Benchmark Linear Road Benchmark (partly) New York Taxi Query Overhead for entire framework: up to 80x Overhead for managed runtime: up to 56x
  10. 10. Steffen Zeuch / DFKI GmbH / 08/27/2019 10/14 What’s the scale-out performance?(reported) An optimized scale-up solution outperforms even 10 node cluster.
  11. 11. Steffen Zeuch / DFKI GmbH / 08/27/2019 11/14 How does Flink scale out? Add new nodes to the system does not solve the problem.
  12. 12. Steffen Zeuch / DFKI GmbH / 08/27/2019 12/14 Why are current SPEs inefficient? Large instruction footprints, virtual function calls, and suboptimal access patterns reduce efficiency.
  13. 13. Steffen Zeuch / DFKI GmbH / 08/27/2019 13/14 What should we do to scale-up? • Avoid managed runtimes • Use a compilation-based approach to produce hardware-tailored code • Avoid queues and use operator fusion • Use late merge instead of partitioning – enables producer/consumer fusing
  14. 14. Steffen Zeuch / DFKI GmbH / 08/27/2019 14/14 Summary • We explore the data-related and processing-related design space. • We show that an up to two orders of magnitude performance improvement is possible. • We derive design changes for streaming systems to exploit modern hardware more efficiently. https://git.io/fjAZg

×