Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gregg, AutoTrader)

Disney+ uses Amazon Kinesis to drive real-time actions like providing title recommendations for customers, sending events across microservices, and delivering logs for operational analytics to improve the customer experience. In this session, you learn how Disney+ built real-time data-driven capabilities on a unified streaming platform. This platform ingests billions of events per hour in Amazon Kinesis Data Streams, processes and analyzes that data in Amazon Kinesis Data Analytics for Apache Flink, and uses Amazon Kinesis Data Firehose to deliver data to destinations without servers or code. Hear how these services helped Disney+ scale its viewing experience to tens of millions of customers with the required quality and reliability. Learn more about re:Invent 2020 at http://bit.ly/3c4NSdY

Presentation at Spark Summit 2015

Daten & Analysen

Visualizing  
Autotrader Traffic
Using Spark Streaming
Jon Gregg, Cox Automotive

Overview
• Cox Automotive and Hadoop
• Spark Streaming application
• Spark roadmap at Cox Automotive

$45B+ vehicle values sold
annually through Manheim
AutoTrader.com has 18M
unique visitors each month
and lists an average of 4M
cars monthly
Kelley Blue Book provides
values for 290M cars annually
and has 18M+ unique visitors
monthly
vAuto has over 2.3M active vehicles
in inventory and started 1.65M
vehicle appraisals in Oct
265K vehicles sold per month (on
average) through a VinSolutions CRM
Cox Automotive
The Journey

Cox Automotive $45B+ vehicle values sold
annually through Manheim
AutoTrader.com has 18M
unique visitors each month
and lists an average of 4M
cars monthly
Kelley Blue Book provides
values for 290M cars annually
and has 18M+ unique visitors
monthly
vAuto has over 2.3M active vehicles
in inventory and started 1.65M
vehicle appraisals in Oct
265K vehicles sold per month (on
average) through a VinSolutions CRM
Cox Automotive
The Journey
• Over 25 companies 
(and growing) 
 
• Facilitate joining data, analyst collaboration
• Hadoop cluster, dedicated ingest team

Use Hadoop where it makes sense
• Joining data from across several companies
• Large amounts of data (Querying and Reporting)
• Build out business logic so it’s shareable

That’s all great…
… but we also have to showcase
what Hadoop can do

Autobowl: Goal
Find which Big Game car commercial led to
the greatest Autotrader trafﬁc increase, as a
proxy for inﬂuence on consumers?

Two solutions
• Hive on MapReduce 
Mature, supported product 
Shows SQL’s capabilities on Hadoop
• Spark 
Started as a POC, no expectations 
How would it work with YARN, Kerberos?

Autobowl Hourly Data
Make Model Hour VDPs Searches
…
Kia Sedona 9pm 300 290
Kia Sedona 10pm 310 320
Kia Sorento 3pm 220 240
Kia Sorento 4pm 210 220
Kia Sorento 5pm 350 380
…
+70%!

Hive vs. Spark: Processing Time
Hive Spark
1.5 min
18 min
Minutes to Process 1hr of Site Activity Data
(And a month  
to spare!)

Spark Streaming for Near Realtime
Visualization of Trafﬁc

High-level architecture diagram
web server
web server
web server
web server
emitter kafka
Hadoop
(Spark)
AWS

• Detecting anomalies in Autotrader metrics after
a site update
Other Visualization use cases

• Detecting anomalies in Autotrader metrics after
a site update
• Executive dashboards
• Visualizations for A/B testing
Other Visualization use cases

• Most BI users use Hive or point-and-click app
• But there’s been a shift - Spark is in use by
analyst teams within Autotrader, KBB using
Python
• Spark used by developers at Autotrader, KBB,
Mannheim, NextGear
Gaining BI Adoption

• Speed improvements with Dataframes,  
Kafka integration
• “Easier sell” than Java/Scala 
Scripting, visualization+analytics packages
• Onboarding users 
Central repository with best practices 
Individual support for BI Spark champions 
Guidelines on setting Spark parameters 
Python: our primary Spark language

Weitere ähnliche Inhalte

Was ist angesagt?

How Disney+ uses fast data ubiquity to improve the customer experience

Martin Zapletal

Streaming data for real time analysis

The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.

Disrupting Big Data with Apache Spark in the Cloud

Jen Aman

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...

DataOps challenges us to build data experiences in a repeatable way. For those with Kafka, this means finding a means of deploying flows in an automated and consistent fashion. The challenge is to make the deployment of Kafka flows consistent across different technologies and systems: the topics, the schemas, the monitoring rules, the credentials, the connectors, the stream processing apps. And ideally not coupled to a particular infrastructure stack. In this talk we will discuss the different approaches and benefits/disadvantages to automating the deployment of Kafka flows including Git operators and Kubernetes operators. We will walk through and demo deploying a flow on AWS EKS with MSK and Kafka Connect using GitOps practices: including a stream processing application, S3 connector with credentials held in AWS Secrets Manager.

DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros ...

How Spark Enables the Internet of Things- Paula Ta-Shma

Kafka is widely positioned as the proverbial "central nervous system" of the enterprise. In this session, we explore how the central nervous system can be used to build a mesh topology & unified catalog of enterprise wide events, enabling development teams to build event driven architectures faster & better. The central theme of this topic is also aligned to seeking idioms from API Management, Service Meshes, Workflow management and Service orchestration. We compare how these approaches can be harmonized with Kafka. We will also touch upon the topic of how this relates to Domain Driven Design, CQRS & other patterns in microservices. Some potential takeaways for the discerning audience: 1. Opportunities in a platform approach to Event Driven Architecture in the enterprise 2. Adopting a product mindset around Data & Event Streams 3. Seeking harmony with allied enterprise applications

Event & Data Mesh as a Service: Industrializing Microservices in the Enterpri...

Are you looking for a cloud-based architecture that includes the best of breed streaming and database technologies? In this session you will learn how to setup and configure the Confluent Cloud with MongoDB Atlas. We'll start the journey learning about the basic connectivity between the two cloud services and end with a brief discovery of what you can do with data once it is in MongoDB Atlas. By the end of this session you will know how to securely setup and configure the MongoDB Atlas connectors in the Confluent Cloud in both a source and sink configuration.

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

Apache Druid is a high-performance distributed analytics store for modern analytics applications. It supports ingesting millions of events per second and sub-second query processing. Druid supports various types of data sources for ingestion, including Apache Kafka. You can immediately query on stream events once they get ingested into Druid. Since Kafka provides scalable and robust data delivery while Druid supports advanced complex analysis on streams, Kafka and Druid are widely used together for BI and operational analytics use cases, which require interactivity, scalability, real-time, and performance. This talk is based on our real-world experiences building out streaming analytics stacks powering production use cases across many industries.

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...

Spark Summit EU talk by Zoltan Zvara

Almost all organizations now have a need for data science and, as such, the main challenge after determining the algorithm is to scale it up and make it operational. Comcast uses several tools and technologies such as Python, R, SaS, H2O and so on. In this session, they’ll show how many common use cases use the common algorithms like Logistic Regression, Random Forest, Decision Trees, Clustering, NLP, etc. Apache Spark has several machine learning algorithms built in and has excellent scalability. Hence, at Comcast, they built a platform to provide DSaaS on top of Spark with REST API as a means of controlling and submitting jobs, so as to abstract most users from the rigor of writing (repeating) code, instead focusing on the actual requirements. Learn how they solved some of the problems of establishing feature vectors, choosing algorithms and then deploying models into production. They’ll also showcase their use of Scala, R and Python to implement models using language of choice yet deploying quickly into production on 500-node Spark clusters.

Netflix Big Data Paris 2017

Jason Flittner

Using SparkML to Power a DSaaS (Data Science as a Service) with Kiran Muglurm...

Building flexible machine learning libraries adapted for Netflix’s use cases is paramount in our continued efforts to better model our users’ behaviors and provide them great personalized video recommendations. This talk introduces one such spark-based stratification library developed at Netflix to aid “Training Set Stratification” in offline machine learning workflows. Originally created to implement user selection algorithms in our data snapshotting infrastructure, the library has evolved to cater to general-purpose stratification use cases in ML pipelines. We will talk about how using the stratification library’s DSL (domain specific language) and its underlying Spark based implementation, one can easily express complex sampling rules and dynamically carve out matching portions of a Spark dataframe. For example, arbitrary rules governing the distributions of user attributes (and combinations there of) such as origin country, video play frequency, tenure etc can be easily enforced when constructing a ML training data set. The demo section of the talk will showcase example usages of the stratification library in a Jupyter notebook.

Apache Spark-Based Stratification Library for Machine Learning Use Cases at N...

With 60+ products and over 24% of the US GDP flowing through it, system integration is a tough problem for Intuit. Seasonality, scale, and massive peaks in products like TurboTax, QuickBooks, and Mint.com add extra layers of difficulty when building shared data services around transaction and user graphs, clickstream processing, a/b testing, and personalization. To reduce complexity and latency, we’ve implemented Kafka as the backbone across these data services. This allows us to asynchronously trigger relevant processing, elegantly scaling up and down as needed around peaks, all without the need for point-to-point integrations. In this talk, we share what we’ve learned about Kafka at Intuit and describe our data services architecture. We found that Kafka is invaluable in achieving a scalable, clean architecture, allowing engineering teams to focus less on integration and more on product development.

Hands On With Spark: Creating A Fast Data Pipeline With Structured Streaming ...

Lightbend

Kafka, Killer of Point-to-Point Integrations, Lucian Lita

confluent

The data science lifecycle consists of multiple iterative steps: data collection, data cleaning/exploration, feature engineering, model training, model deployment and scoring among others. The process is often tedious and error-prone and requires considerable human effort. Apart from these challenges, when it comes to leveraging ML in enterprise applications, especially in regulated environments, the level of scrutiny for data handling, model fairness, user privacy, and debuggability is very high. In this talk, we present the basic features of Flock, an end-to-end platform that facilitates adoption of ML in enterprise applications. We refer to this new class of applications as Enterprise Grade Machine Learning (EGML). Flock leverages MLflow to simplify and automate some of the steps involved in supporting EGML applications, allowing data scientists to spend most of their time on improving their ML models. Flock makes use of MLflow for model and experiment tracking but extends and complements it by providing automatic logging, deeper integration with relational databases that often store confidential data, model optimizations and support for the ONNX model format and the ONNX Runtime for inference. We will also present our ongoing work on automatically tracking lineage between data and ML models which is crucial in regulated environments. We will showcase Flock’s features through a demo using Microsoft’s Azure Data Studio and MLflow.

Improving the Life of Data Scientists: Automating ML Lifecycle through MLflow

Should you consume Kafka in a stream OR batch? When should you choose each one? What is more efficient, and cost effective? In this talk we’ll give you the tools and metrics to decide which solution you should apply when, and show you a real life example with cost & time comparisons. To highlight the differences, we’ll dive into a project we’ve done, transitioning from reading Kafka in a stream to reading it in batch. By turning conventional thinking on its head and reading our multi-petabyte Kafka stream in batch using Spark and Airflow, we’ve achieved a huge cost reduction of 65% while at the same time getting a more scalable and resilient solution. We’ll explore the tradeoffs and give you the metrics and intuition you’ll need to make such decisions yourself. We’ll cover: Costs of processing in stream compared to batch Scaling up for bursts and reprocessing Making the tradeoff between wait times and costs Recovering from outages And much more…

Should You Read Kafka as a Stream or in Batch? Should You Even Care? | Ido Na...

One of the key metrics to monitor when working with Apache Kafka, as a data pipeline or a streaming platform, is Consumer Groups Lag. Lag is the delta between the last produced message and the last committed message of a partition. In other words, lag indicates how far behind your application is in processing up-to-date information. For a long time, we used our own service to keep track of these metrics, collect them and visualize them. But this didn’t scale well. You had to perform many manual operations, redeploy it and to do other tedious manual tasks, but most importantly, the biggest gap for us, was that its output was represented in absolute numbers (e.g - your lag is 30K), which basically tells you nothing as a human being. We understood that we had to find a more suitable solution that will give us better visibility and will allow us to measure the lag in a time-based format that we all understand. In this talk, I’m going to go over the core concepts of Kafka offsets and lags, and explain why lag even matters and is an important KPI to measure. I’ll also talk about the kind of research we did to find the right tool, what the options in the market were at the time, and eventually why we chose Linkedin’s Burrow as the right tool for us. And finally, I’ll take a closer look at Burrow, its building blocks, how we build and deploy it, how we monitor better with it, and eventually the most important improvement - how we transformed its output from numbers to time-based metrics.

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020

KSQL: Open Source Streaming for Apache Kafka

confluent

How Netflix Uses Amazon Kinesis Streams to Monitor and Optimize Large-scale N...

Contact: https://www.linkedin.com/in/brandonjobrien @hakczar Introducing streaming data concepts, Storm cluster architecture, Storm topology architecture, and demonstrate working example of a WordCount topology for SIGKDD Seattle chapter meetup. Presented by Brandon O'Brien Code example: https://github.com/OpenDataMining/brandonobrien Meetup: http://www.meetup.com/seattlesigkdd/events/222955114/

Was ist angesagt? (20)

How Disney+ uses fast data ubiquity to improve the customer experience

Streaming data for real time analysis

Disrupting Big Data with Apache Spark in the Cloud

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...

DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros ...

How Spark Enables the Internet of Things- Paula Ta-Shma

Event & Data Mesh as a Service: Industrializing Microservices in the Enterpri...

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...

Spark Summit EU talk by Zoltan Zvara

Netflix Big Data Paris 2017

Using SparkML to Power a DSaaS (Data Science as a Service) with Kiran Muglurm...

Apache Spark-Based Stratification Library for Machine Learning Use Cases at N...

Hands On With Spark: Creating A Fast Data Pipeline With Structured Streaming ...

Kafka, Killer of Point-to-Point Integrations, Lucian Lita

Improving the Life of Data Scientists: Automating ML Lifecycle through MLflow

Should You Read Kafka as a Stream or in Batch? Should You Even Care? | Ido Na...

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020

KSQL: Open Source Streaming for Apache Kafka

How Netflix Uses Amazon Kinesis Streams to Monitor and Optimize Large-scale N...

Andere mochten auch

Data Science with Spark & Zeppelin

Vinay Shukla

Introduction to Streaming Distributed Processing with Storm

Brandon O'Brien

Contact: https://www.linkedin.com/in/brandonjobrien @hakczar Code examples available at https://github.com/br4nd0n/spark-streaming and https://github.com/br4nd0n/spark-viz A demo and explanation of building a streaming application using Spark Streaming, Node.js and Redis with a real time visualization. Includes discussion of internals of Spark and Spark streaming including RDD partitioning and code and data distribution and cluster resource allocation.

Real Time Data Processing With Spark Streaming, Node.js and Redis with Visual...

Brandon O'Brien

Manual de programacion_con_robots_para_la_escuela

Angel De las Heras

Real time data viz with Spark Streaming, Kafka and D3.js

Ben Laird

Big Data visualization with Apache Spark and Zeppelin

prajods

Sparkly Notebook: Interactive Analysis and Visualization with Spark

felixcss

Andere mochten auch (7)

Data Science with Spark & Zeppelin

Introduction to Streaming Distributed Processing with Storm

Real Time Data Processing With Spark Streaming, Node.js and Redis with Visual...

Manual de programacion_con_robots_para_la_escuela

Real time data viz with Spark Streaming, Kafka and D3.js

Big Data visualization with Apache Spark and Zeppelin

Sparkly Notebook: Interactive Analysis and Visualization with Spark

Ähnlich wie Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gregg, AutoTrader)

Modernizing Cloud and Hyperconverged Infrastructure monitoring

ManageEngine, Zoho Corporation

Maintaining the Front Door to Netflix : The Netflix API

Daniel Jacobson

The concepts and architectures of a data warehouse, a data lake, and data streaming are complementary to solving business problems. Unfortunately, the underlying technologies are often misunderstood, overused for monolithic and inflexible architectures, and pitched for wrong use cases by vendors. Let’s explore this dilemma in a presentation. The slides cover technologies such as Apache Kafka, Apache Spark, Confluent, Databricks, Snowflake, Elasticsearch, AWS Redshift, GCP with Google Bigquery, and Azure Synapse.

Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?

Kai Wähner

Apache Kafka in conjunction with Apache Spark became the de facto standard for processing and analyzing data. Both frameworks are open, flexible, and scalable. Unfortunately, the latter makes operations a challenge for many teams. Ideally, teams can use serverless SaaS offerings to focus on business logic. However, hybrid and multi-cloud scenarios require a cloud-native platform that provides automated and elastic tooling to reduce the operations burden. This session explores different architectures to build serverless Apache Kafka and Apache Spark multi-cloud architectures across regions and continents. We start from the analytics perspective of a data lake and explore its relation to a fully integrated data streaming layer with Kafka to build a modern data Data Lakehouse. Real-world use cases show the joint value and explore the benefit of the "delta lake" integration.

Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture

Kai Wähner

BikersPlanet.pptx

PareshsinhUmeshsinhC

SUGMEA - Sitecore JSS and Performance Optimization - Alex Shyba - Altudo

dharmeshharji

WordPress Café April: Viking motors case

Exove

Hearst Corporation monitors trending content on 250+ sites worldwide, providing metrics to editors and promoting cross-platform content sharing. To facilitate this, Hearst built a clickstream analytics platform on AWS that transmits and processes over 30 TB of data a day using AWS resources such as AWS Elastic Beanstalk, Amazon Kinesis, Spark on Amazon EMR, Amazon S3, Amazon Redshift, and Amazon Elasticsearch. In this session, learn how Hearst designed their clickstream analytics application and how you can use the same architecture to build your own and be ready to handle the changing world of clickstream data. Dive into how to do Spark streaming from an Amazon Kinesis stream, use timestamps to cleanse and validate data coming from diverse sources, and see how the system has evolved as data types have change from HTTP GET to RESTful JSON requests. Finally, see how Hearst's data scientists interact with and use cleansed data provided by the platform to perform ad hoc analyses, develop home-grown algorithms, and create visualizations and dashboards that support Hearst business stakeholders.

(BDT306) How Hearst Publishing Manages Clickstream Analytics with AWS

Top 13 web scraping tools in 2022

Aparna Sharma

Analytics at Carbonite: presentation to Snowplow Meetup Boston April 2016

yalisassoon

Building a Data Warehouse for Business Analytics using Spark SQL-(Blagoy Kalo...

Auto Trader is the UK’s leading digital automotive marketplace. We receive 60 million cross-platform visits each month, while our ML-powered car valuations provide 5.5 million valuations a month to both consumers and dealers. At Auto Trader, our development culture aims to combine a data-driven approach with continuous delivery. As such we are keen to ensure that any new machine learning models we develop fit this way of working; our Data Scientists should be able to directly build, train, re-train and deploy their models with minimal outside input. Reducing the time to live allows for more experimentation and reduces the cost of getting machine learning models into production. This talk will describe how we launched our new “Days to Sell” ML-powered metric, using Databricks notebooks to experiment with and tune our model, MLeap to serve out predictions in real time, and Kubernetes to automate the deployment of new models. We will show how we went from model experimentation and design, right through to deploying our model to a production environment. We will also share our thoughts on how we expect this approach to evolve in the future, including how we expect to integrate with MLflow.

How We Used Databricks, MLeap, and Kubernetes to Productionize Spark ML Faste...

In the middle of 2015, Expedia started using AWS Lambda for serverless computing. We built boilerplate templates in Node.js, Java, and Python so development teams could build and deploy serverless applications into AWS. Currently, we have 300 AWS Lambda functions processing 40 million invocations per day. In this session, we will discuss how development teams use boilerplate templates to create serverless applications with Amazon API Gateway and AWS Lambda and how they deploy them to AWS. We will cover patterns, architectural design choices, and the benefits --- like cost, scale, availability, and operations --- of running serverless applications.

SEO Project

machli

Redis as a High Scale Swiss Army Knife by Rahul Dagar and Abhishek Gupta of G...

Redis Labs

Big problems Big data, simple AWS solution

Jean-Claude Sotto

Big problems Big Data, simple solutions

Claudio Pontili

Keynote SUGCON 2021 - Sitecore and SaaS our shared journey

Pieter Brinkman

Kanban: Performance and control using Varnish

Varnish Software

Move Fast;Stay Safe:Developing & Deploying the Netflix API

Sangeeta Narayanan

AWS re:Invent 2016: Serverless Computing Patterns at Expedia (SVR306) )

In this session we will present a Configurable FPGA-Based Spark SQL Acceleration Architecture. It is target to leverage FPGA highly parallel computing capability to accelerate Spark SQL Query and for FPGA’s higher power efficiency than CPU we can lower the power consumption at the same time. The Architecture consists of SQL query decomposition algorithms, fine-grained FPGA based Engine Units which perform basic computation of sub string, arithmetic and logic operations. Using SQL query decomposition algorithm, we are able to decompose a complex SQL query into basic operations and according to their patterns each is fed into an Engine Unit. SQL Engine Units are highly configurable and can be chained together to perform complex Spark SQL queries, finally one SQL query is transformed into a Hardware Pipeline. We will present the performance benchmark results comparing the queries with FGPA-Based Spark SQL Acceleration Architecture on XEON E5 and FPGA to the ones with Spark SQL Query on XEON E5 with 10X ~ 100X improvement and we will demonstrate one SQL query workload from a real customer.

Ähnlich wie Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gregg, AutoTrader) (20)

Modernizing Cloud and Hyperconverged Infrastructure monitoring

Maintaining the Front Door to Netflix : The Netflix API

Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?

Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture

BikersPlanet.pptx

SUGMEA - Sitecore JSS and Performance Optimization - Alex Shyba - Altudo

WordPress Café April: Viking motors case

(BDT306) How Hearst Publishing Manages Clickstream Analytics with AWS

Top 13 web scraping tools in 2022

Analytics at Carbonite: presentation to Snowplow Meetup Boston April 2016

Building a Data Warehouse for Business Analytics using Spark SQL-(Blagoy Kalo...

How We Used Databricks, MLeap, and Kubernetes to Productionize Spark ML Faste...

SEO Project

Redis as a High Scale Swiss Army Knife by Rahul Dagar and Abhishek Gupta of G...

Big problems Big data, simple AWS solution

Big problems Big Data, simple solutions

Keynote SUGCON 2021 - Sitecore and SaaS our shared journey

Kanban: Performance and control using Varnish

Move Fast;Stay Safe:Developing & Deploying the Netflix API

AWS re:Invent 2016: Serverless Computing Patterns at Expedia (SVR306) )

Mehr von Spark Summit

FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang

In this talk, we’ll present techniques for visualizing large scale machine learning systems in Spark. These are techniques that are employed by Netflix to understand and refine the machine learning models behind Netflix’s famous recommender systems that are used to personalize the Netflix experience for their 99 millions members around the world. Essential to these techniques is Vegas, a new OSS Scala library that aims to be the “missing MatPlotLib” for Spark/Scala. We’ll talk about the design of Vegas and its usage in Scala notebooks to visualize Machine Learning Models.

VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...

This presentation introduces how we design and implement a real-time processing platform using latest Spark Structured Streaming framework to intelligently transform the production lines in the manufacturing industry. In the traditional production line there are a variety of isolated structured, semi-structured and unstructured data, such as sensor data, machine screen output, log output, database records etc. There are two main data scenarios: 1) Picture and video data with low frequency but a large amount; 2) Continuous data with high frequency. They are not a large amount of data per unit. However the total amount of them is very large, such as vibration data used to detect the quality of the equipment. These data have the characteristics of streaming data: real-time, volatile, burst, disorder and infinity. Making effective real-time decisions to retrieve values from these data is critical to smart manufacturing. The latest Spark Structured Streaming framework greatly lowers the bar for building highly scalable and fault-tolerant streaming applications. Thanks to the Spark we are able to build a low-latency, high-throughput and reliable operation system involving data acquisition, transmission, analysis and storage. The actual user case proved that the system meets the needs of real-time decision-making. The system greatly enhance the production process of predictive fault repair and production line material tracking efficiency, and can reduce about half of the labor force for the production lines.

Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu

As common sense would suggest, weather has a definite impact on traffic. But how much? And under what circumstances? Can we improve traffic (congestion) prediction given weather data? Predictive traffic is envisioned to significantly impact how driver’s plan their day by alerting users before they travel, find the best times to travel, and over time, learn from new IoT data such as road conditions, incidents, etc. This talk will cover the traffic prediction work conducted jointly by IBM and the traffic data provider. As a part of this work, we conducted a case study over five large metropolitans in the US, 2.58 billion traffic records and 262 million weather records, to quantify the boost in accuracy of traffic prediction using weather data. We will provide an overview of our lambda architecture with Apache Spark being used to build prediction models with weather and traffic data, and Spark Streaming used to score the model and provide real-time traffic predictions. This talk will also cover a suite of extensions to Spark to analyze geospatial and temporal patterns in traffic and weather data, as well as the suite of machine learning algorithms that were used with Spark framework. Initial results of this work were presented at the National Association of Broadcasters meeting in Las Vegas in April 2017, and there is work to scale the system to provide predictions in over a 100 cities. Audience will learn about our experience scaling using Spark in offline and streaming mode, building statistical and deep-learning pipelines with Spark, and techniques to work with geospatial and time-series data.

Improving Traffic Prediction Using Weather Data with Ramya Raghavendra

Graph is on the rise and it’s time to start learning about scalable graph analytics! In this session we will go over two Spark-based Graph Analytics frameworks: Tinkerpop and GraphFrames. While both frameworks can express very similar traversals, they have different performance characteristics and APIs. In this Deep-Dive by example presentation, we will demonstrate some common traversals and explain how, at a Spark level, each traversal is actually computed under the hood! Learn both the fluent Gremlin API as well as the powerful GraphFrame Motif api as we show examples of both simultaneously. No need to be familiar with Graphs or Spark for this presentation as we’ll be explaining everything from the ground up!

A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...

Building accurate machine learning models has been an art of data scientists, i.e., algorithm selection, hyper parameter tuning, feature selection and so on. Recently, challenges to breakthrough this “black-arts” have got started. In cooperation with our partner, NEC Laboratories America, we have developed a Spark-based automatic predictive modeling system. The system automatically searches the best algorithm, parameters and features without any manual work. In this talk, we will share how the automation system is designed to exploit attractive advantages of Spark. The evaluation with real open data demonstrates that our system can explore hundreds of predictive models and discovers the most accurate ones in minutes on a Ultra High Density Server, which employs 272 CPU cores, 2TB memory and 17TB SSD in 3U chassis. We will also share open challenges to learn such a massive amount of models on Spark, particularly from reliability and stability standpoints. This talk will cover the presentation already shown on Spark Summit SF’17 (#SFds5) but from more technical perspective.

No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...

In Sweden, from the Rise ICE Data Center at www.hops.site, we are providing to reseachers both Spark-as-a-Service and, more recently, Tensorflow-as-a-Service as part of the Hops platform. In this talk, we examine the different ways in which Tensorflow can be included in Spark workflows, from batch to streaming to structured streaming applications. We will analyse the different frameworks for integrating Spark with Tensorflow, from Tensorframes to TensorflowOnSpark to Databrick’s Deep Learning Pipelines. We introduce the different programming models supported and highlight the importance of cluster support for managing different versions of python libraries on behalf of users. We will also present cluster management support for sharing GPUs, including Mesos and YARN (in Hops Hadoop). Finally, we will perform a live demonstration of training and inference for a TensorflowOnSpark application written on Jupyter that can read data from either HDFS or Kafka, transform the data in Spark, and train a deep neural network on Tensorflow. We will show how to debug the application using both Spark UI and Tensorboard, and how to examine logs and monitor training.

Apache Spark and Tensorflow as a Service with Jim Dowling

Apache Spark and Tensorflow as a Service with Jim Dowling

With the rapid growth of available datasets, it is imperative to have good tools for extracting insight from big data. The Spark ML library has excellent support for performing at-scale data processing and machine learning experiments, but more often than not, Data Scientists find themselves struggling with issues such as: low level data manipulation, lack of support for image processing, text analytics and deep learning, as well as the inability to use Spark alongside other popular machine learning libraries. To address these pain points, Microsoft recently released The Microsoft Machine Learning Library for Apache Spark (MMLSpark), an open-source machine learning library built on top of SparkML that seeks to simplify the data science process and integrate SparkML Pipelines with deep learning and computer vision libraries such as the Microsoft Cognitive Toolkit (CNTK) and OpenCV. With MMLSpark, Data Scientists can build models with 1/10th of the code through Pipeline objects that compose seamlessly with other parts of the SparkML ecosystem. In this session, we explore some of the main lessons learned from building MMLSpark. Join us if you would like to know how to extend Pipelines to ensure seamless integration with SparkML, how to auto-generate Python and R wrappers from Scala Transformers and Estimators, how to integrate and use previously non-distributed libraries in a distributed manner and how to efficiently deploy a Spark library across multiple platforms.

MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...

The Next Accelerator Logging Service (NXCALS) is a new Big Data project at CERN aiming to replace the existing Oracle-based service. The main purpose of the system is to store and present Controls/Infrastructure related data gathered from thousands of devices in the whole accelerator complex. The data is used to operate the machines, improve their performance and conduct studies for new beam types or future experiments. During this talk, Jakub will speak about NXCALS requirements and design choices that lead to the selected architecture based on Hadoop and Spark. He will present the Ingestion API, the abstractions behind the Meta-data Service and the Spark-based Extraction API where simple changes to the schema handling greatly improved the overall usability of the system. The system itself is not CERN specific and can be of interest to other companies or institutes confronted with similar Big Data problems.

Next CERN Accelerator Logging Service with Jakub Wozniak

In Between (A mobile App for couples, downloaded 20M in Global), from daily batch for extracting metrics, analysis and dashboard. Spark is widely used by engineers and data analysts in Between, thanks to the performance and expendability of Spark, data operating has become extremely efficient. Entire team including Biz Dev, Global Operation, Designers are enjoying data results so Spark is empowering entire company for data driven operation and thinking. Kevin, Co-founder and Data Team leader of Between will be presenting how things are going in Between. Listeners will know how small and agile team is living with data (how we build organization, culture and technical base) after this presentation.

Powering a Startup with Apache Spark with Kevin Kim

Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra

In many cases, Big Data becomes just another buzzword because of the lack of tools that can support both the technological requirements for developing and deploying of the projects and/or the fluency of communication between the different profiles of people involved in the projects. In this talk, we will present Moriarty, a set of tools for fast prototyping of Big Data applications that can be deployed in an Apache Spark environment. These tools support the creation of Big Data workflows using the already existing functional blocks or supporting the creation of new functional blocks. The created workflow can then be deployed in a Spark infrastructure and used through a REST API. For better understanding of Moriarty, the prototyping process and the way it hides the Spark environment to the Big Data users and developers, we will present it together with a couple of examples based on a Industry 4.0 success cases and other on a logistic success case.

Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...

Large-scale testing of new data products or enhancements to existing products in a research and development environment can be a technical challenge for data scientists. In some cases, tools available to data scientists lack production-level capacity, whereas other tools do not provide the algorithms needed to run the methodology. At Nielsen, the Databricks platform provided a solution to both of these challenges. This breakout session will cover a specific Nielsen business case where two methodology enhancements were developed and tested at large-scale using the Databricks platform. Development and large-scale testing of these enhancements would not have been possible using standard database tools.

How Nielsen Utilized Databricks for Large-Scale Research and Development with...

Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...

Since the invention of SQL and relational databases, data production has been about specifying how data is transformed through queries. While Apache Spark can certainly be used as a general distributed query engine, the power and granularity of Spark’s APIs enables a revolutionary increase in data engineering productivity: goal-based data production. Goal-based data production concerns itself with specifying WHAT the desired result is, leaving the details of HOW the result is achieved to a smart data warehouse running on top of Spark. That not only substantially increases productivity, but also significantly expands the audience that can work directly with Spark: from developers and data scientists to technical business users. With specific data and architecture patterns spanning the range from ETL to machine learning data prep and with live demos, this session will demonstrate how Spark users can gain the benefits of goal-based data production.

Goal Based Data Production with Sim Simeonov

Have you imagined a simple machine learning solution able to prevent revenue leakage and monitor your distributed application? To answer this question, we offer a practical and a simple machine learning solution to create an intelligent monitoring application based on simple data analysis using Apache Spark MLlib. Our application uses linear regression models to make predictions and check if the platform is experiencing any operational problems that can impact in revenue losses. The application monitor distributed systems and provides notifications stating the problem detected, that way users can operate quickly to avoid serious problems which directly impact the company’s revenue and reduce the time for action. We will present an architecture for not only a monitoring system, but also an active actor for our outages recoveries. At the end of the presentation you will have access to our training program source code and you will be able to adapt and implement in your company. This solution already helped to prevent about US$3mi in losses last year.

Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...

Getting Ready to use Redis with Apache Spark is a technical tutorial designed to address integrating Redis with an Apache Spark deployment to increase the performance of serving complex decision models. To set the context for the session, we start with a quick introduction to Redis and the capabilities Redis provides. We cover the basic data types provided by Redis and cover the module system. Using an ad serving use-case, we look at how Redis can improve the performance and reduce the cost of using complex ML-models in production. Attendees will be guided through the key steps of setting up and integrating Redis with Spark, including how to train a model using Spark then load and serve it using Redis, as well as how to work with the Spark Redis module. The capabilities of the Redis Machine Learning Module (redis-ml) will be discussed focusing primarily on decision trees and regression (linear and logistic) with code examples to demonstrate how to use these feature. At the end of the session, developers should feel confident building a prototype/proof-of-concept application using Redis and Spark. Attendees will understand how Redis complements Spark and how to use Redis to serve complex, ML-models with high performance.

Getting Ready to Use Redis with Apache Spark with Dvir Volk

Here we present a general supervised framework for record deduplication and author-disambiguation via Spark. This work differentiates itself by – Application of Databricks and AWS makes this a scalable implementation. Compute resources are comparably lower than traditional legacy technology using big boxes 24/7. Scalability is crucial as Elsevier’s Scopus data, the biggest scientific abstract repository, covers roughly 250 million authorships from 70 million abstracts covering a few hundred years. – We create a fingerprint for each content by deep learning and/or word2vec algorithms to expedite pairwise similarity calculation. These encoders substantially reduce compute time while maintaining semantic similarity (unlike traditional TFIDF or predefined taxonomies). We will briefly discuss how to optimize word2vec training with high parallelization. Moreover, we show how these encoders can be used to derive a standard representation for all our entities namely such as documents, authors, users, journals, etc. This standard representation can simplify the recommendation problem into a pairwise similarity search and hence it can offer a basic recommender for cross-product applications where we may not have a dedicate recommender engine designed. – Traditional author-disambiguation or record deduplication algorithms are batch-processing with small to no training data. However, we have roughly 25 million authorships that are manually curated or corrected upon user feedback. Hence, it is crucial to maintain historical profiles and hence we have developed a machine learning implementation to deal with data streams and process them in mini batches or one document at a time. We will discuss how to measure the accuracy of such a system, how to tune it and how to process the raw data of pairwise similarity function into final clusters. Lessons learned from this talk can help all sort of companies where they want to integrate their data or deduplicate their user/customer/product databases.

Deduplication and Author-Disambiguation of Streaming Records via Supervised M...

The use of large-scale machine learning and data mining methods is becoming ubiquitous in many application domains ranging from business intelligence and bioinformatics to self-driving cars. These methods heavily rely on matrix computations, and it is hence critical to make these computations scalable and efficient. These matrix computations are often complex and involve multiple steps that need to be optimized and sequenced properly for efficient execution. This work presents new efficient and scalable matrix processing and optimization techniques based on Spark. The proposed techniques estimate the sparsity of intermediate matrix-computation results and optimize communication costs. An evaluation plan generator for complex matrix computations is introduced as well as a distributed plan optimizer that exploits dynamic cost-based analysis and rule-based heuristics The result of a matrix operation will often serve as an input to another matrix operation, thus defining the matrix data dependencies within a matrix program. The matrix query plan generator produces query execution plans that minimize memory usage and communication overhead by partitioning the matrix based on the data dependencies in the execution plan. We implemented the proposed matrix techniques inside the Spark SQL, and optimize the matrix execution plan based on Spark SQL Catalyst. We conduct case studies on a series of ML models and matrix computations with special features on different datasets. These are PageRank, GNMF, BFGS, sparse matrix chain multiplications, and a biological data analysis. The open-source library ScaLAPACK and the array-based database SciDB are used for performance evaluation. Our experiments are performed on six real-world datasets are: social network data ( e.g., soc-pokec, cit-Patents, LiveJournal), Twitter2010, Netflix recommendation data, and 1000 Genomes Project sample. Experiments demonstrate that our proposed techniques achieve up to an order-of-magnitude performance.

MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...