Scaling Your Skillset with Your Data with Jarrett Garcia (Nielsen)

•Als PPTX, PDF herunterladen•

1 gefällt mir•779 views

More data is being created than ever before and the rate at which it is being created is not slowing. This means that new techniques are being developed and used to tackle big data challenges. At Nielsen, big data means opportunities for new products like digital measurement, but also exposed a skills gap for dealing with big data. This session will cover Nielsen’s successful implementation of Spark and Databricks which has allowed Nielsen to scale its products and its Data Scientists’ skillsets.

Daten & Analysen

Copyright©2017TheNielsenCompany.Confidentialandproprietary.
Scaling Your Skillset with Your Data
Through the Modernization of Data Science Technology
Jarrett Garcia
October 25, 2017

Copyright©2017TheNielsenCompany.Confidentialand
proprietary.
2
WHAT WE MEASURE
What Consumers WATCH What Consumers BUY
“We are a measurement company. Data is the core of our business. Everything we
do is to help our clients and our company realize the full value of that
measurement”
6.7 BILLION
STORE TRANSACTIONS
EACH MONTH
250K
PANEL
HOUSEHOLDS
1.7 MILLION
STORE VISITS EACH MONTH
1.6 TRILLION
IMPRESSIONS PER YR
1.7 BILLION
VIEWING RECORDS
EACH MONTH
7 MILLION
WEB EVENTS DAILY
FROM MOBILE DEVICES

3
Copyright©2017TheNielsenCompany.Confidentialandproprietary.
CHALLENGE
● Business - sample based data collection to digital scale
● People - Variety of skill sets requires future proof tooling
● Technology - Over dependence on proprietary software ($$$)
● Infrastructure - old, expensive and not scalable

4
Copyright©2017TheNielsenCompany.Confidentialandproprietary.
STRATEGY
Modernize technology stack
Support speed and scale of R&D
Break group silos and
increase collaboration
Attract Data Science talent
Goal: Create a modern Data Science organization that uses best
practices and tools
Python, SQL, Databricks, AWS
,
Cluster computing & data access
SDLC collaboration software
Modern methods & tools

5
Copyright©2017TheNielsenCompany.Confidentialandproprietary.
MODERN TECHNOLOGY STACK
Data Lake
S3
Storage
Bitbucket Databricks Unified Analytics Platform
Spark SQL

6
Copyright©2017TheNielsenCompany.Confidentialandproprietary.
DATABRICKS SCALING DATA SCIENCE
✓ Data Science can utilize Python as a SAS replacement.
Databricks’ notebooks provide a Python coding platform
✓ Data Science can work within the Cloud ecosystem.
Databricks provides the users access to cloud scale and elasticity
✓ Data Science can access data and perform analytics quickly
Databricks democratizes data access

7
Copyright©2017TheNielsenCompany.Confidentialandproprietary.
SUCCESS STORIES
● Databricks has helped Nielsen Data Science reduce its SAS footprint by
40% for an annual savings of just under two million dollars.
● Reduced the runtime of the Viewer Assignment model from 36 days to 4
hours.
● The Databricks accelerated a Machine Learning model by removing
manual classification process and improve the AUC score from .6 to .8.
● Increased Databricks users from 9 to 156 users while partnering with
Databricks to reduce DBU usage by 45%.
The Area Under the Curve (AUC) is a common evaluation metric for binary classification problems. If the classifier is
very good, the area under the curve will be close to 1.

Copyright © 2017 The Nielsen Company. Confidential and proprietary.

10
Copyright©2017TheNielsenCompany.Confidentialandproprietary.
Move to Cloud Computing
Requirements:
● Flexible compute options
● Separate compute and storage
● Parallelization options
● Improved deployment speed

11
Copyright©2017TheNielsenCompany.Confidentialandproprietary.
CHALLENGE
● Legacy servers will not support newer versions of various languages,
packages and tools
● Older and outdated processor and memory capabilities
● Limited space capabilities and costly expansions
● Long computation times for tools and algorithms
● Over dependence on proprietary software ($$$)

Weitere ähnliche Inhalte

Was ist angesagt?

While we frequently talk about how to build interesting products on top of machine and event data, the reality is that collecting, organizing, providing access to, and managing this data is where most people get stuck. In this session, we’ll follow the flow of data through an end to end system built to handle tens of terabytes per day of event-oriented data, providing real time streaming, in-memory, SQL, and batch access to this data. We’ll go into detail on how open source systems such as Hadoop, Kafka, Solr, and Impala/Hive are actually stitched together; describe how and where to perform data transformation and aggregation; provide a simple and pragmatic way of managing event metadata; and talk about how applications built on top of this platform get access to data and extend its functionality. This session is especially recommended for data infrastructure engineers and architects planning, building, or maintaining similar systems.

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...

Data Con LA

HP ships millions of PCs, Printers, and other devices every year to customers in all market segments. More customers are seeking services provided with our products enabling new opportunities for HP to create services from the data we can collect from our devices. Every device we ship is an IoT endpoint with powerful CPU to capture rich data. Insights from this data are used internally to improve our products and focus on customer needs. In this presentation, John will focus on HP’s journey to enabling Big Data analytics from within a large enterprise environment. He will review the challenges and how HP decided on AWS, Apache Spark and Databricks as the foundation for their entry into Big Data Analytics. John will also review how HP uses Spark to build analytic services from the data they generate from their devices.

Unlocking Value in Device Data Using Spark: Spark Summit East talk by John La...

Spark Summit

Big Data Day LA 2016/ NoSQL track - Architecting Real Life IoT Architecture, ...

Data Con LA

Spark in the Enterprise - 2 Years Later by Alan Saldich

Spark Summit

Today’s Software Defined environments attempt to remove the weakness of computing hardware from the operational equation. There is no doubt that this is a natural progress away from overpriced, proprietary compute and storage layers. However, even at the heart of any Software Defined universe is an underlying hardware stack that must be robust, reliable and cost effective. Our 20+ years experience delivering over 2000 clusters and clouds has taught us how to properly design and engineer the right hardware solution for Big Data, Cluster and Cloud environments. This presentation will share this knowledge allowing user to make better design decisions for any deployment.

Big Data Day LA 2016/ Use Case Driven track - From Clusters to Clouds, Hardwa...

Data Con LA

In this webinar, we talk about what role traditional extraction, transformation and load (ETL) tools will play when it comes to big data analytics as the organizing principles of managing big (and small) data are in the midst of being rewritten. We also discuss new trends in the big data market, what’s changing and the impact of Spark, while reviewing some of the use cases we’re seeing for a more flexible integration platform as a service (iPaaS) that can handle multiple styles of ingesting, synchronizing and transforming big data sources. To learn more, visit: http://www.snaplogic.com/solutions/big-data

Webinar: Big Data Integration - Why Same Old, Same Old Won't Cut It

SnapLogic

How did eBay move their ETL computation from conventional RDBMS environment over to Spark? What did it take to go from a strategic vision to a viable solution? This paper will take you through a journey which lead to an implementation of a 1000+ node Spark Cluster running 10,000+ ETL jobs daily, all done in a span of less than 6 months, by a team with limited Spark experience. We will share the vision, technical architecture, critical Management decisions, Challenges and Road ahead. This will be a unique opportunity to look into this awesome Spark success story at eBay!

Moving eBay’s Data Warehouse Over to Apache Spark – Spark as Core ETL Platfor...

Databricks

Trusted, high-quality data and efficient use of data engineers’ time are critical success factors for AI/ML projects. Enterprise data is complex—it comes from several sources, in a variety of formats, and at varied speeds. For your machine learning projects on Apache Spark, you need a holistic approach to data engineering: finding & discovering, ingesting & integrating, server-less processing at scale, and data governance. Stop by this session for an overview on how to set up AI/ML projects for success while Informatica takes the heavy lifting out of your data engineering.

Successful AI/ML Projects with End-to-End Cloud Data Engineering

Databricks

The term “Lambda Architecture” stands for a generic, scalable and fault-tolerant data processing architecture. As the hyper-scale now offers a various PaaS services for data ingestion, storage and processing, the need for a revised, cloud-native implementation of the lambda architecture is arising. In this talk we demonstrate the blueprint for such an implementation in Microsoft Azure, with Azure Databricks — a PaaS Spark offering – as a key component. We go back to some core principles of functional programming and link them to the capabilities of Apache Spark for various end-to-end big data analytics scenarios. We also illustrate the “Lambda architecture in use” and the associated tread-offs using the real customer scenario – Rijksmuseum in Amsterdam – a terabyte-scale Azure-based data platform handles data from 2.500.000 visitors per year.

Lambda Architecture in the Cloud with Azure Databricks with Andrei Varanovich

Databricks

"In the oil and gas industry, utilizing vast amounts of data has long been identified as an important indicator of operational performance. The measurement of key performance indicators is a routine practice in well construction, but a systematic way of statistically analyzing performance against a large data bank of offset wells is not a common practice. The performance of statistical analysis in real-time is even less common. With the adoption of distributed computing platforms, like Apache Spark, new analysis opportunities become available to leverage large-scale time-series data sets to optimize performance. Two case studies are presented in this talk: the rate of penetration (ROP) and the amount of vibration per run. By collecting real-time, telemetry data and comparing it with historic sample datasets within the Databricks Unified Analytics Platform, the optimization team was able to quickly determine whether the performance being delivered matched or exceeded past performance with statistical certainty. This is extremely important while trying new techniques with data that is highly variable. By substituting anecdotal evidence with statistical analysis, decision making is more precise and better informed. In this talk we'll share how we accomplished this and the lessons learned along the way."

Unifying Streaming and Historical Telemetry Data For Real-time Performance Re...

Databricks

Machine learning algorithm itself is rarely the main barrier in building AI applications. Instead, the real culprit is the set of complex systems that prepares large-scale training and test data for the ML algorithms. Apache Spark is a huge leap forward in democratizing AI. However, it does not solve all the problems. Databricks CEO Ali Ghodsi explains how Databricks democratizes AI by making it easier to build end-to-end machine learning pipelines with Apache Spark.

Spark Summit Europe 2016 Keynote - Databricks CEO

Databricks

In the race to invent multi-million dollar business opportunities with exclusive insights, data scientists and engineers are hampered by a multitude of challenges just to make one use case a reality – the need to ingest data from multiple sources, apply real-time analytics, build machine learning algorithms, and intermix different data processing models, all while navigating around their legacy data infrastructure that is just not up to the task. This need has created the demand for Virtual Analytics, where the complexities of disparate data and technology silos have been abstracted away, coupled with a powerful range of analytics and processing horsepower, all in one unified data platform. This talk describes how Databricks is powering this revolutionary new trend with Apache Spark.

Virtualizing Analytics with Apache Spark: Keynote by Arsalan Tavakoli

Spark Summit

A couple of thousands of servers for a big data system is also a big investment. Microsoft Bing has figured out a way to fulfill our needs without signing a huge check. We have the technology to harvest spare cycles on underutilized servers. And we tweak the configurations in Hadoop and Spark to fit the flexible capacity base. We have saved hundreds of millions of dollars per year. Bing is adopting open source big data technologies for our offline data processing system. It requires a massive amount of capacity, which implies a significant bill. With collaboration with Windows and Azure, Research teams, we can harvest most of the needed capacity from our existing server fleet. We make use of the capacity on reserve servers while keeping them instantly available for emergency use; we allocate compute and storage to servers when they are not fully occupied. We updated Hadoop node decommission, HDFS block placement, YARN node label mapping, and a few other policies so that they can adapt to the capacity that is even less reliable than commodity servers. We brought open source capacity to Bing product with less than 1 percent of the cost we had done it through normal approach. We also extend the YARN and Spark framework to better fit the need of deep learning training and inferencing workloads in our system. This extension is equipping Bing with direct questioning and answering type of interactive query features. Big does not mean expensive. The audience can learn about the approach from Bing that they can make better use of their existing servers to do additional big data systems.

Free Servers to Build Big Data System on: Bing’s Approach

DataWorks Summit

Hadoop and Friends as Key Enabler of the IoE - Continental's Dynamic eHorizon

DataWorks Summit/Hadoop Summit

The Life of an Internet of Things Electron

DataWorks Summit/Hadoop Summit

Supercharging Data Performance for Real-Time Data Analysis

Ryft

Technical Leads and Databricks Champions Darren Fuller & Sandy May will give a fast paced view of how they have productionised Data Quality Pipelines across multiple enterprise customers. Their vision to empower business decisions on data remediation actions and self healing of Data Pipelines led them to build a library of Data Quality rule templates and accompanying reporting Data Model and PowerBI reports. With the drive for more and more intelligence driven from the Lake and less from the Warehouse, also known as the Lakehouse pattern, Data Quality at the Lake layer becomes pivotal. Tools like Delta Lake become building blocks for Data Quality with Schema protection and simple column checking, however, for larger customers they often do not go far enough. Notebooks will be shown in quick fire demos how Spark can be leverage at point of Staging or Curation to apply rules over data. Expect to see simple rules such as Net sales = Gross sales + Tax, or values existing with in a list. As well as complex rules such as validation of statistical distributions and complex pattern matching. Ending with a quick view into future work in the realm of Data Compliance for PII data with generations of rules using regex patterns and Machine Learning rules based on transfer learning.

Building Data Quality pipelines with Apache Spark and Delta Lake

Databricks

Hadoop Hadoop & Spark meetup - Altiscale

Mark Kerzner

Life occurs in real-time, and not surprisingly, more solutions are being built using streaming technologies. Event-based architectures are becoming the norm, and customers are expecting immediate access to their data. This new world offers many exciting opportunities, but also some new challenges. What do you do when your streaming data is not complete? What if it relies on another data source? Does the dependent data exist yet, and does it come from a 3rd party? How do we merge a complete picture of data when data is sourcing from multiple places at the same time? A new norm in the world of distributed services. Join us as we dive deep into the technical details around these scenarios and more. Expect to learn about stream-stream joins, enriching stream data using local or remote data, and ways to anticipate and correct errors within the stream. Leave with a better understanding of managing data dependencies within a Spark Structured Streaming application.

Life is but a Stream

Databricks

Lance Olson. Cortana Analytics is a fully managed big data and advanced analytics suite that helps you transform your data into intelligent action. Come to this two-part session to learn how you can do "big data" processing and storage in Cortana Analytics. In the first part, we will provide an overview of the processing and storage services. We will then talk about the patterns and use cases which make up most big data solutions. In the second part, we will go hands-on, showing you how to get started today with writing batch/interactive queries, real-time stream processing, or NoSQL transactions all over the same repository of data. Crunch petabytes of data by scaling out your computation power to any sized cluster. Store any amount of unstructured data in its native format with no limits to file or account size. All of this can be done with no hardware to acquire or maintain and minimal time to setup giving you the value of "big data" within minutes. Go to https://channel9.msdn.com/ to find the recording of this session.

Cortana Analytics Workshop: The "Big Data" of the Cortana Analytics Suite, Pa...

MSAdvAnalytics

Was ist angesagt? (20)

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...

Unlocking Value in Device Data Using Spark: Spark Summit East talk by John La...

Big Data Day LA 2016/ NoSQL track - Architecting Real Life IoT Architecture, ...

Spark in the Enterprise - 2 Years Later by Alan Saldich

Big Data Day LA 2016/ Use Case Driven track - From Clusters to Clouds, Hardwa...

Webinar: Big Data Integration - Why Same Old, Same Old Won't Cut It

Moving eBay’s Data Warehouse Over to Apache Spark – Spark as Core ETL Platfor...

Successful AI/ML Projects with End-to-End Cloud Data Engineering

Lambda Architecture in the Cloud with Azure Databricks with Andrei Varanovich

Unifying Streaming and Historical Telemetry Data For Real-time Performance Re...

Spark Summit Europe 2016 Keynote - Databricks CEO

Virtualizing Analytics with Apache Spark: Keynote by Arsalan Tavakoli

Free Servers to Build Big Data System on: Bing’s Approach

Hadoop and Friends as Key Enabler of the IoE - Continental's Dynamic eHorizon

The Life of an Internet of Things Electron

Supercharging Data Performance for Real-Time Data Analysis

Building Data Quality pipelines with Apache Spark and Delta Lake

Hadoop Hadoop & Spark meetup - Altiscale

Life is but a Stream

Cortana Analytics Workshop: The "Big Data" of the Cortana Analytics Suite, Pa...

Andere mochten auch

We demonstrate how to use Spark Streaming to build a global News Scanner that scrapes news in near real time, and uses sophisticated text analysis, SimHash, Random Indexing and Streaming K-Means to produce a geopolitical monitoring tool that allows users to track major world events as they unfold. We highlight advanced spark techniques for scaling, including: using Apache NIFI to deliver data to Spark Streaming, using the Goose library with Spark to build web scrapers, how to de-duplicate streamed documents at scale using advanced techniques like SimHash, Random Indexing, and Streaming K-Means in order to detect, track and visualise “global media conversations” as they mutate over time.

Story Deduplication and Mutation with Antoine Amend and Andrew Morgan

Spark Summit

The LHC performance has reached new heights, in-turn the amount of data produced by the underlying infrastructure and observation systems has increased by an order of magnitude putting unprecedented load on the current data Logging Service. In this talk Jakub will introduce you to the core business of beam production at CERN and the motivations behind the evolution of the current CERN Accelerator Logging Service towards Hadoop and Apache Spark.

The Next CERN Accelerator Logging Service—A Road to Big Data with Jakub Wozni...

Spark Summit

The T-Digest has earned a reputation as a highly efficient and versatile sketching data structure; however, its applications as a fast generative model are less appreciated. Several common algorithms from machine learning use randomization of feature columns as a building block. Column randomization is an awkward and expensive operation when performed directly, but when implemented with generative T-Digests, it can be accomplished elegantly in a single pass that also parallelizes across Spark data partitions. In this talk Erik will review the principles of T-Digest sketching, and how T-Digests can be applied as generative models. He will explain how generative T-Digests can be used to implement fast randomization of columnar data, and conclude with demonstrations of T-Digest randomization applied to Variable Importance, Random Forest Clustering and Feature Reduction. Attendees will leave this talk with an understanding of T-Digest sketching, how T-Digests can be used as generative models, and insights into applying generative T-Digests to accelerate their own data science projects.

One-Pass Data Science In Apache Spark With Generative T-Digests with Erik Erl...

Spark Summit

In this talk, we’ll present techniques for visualizing large scale machine learning systems in Spark. These are techniques that are employed by Netflix to understand and refine the machine learning models behind Netflix’s famous recommender systems that are used to personalize the Netflix experience for their 99 millions members around the world. Essential to these techniques is Vegas, a new OSS Scala library that aims to be the “missing MatPlotLib” for Spark/Scala. We’ll talk about the design of Vegas and its usage in Scala notebooks to visualize Machine Learning Models.

VEGAS: The Missing Matplotlib for Scala/Apache Spark with Roger Menezes and D...

Spark Summit

Skewed data is the enemy when joining tables using Spark. It shuffles a large proportion of the data onto a few overloaded nodes, bottlenecking Spark’s parallelism and resulting in out of memory errors. The go-to answer is to use broadcast joins; leaving the large, skewed dataset in place and transmitting a smaller table to every machine in the cluster for joining. But what happens when your second table is too large to broadcast, and does not fit into memory? Or even worse, when a single key is bigger than the total size of your executor? Firstly, we will give an introduction into the problem. Secondly, the current ways of fighting the problem will be explained, including why these solutions are limited. Finally, we will demonstrate a new technique – the iterative broadcast join – developed while processing ING Bank’s global transaction data. This technique, implemented on top of the Spark SQL API, allows multiple large and highly skewed datasets to be joined successfully, while retaining a high level of parallelism. This is something that is not possible with existing Spark join types.

Working with Skewed Data: The Iterative Broadcast with Fokko Driesprong Rob K...

Spark Summit

Reverse geocoding is one of HERE Technologies most heavily used services. From its access logs geocodes can be extracted and then counted with respect to some cellulation of the earth–-creating a sparse heat map. For our Place & Address Search products we use such a heat map to define a notion of relative place importance to rank and index addresses and places. However, the large data size, sparsity, and variations in traffic from which different global heat maps may be derived, makes faithful visualization and comparison a challenge. Additionally, common implementations of spatial image processing techniques that can help address the aforementioned challenges don’t map directly onto Spark’s computing engine. In this talk Arvind Rao will describe implementations of histogram equalization and kernel-based sparse image processing methods on Spark. Histogram equalization, which is best known as a method of contrast enhancement, automatically normalizes images, facilitating comparison. Along the way, Arvind will talk about how HERE uses heat maps as a feature in their autocompletion service, and say just enough about perception of contrast to put histogram equalization in context.

Histogram Equalized Heat Maps from Log Data via Apache Spark with Arvind Rao

Spark Summit

Handling Data Skew Adaptively In Spark Using Dynamic Repartitioning

Spark Summit

The ad tech industry processes large volumes of pixel and server-to-server data for each online user’s click, impression, and conversion data. At Neustar, we process 10+ billion events per day, and all of our events are fed through a number of Spark ETL batch jobs. Many of our Spark jobs process over 100 terabytes of data per run, each job runs to completion in around 3.5 hours. This means we needed to optimize our jobs in specific ways to achieve massive parallelization while keeping the memory usage (and cost) as low as possible. Our talk is focused on strategies dealing with extremely large data. We will talk about the things we learned and the mistakes we made. This includes: – Optimizing memory usage using Ganglia – Optimizing partition counts for different types of stages and effective joins – Counterintuitive strategies for materializing data to maximize efficiency – Spark default settings specific to large scale jobs, and how they matter – Running Spark using Amazon EMR with more than 3200 cores – Review different types of errors and stack traces that occur with large-scale jobs and how to read and handle them – How to deal with large number of map output status when there are 100k partitions joining with 100k partitions – How to prevent serialization buffer overflow as well as map out status buffer overflow. This can easily happen when data is extremely large – How to effectively use partitioners to combine stages and minimize shuffle.

Optimal Strategies for Large Scale Batch ETL Jobs with Emma Tang

Databricks

You have the perfect use case for your Spark applications – whether it be batch processing or super fast near-real time streaming — Now, where to store your valuable data!? In this talk we take a look at four storage options; HDFS, HBase, Solr and Kudu. With so many to choose from, which will fit your use case? What considerations should be taken into account? What are the pros and cons, what are the similarities and differences and how do they fit in with your Spark application? Learn the answers to these questions and more with a look at design patterns and techniques, and sample code to integrate into your application immediately. Walk away with the confidence to propose the right architecture for your use cases and the development know-how to implement and deliver with success.

Storage Engine Considerations for Your Apache Spark Applications with Mladen ...

Spark Summit

As a Spark developer, do you want to quickly develop your Spark workflows? Do you want to test your workflows in a sandboxed environment similar to production? Do you want to write end-to-end tests for your workflows and add assertions on top of it? In just a few years, the number of users writing Spark jobs at LinkedIn have grown from tens to hundreds, and the number of jobs running every day has grown from hundreds to thousands. With the ever increasing number of users and jobs, it becomes crucial to reduce the development time for these jobs. It is also important to test these jobs thoroughly before they go to production. Currently, there is no way users can test their spark jobs end-to-end. The only way is to divide the spark jobs into functions and unit test the functions. We’ve tried to address these issues by creating a testing framework for Spark workflows. The testing framework enables the users to run their jobs in an environment similar to the production environment and on the data which is sampled from the original data. The testing framework consists of a test deployment system, a data generation pipeline to generate the sampled data, a data management system to help users manage and search the sampled data and an assertion engine to validate the test output. In this talk, we will discuss the motivation behind the testing framework before deep diving into its design. We will further discuss how the testing framework is helping the Spark users at LinkedIn to be more productive.

Beyond unit tests: Testing for Spark/Hadoop Workflows with Shankar Manian Ana...

Spark Summit

Catalyst is an excellent optimizer in SparkSQL, provides open interface for rule-based optimization in planning stage. However, the static (rule-based) optimization will not consider any data distribution at runtime. A technology called Adaptive Execution has been introduced since Spark 2.0 and aims to cover this part, but still pending in early stage. We enhanced the existing Adaptive Execution feature, and focus on the execution plan adjustment at runtime according to different staged intermediate outputs, like set partition numbers for joins and aggregations, avoid unnecessary data shuffling and disk IO, handle data skew cases, and even optimize the join order like CBO etc.. In our benchmark comparison experiments, this feature save huge manual efforts in tuning the parameters like the shuffled partition number, which is error-prone and misleading. In this talk, we will expose the new adaptive execution framework, task scheduling, failover retry mechanism, runtime plan switching etc. At last, we will also share our experience of benchmark 100 -300 TB scale of TPCx-BB in a hundreds of bare metal Spark cluster.

An Adaptive Execution Engine for Apache Spark with Carson Wang and Yucai Yu

Databricks

Apache Spark is an excellent tool to accelerate your analytics, whether you’re doing ETL, Machine Learning, or Data Warehousing. However, to really make the most of Spark it pays to understand best practices for data storage, file formats, and query optimization. This talk will cover best practices I’ve applied over years in the field helping customers write Spark applications as well as identifying what patterns make sense for your use case.

Lessons from the Field: Applying Best Practices to Your Apache Spark Applicat...

Databricks

This talk is about methods and tools for troubleshooting Spark workloads at scale and is aimed at developers, administrators and performance practitioners. You will find examples illustrating the importance of using the right tools and right methodologies for measuring and understanding performance, in particular highlighting the importance of using data and root cause analysis to understand and improve the performance of Spark applications. The talk has a strong focus on practical examples and on tools for collecting data relevant for performance analysis. This includes tools for collecting Spark metrics and tools for collecting OS metrics. Among others, the talk will cover sparkMeasure, a tool developed by the author to collect Spark task metric and SQL metrics data, tools for analysing I/O and network workloads, tools for analysing CPU usage and memory bandwidth, tools for profiling CPU usage and for Flame Graph visualization.

Apache Spark Performance Troubleshooting at Scale, Challenges, Tools, and Met...

Databricks

For predicting vehicle defects at BMW, a machine learning pipeline evaluating several thousand features was implemented. As important features can be useful for evaluating specific defects, a feature selection approach has been used. For further evaluating the importance of features, several feature selection techniques (filters and wrappers) have been implemented as ml PipelineStages for usage on dataframes for incorporation in a complete Spark ml Pipeline, including preprocessing and classification. The general steps for building custom Spark ml Estimators are presented. The API of the newly implemented feature selection techniques is demonstrated and results of a performance analysis are shown. Besides that, experiences gained and pitfalls that should be avoided are shared.

Building Custom ML PipelineStages for Feature Selection with Marc Kaminski

Spark Summit

There are lots of reasons why you might want to implement your own machine learning algorithms on Spark: you might want to experiment with a new idea, try and reproduce results from a recent research paper, or simply to use an existing technique that isn’t implemented in MLlib. In this talk, we’ll walk through the process of developing a new machine learning model for Spark. We’ll start with the basics, by considering how we’d design a parallel implementation of a particular unsupervised learning technique. The bulk of the talk will focus on the details you need to know to turn an algorithm design into an efficient parallel implementation on Spark: we’ll start by reviewing a simple RDD-based implementation, show some improvements, point out some pitfalls to avoid, and iteratively extend our implementation to support contemporary Spark features like ML Pipelines and structured query processing. You’ll leave this talk with everything you need to build a new machine learning technique that runs on Spark.

Building Machine Learning Algorithms on Apache Spark with William Benton

Spark Summit

The opportunity in accelerating Spark by improving its network data transfer facilities has been under much debate in the last few years. RDMA (remote direct memory access) is a network acceleration technology that is very prominent in the HPC (high-performance computing) world, but has not yet made its way to mainstream Apache Spark. Proper implementation of RDMA in network-oriented applications can improve scalability, throughput, latency and CPU utilization. In this talk we are going to present a new RDMA solution for Apache Spark that shows amazing improvements in multiple Spark use cases. The solution is under development in our labs, and is going to be released to the public as an open-source plug-in.

Accelerating Shuffle: A Tailor-Made RDMA Solution for Apache Spark with Yuval...

Spark Summit

Leveraging your data to make better decisions is something every business wants to do, but doing it correctly involves weeks or months of work from highly skilled, hard to find individuals. However, many of the laborious steps a machine learning specialist follows in creating a custom application can be automated with enough compute power and flexibility. By leveraging Spark to do machine learning at scale, Salesforce Einstein has created a system that lets individuals with domain knowledge, but no machine learning expertise, create high quality, high impact machine learning applications.

Low Touch Machine Learning with Leah McGuire (Salesforce)

Spark Summit

Feature hashing is a powerful technique for handling high-dimensional features in machine learning. It is fast, simple, memory-efficient, and well suited to online learning scenarios. While an approximation, it has surprisingly low accuracy tradeoffs in many machine learning problems. Feature hashing has been made somewhat popular by libraries such as Vowpal Wabbit and scikit-learn. In Spark MLlib, it is mostly used for text features; however, its use cases extend more broadly. Many Spark users are not familiar with the ways in which feature hashing might be applied to their problems. In this talk, I will cover the basics of feature hashing, and how to use it for all feature types in machine learning. I will also introduce a more flexible and powerful feature hashing transformer for use within Spark ML pipelines. Finally, I will explore the performance and scalability tradeoffs of feature hashing on various datasets.

Feature Hashing for Scalable Machine Learning with Nick Pentreath

Spark Summit

Big data tools are challenging to combine into a larger application: ironically, big data applications themselves do not tend to scale very well. These issues of integration and data management are only magnified by increasingly large volumes of data. Apache Spark provides strong building blocks for batch processes, streams and ad-hoc interactive analysis. However, users face challenges when putting together a single coherent pipeline that could involve hundreds of transformation steps, especially when confronted by the need of rapid iterations. This talk explores these issues through the lens of functional programming. It presents an experimental framework that provides full-pipeline guarantees by introducing more laziness to Apache Spark. This framework allows transformations to be seamlessly composed and alleviates common issues, thanks to whole program checks, auto-caching, and aggressive computation parallelization and reuse.

From Pipelines to Refineries: scaling big data applications with Tim Hunter

Databricks

Experimental Design for Distributed Machine Learning with Myles Baker

Databricks

Andere mochten auch (20)