A Hybrid Recommender with Yelp Challenge Data

•

2 gefällt mir•997 views

Developed by Chao Shi, Sam O'Mullane, Sean Kickham, Reza Rad and Andrew Rubino Watch the project presentation: https://youtu.be/gkKGnnBenyk This project was completed by students from NYC Data Science Academy's 12-Week Bootcamp. Learn more about the bootcamp: http://nycdatascience.com/data-science-bootcamp/ People make decisions on where to eat based on friends’ recommendations. Since they know you, their suggestions matter more than those of strangers. For the capstone project, we built a hybrid Yelp recommendation system that can provide individualized recommendations based on your friend’s reviews on the social network. We built the machine learning models using Spark, and set up a Flask-Kafka-RDS-Databricks pipeline that allows a continuous stream of user requests. During the presentation, we will talk about the development framework and technical implementation of the pipeline. Read on their project posts and code: https://blog.nycdatascience.com/student-works/capstone/yelp-recommender-part-1/ https://blog.nycdatascience.com/student-works/yelp-recommender-part-2/

Daten & Analysen

A Hybrid Recommender with
Yelp Challenge Data
S. Kickham, A. Rubino, R. Rad, S. O’Mullane, C. Shi

● Intro
● Data Preparation & Exploratory Analysis
● Hybrid Recommendation System
● Data Pipeline
● Demo
● Experience & Future Work
Outline
Background
EDA
Recommendation
Algorithms
App Pipeline
Demo

Motivation
Business Need
Big Data
Machine
Learning
Yelper
Helper

Amazon RDS - MySQL Database
● RDS = Relational Database Service
● Main tables include:
○ Reviews - 4.1M
○ Tips - 947K
○ Businesses - 144K
○ Checkins - 125K
○ Users - 1M
● The main tables allow for simple subtable creation, which allows for fast
requests and data loads.

MySQL Schema Viz
● Primary Keys for all tables:
○ Businesses: business_id
○ Users: user_id
○ Reviews: review_id
○ Tips: user_id
○ Checkins: business_id
● RDS allowed us to create
relationships between tables.

Sentiment Analysis
Pre-
Processing
Neural
Network
Sentiment
bit.ly/sent_analysis
1. Binarize
2. Tokenize
3. Texts_to_Seq
1. Supervised
2. Convolution
3. Pooling
1. 94% Accuracy
2. Spot Check
3. Base Rec.

Cosine Similarity using Word2Vec
Pre-
Processing
Neural
Network
Similarity
bit.ly/word_2_vec
1. Tokenize
2. Stop Word
Removal
1. Unsupervised
2. Skip-gram
3. Hyperparameters
1. Cosine
Similarity
2. Rank

Word Similarity
https://adriancolyer.files.wordpress.com/2016/04/word2vec-king-queen-vectors.png?w=600

Word Algebra
https://adriancolyer.files.wordpress.com/2016/04/word2vec-king-queen-composition.png?w=600

Word Algebra: Food
https://adriancolyer.files.wordpress.com/2016/04/word2vec-king-queen-composition.png?w=600

Word Algebra: Examples
'filet_mignon' - 'beef' + 'seafood' =
'fork' + 'soup' =
'drink' + 'barley' =

Word Algebra: Examples
'filet_mignon' - 'beef' + 'seafood' = lobster tail
'fork' + 'soup' = spoon
'drink' + 'barley' = beer

NLP Summary
Sentiment
Analysis
Cosine
Similarity
Base Rec. Content Rec.

Collaborative Filtering
https://en.wikipedia.org/wiki/Collaborative_filtering

Collaborative Filtering
Matrix
Factorization
Cost Function

Review-level adjustments
How relevant is a 10 year old review?
Newest 60% of reviews unaffected by time-weighting
Reviews binned by years from present

Business-level adjustments
Log
Log transformations for popularity measures
Check Ins
Reviews
Multiplicative factor based
on NLP sentiment

Stars Percentile
99th (top 40)
95th (top 300)
83rd
60th
31st
5th
Final Location-only Ratings

Recommendation Engine
Radial Filter:
Haversine Distance
Location
Latitude
Longitude
User Info
User ID
User Interest
Keyword
User Requests
Required
Optional
Optional
Optional
Dynamic generation of up to 40 recommendations within 20s
Social Network Data Mining

Data Pipeline: 1. Flask
User ID
Keyword
Longitude
Latitude
Search Radius

Components:
1. Brokers
2. Topics
3. Producers
4. Consumers
Data Pipeline: 2. Kafka

Data Pipeline: 2. Kafka
Consumer
Producer Consumer
Producer
Topic 1
Topic 2
Kafka Server (Running on Zookeeper)

Data Pipeline: 3. Spark
Spark’s Structured Streaming

Weitere ähnliche Inhalte

Andere mochten auch

Hack session for NYTimes Dialect Map Visualization( developed by R Shiny)

Vivian S. Zhang

This project was completed by Scott Dobbins and Rachel Kogan, who enrolled in the NYC Data Science Academy's 12-Week Data Science Bootcamp. Learn more about the program: http://nycdatascience.com/data-science-bootcamp/ Given that both Wikipedia and comments sections of most websites are freely open to anyone to edit at any time, how has Wikipedia managed to remain such a useful resource while most comments sections are ridden with vandalism, ads, and other counterproductive user behavior? We believe the answer is two-fold: 1) Wikipedia has an army of bots that quickly identify and revert vandalism so that the worst edits are usually never seen by people and the site generally maintains itself in a well-kempt state, and 2) Wikipedia has a strong community of administrators and other contributors who routinely clean the site’s flagged contents. Vandalism is relatively easy to flag, though a few clever edits manage to stay on the site for a long time. What about site content problems that are more subjective, like bias? Wikipedia users do routinely manually flag pages with point-of-view (POV) issues, though with millions of pages and no machine-based approaches, the site can only manage to confidently maintain neutrality on the more well-trafficked pages. Here we propose a solution to solve some of the more intractable content issues for Wikipedia and other sites using Natural Language Processing (NLP) and machine learning approaches. The sheer quantity of data managed by Wikipedia and similar sites requires distributed computing approaches, so we show here how Apache Spark can upgrade common algorithms to run on massive data sets.

Wikipedia: Tuned Predictions on Big Data

Vivian S. Zhang

We're so skewed_presentation

Vivian S. Zhang

Introducing natural language processing(NLP) with r

Vivian S. Zhang

Data Science is concerned with the analysis of large amounts of data. When the volume of data is really large, it requires the use of cooperating, distributed machines. The most popular method of doing this is Hadoop, a collection of programs to perform computations on connected machines in a cluster. Hadoop began life as an open-source implementation of MapReduce, an idea first developed and implemented by Google for its own clusters. Though Hadoop's MapReduce is Java-based, and quite complex, this talk focuses on the "streaming" facility, which allows Python programmers to use MapReduce in a clean and simple way. We will present the core ideas of MapReduce and show you how to implement a MapReduce computation using Python streaming. The presentation will also include an overview of the various components of the Hadoop "ecosystem." NYC Data Science Academy is excited to welcome Sam Kamin who will be presenting an Introduction to Hadoop for Python Programmers a well as a discussion of MapReduce with Streaming Python. Sam Kamin was a professor in the University of Illinois Computer Science Department. His research was in programming languages, high-performance computing, and educational technology. He taught a wide variety of courses, and served as the Director of Undergraduate Programs. He retired as Emeritus Associate Professor, and worked at Google until taking his current position as VP of Data Engineering in NYC Data Science Academy. -------------------------------------- Our fall 12-Week Data Science bootcamp starts on Sept 21st,2015. Apply now to get a spot! If you are hiring Data Scientists, call us at (1)888-752-7585 or reach info@nycdatascience.com to share your openings and set up interviews with our excellent students.

Streaming Python on Hadoop

Vivian S. Zhang

Bayesian models in r

Vivian S. Zhang

Twitter: @NycDataSci Learn with our NYC Data Science Program (weekend courses for working professionals and 12 week full time for whom are advancing their career into Data Science) Our next 12-Week Data Science Bootcamp starts in Jun. (Deadline to apply is May 1st, all decisions will be made by May 15th.) ==================================== Max Kuhn, Director is Nonclinical Statistics of Pfizer and also the author of Applied Predictive Modeling. He will join us and share his experience with Data Mining with R. Max is a nonclinical statistician who has been applying predictive models in the diagnostic and pharmaceutical industries for over 15 years. He is the author and maintainer for a number of predictive modeling packages, including: caret, C50, Cubist and AppliedPredictiveModeling. He blogs about the practice of modeling on his website at ttp://appliedpredictivemodeling.com/blog --------------------------------------------------------- His Feb 18th course can be RSVP at NYC Data Science Academy. Syllabus Predictive Modeling using R Description This class will get attendees up to speed in predictive modeling using the R programming language. The goal of the course is to understand the general predictive modeling process and how it can be implemented in R. A selection of important models (e.g. tree-based models, support vector machines) will be described in an intuitive manner to illustrate the process of training and evaluating models. Prerequisites: Attendees should have a working knowledge of basic R data structures (e.g. data frames, factors etc) and language fundamentals such as functions and subsetting data. Understanding of the content contained in Appendix B sections B1 though B8 of Applied Predictive Modeling (free PDF from publisher [1]) should suffice. Outline: - An introduction to predictive modeling - R and predictive modeling: the good and bad - Illustrative example - Measuring performance - Data splitting and resampling - Data pre-processing - Classification trees - Boosted trees - Support vector machines If time allows, the following topics will also be covered - Parallel processing - Comparing models - Feature selection - Common pitfalls Materials: Attendees will be provided with a copy of Applied Predictive Modeling[2] as well as course notes, code and raw data. Participants will be able to reproduce the examples described in the workshop. Attendees should have a computer with a relatively recent version of R installed. About the Instructor: More about Max's work: [1] http://rd.springer.com/content/pdf/bbm%3A978-1-4614-6849-3%2F1.pdf [2] http://appliedpredictivemodeling.com

Max Kuhn's talk on R machine learning

Vivian S. Zhang

This project was completed by students graduated from NYC Data Science Academy 12-week Data Science Bootcamp. Learn more about the bootcamp: http://nycdatascience.com/data-science-bootcamp/ Watch the project presentation: https://youtu.be/W530d2ZdbJE Ranked #15 out of 3,274 teams on Kaggle Team Members - Brandy Freitas, Chase Edge and Grant Webb Given 4 years of housing price data in a foreign market, predicting the following year’s prices should be pretty straightforward, right? But what if in that last year of data, the country’s stock market, the value of its currency and the price of its number 1 export, all dropped by nearly 50%. And on top of all that, the country was slapped with economic sanctions by the EU and the US. This was Moscow in 2014 and as you can see, it was anything but straightforward. We were able to overcome these challenges and in the two weeks of working together, were able to achieve a top 1% ranking on Kaggle. Our success is a product of our in depth data cleaning, feature engineering and our approach to modeling. With a focus on interpretability and simplicity, we begin modeling using linear regression and decision trees which gave us a better understanding of the data. We then utilized more complicated models such as random forests and XGBoost which ultimately resulted in our top submission.

Kaggle Top1% Solution: Predicting Housing Prices in Moscow

Vivian S. Zhang

Data mining with caret package

Vivian S. Zhang

This talk was presented to NYC Open Data Meetup Group on Nov 11, 2014. Speaker: Daeil Kim is currently a data scientist at the Times and is finishing up his Ph.D at Brown University on work related to developing scalable inference algorithms for Bayesian Nonparametric models. His work at the Times spans a variety of problems related to the company's business interests, audience development, as well as developing tools to aid journalism. Topic: This talk will focus mostly on how machine learning can help problems that prop up in journalism. We'll begin first by talking about using popular supervised learning algorithms such as regularized Logistic Regression to help assist a journalist's work in uncovering insights into a story regarding the recall of Takata airbags in cars. Afterwards, we'll think about using topic modeling to deal with large document dumps generated from FOIA (Freedom of Information Act) requests and Refinery, a simple web based tool to ease the implementation of such tasks. Finally, if there is time, we will go over how topic models have been extended to assist in the problem of designing an efficient recommendation engine for text-based content.

Using Machine Learning to aid Journalism at the New York Times

Vivian S. Zhang

Winning data science competitions, presented by Owen Zhang

Vivian S. Zhang

Kaggle Winning Solution Xgboost algorithm -- Let us learn from its author

Vivian S. Zhang

Tong is a data scientist in Supstat Inc and also a master students of Data Mining. He has been an active R programmer and developer for 5 years. He is the author of the R package of XGBoost, one of the most popular and contest-winning tools on kaggle.com nowadays. Agenda: Introduction of Xgboost Real World Application Model Specification Parameter Introduction Advanced Features Kaggle Winning Solution

Xgboost

Vivian S. Zhang

Tips for data science competitions

Owen Zhang

Andere mochten auch (14)

Hack session for NYTimes Dialect Map Visualization( developed by R Shiny)

Wikipedia: Tuned Predictions on Big Data

We're so skewed_presentation

Introducing natural language processing(NLP) with r

Streaming Python on Hadoop

Bayesian models in r

Max Kuhn's talk on R machine learning

Kaggle Top1% Solution: Predicting Housing Prices in Moscow

Data mining with caret package

Using Machine Learning to aid Journalism at the New York Times

Winning data science competitions, presented by Owen Zhang

Kaggle Winning Solution Xgboost algorithm -- Let us learn from its author

Xgboost

Tips for data science competitions

Ähnlich wie A Hybrid Recommender with Yelp Challenge Data

In this retail-focused workshop, we review and solve some of the common technical challenges that retailers face. These include scaling their backend databases to accommodate fluctuating demand and enabling full-text product search to achieve more relevant product search results. Bring your laptops, because after reviewing the proposed solutions, you can get hands-on with Amazon ElastiCache for Redis and see how easy it is to reduce the cost and pressure to your backend database, while dramatically improving the performance. We also show how you can leverage the Amazon Elasticsearch Service for building a full-text search solution.

RET305-Turbo Charge Your E-Commerce Site wAmazon Cache and Search Solutions.pdf

Amazon Web Services

One of the trickiest problems with microservices is dealing with data as it becomes spread across many different bounded contexts. An event architecture and event-streaming platform like Kafka provide a respite to this problem. Event-first thinking has a plethora of other advantages too, pulling in concepts from event sourcing, stream processing, and domain-driven design. In this talk, Ben and Cornelia will tackle how to do the following: ● Transform the data monolith to microservices ● Manage bounded contexts for data fields that overlap ● Use event architectures that apply streaming technologies like Kafka to address the challenges of distributed data Speakers: Cornelia Davis, Author & VP, Technology, Pivotal Ben Stopford, Author & Technologist, Office of CTO, Confluent

Microservices, Events, and Breaking the Data Monolith with Kafka

VMware Tanzu

Lunch && Learn DevHub - 6 Things to Learn to become an AWS Genius

Andrew Brown

PHP Experience 2016 - [Workshop] Elastic Search: Turbinando sua aplicação PHP

iMasters

Fraud Detection and Prevention on AWS Speaker: Andrew Kane, Solutions Architect, AWS AI and real-time analytics are increasingly being used to quickly identify financial fraud. In this session, we will also give an overview of how to build and train a model to identify fraudulent card transactions from bona fide ones. We will also describe an architecture that allows you to feed live transaction data into a Machine Learning model to provide a real-time authentic / fraudulent classification of that transaction.

Fraud Detection and Prevention on AWS using Machine Learning

Amazon Web Services

Real-time predictive applications can demand continuous and agile development, with new models constantly being trained, tested, and then deployed. Training and testing are done by replaying stored event logs, running new models in the context of historical data in a form of backtesting or ``what if?'' analysis. To replay weeks or months of logs while developers wait, we need systems that can stream event logs through prediction logic many times faster than the real-time rate. A challenge with high-speed replay is preserving sequential semantics while harnessing parallel processing power. The crux of the problem lies with causal dependencies inherent in the sequential semantics of log replay. We introduce an execution engine that produces serial-equivalent output while accelerating throughput with pipelining and distributed parallelism. This is made possible by optimizing for high throughput rather than the traditional stream processing goal of low latency, and by aggressive sharing of versioned state, a technique we term Multi-Versioned Parallel Streaming (MVPS). In experiments we see that this engine, which we call ReStream, performs as well as batch processing and more than an order of magnitude better than a single-threaded implementation.

ReStream: Accelerating Backtesting and Stream Replay with Serial-Equivalent P...

Johann Schleier-Smith

Detective Controls: Gain Visibility and Record Change

Amazon Web Services

MongoDB.local Sydney: MongoDB Atlas for Your Enterprise

MongoDB

Planning Application Resilience

Jennifer Davis

The need for Natural Language Processing (NLP) is gaining more importance as the amount of unstructured text data doubles every 18 months and customers are looking to extend their existing analytics workloads to include natural language capabilities. Historically, this data had been prohibitively expensive to store and early manual processing evolved into rule-based systems, which were expensive to operate and inflexible. In this session we will show you how you can address this problem using Amazon Comprehend.

NEW LAUNCH! Natural Language Processing for Data Analytics - MCL343 - re:Inve...

Amazon Web Services

MyHeritage Cassandra meetup 2016

Ran Peled

Which database is best suited for your use case? Should you choose a relational database or NoSQL or a data warehouse for your workload? Would a managed service like Amazon RDS, Amazon DynamoDB, or Amazon Redshift work better for you, or would it be better to run your own database on Amazon EC2? FanDuel has been running its fantasy sports service on Amazon Web Services (AWS) since 2012. You will learn best practices and insights from FanDuel’s successful migrations from self-managed databases on EC2 to fully-managed database services.

AWS re:Invent 2016: Introduction to Managed Database Services on AWS (DAT307)

Amazon Web Services

Leveraging an in-house modeling framework for fun and profit

Carl Anderson

Data infrastructure architecture for medium size organization: tips for colle...

DataWorks Summit/Hadoop Summit

Redis overview for Software Architecture Forum

Christopher Spring

Fraud Detection with Amazon Machine Learning on AWS

Amazon Web Services

Custom rules created with AWS Config and AWS Lambda enables organizations to inspect, assess, and remediate changes to AWS resources. These tools provide the development speed and flexibility required for your team to quickly start and finish a job before it becomes an issue for your client. In this workshop, you practice using AWS Lambda to design and implement the AWS Config rules that you think an organization should have ready at a moment’s notice before their next client contacts them about an issue.

SRV334-Making Things Right with AWS Config Rules and AWS Lambda

Amazon Web Services

Visual analysis of 20 unique developer communities found within @GitHub US audience (Twitter followers) using Affinio. Insights for each community include: - Top coding languages - Top brands - Top influencers - Top media & entertainment (shows, networks, channels) - Top publishers - Top events, organizations, etc. - Top conversations & hashtags - Top links, articles, images, and media By the numbers: - 127k followers analyzed - 20 clusters - 1bn+ relationships mapped

US Software Developers - Github Audience Analysis

Affinio

This session is especially tailored for technology and consulting partners, looking to learn more about big data and analytics on AWS. As individuals and commerce move online, companies have unprecedented access to data to improve customer experience and take advantage of new market opportunities. However, organizations often struggle with turning data into actionable insights to drive their business. Learn how AWS and big data APN partners are helping companies enable a broad range of analytic capabilities, to deliver better business results and better serve their customers. We discuss key big data and analytics use cases, and programs to enable partners to get to market with these solutions.

GPSBUS202_Driving Customer Value with Big Data Analytics

Amazon Web Services

MongoDB.local Austin 2018: MongoDB Atlas for Your Enterprise

MongoDB

Ähnlich wie A Hybrid Recommender with Yelp Challenge Data (20)

RET305-Turbo Charge Your E-Commerce Site wAmazon Cache and Search Solutions.pdf

Microservices, Events, and Breaking the Data Monolith with Kafka

Lunch && Learn DevHub - 6 Things to Learn to become an AWS Genius

PHP Experience 2016 - [Workshop] Elastic Search: Turbinando sua aplicação PHP

Fraud Detection and Prevention on AWS using Machine Learning

ReStream: Accelerating Backtesting and Stream Replay with Serial-Equivalent P...

Detective Controls: Gain Visibility and Record Change

MongoDB.local Sydney: MongoDB Atlas for Your Enterprise

Planning Application Resilience

NEW LAUNCH! Natural Language Processing for Data Analytics - MCL343 - re:Inve...

MyHeritage Cassandra meetup 2016

AWS re:Invent 2016: Introduction to Managed Database Services on AWS (DAT307)

Leveraging an in-house modeling framework for fun and profit

Data infrastructure architecture for medium size organization: tips for colle...

Redis overview for Software Architecture Forum

Fraud Detection with Amazon Machine Learning on AWS

SRV334-Making Things Right with AWS Config Rules and AWS Lambda

US Software Developers - Github Audience Analysis

GPSBUS202_Driving Customer Value with Big Data Analytics

MongoDB.local Austin 2018: MongoDB Atlas for Your Enterprise

Mehr von Vivian S. Zhang

Why NYC DSA.pdf

Vivian S. Zhang

Career services workshop- Roger Ren

Vivian S. Zhang

Nycdsa wordpress guide book

Vivian S. Zhang

Our fall 12-Week Data Science bootcamp starts on Sept 21st,2015. Apply now to get a spot! If you are hiring Data Scientists, call us at (1)888-752-7585 or reach info@nycdatascience.com to share your openings and set up interviews with our excellent students. --------------------------------------------------------------- Come join our meet-up and learn how easily you can use R for advanced Machine learning. In this meet-up, we will demonstrate how to understand and use Xgboost for Kaggle competition. Tong is in Canada and will do remote session with us through google hangout. --------------------------------------------------------------- Speaker Bio: Tong is a data scientist in Supstat Inc and also a master students of Data Mining. He has been an active R programmer and developer for 5 years. He is the author of the R package of XGBoost, one of the most popular and contest-winning tools on kaggle.com nowadays. Pre-requisite(if any): R /Calculus Preparation: A laptop with R installed. Windows users might need to have RTools installed as well. Agenda: Introduction of Xgboost Real World Application Model Specification Parameter Introduction Advanced Features Kaggle Winning Solution Event arrangement: 6:45pm Doors open. Come early to network, grab a beer and settle in. 7:00-9:00pm XgBoost Demo Reference: https://github.com/dmlc/xgboost

Xgboost

Vivian S. Zhang

Scikit-learn is a machine learning library in Python, that has become a valuable tool for many data science practitioners. This talk will cover some of the more advanced aspects of scikit-learn, such as building complex machine learning pipelines, model evaluation, parameter search, and out-of-core learning. Apart from metrics for model evaluation, we will cover how to evaluate model complexity, and how to tune parameters with grid search, randomized parameter search, and what their trade-offs are. We will also cover out of core text feature processing via feature hashing. --------------------------------------------------------- Andreas is an Assistant Research Scientist at the NYU Center for Data Science, building a group to work on open source software for data science. Previously he worked as a Machine Learning Scientist at Amazon, working on computer vision and forecasting problems. He is one of the core developers of the scikit-learn machine learning library, and maintained it for several years. Material will be posted here: https://github.com/amueller/pydata-nyc-advanced-sklearn Blog: peekaboo-vision.blogspot.com Twitter: https://twitter.com/t3kcit

Nyc open-data-2015-andvanced-sklearn-expanded

Vivian S. Zhang

Nycdsa ml conference slides march 2015

Vivian S. Zhang

THE HACK ON JERSEY CITY CONDO PRICES explore trends in public data

Vivian S. Zhang

Natural Language Processing(SupStat Inc)

Vivian S. Zhang

Data Science Academy Student Demo day--Moyi Dang, Visualizing global public c...

Vivian S. Zhang

Data Science Academy Student Demo day--Divyanka Sharma, Businesses in nyc

Vivian S. Zhang

Data Science Academy Student Demo day--Chang Wang, dogs breeds in nyc

Vivian S. Zhang

Data Science Academy Student Demo day--Richard Sheng, kinvolved school attend...

Vivian S. Zhang

Data Science Academy Student Demo day--Peggy sobolewski,analyzing transporati...

Vivian S. Zhang

Data Science Academy Student Demo day--Michael blecher,the importance of clea...

Vivian S. Zhang

Data Science Academy Student Demo day--Shelby Ahern, An Exploration of Non-Mi...

Vivian S. Zhang

R003 laila restaurant sanitation report(NYC Data Science Academy, Data Scienc...

Vivian S. Zhang

R003 jiten south park episode popularity analysis(NYC Data Science Academy, D...

Vivian S. Zhang

Mehr von Vivian S. Zhang (17)