A Pipeline for Distributed Topic and Sentiment Analysis of Tweets on Pivotal Greenplum Database

1. Big Data Pipeline for Topic and Sentiment Analysis, with Applications Srivatsan Ramanujam (@being_bayesian) Senior Data Scientist, Pivotal 11 Jan 2014 © Copyright 2013 Pivotal. All rights reserved. 1

2. Agenda Introduction The Problem The Platform The Pipeline Live Demo: Topic and Sentiment Analysis Engine Applications in real world customer engagements © Copyright 2013 Pivotal. All rights reserved. 2

3. Pivotal: A New Platform for a New Era Data-Driven Application Development Pivotal Data Science Labs App Fabric Data Fabric “The new Middleware” “The new Database” Cloud Fabric “The new OS” ...ETC “The new Hardware” © Copyright 2013 Pivotal. All rights reserved. 3

5. The Problem Make sense of large volumes of unstructured text and integrate this with the structured sources of data to make better predictions Approaches – Topic Analysis – Sentiment Analysis © Copyright 2013 Pivotal. All rights reserved. 5

7. Pivotal Greenplum MPP DB Think of it as multiple PostGreSQL servers Master Segments/Workers Rows are distributed across segments by a particular field (or randomly) © Copyright 2013 Pivotal. All rights reserved. 7

9. Data Parallelism Vs. Task Parallelism Data Parallelism: Little or no effort is required to break up the problem into a number of parallel tasks, and there exists no dependency (or communication) between those parallel tasks. – Ex: Build one Churn model for each state in the US simultaneously, when customer data is distributed by state code. Task Parallelism: Split the problem into independent sub-tasks which can executed in parallel. – Ex: Build one Churn model in parallel for the entire US, though customer data is distributed by state code. © Copyright 2013 Pivotal. All rights reserved. 9

10. User-Defined Functions (UDFs) PostgreSQL/Greenplum provide lots of flexibility in defining your own functions. Simple UDFs are SQL queries with calling arguments and return types. Definition: Execution: CREATE FUNCTION times2(INT) RETURNS INT AS $$ SELECT 2 * $1 $$ LANGUAGE sql; SELECT times2(1); times2 -------2 (1 row) © Copyright 2013 Pivotal. All rights reserved. 10

11. PL/X : X in {pgsql, R, Python, Java, Perl, C etc.} • Allows users to write Greenplum/PostgreSQL functions in the R/Python/Java, Perl, pgsql or C languages SQL Master Host The interpreter/VM of the language ‘X’ is installed on each node of the Greenplum Database Cluster • Data Parallelism: - PL/X piggybacks on Greenplum’s MPP architecture © Copyright 2013 Pivotal. All rights reserved. Standby Master Interconnect Segment Host Segment Segment Segment Host Segment Segment Segment Host Segment Segment Segment Host Segment Segment … 11

12. Going Beyond Data Parallelism Data Parallel computation via PL/Python libraries only allow us to run ‘n’ models in parallel. This works great when we are building one model for each value of the group by column, but we need parallelized algorithms to be able to build a single model on all the available data For this, we use MADlib – an open source library of parallel in-database machine learning algorithms. © Copyright 2013 Pivotal. All rights reserved. 12

13. Scalable, in-database ML • • • Open Source!https://github.com/madlib/madlib Works on Greenplum DB and PostgreSQL Active development by Pivotal - • © Copyright 2013 Pivotal. All rights reserved. Latest Release : 1.4 (Dec 2014) Downloads and Docs: http://madlib.net/ 13

14. MADlib In-Database Functions Descriptive Statistics Predictive Modeling Library Generalized Linear Models • Linear Regression • Logistic Regression • Multinomial Logistic Regression • Cox Proportional Hazards • Regression • Elastic Net Regularization • Sandwich Estimators (Huber white, clustered, marginal effects) Matrix Factorization • Single Value Decomposition (SVD) • Low-Rank © Copyright 2013 Pivotal. All rights reserved. Machine Learning Algorithms • Principal Component Analysis (PCA) • Association Rules (Affinity Analysis, Market Basket) • Topic Modeling (Parallel LDA) • Decision Trees • Ensemble Learners (Random Forests) • Support Vector Machines • Conditional Random Field (CRF) • Clustering (K-means) • Cross Validation Linear Systems • Sparse and Dense Solvers Sketch-based Estimators • CountMin (CormodeMuthukrishnan) • FM (Flajolet-Martin) • MFV (Most Frequent Values) Correlation Summary Support Modules Array Operations Sparse Vectors Random Sampling Probability Functions 14

15. Architecture User Interface “Driver” Functions (outer loops of iterative algorithms, optimizer invocations) High-level Abstraction Layer (iteration controller, ...) RDBMS Built-in Functions SQL, generated from specification Python with templated SQL Python Functions for Inner Loops (for streaming algorithms) Low-level Abstraction Layer (matrix operations, C++ to RDBMS type bridge, …) C++ RDBMS Query Processing (Greenplum, PostgreSQL, …) © Copyright 2013 Pivotal. All rights reserved. 15

16. MADlib on Hadoop • A subset of algorithms from MADlib on Pivotal Greenplum DB, work out of the box on HAWQ. • Other functions are being ported. • With the general availability and support for User Defined Functions in HAWQ, MADlib will attain full parity with GPDB © Copyright 2013 Pivotal. All rights reserved. 16

18. The Pipeline Tweet Stream D3.js Stored on HDFS Topic Analysis through MADlib pLDA (gpfdist) Loaded as external tables into GPDB © Copyright 2013 Pivotal. All rights reserved. Parallel Parsing of JSON and extraction of fields using PL/Python Sentiment Analysis through custom PL/Python functions 18

19. Topic Analysis – MADlib pLDA Natural Language Processing - GPText Filter relevant content Align Data Social Media Tokenizer Stemming, frequency filtering Prepare dataset for Topic Modeling Topic Graph Topic composition MADlib Topic Model Topic Clouds © Copyright 2013 Pivotal. All rights reserved. 19

20. Sentiment Analysis We don’t have labeled data for our problem (Tweets aren’t tagged with Sentiment) “Unpredictable” Semi-Supervised Sentiment Prediction can be achieved by dictionary look-ups of tokens in a Tweet, but without Context, Sentiment Prediction is futile! “Breakthrough” © Copyright 2013 Pivotal. All rights reserved. 20

21. Sentiment Analysis – PL/X Functions Break-up Tweets into tokens and tag their parts-of-speech Part-of-speech tagger1 1: Semi-Supervised Sentiment Classification Phrase Extraction Phrasal Polarity Scoring Use learned phrasal polarities to score sentiment of new tweets Sentiment Scored Tweets Parts-of-speech Tagger : Gp-Ark-Tweet-NLP (http://vatsan.github.io/gp-ark-tweet-nlp/) © Copyright 2013 Pivotal. All rights reserved. 21

24. Churn Models for Telecom Industry Goal – Identify and prevent customers who are likely to churn. Challenges – – – – Cost of acquiring new customers is high Recouping cost of customer acquisition high if customer is not retained long enough Lower barrier to switching subscribers With mobile number portability, barrier to switching even lower Good News – Cost of retaining existing customers is lower! © Copyright 2013 Pivotal. All rights reserved. 24

25. Structured Features for Churn Models The problem is extensively studied with a rich set of approaches in the literature Device Texting Stats Call Stats Rate Plans Customer Demographics These features are great, but the models soon hit a plateau with structured features! © Copyright 2013 Pivotal. All rights reserved. 25

26. Blending the Unstructured with the Structured What other sources of previously untapped data could we use ? Are our customers happy ? Where ? What segments ? What are the common topics in their conversations online ? © Copyright 2013 Pivotal. All rights reserved. 26

27. Sentiment Analysis and Topic Models MORE ACCURATE LIKELIHOOD TO CHURN Unstructured Data External Internal Sentiment Analysis Engine (Classifier) Topic Engine (LDA) Structured Data: EDW © Copyright 2013 Pivotal. All rights reserved. Topic Dashboard 27

28. Predicting Commodity Futures through Twitter Customer A major a agri-business cooperative Business Problem Predict price of commodity futures through Twitter Solution Built Sentiment Analysis and Text Regression algorithms to predict commodity futures from Tweets Established the foundation for blending the structured data (market fundamentals) with unstructured data (tweets) Challenges Language on Twitter does not adhere to rules of grammar and has poor structure No domain specific label corpus of tweet sentiment – problem is semi-supervised © Copyright 2013 Pivotal. All rights reserved. 28

29. The Approach • Tweets alone had significant predictive power for the commodity of interest to us. When blended with structured features like weather data we expect to see much better results. © Copyright 2013 Pivotal. All rights reserved. 29

31. Pivotal Open Source Contributions http://gopivotal.com/pivotal-products/open-source-software • PyMADlib – Python Wrapper for MADlib - https://github.com/gopivotal/pymadlib • PivotalR – R wrapper for MADlib - https://github.com/madlib-internal/PivotalR • Part-of-speech tagger for Twitter via SQL - http://vatsan.github.io/gp-ark-tweet-nlp/ Questions? @being_bayesian © Copyright 2013 Pivotal. All rights reserved. 31

32. BUILT FOR THE SPEED OF BUSINESS

A Pipeline for Distributed Topic and Sentiment Analysis of Tweets on Pivotal Greenplum Database

A Pipeline for Distributed Topic and Sentiment Analysis of Tweets on Pivotal Greenplum Database

Recomendados

Más contenido relacionado

Was ist angesagt?

Was ist angesagt?(20)

Destacado

Destacado(20)

Similar a A Pipeline for Distributed Topic and Sentiment Analysis of Tweets on Pivotal Greenplum Database

Similar a A Pipeline for Distributed Topic and Sentiment Analysis of Tweets on Pivotal Greenplum Database(20)

Más de Srivatsan Ramanujam

Más de Srivatsan Ramanujam(7)

Último

Último(20)

A Pipeline for Distributed Topic and Sentiment Analysis of Tweets on Pivotal Greenplum Database

Hinweis der Redaktion