Spark tutorial py con 2016 part 2

David Taieb
STSM - IBM Cloud Data Services
Developer advocate
david_taieb@us.ibm.com
HANDS-ON SESSION:
DEVELOPING ANALYTIC APPLICATIONS
USING APACHE SPARK™ AND PYTHON

Part 2: Analyzing car twiQer data with Spark
and DashDb
PyCon 2016, Portland

©2016 IBM Corpora6on

Agenda
•  Provision the applica6on services on Bluemix:
Spark, DashDb, IBM Insight for TwiJer
•  Load car related tweets from IBM Insight for
TwiJer into DashDb warehouse
•  Run Analy6cs in Python Notebook and
discover new insights


Sign up for Bluemix
•  Access IBM Bluemix website on hJps://console.ng.bluemix.net
•  Click on Get Started for Free
•  Complete the form and click Create account
•  Look for conﬁrma6on email and click on conﬁrm you account link
Create new Space


Create a new space on Bluemix
In prepara6on for running the project, we create a new space on Bluemix
Create a Spark Instance
Op6onal: You can skip this step if you already have a
space with Spark instance that you would like to reuse


Create a Spark Instance


Create New Spark Instance


Acquiring the data
•  In the next sec6on, we show how to acquire
the twiJer data and store it into DashDb.
•  We use the TwiJer loading connector
available as a menu in DashDb console
Create a DashDb instance


Create an instance of IBM Dash DB on Bluemix
Create an IBM Insight for TwiJer instance


Create an instance of IBM Insight for Twitter on Bluemix


Launch DashDb Console
Click on the DashDb Service 6le to open this dashboard, then click on Launch buJon
Load TwiJer Data


Load Twitter Data
DashDb Console oﬀered mul6ple data
connectors including a TwiJer
connector that automa6cally
connects to IBM Insight for TwiJer
Connect to TwiJer


Connect to Twitter
Reusing the TwiJer
service instance
created in previous
step


Select the data to be loaded
TwiJer Query being used:
posted:2015-01-01,2015-12-31 followers_count:2000 listed_count:1000 (volkswagen OR vw OR toyota
OR daimler OR mercedes OR bmw OR gm OR "general motors" OR tesla)
Specify twiJer query
Provide preview count of
output data


Select the DashDb Table
Name of the schema under which the tables will be created
Preﬁx (Namespace) for the created tables
List of tables that will be created


Loading data monitoring page
Warning: loading 6me may vary
based on bandwidth. It may take
between 15 mns and 1hour


Complete the load: Statistics


Complete the load: explore the data


Get connection information
Copy the User id, password and jdbc url, you’ll need this informa6on later


Create new Notebook from URL
Import required Python packages
• Create notebook from URL
• Use hJps://github.com/ibm-cds-labs/spark.samples/raw/master/notebook/
DashDB%20TwiJer%20Car%202015%20Python%20Notebook.ipynb


Step 1: Import Python Packages
• Install nltk package (Natural language toolkit)
• We will use it to ﬁlter stop words later in the tutorial


Import Python modules and setup the SQLContext


Step 2: Define global Variables
Set up various data structures we’ll
need throughout the Notebook
This is the SCHEMA and PREFIX you used in
Step 3 of the TwiJer connector wizard


Set up some global helper functions
JavaScript Google map visualiza6on
Misc helper that ﬁll in missing dates


Step 3: Acquire the data from DashDB
User ID and password from Connec6on page
User ID and password from Connec6on page


Join the Tweets and Sentiment Table
In this step, we want to add a sen6ment score for each tweet record:
•  Join the Tweets and Sen6ments table
•  Encode the sen6ment into a number e.g. POSITIVE=+1, NEGATIVE=-1, AMBIVALENT=0
•  Create an average for each sen6ment associated with a tweet
•  %6me instruments the code to provide proﬁle execu6on stats.


Step 4: Transform the data
Create a clean Working dataframe
that will be easier to use in our
analy6cs


Step 5: Geographic distribution of tweets
GroupBy countries and aggregate the tweets count
Convert Spark SQL dataframe to Pandas data
structure for visualiza6on


Bar chart visualization of Tweet distribution by Geo


Google map visualization of tweet distribution by Geos
Call GeoChart Helper that set up the JavaScript code


Clean up memory before next analytics
Resources including memory on the Spark Driver machine are not inﬁnite.
It is good prac6ce to clean up when data is not needed anymore


Step 6: Analyzing tweets sentiment
GroupBy Sen6ments and aggregate the tweets count


Sentiment visualization
Use Matplot pie chart


Step 7: Analyze Tweet timeline
GroupBy Pos6ng 6me and sen6ment tuples
Aggregate the tweet counts
GroupBy Pos6ng 6me and sen6ment tuples
Aggregate the sum of the tweet counts


Prepare the timeline data structures


Time series visualization for all tweets


Deep dive into car manufacturers
Create new DataFrame that enrich tweets with
extra metadata:
-Boolean for each car manufacturer
-Boolean for electric car
-Boolean for self driving car


Re-analyze tweeter timeline for each car manufacturer
Create new DataFrame for each car manufacturer
Aggregate the tweet counts, order by pos6ng 6me


Timeline series visualization
No6ce the peak of tweets for VW between
September and October 2015


Explain why the peak of tweets for VW between September and October 2015
Filter for all VW tweets between Sept and Oct 2015
Pie chart visualiza6on of the top 10 words being
used in these tweets
Create map count of all non-stop words
used in the tweets
Use NLTK stopwords module to ﬁlter out stop words


Peak explained
We can clearly see from the list of
most used words that the peak
correspond to the VW scandal
around fraudulent emissions
tes6ng


Follow the notebook for many more
interesting analytics


Resource
•  hJps://developer.ibm.com/clouddataservices/
•  hJps://github.com/ibm-cds-labs/simple-data-pipe
•  hJps://github.com/ibm-cds-labs/pipes-connector-ﬂightstats
•  hJp://spark.apache.org/docs/latest/mllib-guide.html
•  hJps://console.ng.bluemix.net/data/analy6cs/


Thank You

Spark tutorial py con 2016 part 2

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie Spark tutorial py con 2016 part 2

Ähnlich wie Spark tutorial py con 2016 part 2 (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Spark tutorial py con 2016 part 2