weekly AI online tech talk: https://learn.xnextcon.com/event/eventdetails/W19071110

1. NLP IN PRACTICE
STREAMING
ANALYSIS OF
LIVE EVENTS
ANDREW LARIMER
@ANDREWLARIMER

3. If we produced these event,
what would we want to know?
3

4. 4
What did people think?
What did they like and what didn’t
they like?
What were people most excited
about?

5. 5
What were most people saying (i.e.
what were the trends in
conversation)?

6. 6
What characters/players stood out
to the fans?

7. With SENTIMENT ANALYSIS we can
tackle the first block of questions:
7
What did people think?
What did they like and what didn’t they like?
What were people most excited about?

8. 8
SENTIMENT ANALYSIS
vaderSentiment is:
- rule and lexicon-based
- easy to use
- assigns polarity and intensity
- handles social media usage & emojis
- handles negation, i.e.
“VADER is not smart, handsome, nor
funny.”

9. 9
SENTIMENT ANALYSIS
-1 1
MOST NEGATIVE MOST POSITIVE

10. 10
SENTIMENT ANALYSIS
pip install vaderSentiment

11. 11
SENTIMENT ANALYSIS
from vaderSentiment.vaderSentiment
import SentimentIntensityAnalyzer
analyzer = SentimentIntensityAnalyzer()
analyzer.polarity_scores(your_text_here)

12. 12
SENTIMENT ANALYSIS
{'neg': 0.0,
'neu': 1.0,
'pos': 0.0,
‘compound': 0.0}
“You’re my queen.”

13. 13
SENTIMENT ANALYSIS
{'neg': 0.379,
'neu': 0.621,
'pos': 0.0,
‘compound': -0.0572}
“I don’t want it.”

14. 14
SENTIMENT ANALYSIS
{'neg': 0.602,
'neu': 0.398,
'pos': 0.0,
‘compound': -0.6739}
“Steph Curry is a FRAUD.”

15. 15
SENTIMENT ANALYSIS
BUT let’s also explore its limitations

16. 16
SENTIMENT ANALYSIS
“Steph Curry got outscored by Fred VanVleet & Kyle
Lowry at home in an elimination game in the NBA Finals.”
{'neg': 0.0,
'neu': 1.0,
'pos': 0.0,
‘compound': 0.0}

17. 17
SENTIMENT ANALYSIS
“I love Jamie Lannister.”
{'neg': 0.0,
'neu': 0.323,
'pos': 0.677,
'compound': 0.6369}

18. 18
SENTIMENT ANALYSIS
“I love that Jaime Lannister got what he deserved.”
{'neg': 0.0,
'neu': 0.625,
'pos': 0.375,
'compound': 0.6369}

19. 19

20. 20

21. 21

22. 22

23. 23
With TOPIC CLUSTERING we can
tackle the second block of questions:
What were most people saying (i.e. what were the
trends in conversation)?

24. 24
TOPIC CLUSTERING
BERT:
- Bidirectional Encoder Representations
from Transformer
- Generates Word Embeddings
Word Embeddings
- Multi-dimensional numerical
representations of the context in which a
word is found

25. 25
TOPIC CLUSTERING
768
DIMENSIONAL REPRESENTATIONS

26. 26
TOPIC CLUSTERING
Github: hanxiao/bert-as-service
docker build -t bert-as-service -f ./docker/Dockerfile .
NUM_WORKER=1
PATH_MODEL=/PATH_TO/_YOUR_MODEL/
docker run --runtime nvidia -dit -p 5555:5555 -p 5556:5556
-v $PATH_MODEL:/model -t bert-as-service $NUM_WORKER

27. 27
TOPIC CLUSTERING
Github: hanxiao/bert-as-service
pip install bert-serving-client
from bert_serving.client import BertClient
bc = BertClient()
bc.encode(['First do it', 'then do it right', 'then do it
better'])

28. 28
TOPIC CLUSTERING

29. 29
Now, streaming…

30. 30
UNBOUNDED DATA
STREAMING
WINDOW
FINITE DATA
BATCH PROCESSING
STREAMING
WINDOW
STREAMING
WINDOW

31. 31
Spark
Apache Beam + Dataflow

32. MESSAGES
(WITH TIMESTAMP)
32
TWEETS (VIA
TWEEPY)
CLOUD
PUB/SUB
WINDOW
SENTIMENT
ANALYSIS
AGGREGATE
STATISTICS
WRITE TO
BIG QUERY
DATAFLOW
KUBERNETES
CLUSTER

33. 33
messages = []
for line in data_lines:
pub = base64.urlsafe_b64encode(line)
messages.append({'data': pub})
body = {'messages': messages}
resp = client.projects().topics().publish(
topic=pubsub_topic, body=body).execute(
num_retries=NUM_RETRIES)

34. 34
with Pipeline(options=options) as p:
results = (p | 'read_from_topic' >> ReadFromPubSub(topic=PUBSUB_TOPIC,
with_attributes=False,
timestamp_attribute='created_at')
| 'Window' >> WindowInto(window.FixedWindows(60))
| 'Emit_needed_values' >> FlatMap(emit_values,entity_map)
| 'Combine' >> CombinePerKey(EntityScoreCombine())
| 'Add Window Timestamp' >> beam.ParDo(AddWindowTimestampFn())
| 'FormatForWrite' >> Map(format_for_write)
| 'Write' >> WriteToBigQuery('streaming_scores',
dataset=BQ_DATASET,
project=PROJECT_ID,
create_disposition='CREATE_IF_NEEDED',
write_disposition='WRITE_APPEND',
batch_size=10)
)

35. MESSAGES
(WITH TIMESTAMP)
35
TWEETS (VIA
TWEEPY)
CLOUD
PUB/SUB
WINDOW
SENTIMENT
ANALYSIS
AGGREGATE
STATISTICS
WRITE TO
BIG QUERY
DATAFLOW
KUBERNETES
CLUSTER

37. THANKS!
Any questions?
You can find me at:
@andrewlarimer
andrew.larimer@springml.com
37