SlideShare ist ein Scribd-Unternehmen logo

stackconf 2023 | Analyzing Public Conversation using LDA and Topic Modeling, in Python by Sveta Gimpelson.pdf

NETWAYS
NETWAYS

In this talk, we will build “Choir”. An OSINT (Open-source intelligent) project focused on gathering context-based connections between social profiles using AI models like LDA and topic modeling, written in Python to explain what the world discusses over a specific domain and by high-ranking influencers in that domain and focus on what’s going on at the margins.

1 von 31
Downloaden Sie, um offline zu lesen
Analyzing Public Conversations
using LDA and Topic Modeling
Introducing Myself
2
Sveta Gimpelson
● CDO and Co-Founder at Memphis.dev
● Software engineer
● Excited about networks. And graphs
● Playing football on mom’s team
● Harry Potter fan
Analyzing Public Conversations of Influencers
3
Visualize the output
Analyze collected data
Collect data from social
networks
I want to know the highlights of what is happening in
the football domain.
Why Social Media
4 Source: Global social media statistics research summary 2023
Why Social Media
5 Source: Global social media statistics research summary 2023
Social Network Analysis (SNA)
6
A network is a number of points (or ‘nodes’) that are
connected by links.
Generally in social network analysis, the nodes are people
and the links are any social connection between them –
for example, friendship, marital/family ties, or financial ties.
follows/ retweeted/ liked a
tweet
replied to a question/ have >1
groups in common
?
Social Network Analysis (SNA)
7
Social network analysis aims to understand a community by
mapping the relationships that connect them as a network,
and then trying to draw out key individuals, groups within the
network (‘components’), and/or associations between the
individuals.
Source:
https://digi.uga.edu/network-graphs/
Social Network - Representation
8
● Graph - Network
● Nodes - People
● Links - Relationships
● Sub-Graph - Communities
9
Collect data - Find the influencers
10
Begin with 10 accounts
1. GOAL – @goal
2. ESPN FC – @ESPNFC
3. FourFourTwo – @FourFourTwo
4. BBC Sport – @BBCSport
5. WhoScored.com – @WhoScored
6. Squawka – @Squawka
7. OptaJoe – @OptaJoe
8. Bleacher Report Football – @brfootball
9. Sky Sports News – @SkySportsNews
10. Transfermarkt – @Transfermarkt
Collect data - Find the influencers
11
Ranking mechanism:
Based on the following parameters
1. Accounts that this accounts follow
2. Use keywords related to football
3. Have many followers
4. Accounts that been retweeted by our network
Follows
Ranking mechanism
Decision
IN
OUT
NEEDS REVIEW
The Expansion
14
15
Architecture
16
Architecture
17
Architecture
18
Collector
19
Rate Limit
Deduplication
Memphis’ Station
20
Memphis’ Station
21
Analyzer
22
Consumer
23
Preprocessing
24
Run Model
25
Upload To BigQuery
26
Visualize
27
Enhancements
28
Collector
● Produce to different partitions based on source/ account
● Collect from different sources - Facebook, Reddit, Telegram
● Enforce schema to the stations
● Use storage tiering - to enable batch processing (Apache iceberg,
Spark..)
Enhancements
29
Analyzer
● Consume from specific partition/ entire station/ communities
● Fine tuning to the LDA model
● Use other models/ combinations
● Use GPT to show more “readable” output
Enhancements
30
Visualization
● Other visualization tools
● Topics over time (TOT) - Evolution
● Create metrics
Thank you!
sveta@memphis.dev
https://twitter.com/memphisveta
Source code: HERE!

Recomendados

Τweetfix: Data Analytics on Match Fixing
Τweetfix: Data Analytics on Match FixingΤweetfix: Data Analytics on Match Fixing
Τweetfix: Data Analytics on Match FixingAntigoni-Maria Founta
 
User behavior model & recommendation on basis of social networks
User behavior model & recommendation on basis of social networks User behavior model & recommendation on basis of social networks
User behavior model & recommendation on basis of social networks Shah Alam Sabuj
 
Approaching Big Data: Lesson Plan
Approaching Big Data: Lesson Plan Approaching Big Data: Lesson Plan
Approaching Big Data: Lesson Plan Bessie Chu
 
Tackling the challenge of data Visualisation
Tackling the challenge of data VisualisationTackling the challenge of data Visualisation
Tackling the challenge of data VisualisationScott Abbott
 
Social media tools for audience research and measurement and relevant influen...
Social media tools for audience research and measurement and relevant influen...Social media tools for audience research and measurement and relevant influen...
Social media tools for audience research and measurement and relevant influen...Brilliant Noise
 
Virtual Class: Raising Visibility // Week 4
Virtual Class: Raising Visibility // Week 4Virtual Class: Raising Visibility // Week 4
Virtual Class: Raising Visibility // Week 4KDMC
 
FIX the Fixing PROactive quelling of SPORTS EVENTS manipulation
FIX the Fixing PROactive quelling of SPORTS EVENTS manipulationFIX the Fixing PROactive quelling of SPORTS EVENTS manipulation
FIX the Fixing PROactive quelling of SPORTS EVENTS manipulationapolon1hermes1
 

Más contenido relacionado

Ähnlich wie stackconf 2023 | Analyzing Public Conversation using LDA and Topic Modeling, in Python by Sveta Gimpelson.pdf

5 Latest Research Papers On ML You Must Read In 2020
5 Latest Research Papers On ML You Must Read In 20205 Latest Research Papers On ML You Must Read In 2020
5 Latest Research Papers On ML You Must Read In 2020Steven Wallach
 
[Brighton SEO] Audience Intelligence & SEO: How to integrate data sources to ...
[Brighton SEO] Audience Intelligence & SEO: How to integrate data sources to ...[Brighton SEO] Audience Intelligence & SEO: How to integrate data sources to ...
[Brighton SEO] Audience Intelligence & SEO: How to integrate data sources to ...Rory Hope
 
Social Media Analytics for Official Statistics
Social Media Analytics for Official StatisticsSocial Media Analytics for Official Statistics
Social Media Analytics for Official StatisticsIsmail Fahmi
 
20120706 dir res_pres4_02
20120706 dir res_pres4_0220120706 dir res_pres4_02
20120706 dir res_pres4_02robbratney
 
20120706 dir res_pres4_03
20120706 dir res_pres4_0320120706 dir res_pres4_03
20120706 dir res_pres4_03robbratney
 
Managing Entropy - Clarity '13 - Keith Goode
Managing Entropy - Clarity '13 - Keith GoodeManaging Entropy - Clarity '13 - Keith Goode
Managing Entropy - Clarity '13 - Keith GoodeKeith Goode
 
Measuring Social Media Return on Investment: Advanced social media analytics
Measuring Social Media Return on Investment: Advanced social media analyticsMeasuring Social Media Return on Investment: Advanced social media analytics
Measuring Social Media Return on Investment: Advanced social media analyticsHugh Stephens
 
Federated Ontology for Sports- Paper
Federated Ontology for Sports- PaperFederated Ontology for Sports- Paper
Federated Ontology for Sports- PaperGeorge Sam
 
'Stories & Numbers' - A Framework for Measuring Engagement
'Stories & Numbers' - A Framework for Measuring Engagement'Stories & Numbers' - A Framework for Measuring Engagement
'Stories & Numbers' - A Framework for Measuring EngagementJason Ryan
 
Raising visibility, awareness and reach for your online project
Raising visibility, awareness and reach  for your online projectRaising visibility, awareness and reach  for your online project
Raising visibility, awareness and reach for your online projectKDMC
 
Technology and open knowledge in sports statistics
Technology and open knowledge in sports statisticsTechnology and open knowledge in sports statistics
Technology and open knowledge in sports statisticsdwiederman
 
Pathways Summary Brief 25 Aug2010
Pathways Summary Brief   25 Aug2010Pathways Summary Brief   25 Aug2010
Pathways Summary Brief 25 Aug2010jmorriso
 
Raising Visibility of Your Project
Raising Visibility of Your ProjectRaising Visibility of Your Project
Raising Visibility of Your ProjectKDMC
 
Big social data analytics - social network analysis
Big social data analytics - social network analysis Big social data analytics - social network analysis
Big social data analytics - social network analysis Jari Jussila
 
How To Prepare For The Future Of Search
How To Prepare For The Future Of SearchHow To Prepare For The Future Of Search
How To Prepare For The Future Of SearchCharlene Li
 
Rubrics for DMPs
Rubrics for DMPsRubrics for DMPs
Rubrics for DMPsJisc RDM
 

Ähnlich wie stackconf 2023 | Analyzing Public Conversation using LDA and Topic Modeling, in Python by Sveta Gimpelson.pdf (20)

5 Latest Research Papers On ML You Must Read In 2020
5 Latest Research Papers On ML You Must Read In 20205 Latest Research Papers On ML You Must Read In 2020
5 Latest Research Papers On ML You Must Read In 2020
 
[Brighton SEO] Audience Intelligence & SEO: How to integrate data sources to ...
[Brighton SEO] Audience Intelligence & SEO: How to integrate data sources to ...[Brighton SEO] Audience Intelligence & SEO: How to integrate data sources to ...
[Brighton SEO] Audience Intelligence & SEO: How to integrate data sources to ...
 
Social Media Analytics for Official Statistics
Social Media Analytics for Official StatisticsSocial Media Analytics for Official Statistics
Social Media Analytics for Official Statistics
 
20120706 dir res_pres4_02
20120706 dir res_pres4_0220120706 dir res_pres4_02
20120706 dir res_pres4_02
 
20120706 dir res_pres4_03
20120706 dir res_pres4_0320120706 dir res_pres4_03
20120706 dir res_pres4_03
 
Managing Entropy - Clarity '13 - Keith Goode
Managing Entropy - Clarity '13 - Keith GoodeManaging Entropy - Clarity '13 - Keith Goode
Managing Entropy - Clarity '13 - Keith Goode
 
Measuring Social Media Return on Investment: Advanced social media analytics
Measuring Social Media Return on Investment: Advanced social media analyticsMeasuring Social Media Return on Investment: Advanced social media analytics
Measuring Social Media Return on Investment: Advanced social media analytics
 
Federated Ontology for Sports- Paper
Federated Ontology for Sports- PaperFederated Ontology for Sports- Paper
Federated Ontology for Sports- Paper
 
'Stories & Numbers' - A Framework for Measuring Engagement
'Stories & Numbers' - A Framework for Measuring Engagement'Stories & Numbers' - A Framework for Measuring Engagement
'Stories & Numbers' - A Framework for Measuring Engagement
 
Raising visibility, awareness and reach for your online project
Raising visibility, awareness and reach  for your online projectRaising visibility, awareness and reach  for your online project
Raising visibility, awareness and reach for your online project
 
Technology and open knowledge in sports statistics
Technology and open knowledge in sports statisticsTechnology and open knowledge in sports statistics
Technology and open knowledge in sports statistics
 
Pathways Summary Brief 25 Aug2010
Pathways Summary Brief   25 Aug2010Pathways Summary Brief   25 Aug2010
Pathways Summary Brief 25 Aug2010
 
nm
nmnm
nm
 
Raising Visibility of Your Project
Raising Visibility of Your ProjectRaising Visibility of Your Project
Raising Visibility of Your Project
 
Big social data analytics - social network analysis
Big social data analytics - social network analysis Big social data analytics - social network analysis
Big social data analytics - social network analysis
 
Proposal.docx
Proposal.docxProposal.docx
Proposal.docx
 
How To Prepare For The Future Of Search
How To Prepare For The Future Of SearchHow To Prepare For The Future Of Search
How To Prepare For The Future Of Search
 
Project Report 05_06_13
Project Report 05_06_13Project Report 05_06_13
Project Report 05_06_13
 
Rubrics for DMPs
Rubrics for DMPsRubrics for DMPs
Rubrics for DMPs
 
ONS Local presents: Explore Subnational Statistics
ONS Local presents: Explore Subnational StatisticsONS Local presents: Explore Subnational Statistics
ONS Local presents: Explore Subnational Statistics
 

Último

Reflection Vs Technical Rationality - By Dr. Cherinet Aytenfsu Weldearegay (2...
Reflection Vs Technical Rationality - By Dr. Cherinet Aytenfsu Weldearegay (2...Reflection Vs Technical Rationality - By Dr. Cherinet Aytenfsu Weldearegay (2...
Reflection Vs Technical Rationality - By Dr. Cherinet Aytenfsu Weldearegay (2...Dr. Cherinet Aytenfsu Weldearegay
 
Monthly HSE Report March for overall HSE
Monthly HSE Report March for overall HSEMonthly HSE Report March for overall HSE
Monthly HSE Report March for overall HSEOlgaOliveaJohn
 
Freeman_Abigail Personal Brand Exploration
Freeman_Abigail Personal Brand ExplorationFreeman_Abigail Personal Brand Exploration
Freeman_Abigail Personal Brand Explorationabbytoliver
 
TheSimpsons_Fandom_Assignment_4.5pc.pptx
TheSimpsons_Fandom_Assignment_4.5pc.pptxTheSimpsons_Fandom_Assignment_4.5pc.pptx
TheSimpsons_Fandom_Assignment_4.5pc.pptxStevenLuker3
 
Vazquez_Gabriel_SPCASTNG_PB1_2024-02.pdf
Vazquez_Gabriel_SPCASTNG_PB1_2024-02.pdfVazquez_Gabriel_SPCASTNG_PB1_2024-02.pdf
Vazquez_Gabriel_SPCASTNG_PB1_2024-02.pdfgabev3104
 
Exploring Wisdom-Based Leadership.pptx
Exploring Wisdom-Based Leadership.pptxExploring Wisdom-Based Leadership.pptx
Exploring Wisdom-Based Leadership.pptxAkash Das
 
Garcia_RobertDaniel_SPCSTA_PB1_2024-02.pptx
Garcia_RobertDaniel_SPCSTA_PB1_2024-02.pptxGarcia_RobertDaniel_SPCSTA_PB1_2024-02.pptx
Garcia_RobertDaniel_SPCSTA_PB1_2024-02.pptx0461620
 
Issues affecting LGBT as they grow older.pptx
Issues affecting LGBT as they grow older.pptxIssues affecting LGBT as they grow older.pptx
Issues affecting LGBT as they grow older.pptxbill846304
 
Relationships in Teaching By Dr. Cherinet Aytenfsu Weldearegay (2023).pdf
Relationships in Teaching By Dr. Cherinet Aytenfsu Weldearegay (2023).pdfRelationships in Teaching By Dr. Cherinet Aytenfsu Weldearegay (2023).pdf
Relationships in Teaching By Dr. Cherinet Aytenfsu Weldearegay (2023).pdfDr. Cherinet Aytenfsu Weldearegay
 

Último (10)

Tethex Cards - Crypto VISA Card - Invest 2 ENGLISH
Tethex Cards - Crypto VISA Card - Invest 2 ENGLISHTethex Cards - Crypto VISA Card - Invest 2 ENGLISH
Tethex Cards - Crypto VISA Card - Invest 2 ENGLISH
 
Reflection Vs Technical Rationality - By Dr. Cherinet Aytenfsu Weldearegay (2...
Reflection Vs Technical Rationality - By Dr. Cherinet Aytenfsu Weldearegay (2...Reflection Vs Technical Rationality - By Dr. Cherinet Aytenfsu Weldearegay (2...
Reflection Vs Technical Rationality - By Dr. Cherinet Aytenfsu Weldearegay (2...
 
Monthly HSE Report March for overall HSE
Monthly HSE Report March for overall HSEMonthly HSE Report March for overall HSE
Monthly HSE Report March for overall HSE
 
Freeman_Abigail Personal Brand Exploration
Freeman_Abigail Personal Brand ExplorationFreeman_Abigail Personal Brand Exploration
Freeman_Abigail Personal Brand Exploration
 
TheSimpsons_Fandom_Assignment_4.5pc.pptx
TheSimpsons_Fandom_Assignment_4.5pc.pptxTheSimpsons_Fandom_Assignment_4.5pc.pptx
TheSimpsons_Fandom_Assignment_4.5pc.pptx
 
Vazquez_Gabriel_SPCASTNG_PB1_2024-02.pdf
Vazquez_Gabriel_SPCASTNG_PB1_2024-02.pdfVazquez_Gabriel_SPCASTNG_PB1_2024-02.pdf
Vazquez_Gabriel_SPCASTNG_PB1_2024-02.pdf
 
Exploring Wisdom-Based Leadership.pptx
Exploring Wisdom-Based Leadership.pptxExploring Wisdom-Based Leadership.pptx
Exploring Wisdom-Based Leadership.pptx
 
Garcia_RobertDaniel_SPCSTA_PB1_2024-02.pptx
Garcia_RobertDaniel_SPCSTA_PB1_2024-02.pptxGarcia_RobertDaniel_SPCSTA_PB1_2024-02.pptx
Garcia_RobertDaniel_SPCSTA_PB1_2024-02.pptx
 
Issues affecting LGBT as they grow older.pptx
Issues affecting LGBT as they grow older.pptxIssues affecting LGBT as they grow older.pptx
Issues affecting LGBT as they grow older.pptx
 
Relationships in Teaching By Dr. Cherinet Aytenfsu Weldearegay (2023).pdf
Relationships in Teaching By Dr. Cherinet Aytenfsu Weldearegay (2023).pdfRelationships in Teaching By Dr. Cherinet Aytenfsu Weldearegay (2023).pdf
Relationships in Teaching By Dr. Cherinet Aytenfsu Weldearegay (2023).pdf
 

stackconf 2023 | Analyzing Public Conversation using LDA and Topic Modeling, in Python by Sveta Gimpelson.pdf