SlideShare ist ein Scribd-Unternehmen logo
1 von 30
Downloaden Sie, um offline zu lesen
Context Suggestion:
Empirical Evaluations vs User Studies
Yong Zheng
School of Applied Technology
Illinois Institute of Technology
Chicago, IL, 60616, USA
The 2017 IEEE/WIC/ACM Conference on Web Intelligence (WI)
August 23-26, 2017, Leipzig, Germany
Agenda
• Intro: Context-aware Recommender Systems
• Motivations: Context Suggestion
• Methodologies and Research Problems
– Direct Context Prediction
– Indirect Context Suggestion
• Experimental Results and Findings
• Conclusions and Future Work
2
Agenda
• Intro: Context-aware Recommender Systems
• Motivations: Context Suggestion
• Methodologies and Research Problems
– Direct Context Prediction
– Indirect Context Suggestion
• Experimental Results and Findings
• Conclusions and Future Work
3
Recommender System (RS)
• RS: item recommendations tailored to user tastes
4
Context-Aware Recommendation
5
Companion
User’s decision may vary from contexts to contexts
• Examples:
➢ Travel destination: in winter vs in summer
➢ Movie watching: with children vs with partner
➢ Restaurant: quick lunch vs business dinner
➢ Music: for workout vs for study
Terminology in CARS
6
• Example of Multi-dimensional Context-aware Data set
➢Context Dimension: time, location, companion
➢Context Condition: Weekend/Weekday, Home/Cinema
➢Context Situation: {Weekend, Home, Kids}
User Item Rating Time Location Companion
U1 T1 3 Weekend Home Kids
U1 T2 5 Weekday Home Partner
U2 T2 2 Weekend Cinema Partner
U2 T3 3 Weekday Cinema Family
U1 T3 ? Weekend Cinema Kids
What is Context?
7
The most common contextual variables:
➢Time and Location
➢User intent or purpose
➢User emotional states
➢Devices
➢Topics of interests, e.g., apple vs. Apple
➢Others: companion, weather, budget, etc
Usually, the selection/definition of contexts is a domain-specific problem
Agenda
• Intro: Context-aware Recommender Systems
• Motivations: Context Suggestion
• Methodologies and Research Problems
– Direct Context Prediction
– Indirect Context Suggestion
• Experimental Results and Findings
• Conclusions and Future Work
8
Motivations: Context-Drive Applications
1) Context is necessary to maximize the user
experience. A list of good item
recommendations is NOT enough.
2) It is difficult to collect context information on
the Web!!!! Context suggestion provides a way
for context acquisition
9
Motivation: User Experience
10
• San Diego Zoo • San Diego Zoo Safari Park
Motivation: User Experience
11
Motivation: User Experience
12
13
Motivation: Context Acquisition
• Google Music
Agenda
• Intro: Context-aware Recommender Systems
• Motivations: Context Suggestion
• Methodologies and Research Problems
– Direct Context Prediction
– Indirect Context Suggestion
• Experimental Results and Findings
• Conclusions and Future Work
14
Solution for Context Suggestion
• Direct Context Prediction
The output is a binary prediction
The value “1” indicates appropriate suggestion
The value “0” tells inappropriate suggestion
• Indirect Context Suggestion
The output is a top-N recommendations
Task: Suggest appropriate contexts for a user to
enjoy a given item
15
Direct Context Prediction
16
Treat context conditions as binary labels
Utilize multi-label classification as solution
Indirect Context Suggestion
17
• Context-aware Recommendation
Task:
Given a user and context info
Recommend a list of items
Indirect Context Suggestion
18
• Indirect Context Suggestion
Task:
Given a user and an item
Recommend a list of appropriate
contexts for the users to enjoy
the items
Item-Aware Context Recommendation
Research Problems
• Direct Context Prediction was explored in WI’14,
but Indirect Context Suggestion was never
discussed and compared with the direct context
prediction
• Previous research infers user preferences on
contexts from contextual ratings on the items – it is
not validated that whether contextual ratings can
tell whether a user prefers a given contexts to enjoy
the items
• There are no evaluation standards for context
suggestion
19
Agenda
• Intro: Context-aware Recommender Systems
• Motivations: Context Suggestion
• Methodologies and Research Problems
– Direct Context Prediction
– Indirect Context Suggestion
• Experimental Results and Findings
• Conclusions and Future Work
20
Data Collection
• It’s first time to collect user’s tastes on contexts
• 5043 ratings by 97 users on 79 movies
21
Experimental Settings
• 5-fold Cross Validation
• Direct Context Prediction
– Classification Chains (MLC_CC)
– Label Powerset (MLC_LP)
• Indirect Context Suggestion
– Tensor Factorization (TF)
– Context-aware Matrix Factorization (CAMF)
– Contextual Sparse Linear Methods (CSLIM)
22
Evaluation Mechanisms
We propose the evaluation standards for context suggestion
• Top-N Context Prediction
– N varies from 1 to the number of context conditions
– Any available N value is fine
– It does not matter if two contexts from a same variable are
suggested. For example, {weekend, weekday, home, kids}
• Exact Context Suggestion
– Top-N, but N = the number of context condition
– For each dimension, we only suggest one condition
For example, {weekend, home, kids}
23
Results (Top-N Suggestion)
24
By using contextual ratings as ground truth for evaluation purpose
Results (Top-N Suggestion)
25
By using user tastes on contexts (from user surveys) as ground truth
Results (Exact Suggestion)
26
Results (Exact Suggestion)
27
Conclusions and Findings
28
• “General” indicates user’s general preferences on
contexts for movie watching, without considering
which movie it is
• Personalization is required, since many algorithms
outperform the “General” method
• UISplitting and TF are the best ones
• Indirect context suggestion is better to offer better
suggestions than direct context prediction
• The results by using contextual ratings and user
tastes on contexts are consistent.
Future Work
29
• We will try to collect more data and evaluate these
solutions on larger data set
• We will try to utilize the context suggestion
methods to predict emotional states
• We will seek solutions to improve the indirect
context prediction
Context Suggestion:
Empirical Evaluations vs User Studies
Yong Zheng
School of Applied Technology
Illinois Institute of Technology
Chicago, IL, 60616, USA
The 2017 IEEE/WIC/ACM Conference on Web Intelligence (WI)
August 23-26, 2017, Leipzig, Germany

Weitere ähnliche Inhalte

Was ist angesagt?

Techniques for Context-Aware and Cold-Start Recommendations
Techniques for Context-Aware and Cold-Start RecommendationsTechniques for Context-Aware and Cold-Start Recommendations
Techniques for Context-Aware and Cold-Start RecommendationsMatthias Braunhofer
 
Context-aware Recommendation: A Quick View
Context-aware Recommendation: A Quick ViewContext-aware Recommendation: A Quick View
Context-aware Recommendation: A Quick ViewYONG ZHENG
 
Hybrid Solution of the Cold-Start Problem in Context-Aware Recommender Systems
Hybrid Solution of the Cold-Start Problem in Context-Aware Recommender SystemsHybrid Solution of the Cold-Start Problem in Context-Aware Recommender Systems
Hybrid Solution of the Cold-Start Problem in Context-Aware Recommender SystemsMatthias Braunhofer
 
Hybridisation Techniques for Cold-Starting Context-Aware Recommender Systems
Hybridisation Techniques for Cold-Starting Context-Aware Recommender SystemsHybridisation Techniques for Cold-Starting Context-Aware Recommender Systems
Hybridisation Techniques for Cold-Starting Context-Aware Recommender SystemsMatthias Braunhofer
 
Recommender system introduction
Recommender system   introductionRecommender system   introduction
Recommender system introductionLiang Xiang
 
Movie lens movie recommendation system
Movie lens movie recommendation systemMovie lens movie recommendation system
Movie lens movie recommendation systemGaurav Sawant
 
Contextual Information Elicitation in Travel Recommender Systems
Contextual Information Elicitation in Travel Recommender SystemsContextual Information Elicitation in Travel Recommender Systems
Contextual Information Elicitation in Travel Recommender SystemsMatthias Braunhofer
 
Information Retrieval Models for Recommender Systems - PhD slides
Information Retrieval Models for Recommender Systems - PhD slidesInformation Retrieval Models for Recommender Systems - PhD slides
Information Retrieval Models for Recommender Systems - PhD slidesDaniel Valcarce
 
[SAC 2015] Improve General Contextual SLIM Recommendation Algorithms By Facto...
[SAC 2015] Improve General Contextual SLIM Recommendation Algorithms By Facto...[SAC 2015] Improve General Contextual SLIM Recommendation Algorithms By Facto...
[SAC 2015] Improve General Contextual SLIM Recommendation Algorithms By Facto...YONG ZHENG
 
Recommendation and Information Retrieval: Two Sides of the Same Coin?
Recommendation and Information Retrieval: Two Sides of the Same Coin?Recommendation and Information Retrieval: Two Sides of the Same Coin?
Recommendation and Information Retrieval: Two Sides of the Same Coin?Arjen de Vries
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender SystemsLior Rokach
 
HT2014 Tutorial: Evaluating Recommender Systems - Ensuring Replicability of E...
HT2014 Tutorial: Evaluating Recommender Systems - Ensuring Replicability of E...HT2014 Tutorial: Evaluating Recommender Systems - Ensuring Replicability of E...
HT2014 Tutorial: Evaluating Recommender Systems - Ensuring Replicability of E...Alejandro Bellogin
 
[Final]collaborative filtering and recommender systems
[Final]collaborative filtering and recommender systems[Final]collaborative filtering and recommender systems
[Final]collaborative filtering and recommender systemsFalitokiniaina Rabearison
 
[Decisions2013@RecSys]The Role of Emotions in Context-aware Recommendation
[Decisions2013@RecSys]The Role of Emotions in Context-aware Recommendation[Decisions2013@RecSys]The Role of Emotions in Context-aware Recommendation
[Decisions2013@RecSys]The Role of Emotions in Context-aware RecommendationYONG ZHENG
 
Summary of a Recommender Systems Survey paper
Summary of a Recommender Systems Survey paperSummary of a Recommender Systems Survey paper
Summary of a Recommender Systems Survey paperChangsung Moon
 
[RecSys 2014] Deviation-Based and Similarity-Based Contextual SLIM Recommenda...
[RecSys 2014] Deviation-Based and Similarity-Based Contextual SLIM Recommenda...[RecSys 2014] Deviation-Based and Similarity-Based Contextual SLIM Recommenda...
[RecSys 2014] Deviation-Based and Similarity-Based Contextual SLIM Recommenda...YONG ZHENG
 
Collaborative Filtering using KNN
Collaborative Filtering using KNNCollaborative Filtering using KNN
Collaborative Filtering using KNNŞeyda Hatipoğlu
 
Recommender Systems! @ASAI 2011
Recommender Systems! @ASAI 2011Recommender Systems! @ASAI 2011
Recommender Systems! @ASAI 2011Ernesto Mislej
 
Recommender system a-introduction
Recommender system a-introductionRecommender system a-introduction
Recommender system a-introductionzh3f
 
Active Learning in Collaborative Filtering Recommender Systems : a Survey
Active Learning in Collaborative Filtering Recommender Systems : a SurveyActive Learning in Collaborative Filtering Recommender Systems : a Survey
Active Learning in Collaborative Filtering Recommender Systems : a SurveyUniversity of Bergen
 

Was ist angesagt? (20)

Techniques for Context-Aware and Cold-Start Recommendations
Techniques for Context-Aware and Cold-Start RecommendationsTechniques for Context-Aware and Cold-Start Recommendations
Techniques for Context-Aware and Cold-Start Recommendations
 
Context-aware Recommendation: A Quick View
Context-aware Recommendation: A Quick ViewContext-aware Recommendation: A Quick View
Context-aware Recommendation: A Quick View
 
Hybrid Solution of the Cold-Start Problem in Context-Aware Recommender Systems
Hybrid Solution of the Cold-Start Problem in Context-Aware Recommender SystemsHybrid Solution of the Cold-Start Problem in Context-Aware Recommender Systems
Hybrid Solution of the Cold-Start Problem in Context-Aware Recommender Systems
 
Hybridisation Techniques for Cold-Starting Context-Aware Recommender Systems
Hybridisation Techniques for Cold-Starting Context-Aware Recommender SystemsHybridisation Techniques for Cold-Starting Context-Aware Recommender Systems
Hybridisation Techniques for Cold-Starting Context-Aware Recommender Systems
 
Recommender system introduction
Recommender system   introductionRecommender system   introduction
Recommender system introduction
 
Movie lens movie recommendation system
Movie lens movie recommendation systemMovie lens movie recommendation system
Movie lens movie recommendation system
 
Contextual Information Elicitation in Travel Recommender Systems
Contextual Information Elicitation in Travel Recommender SystemsContextual Information Elicitation in Travel Recommender Systems
Contextual Information Elicitation in Travel Recommender Systems
 
Information Retrieval Models for Recommender Systems - PhD slides
Information Retrieval Models for Recommender Systems - PhD slidesInformation Retrieval Models for Recommender Systems - PhD slides
Information Retrieval Models for Recommender Systems - PhD slides
 
[SAC 2015] Improve General Contextual SLIM Recommendation Algorithms By Facto...
[SAC 2015] Improve General Contextual SLIM Recommendation Algorithms By Facto...[SAC 2015] Improve General Contextual SLIM Recommendation Algorithms By Facto...
[SAC 2015] Improve General Contextual SLIM Recommendation Algorithms By Facto...
 
Recommendation and Information Retrieval: Two Sides of the Same Coin?
Recommendation and Information Retrieval: Two Sides of the Same Coin?Recommendation and Information Retrieval: Two Sides of the Same Coin?
Recommendation and Information Retrieval: Two Sides of the Same Coin?
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender Systems
 
HT2014 Tutorial: Evaluating Recommender Systems - Ensuring Replicability of E...
HT2014 Tutorial: Evaluating Recommender Systems - Ensuring Replicability of E...HT2014 Tutorial: Evaluating Recommender Systems - Ensuring Replicability of E...
HT2014 Tutorial: Evaluating Recommender Systems - Ensuring Replicability of E...
 
[Final]collaborative filtering and recommender systems
[Final]collaborative filtering and recommender systems[Final]collaborative filtering and recommender systems
[Final]collaborative filtering and recommender systems
 
[Decisions2013@RecSys]The Role of Emotions in Context-aware Recommendation
[Decisions2013@RecSys]The Role of Emotions in Context-aware Recommendation[Decisions2013@RecSys]The Role of Emotions in Context-aware Recommendation
[Decisions2013@RecSys]The Role of Emotions in Context-aware Recommendation
 
Summary of a Recommender Systems Survey paper
Summary of a Recommender Systems Survey paperSummary of a Recommender Systems Survey paper
Summary of a Recommender Systems Survey paper
 
[RecSys 2014] Deviation-Based and Similarity-Based Contextual SLIM Recommenda...
[RecSys 2014] Deviation-Based and Similarity-Based Contextual SLIM Recommenda...[RecSys 2014] Deviation-Based and Similarity-Based Contextual SLIM Recommenda...
[RecSys 2014] Deviation-Based and Similarity-Based Contextual SLIM Recommenda...
 
Collaborative Filtering using KNN
Collaborative Filtering using KNNCollaborative Filtering using KNN
Collaborative Filtering using KNN
 
Recommender Systems! @ASAI 2011
Recommender Systems! @ASAI 2011Recommender Systems! @ASAI 2011
Recommender Systems! @ASAI 2011
 
Recommender system a-introduction
Recommender system a-introductionRecommender system a-introduction
Recommender system a-introduction
 
Active Learning in Collaborative Filtering Recommender Systems : a Survey
Active Learning in Collaborative Filtering Recommender Systems : a SurveyActive Learning in Collaborative Filtering Recommender Systems : a Survey
Active Learning in Collaborative Filtering Recommender Systems : a Survey
 

Andere mochten auch

[UMAP 2015] Integrating Context Similarity with Sparse Linear Recommendation ...
[UMAP 2015] Integrating Context Similarity with Sparse Linear Recommendation ...[UMAP 2015] Integrating Context Similarity with Sparse Linear Recommendation ...
[UMAP 2015] Integrating Context Similarity with Sparse Linear Recommendation ...YONG ZHENG
 
[CIKM 2014] Deviation-Based Contextual SLIM Recommenders
[CIKM 2014] Deviation-Based Contextual SLIM Recommenders[CIKM 2014] Deviation-Based Contextual SLIM Recommenders
[CIKM 2014] Deviation-Based Contextual SLIM RecommendersYONG ZHENG
 
[WISE 2015] Similarity-Based Context-aware Recommendation
[WISE 2015] Similarity-Based Context-aware Recommendation[WISE 2015] Similarity-Based Context-aware Recommendation
[WISE 2015] Similarity-Based Context-aware RecommendationYONG ZHENG
 
[SAC2014]Splitting Approaches for Context-Aware Recommendation: An Empirical ...
[SAC2014]Splitting Approaches for Context-Aware Recommendation: An Empirical ...[SAC2014]Splitting Approaches for Context-Aware Recommendation: An Empirical ...
[SAC2014]Splitting Approaches for Context-Aware Recommendation: An Empirical ...YONG ZHENG
 
[IUI2015] A Revisit to The Identification of Contexts in Recommender Systems
[IUI2015] A Revisit to The Identification of Contexts in Recommender Systems[IUI2015] A Revisit to The Identification of Contexts in Recommender Systems
[IUI2015] A Revisit to The Identification of Contexts in Recommender SystemsYONG ZHENG
 
Matrix Factorization In Recommender Systems
Matrix Factorization In Recommender SystemsMatrix Factorization In Recommender Systems
Matrix Factorization In Recommender SystemsYONG ZHENG
 

Andere mochten auch (6)

[UMAP 2015] Integrating Context Similarity with Sparse Linear Recommendation ...
[UMAP 2015] Integrating Context Similarity with Sparse Linear Recommendation ...[UMAP 2015] Integrating Context Similarity with Sparse Linear Recommendation ...
[UMAP 2015] Integrating Context Similarity with Sparse Linear Recommendation ...
 
[CIKM 2014] Deviation-Based Contextual SLIM Recommenders
[CIKM 2014] Deviation-Based Contextual SLIM Recommenders[CIKM 2014] Deviation-Based Contextual SLIM Recommenders
[CIKM 2014] Deviation-Based Contextual SLIM Recommenders
 
[WISE 2015] Similarity-Based Context-aware Recommendation
[WISE 2015] Similarity-Based Context-aware Recommendation[WISE 2015] Similarity-Based Context-aware Recommendation
[WISE 2015] Similarity-Based Context-aware Recommendation
 
[SAC2014]Splitting Approaches for Context-Aware Recommendation: An Empirical ...
[SAC2014]Splitting Approaches for Context-Aware Recommendation: An Empirical ...[SAC2014]Splitting Approaches for Context-Aware Recommendation: An Empirical ...
[SAC2014]Splitting Approaches for Context-Aware Recommendation: An Empirical ...
 
[IUI2015] A Revisit to The Identification of Contexts in Recommender Systems
[IUI2015] A Revisit to The Identification of Contexts in Recommender Systems[IUI2015] A Revisit to The Identification of Contexts in Recommender Systems
[IUI2015] A Revisit to The Identification of Contexts in Recommender Systems
 
Matrix Factorization In Recommender Systems
Matrix Factorization In Recommender SystemsMatrix Factorization In Recommender Systems
Matrix Factorization In Recommender Systems
 

Ähnlich wie [WI 2017] Context Suggestion: Empirical Evaluations vs User Studies

Overview of recommender system
Overview of recommender systemOverview of recommender system
Overview of recommender systemStanley Wang
 
Target-Based Sentiment Anaysis as a Sequence-Tagging Task
Target-Based Sentiment Anaysis as a Sequence-Tagging TaskTarget-Based Sentiment Anaysis as a Sequence-Tagging Task
Target-Based Sentiment Anaysis as a Sequence-Tagging Taskjcscholtes
 
Recommandation systems -
Recommandation systems - Recommandation systems -
Recommandation systems - Yousef Fadila
 
Recommender systems
Recommender systemsRecommender systems
Recommender systemsTamer Rezk
 
Revisiting Offline Evaluation for Implicit-Feedback Recommender Systems (Doct...
Revisiting Offline Evaluation for Implicit-Feedback Recommender Systems (Doct...Revisiting Offline Evaluation for Implicit-Feedback Recommender Systems (Doct...
Revisiting Offline Evaluation for Implicit-Feedback Recommender Systems (Doct...Olivier Jeunen
 
Language Models for Collaborative Filtering Neighbourhoods [ECIR '16 Slides]
Language Models for Collaborative Filtering Neighbourhoods [ECIR '16 Slides]Language Models for Collaborative Filtering Neighbourhoods [ECIR '16 Slides]
Language Models for Collaborative Filtering Neighbourhoods [ECIR '16 Slides]Daniel Valcarce
 
Optimal Recommendations under Attraction, Aversion, and Social Influence
Optimal Recommendations under Attraction, Aversion, and Social InfluenceOptimal Recommendations under Attraction, Aversion, and Social Influence
Optimal Recommendations under Attraction, Aversion, and Social InfluenceWei Lu
 
The “Bellwether” Effect and Its Implications to Transfer Learning
The “Bellwether” Effect and Its Implications to Transfer LearningThe “Bellwether” Effect and Its Implications to Transfer Learning
The “Bellwether” Effect and Its Implications to Transfer LearningRahul Krishna
 
Linked Administrative Data and Adaptive Design
Linked Administrative Data and Adaptive DesignLinked Administrative Data and Adaptive Design
Linked Administrative Data and Adaptive DesignMickeyJackson3
 
ACM ICTIR 2019 Slides - Santa Clara, USA
ACM ICTIR 2019 Slides -  Santa Clara, USAACM ICTIR 2019 Slides -  Santa Clara, USA
ACM ICTIR 2019 Slides - Santa Clara, USAIadh Ounis
 
Empirical Evaluation of Active Learning in Recommender Systems
Empirical Evaluation of Active Learning in Recommender SystemsEmpirical Evaluation of Active Learning in Recommender Systems
Empirical Evaluation of Active Learning in Recommender SystemsUniversity of Bergen
 
Past, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspectivePast, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspectiveXavier Amatriain
 
Continuous Evaluation of Collaborative Recommender Systems in Data Stream Man...
Continuous Evaluation of Collaborative Recommender Systems in Data Stream Man...Continuous Evaluation of Collaborative Recommender Systems in Data Stream Man...
Continuous Evaluation of Collaborative Recommender Systems in Data Stream Man...Dr. Cornelius Ludmann
 
Rokach-GomaxSlides.pptx
Rokach-GomaxSlides.pptxRokach-GomaxSlides.pptx
Rokach-GomaxSlides.pptxJadna Almeida
 
Rokach-GomaxSlides (1).pptx
Rokach-GomaxSlides (1).pptxRokach-GomaxSlides (1).pptx
Rokach-GomaxSlides (1).pptxJadna Almeida
 

Ähnlich wie [WI 2017] Context Suggestion: Empirical Evaluations vs User Studies (20)

Overview of recommender system
Overview of recommender systemOverview of recommender system
Overview of recommender system
 
Target-Based Sentiment Anaysis as a Sequence-Tagging Task
Target-Based Sentiment Anaysis as a Sequence-Tagging TaskTarget-Based Sentiment Anaysis as a Sequence-Tagging Task
Target-Based Sentiment Anaysis as a Sequence-Tagging Task
 
Recommandation systems -
Recommandation systems - Recommandation systems -
Recommandation systems -
 
Recommender systems
Recommender systemsRecommender systems
Recommender systems
 
master_thesis.pdf
master_thesis.pdfmaster_thesis.pdf
master_thesis.pdf
 
Contextual information elicitation in travel recommender systems
Contextual information elicitation in travel recommender systemsContextual information elicitation in travel recommender systems
Contextual information elicitation in travel recommender systems
 
Revisiting Offline Evaluation for Implicit-Feedback Recommender Systems (Doct...
Revisiting Offline Evaluation for Implicit-Feedback Recommender Systems (Doct...Revisiting Offline Evaluation for Implicit-Feedback Recommender Systems (Doct...
Revisiting Offline Evaluation for Implicit-Feedback Recommender Systems (Doct...
 
Language Models for Collaborative Filtering Neighbourhoods [ECIR '16 Slides]
Language Models for Collaborative Filtering Neighbourhoods [ECIR '16 Slides]Language Models for Collaborative Filtering Neighbourhoods [ECIR '16 Slides]
Language Models for Collaborative Filtering Neighbourhoods [ECIR '16 Slides]
 
Optimal Recommendations under Attraction, Aversion, and Social Influence
Optimal Recommendations under Attraction, Aversion, and Social InfluenceOptimal Recommendations under Attraction, Aversion, and Social Influence
Optimal Recommendations under Attraction, Aversion, and Social Influence
 
The “Bellwether” Effect and Its Implications to Transfer Learning
The “Bellwether” Effect and Its Implications to Transfer LearningThe “Bellwether” Effect and Its Implications to Transfer Learning
The “Bellwether” Effect and Its Implications to Transfer Learning
 
MORS22.pdf
MORS22.pdfMORS22.pdf
MORS22.pdf
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender Systems
 
User Personality and the New User Problem in a Context-­‐Aware POI Recommende...
User Personality and the New User Problem in a Context-­‐Aware POI Recommende...User Personality and the New User Problem in a Context-­‐Aware POI Recommende...
User Personality and the New User Problem in a Context-­‐Aware POI Recommende...
 
Linked Administrative Data and Adaptive Design
Linked Administrative Data and Adaptive DesignLinked Administrative Data and Adaptive Design
Linked Administrative Data and Adaptive Design
 
ACM ICTIR 2019 Slides - Santa Clara, USA
ACM ICTIR 2019 Slides -  Santa Clara, USAACM ICTIR 2019 Slides -  Santa Clara, USA
ACM ICTIR 2019 Slides - Santa Clara, USA
 
Empirical Evaluation of Active Learning in Recommender Systems
Empirical Evaluation of Active Learning in Recommender SystemsEmpirical Evaluation of Active Learning in Recommender Systems
Empirical Evaluation of Active Learning in Recommender Systems
 
Past, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspectivePast, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspective
 
Continuous Evaluation of Collaborative Recommender Systems in Data Stream Man...
Continuous Evaluation of Collaborative Recommender Systems in Data Stream Man...Continuous Evaluation of Collaborative Recommender Systems in Data Stream Man...
Continuous Evaluation of Collaborative Recommender Systems in Data Stream Man...
 
Rokach-GomaxSlides.pptx
Rokach-GomaxSlides.pptxRokach-GomaxSlides.pptx
Rokach-GomaxSlides.pptx
 
Rokach-GomaxSlides (1).pptx
Rokach-GomaxSlides (1).pptxRokach-GomaxSlides (1).pptx
Rokach-GomaxSlides (1).pptx
 

Mehr von YONG ZHENG

[WI 2014]Context Recommendation Using Multi-label Classification
[WI 2014]Context Recommendation Using Multi-label Classification[WI 2014]Context Recommendation Using Multi-label Classification
[WI 2014]Context Recommendation Using Multi-label ClassificationYONG ZHENG
 
[UMAP2013]Tutorial on Context-Aware User Modeling for Recommendation by Bamsh...
[UMAP2013]Tutorial on Context-Aware User Modeling for Recommendation by Bamsh...[UMAP2013]Tutorial on Context-Aware User Modeling for Recommendation by Bamsh...
[UMAP2013]Tutorial on Context-Aware User Modeling for Recommendation by Bamsh...YONG ZHENG
 
[UMAP2013] Recommendation with Differential Context Weighting
[UMAP2013] Recommendation with Differential Context Weighting[UMAP2013] Recommendation with Differential Context Weighting
[UMAP2013] Recommendation with Differential Context WeightingYONG ZHENG
 
[SOCRS2013]Differential Context Modeling in Collaborative Filtering
[SOCRS2013]Differential Context Modeling in Collaborative Filtering[SOCRS2013]Differential Context Modeling in Collaborative Filtering
[SOCRS2013]Differential Context Modeling in Collaborative FilteringYONG ZHENG
 
Slope one recommender on hadoop
Slope one recommender on hadoopSlope one recommender on hadoop
Slope one recommender on hadoopYONG ZHENG
 
A manual for Ph.D dissertation
A manual for Ph.D dissertationA manual for Ph.D dissertation
A manual for Ph.D dissertationYONG ZHENG
 
Attention flow by tagging prediction
Attention flow by tagging predictionAttention flow by tagging prediction
Attention flow by tagging predictionYONG ZHENG
 
[CARS2012@RecSys]Optimal Feature Selection for Context-Aware Recommendation u...
[CARS2012@RecSys]Optimal Feature Selection for Context-Aware Recommendation u...[CARS2012@RecSys]Optimal Feature Selection for Context-Aware Recommendation u...
[CARS2012@RecSys]Optimal Feature Selection for Context-Aware Recommendation u...YONG ZHENG
 
[ECWEB2012]Differential Context Relaxation for Context-Aware Travel Recommend...
[ECWEB2012]Differential Context Relaxation for Context-Aware Travel Recommend...[ECWEB2012]Differential Context Relaxation for Context-Aware Travel Recommend...
[ECWEB2012]Differential Context Relaxation for Context-Aware Travel Recommend...YONG ZHENG
 
[HetRec2011@RecSys]Experience Discovery: Hybrid Recommendation of Student Act...
[HetRec2011@RecSys]Experience Discovery: Hybrid Recommendation of Student Act...[HetRec2011@RecSys]Experience Discovery: Hybrid Recommendation of Student Act...
[HetRec2011@RecSys]Experience Discovery: Hybrid Recommendation of Student Act...YONG ZHENG
 

Mehr von YONG ZHENG (10)

[WI 2014]Context Recommendation Using Multi-label Classification
[WI 2014]Context Recommendation Using Multi-label Classification[WI 2014]Context Recommendation Using Multi-label Classification
[WI 2014]Context Recommendation Using Multi-label Classification
 
[UMAP2013]Tutorial on Context-Aware User Modeling for Recommendation by Bamsh...
[UMAP2013]Tutorial on Context-Aware User Modeling for Recommendation by Bamsh...[UMAP2013]Tutorial on Context-Aware User Modeling for Recommendation by Bamsh...
[UMAP2013]Tutorial on Context-Aware User Modeling for Recommendation by Bamsh...
 
[UMAP2013] Recommendation with Differential Context Weighting
[UMAP2013] Recommendation with Differential Context Weighting[UMAP2013] Recommendation with Differential Context Weighting
[UMAP2013] Recommendation with Differential Context Weighting
 
[SOCRS2013]Differential Context Modeling in Collaborative Filtering
[SOCRS2013]Differential Context Modeling in Collaborative Filtering[SOCRS2013]Differential Context Modeling in Collaborative Filtering
[SOCRS2013]Differential Context Modeling in Collaborative Filtering
 
Slope one recommender on hadoop
Slope one recommender on hadoopSlope one recommender on hadoop
Slope one recommender on hadoop
 
A manual for Ph.D dissertation
A manual for Ph.D dissertationA manual for Ph.D dissertation
A manual for Ph.D dissertation
 
Attention flow by tagging prediction
Attention flow by tagging predictionAttention flow by tagging prediction
Attention flow by tagging prediction
 
[CARS2012@RecSys]Optimal Feature Selection for Context-Aware Recommendation u...
[CARS2012@RecSys]Optimal Feature Selection for Context-Aware Recommendation u...[CARS2012@RecSys]Optimal Feature Selection for Context-Aware Recommendation u...
[CARS2012@RecSys]Optimal Feature Selection for Context-Aware Recommendation u...
 
[ECWEB2012]Differential Context Relaxation for Context-Aware Travel Recommend...
[ECWEB2012]Differential Context Relaxation for Context-Aware Travel Recommend...[ECWEB2012]Differential Context Relaxation for Context-Aware Travel Recommend...
[ECWEB2012]Differential Context Relaxation for Context-Aware Travel Recommend...
 
[HetRec2011@RecSys]Experience Discovery: Hybrid Recommendation of Student Act...
[HetRec2011@RecSys]Experience Discovery: Hybrid Recommendation of Student Act...[HetRec2011@RecSys]Experience Discovery: Hybrid Recommendation of Student Act...
[HetRec2011@RecSys]Experience Discovery: Hybrid Recommendation of Student Act...
 

Kürzlich hochgeladen

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobeapidays
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontologyjohnbeverley2021
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...apidays
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Jeffrey Haguewood
 
Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​Bhuvaneswari Subramani
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous
 
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdfSandro Moreira
 
WSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024The Digital Insurer
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...DianaGray10
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Victor Rentea
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherRemote DBA Services
 
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityWSO2
 
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...apidays
 
Vector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxVector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxRemote DBA Services
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Victor Rentea
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAndrey Devyatkin
 

Kürzlich hochgeladen (20)

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontology
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf
 
WSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering Developers
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital Adaptability
 
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
 
Vector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxVector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptx
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 

[WI 2017] Context Suggestion: Empirical Evaluations vs User Studies

  • 1. Context Suggestion: Empirical Evaluations vs User Studies Yong Zheng School of Applied Technology Illinois Institute of Technology Chicago, IL, 60616, USA The 2017 IEEE/WIC/ACM Conference on Web Intelligence (WI) August 23-26, 2017, Leipzig, Germany
  • 2. Agenda • Intro: Context-aware Recommender Systems • Motivations: Context Suggestion • Methodologies and Research Problems – Direct Context Prediction – Indirect Context Suggestion • Experimental Results and Findings • Conclusions and Future Work 2
  • 3. Agenda • Intro: Context-aware Recommender Systems • Motivations: Context Suggestion • Methodologies and Research Problems – Direct Context Prediction – Indirect Context Suggestion • Experimental Results and Findings • Conclusions and Future Work 3
  • 4. Recommender System (RS) • RS: item recommendations tailored to user tastes 4
  • 5. Context-Aware Recommendation 5 Companion User’s decision may vary from contexts to contexts • Examples: ➢ Travel destination: in winter vs in summer ➢ Movie watching: with children vs with partner ➢ Restaurant: quick lunch vs business dinner ➢ Music: for workout vs for study
  • 6. Terminology in CARS 6 • Example of Multi-dimensional Context-aware Data set ➢Context Dimension: time, location, companion ➢Context Condition: Weekend/Weekday, Home/Cinema ➢Context Situation: {Weekend, Home, Kids} User Item Rating Time Location Companion U1 T1 3 Weekend Home Kids U1 T2 5 Weekday Home Partner U2 T2 2 Weekend Cinema Partner U2 T3 3 Weekday Cinema Family U1 T3 ? Weekend Cinema Kids
  • 7. What is Context? 7 The most common contextual variables: ➢Time and Location ➢User intent or purpose ➢User emotional states ➢Devices ➢Topics of interests, e.g., apple vs. Apple ➢Others: companion, weather, budget, etc Usually, the selection/definition of contexts is a domain-specific problem
  • 8. Agenda • Intro: Context-aware Recommender Systems • Motivations: Context Suggestion • Methodologies and Research Problems – Direct Context Prediction – Indirect Context Suggestion • Experimental Results and Findings • Conclusions and Future Work 8
  • 9. Motivations: Context-Drive Applications 1) Context is necessary to maximize the user experience. A list of good item recommendations is NOT enough. 2) It is difficult to collect context information on the Web!!!! Context suggestion provides a way for context acquisition 9
  • 10. Motivation: User Experience 10 • San Diego Zoo • San Diego Zoo Safari Park
  • 14. Agenda • Intro: Context-aware Recommender Systems • Motivations: Context Suggestion • Methodologies and Research Problems – Direct Context Prediction – Indirect Context Suggestion • Experimental Results and Findings • Conclusions and Future Work 14
  • 15. Solution for Context Suggestion • Direct Context Prediction The output is a binary prediction The value “1” indicates appropriate suggestion The value “0” tells inappropriate suggestion • Indirect Context Suggestion The output is a top-N recommendations Task: Suggest appropriate contexts for a user to enjoy a given item 15
  • 16. Direct Context Prediction 16 Treat context conditions as binary labels Utilize multi-label classification as solution
  • 17. Indirect Context Suggestion 17 • Context-aware Recommendation Task: Given a user and context info Recommend a list of items
  • 18. Indirect Context Suggestion 18 • Indirect Context Suggestion Task: Given a user and an item Recommend a list of appropriate contexts for the users to enjoy the items Item-Aware Context Recommendation
  • 19. Research Problems • Direct Context Prediction was explored in WI’14, but Indirect Context Suggestion was never discussed and compared with the direct context prediction • Previous research infers user preferences on contexts from contextual ratings on the items – it is not validated that whether contextual ratings can tell whether a user prefers a given contexts to enjoy the items • There are no evaluation standards for context suggestion 19
  • 20. Agenda • Intro: Context-aware Recommender Systems • Motivations: Context Suggestion • Methodologies and Research Problems – Direct Context Prediction – Indirect Context Suggestion • Experimental Results and Findings • Conclusions and Future Work 20
  • 21. Data Collection • It’s first time to collect user’s tastes on contexts • 5043 ratings by 97 users on 79 movies 21
  • 22. Experimental Settings • 5-fold Cross Validation • Direct Context Prediction – Classification Chains (MLC_CC) – Label Powerset (MLC_LP) • Indirect Context Suggestion – Tensor Factorization (TF) – Context-aware Matrix Factorization (CAMF) – Contextual Sparse Linear Methods (CSLIM) 22
  • 23. Evaluation Mechanisms We propose the evaluation standards for context suggestion • Top-N Context Prediction – N varies from 1 to the number of context conditions – Any available N value is fine – It does not matter if two contexts from a same variable are suggested. For example, {weekend, weekday, home, kids} • Exact Context Suggestion – Top-N, but N = the number of context condition – For each dimension, we only suggest one condition For example, {weekend, home, kids} 23
  • 24. Results (Top-N Suggestion) 24 By using contextual ratings as ground truth for evaluation purpose
  • 25. Results (Top-N Suggestion) 25 By using user tastes on contexts (from user surveys) as ground truth
  • 28. Conclusions and Findings 28 • “General” indicates user’s general preferences on contexts for movie watching, without considering which movie it is • Personalization is required, since many algorithms outperform the “General” method • UISplitting and TF are the best ones • Indirect context suggestion is better to offer better suggestions than direct context prediction • The results by using contextual ratings and user tastes on contexts are consistent.
  • 29. Future Work 29 • We will try to collect more data and evaluate these solutions on larger data set • We will try to utilize the context suggestion methods to predict emotional states • We will seek solutions to improve the indirect context prediction
  • 30. Context Suggestion: Empirical Evaluations vs User Studies Yong Zheng School of Applied Technology Illinois Institute of Technology Chicago, IL, 60616, USA The 2017 IEEE/WIC/ACM Conference on Web Intelligence (WI) August 23-26, 2017, Leipzig, Germany