Understanding and Measuring User
Engagement and Attention
in Online News Reading
Dmitry Lagun and Mounia Lalmas
1Thanks to...
User Engagement in Online News Reading
2
User engagement:
“emotional, cognitive and behavioral
connection that exists betw...
Measuring user engagement with news content
3
Method PROS CONS
Dwell time (click duration)
(Agichtein et al., 2006)
scalab...
Our Method: Viewport Tracking
4
viewport
time spent at i-th scroll
position
i-th viewport defined by a
rectangle (left, to...
Research questions
● Where do users spend their time during news article viewing?
● Does media image and video content aff...
6
Overall Pattern of Viewport Time
(proxy for user attention)
Many users spend significantly
smaller amount of time at low...
Image and Video do matter … for the first screen
7
Video Image
How do users browse through the article?
8
comment
header
top
middle
bottom
articlebody
start
top
middle
bottom
comment
le...
Mixture of Markov Chains Model
9
Single markov model:
Mixture of K markov models:
probability of starting at state v1
prob...
Patterns of Attention in News Reading
10
Engagement Depth
most probable sequence
Engagement depth: Four User Engagement Classes
11
EngagementDepth
12
Engagement depth: Four User Engagement Classes
EngagementDepth
13
Engagement depth: Four User Engagement Classes
EngagementDepth
14
Engagement depth: Four User Engagement Classes
EngagementDepth
Distribution of Attention is Different across Engagement Classes
15
Modeling of User Engagement from Article Content
16
?
news article
%Bounce
%Shallow
%Deep
%Complete
user engagement profile
TUNE: Topics of User Engagement with News
17
TUNE
news article
%Bounce
%Shallow
%Deep
%Complete
user engagement profile
Un...
Experimental Setting
● Task
○ Predict User Engagement Level Profile
● Model
○ Linear regression
● Features
○ Number of wor...
Results: Baselines
19
Feature Set %Bounce %Shallow %Deep %Complete
NumWords 0.063 0.494 0.370 0.017
NumWords + Media (M) 0...
Results: Baselines
20
Feature Set %Bounce %Shallow %Deep %Complete
NumWords 0.063 0.494 0.370 0.017
NumWords + Media (M) 0...
Results: Baselines vs. TUNE
21
Feature Set %Bounce %Shallow %Deep %Complete
NumWords 0.063 0.494 0.370 0.017
NumWords + Me...
Conclusions
● Unlike in search, user attention in news reading is not constantly decaying
with vertical position (e.g., ca...
Summary: Viewport time attention as proxy of user engagement
23
Effect of position and content on viewport time at
vertica...
Appendix
24
User Attention vs. Engagement Classes
25
Metric
Bounce
(N=26542)
Shallow
(N=63982)
Deep
(N=164197)
Complete
(N=12489)
dwel...
User Engagement Classes and
User Attention
26
Dwell time and viewport time on head, body and comment increase from
Bounce ...
Nächste SlideShare
Wird geladen in …5
×

Understanding and Measuring User Engagement and Attention in Online News Reading

1.727 Aufrufe

Veröffentlicht am

Prior work on user engagement with online news sites identified dwell time as a key engagement metric. Whereas on average, dwell time gives a reasonable estimate of user engagement with a news article, it does not capture user engagement with the news article at sub-document level nor it allows to measure the proportion of article read by the user.

In this paper, we analyze online news reading patterns using large-scale viewport data collected from 267,210 page views on 1,971 news articles on a major online news website. We propose four engagement metrics that, unlike dwell time, more accurately reflect how users engage with and attend to the news content. The four metrics capture different levels of engagement, ranging from bounce to complete, providing clear and interpretable characterizations of user engagement with online news. Furthermore, we develop a probabilistic model that combines both an article textual content and level of user engagement information in a joint model. In our experiments we show that our model, called TUNE, is able to predict future level of user engagement based on textual content alone and outperform currently available methods.

Slides of our paper presented at The 9th ACM International Conference on Web Search and Data Mining (WSDM 2016),
San Francisco, California, USA. February 22-25, 2016.

Veröffentlicht in: Internet
  • Als Erste(r) kommentieren

Understanding and Measuring User Engagement and Attention in Online News Reading

  1. 1. Understanding and Measuring User Engagement and Attention in Online News Reading Dmitry Lagun and Mounia Lalmas 1Thanks to Yahoo Faculty Research and Engagement Program for supporting this work.
  2. 2. User Engagement in Online News Reading 2 User engagement: “emotional, cognitive and behavioral connection that exists between a user and a resource” (Attfield et al., 2011) Stickiness: concerned with users spending time on a news site. User Attention in Online News Reading Challenge II: identifying which aspects of the online interaction influence user engagement the most. Challenge I: attract large shares of online attention by keeping users engaged.
  3. 3. Measuring user engagement with news content 3 Method PROS CONS Dwell time (click duration) (Agichtein et al., 2006) scalable; captures engagement at coarse level cannot distinguish time spent on parts of the article Eye tracking (Arapakis et al., 2014) very detailed small scale; very expensive Mouse cursor movement (Huang et al., 2011) scalable; more fine grained than dwell time cursor is often kept still during article reading, when no pointing action is required coarse but more robust instrument to measure user attention at large scale during news reading VIEWPORT TRACKING
  4. 4. Our Method: Viewport Tracking 4 viewport time spent at i-th scroll position i-th viewport defined by a rectangle (left, top, width, height) viewport
  5. 5. Research questions ● Where do users spend their time during news article viewing? ● Does media image and video content affect time spent at a vertical position? ● What are typical patterns of news article reading? ● Can we accurately predict user engagement from textual content? 5 ● 1,971 Yahoo news articles ● 267,210 page views on desktopDATASET
  6. 6. 6 Overall Pattern of Viewport Time (proxy for user attention) Many users spend significantly smaller amount of time at lower scroll positions. Some users find the article interesting enough to spend significant amount of time at the lower part of the article. Some articles entice users to deeply engage with their content.
  7. 7. Image and Video do matter … for the first screen 7 Video Image
  8. 8. How do users browse through the article? 8 comment header top middle bottom articlebody start top middle bottom comment leave Markov States beginning of a page view top area occupies most of the viewport middle area occupies most of the viewport bottom area occupies most of the viewport comment area occupies most of the viewport user leaves the page V1 V2 ... Vn
  9. 9. Mixture of Markov Chains Model 9 Single markov model: Mixture of K markov models: probability of starting at state v1 probability of transition from state Vi to V(i-1) Markov States: {Start, Top, Middle, Bottom, Comment, Leave} weight of k-th mixture component K=6 is optimal
  10. 10. Patterns of Attention in News Reading 10 Engagement Depth most probable sequence
  11. 11. Engagement depth: Four User Engagement Classes 11 EngagementDepth
  12. 12. 12 Engagement depth: Four User Engagement Classes EngagementDepth
  13. 13. 13 Engagement depth: Four User Engagement Classes EngagementDepth
  14. 14. 14 Engagement depth: Four User Engagement Classes EngagementDepth
  15. 15. Distribution of Attention is Different across Engagement Classes 15
  16. 16. Modeling of User Engagement from Article Content 16 ? news article %Bounce %Shallow %Deep %Complete user engagement profile
  17. 17. TUNE: Topics of User Engagement with News 17 TUNE news article %Bounce %Shallow %Deep %Complete user engagement profile Unlike LDA, in TUNE topic is a combination of word co-occurrence and similarity of user engagement profile. Distribution of user engagement level
  18. 18. Experimental Setting ● Task ○ Predict User Engagement Level Profile ● Model ○ Linear regression ● Features ○ Number of words in the article ○ Presence of media content (e.g., image and video) ○ Distribution of article topics with LDA ○ Distribution of article topics with TUNE (our model) ● Evaluation Metric ○ Pearson’s correlation between ground truth and predicted value ○ Ten fold cross-validation 18 %Bounce %Shallow %Deep %Complete
  19. 19. Results: Baselines 19 Feature Set %Bounce %Shallow %Deep %Complete NumWords 0.063 0.494 0.370 0.017 NumWords + Media (M) 0.071 0.571 0.410 0.185
  20. 20. Results: Baselines 20 Feature Set %Bounce %Shallow %Deep %Complete NumWords 0.063 0.494 0.370 0.017 NumWords + Media (M) 0.071 0.571 0.410 0.185 NumWords + M + LDA (T=5) 0.119 0.597 0.466 0.328 NumWords + M + LDA (T=10) 0.110 0.606 0.497 0.379 NumWords + M + LDA (T=20) 0.150 0.626 0.531 0.402 NumWords + M + LDA (T=50) 0.143 0.629 0.538 0.405
  21. 21. Results: Baselines vs. TUNE 21 Feature Set %Bounce %Shallow %Deep %Complete NumWords 0.063 0.494 0.370 0.017 NumWords + Media (M) 0.071 0.571 0.410 0.185 NumWords + M + LDA (T=5) 0.119 0.597 0.466 0.328 NumWords + M + LDA (T=10) 0.110 0.606 0.497 0.379 NumWords + M + LDA (T=20) 0.150 0.626 0.531 0.402 NumWords + M + LDA (T=50) 0.143 0.629 0.538 0.405 NumWords + M + TUNE (T=5) 0.079 0.648 0.544 0.282 NumWords + M + TUNE (T=10) 0.311 0.713 0.660 0.400 NumWords + M + TUNE (T=20) 0.349 0.724 0.682 0.409 NumWords + M + TUNE (T=50) 0.333 (+132%) 0.742 (+18%) 0.697 (+29%) 0.428 (+6%) NumWords + M + LDA + TUNE 0.334 0.730 0.696 0.442 Dwell 0.392 0.203 0.128 0.351
  22. 22. Conclusions ● Unlike in search, user attention in news reading is not constantly decaying with vertical position (e.g., can be bi-modal) ● Engagement with a news article can be categorized by depth of examination (Bounce, Shallow, Deep & Complete) ● The proposed engagement metrics go beyond “dwell time” as they capture user attention and engagement at sub-document level ● We can obtain accurate prediction of article engagement profile purely from its textual content 22
  23. 23. Summary: Viewport time attention as proxy of user engagement 23 Effect of position and content on viewport time at vertical position V1 V2 ... Vn Article examination can be categorized by depth of examination Four engagement classes: Bounce, Shallow, Deep and Complete Joint model of article topics and user engagement classes improves prediction accuracy: ● Bounce (+140%) ● Shallow (+18%) ● Deep (29%) ● Complete (+9%)
  24. 24. Appendix 24
  25. 25. User Attention vs. Engagement Classes 25 Metric Bounce (N=26542) Shallow (N=63982) Deep (N=164197) Complete (N=12489) dwell 6.17 (0.02) 63.75 (0.37) 99.02 (0.22) 228.35 (1.48) header time 2.99 (0.03) 15.39 (0.14) 18.48 (0.08) 17.41 (0.25) body time 5.06 (0.02) 35.13 (0.21) 86.24 (0.20) 85.00 (0.70) comment time 0.56 (0.01) 17.27 (0.23) 9.72 (0.07) 110.90 (0.89) % header time 0.31 (0.00) 0.23 (0.00) 0.17 (0.00) 0.09 (0.00) % body time 0.62 (0.00) 0.58 (0.00) 0.76 (0.00) 0.40 (0.00) % comment time 0.07 (0.00) 0.20 (0.00) 0.07 (0.00) 0.51 (0.00) % article read 0.12 (0.00) 0.23 (0.00) 0.83 (0.00) 0.84 (0.00) # comment clicks 0.01 (0.00) 0.43 (0.01) 0.00 (0.00 3.14 (0.03)
  26. 26. User Engagement Classes and User Attention 26 Dwell time and viewport time on head, body and comment increase from Bounce to Complete. Viewport time on head steadily decreases from Bounce to Complete: users spend an increasing amount of time reading content deeper in article. Percentage of article read steadily increases from Bounce to Complete, as expected. Deep and Complete correspond to the situations when the majority (83%) of the article was read. Number of comment clicks is highest for Complete and then Shallow: users may engage with comments even if they do not read a large proportion of the article. comment header top middle bottom articlebody header

×