Presentation of the "Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective" presented on the 42nd European Conference on Information Retrieval (ECIR 2020)
ECIR 2020 - Axiomatic Analysis of Contact Recommendation Methods in Social Networks: An IR perspective
1. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Axiomatic Analysis of Contact
Recommendation Methods in Social Networks:
An IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
Javier Sanz-Cruzado1, Craig Macdonald2, Iadh Ounis2 and Pablo Castells1
@JavierSanzCruza, @Craig_macdonald, @iadh, @pcastells
1 Universidad Autónoma de Madrid
http://ir.ii.uam.es
2University of Glasgow
http://terrierteam.dcs.gla.ac.uk/
April 15th 2020
2. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Contact recommendation
?
- 1 1
- 2
1 ? - ? 1
3 -
1 4 -
Users
Users
Rating matrix = adjacency matrix
1
3. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Motivation
Search Contact
recommendation
Models (BM25, Query Likelihood…)
IR axioms
Evaluation (Metrics,…)
2
4. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Space mapping
Previous work (Sanz-Cruzado & Castells, ECIR
2019):
Search Contact recommendation
?
Relevant
result
DocumentQuery
Term
Relevant
link
Candidate
user
Target
user
Neighbor
user
3
5. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Space mapping
Previous work (Sanz-Cruzado & Castells, ECIR
2019):
Search Contact recommendation
?
Relevant
result
DocumentQuery
Term
Relevant
link
Candidate
user
Target
user
Neighbor
user
4
6. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Space mapping
Previous work (Sanz-Cruzado & Castells, ECIR
2019):
Search Contact recommendation
?
Relevant
result
DocumentQuery
Term
Relevant
link
Candidate
user
Target
user
Neighbor
user
5
7. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Space mapping
Previous work (Sanz-Cruzado & Castells, ECIR
2019):
Search Contact recommendation
?
Relevant
result
DocumentQuery
Term
Relevant
link
Candidate
user
Target
user
Neighbor
user
6
8. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Model mapping: e.g. BM25
BM25 for search
Search Contact recommendation
– Query 𝑞 Target user neighborhood
Γ 𝑢
– Document 𝑑 Candidate user neighborhood
Γ 𝑣
– Term 𝑡 ∈ 𝑑 ∩ 𝑞 Neighbor 𝑡 ∈ Γ 𝑢 ∩ Γ 𝑣
– Term frequency in 𝑑, freq 𝑡, 𝑑 Weight of the 𝑡, 𝑣 edge, w 𝑡, 𝑑
– Document 𝑑 length,
𝑑
Candidate user 𝑣 length, len v = 𝑥∈Γ 𝑣 𝑤 𝑥, 𝑣
7
9. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Model mapping: e.g. BM25
BM25 for search
BM25 for contact recommendation
8
10. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Goal: Determine a set of heuristics (axioms) that a good IR system should satisfy
Many constraints have been proposed addressing many properties:
– Term frequency
– Term discrimination
– Length normalization
– …
Useful for:
– Development of effective IR models
– Diagnose of their performance
Axiomatic thinking in IR
Are IR axioms useful for
contact recommendation?
Fang et al. SIGIR 2004, Fang et al. ACM TOIS 2011
9
11. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Axiomatic thinking in IR
Are IR axioms useful for
contact recommendation?
Adaptation
Empirical
observation
10
12. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Axioms in text search
Fang et al. SIGIR 2004, Fang et al. ACM TOIS 2011
TFC1: 𝑓𝑞 𝑑 grows
with freq 𝑡, 𝑑
TFC3: frequency distributed over many
terms better than concentrated on few
terms
TFC2: dampened
growth on freq 𝑡, 𝑑
Term frequency constraints (TFCs)
11
13. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Axioms in contact recommendation
EWC1: 𝑓𝑢 𝑣 grows
with w 𝑡, 𝑣
EWC3: weight distributed over many
neighbors better than concentrated on few
neighbors
EWC2: dampened
growth on w 𝑡, 𝑣
Edge weight constraints (EWCs)
12
14. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Other axioms in contact recommendation
EWC1 EWC3
EWC2
Neighbor discrimination constraint (NDC)
NDC: Popular common
neighbors have less
importance
Candidate length normalization constraint (CLNC)
CLNC: Penalize candidate users
with many uncommon neighbors
13
15. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Applicability to contact recommendation approaches
IR axioms can also be analyzed on other contact recommendation algorithms
Only if the algorithm recommends people at distance 2!
?
?
Queries Terms Documents
Distance 2
Adapted IR models
Adamic-Adar algorithms
Similarity-based
···
kNN
Matrix factorization
PageRank
··· 14
16. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Experimental setup
Facebook
Twitter-1month Twitter 200-tweets
Interaction Follows Interaction Follows
# users 4,039 9,528 9,770 9,985 9,964
Directed ✘ ✔ ✔ ✔ ✔
Weighted ✘ ✔ ✘ ✔ ✘
Train/Test split Random Temporal Temporal Temporal Temporal
15
17. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Procedure
1. Take an IR model
2. Remove pieces related to the axiom
3. Compare
EWC1 EWC3
EWC2
NDC
CLNC
16
18. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Edge weight constraint EWC1
Are edge weights good?
Conclusion
NO
But the best
algorithm (BM25)
uses weights…
nDCG@10
nDCG@10
Point in the plot = recommendation algorithm
17
19. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Edge weight constraint EWC2
Conclusion
nDCG@10
nDCG@10
YES
Is it good to dampen growth with edge weight?
Point in the plot = BM25 with different parameter
setting
18
20. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Neighbor discrimination constraint NDC
Neighbor
discrimination
isbetter
Neighbor
discrimination
isworse
Point in the plot = different parameter setting of
algorithm
Is good to discriminate popular common neighbors?
Conclusion
YES
Δ nDCG@10
With vs. Without
neighbor discrimination
19
21. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Candidate length normalization constraint CLNC
Length
normalization
isbetter
Length
normalization
isworse
Point in the plot = different parameter setting of
algorithm
Is it bad to recommend popular people?
Conclusion
NoΔ nDCG@10
With vs. Without
length normalization
Because of preferential attachment
behaviour in social networks
20
22. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Conclusions
In general, IR axioms correlate with higher accuracy in contact
recommendation.
Exception: Length normalization. Collision with preferential
attachment
Future work:
– Extend this to algorithms recommending at distance greater than 2
– Analyze to what extent those algorithms satisfy the constraints
21
23. IRGIR Group @UAM
Axiomatic Analysis of Contact Recommendation Methods in Social Networks: an IR perspective
42nd European Conference on Information Retrieval (ECIR 2020)
15 April 2019
Thank you for your attention!
Questions?
Editor's Notes
IDEA: Highlight tha fact that, previously, we adapted the textual search task from IR for the task of recommending people in social networks. Now, we have gone further, and we adapt IR axioms to check whether they are useful or not. This has two steps: first, an adaptation of the IR models for the contact recommendation task, and second, an observation of them to determine if they are good or not for the contact recommendation task.
So, the presentation starts HERE:
Hello, my name is Javier Sanz-Cruzado, and I’m going to present this work, in which we explore the utility of adapting of information retrieval axioms for the contact recommendation task. The present work is the result of a collaboration between the University of Glasgow, represented by Craig Macdonald and Iadh Ounis, and Universidad Autónoma de Madrid, with Pablo Castells and myself.
Our work focuses on a particular perspective of recommender systems known as contact recommendation.
Given a social network representing the different relations between users in our system, contact recommendation’s goal is to find people in the network with whom the different users might be interested in connecting with.
Our work continues a research line that we iniciated in last year’s ECIR, in which we explore the relationship between textual search and contact recommendation.
Last year, we explored how to adapt information retrieval models such as BM25 or query likelihood for recommending people in social networks in a collaborative filtering manner. In addition, we also used the methodology and metrics applied for textual search to evaluate such models.
This year, we wanted to go a step further, by adapting the use of axiomatic thinking techniques from the textual search task to the contact recommendation field.
All these past and present adaptations from the search task to contact recommendation are built on top of a mapping of spaces, where we map the three spaces in the IR task (queries, documents and terms) to a single space in contact recommendation: the space of users in the social network.
In that mapping, we first match the documents and the candidate users, since they both represent the elements that we want to retrieve.
Then, we match the query and the target user, since they represent the input of our systems: the explicit information need in the case of the query, and the profile of the user representing his/her tastes.
And, finally, as we want to explore everything from a collaborative filtering perspective, we match the terms with the connections of the target and candidate users: their neighbors.
Using that map, we are ready to translate the different IR models for contact recommendation. Here, we show an example with one of the most well-known approaches: BM25. First, we have its formulation for the search case.
Then, for adapting it to a contact recommendation approach, we just have to change the different pieces in search for pieces in contact recommendation. For example, we take the query, and we change it by the neighborhood of the target user. We take the neighborhood of the candidate instead of the document, and the common neighbors between the target and candidate users instead of the common terms between the query and the document.
Other elements we change are the frequency of the terms in the documents, which is now taken as the weight of the edges between the candidate user and their neighbors, and the length of the document, now transformed in the length of the candidate user (or, in other words, the sum of the weights of the edges shared with his neighbors).
With all of this, we can easily adapt the model for the contact recommendation case.
As we explained earlier, beyond IR models, in this work we wanted to adapt the information retrieval axioms. But, what do we want to say when we talk about IR axioms?
In textual search, axiomatic thinking is a research line that tries to formalize the properties that a model should satisfy to provide good results. That formal properties are known as axioms, and they target things like how we should treat long documents or whether we should give more importance to query terms appearing in only a few documents.
These axioms or constraints are useful for the development of new and effective IR models, as well as for diagnosing the performance of the existing ones.
In this work, we wondered if, under the previous mapping, these axioms could play a similar role for the contact recommendation task, or, in other words, if they are useful for that. In order to check this, we did two things: first, a theoretical work, in which we explored the adaptation of the IR axioms for the task, and, second, an observation on whether satisfying them leads to better models.
In order to check this, we did two things: first, a theoretical work, in which we explored the adaptation of the IR axioms for the task, and, second, an observation on whether satisfying them leads to better models.
We selected for adaptation the basic and fundamental axioms in textual search, proposed by Fang and colleagues in 2004 and improved in 2011. They categorized them in several groups, according to the elements that the axioms where related to. We start by adapting the term frequency constraints, that explian how the overall score should vary when the frequency of the terms in documents varies.
There are three axioms within this category. The first one indicates that the score should increase as the frequency of the term does. In BM25, this is satisfied thanks to the element we highlight in yellow. The second tells that this growth should be done in a diminishing returns pattern, so the difference in score between increasing from 1 appearance of the term and 2 appearances should be greater than between 100 and 101. For satisfying this, BM25 has the small element in the denominator than we highlight.
And, finally, the third one says that distributing a large frequency value over many common terms should be better than concentrating it on a single term. And this is true thanks to the sum over the common terms.
Using the mapping that we explained earlier, adapting these axioms to contact recommendation is straightforward. For the term frequency constraints, for example, we just change the frequency of the terms in the documents by the weight of the edges between the candidate users and the neighbors shared with the target user.
As the original model for textual search already satisfies the axioms for such task, when we adapt them for the contact recommendation task, IR models still keep the properties and satisfy them.
This adaptation can be extended to other axioms. For example, other adapted contact recommendation axioms we use in our work are the neighbor discrimination constraint, that indicates that, the more popular the common terms between the target and candidate users are, the les discriminative power they have, and they should be given les importance, and the candidate length normalization constraint, that says that we should not favor popular users with many uncommon neighbors with the target user.
Then, our study can be straightforwardly applied to IR model adaptations. But, what about other people recommendation algorithms? We observed that, as IR axioms asume a structure like the one shown in the slide, where there are two steps between the queries and the documents, we could extend our analysis to other recommendation approaches that recommend people at distance 2 in the network. This, beyond IR models includes others such as Adamic-Adar.
However, other well-known and effective approaches, such as matrx factorization or PageRank, which are able to recommend people at distance further than 2 cannot be analyzed using these constraints.
Until now, we have just focused on the adaptation of the IR axioms. However, even when their translation is straightforward, there is something that changes: the semantics of those axioms for the new task. So, in order to see if they are really useful for the contact recommendation task, we had to carry several experiments to see if satisfying them is related an improvement on the accuracy of the algorithms. We carried them over data from different social networks: an undirected and unweighted Facebook graph, two directed and weighted interaction Twitter networks, and two directed and unweighted Twitter follow networks.
Since each axiom is related to a particular element in the model formulation, the procedure we followed was the following: for each of the axioms, we took IR models satisfying such axiom, and we removed the element that made them satisfy it. Then, we compared their accuracy, and studied whether keeping that piece was Good or not. We show the results for some of these axioms:
First, we started with the first edge weight axiom. For this, we wanted to check if the use of edge weights is good.
For this, we compared IR models using weights against IR models using binary weights. Each point in the plot represents a different recommendation algorithm, and the x axis indicates its nDCG@10 value for the unweighted models, whereas the y axis shows the same for the weighted models.
It can be observed that, in general, unweighted models achieve a better accuracy, so we might say that the first axiom is not useful. However, if we look it closely, we observe that the best model (the red point above) uses the weights, so we could not fully discard this constraint.
So, for the second constraint, we took this best model, BM25, and we wondered if dampening the growth of the edges was good. So, we removed the dampening term, and we compared several parameter settings of BM25 with and without that dampening. In this plot, the nDCG value for the original algorithm is shown in the X axis, revealing that moderating the growth of the score provides positive results.
Next, we explore the neighbor discrimination concern, asking if it is good giving more weight to unpopular common users or not. We took for that several IR models, and we removed their discrimination terms. We changed here the representation, so each point represents the difference in nDCG for a parameter setting of an algorithm. We sorted the settings by difference, and values greater than 0 represent settings for which applying neighborhood discrimination is better and values smaller than 0, settings for which removing it improved the accuracy.
As we could observe, it seems clear that applying the neighborhood discrimination provides much better results than removing it.
And, finally, we explored the effect of the length normalization, wondering if is it bad or not to recommend bad people. Following the same scheme as we did for the previous constraint, we show here that most values in the plot are below 0, thus showing that recommending popular people is not bad, and contradicting our axiom. We found that this is because the constraint contradicts the preferential attachment behaviour in social networks, where popular users are more likely to receive new connections.
In conclusion, we can say that, in general, satisfying IR axioms leads to improvements on the accuracy of the recommenders, with the exception of length normalization ones, since they collide with preferential attachment evolutionary patterns of social networks. As future work, we would like to extend this work for analyzing the rest of approaches.
Thank you for your attention and, if you have any questions….
