Controversial Users demand Local Trust Metrics: an Experimental Study on Epinions.com Community
1. Controversial Users demand
Local Trust Metrics: an
Experimental
Study on Epinions.com
Community
Paolo Massa
PhD student in ICT
ITC/iRST and University of Trento
Blog: http://moloko.itc.it/paoloblog/
(Joint Work with Paolo Avesani)
(Thanks Epinions.com for providing data)
Slides licenced under CreativeCommons AttributionShareAlike (see last slide for more info)
3. Motivation
In a society, some peers are unknown to you (ebay,
p2p, ...)
Q: “Should I trust peer A?” [decentralization>relevant]
Most papers assume a peer has a unique quality value
(there are good peers and bad peers, goal is to spot bad)
IRREALISTIC assumption (Evidence from real online
community of 150.000 users).
Consequence: we need Local Trust Metrics
(personalized) [But most papers propose Global Metrics]
4. Epinions.com
What is Epinions.com?
Community web site where users can
Write reviews about items and give them ratings
Express their Web of Trust (“Users whose reviews and
ratings you have consistently found to be valuable”)
Express their Block List (“Users whose reviews and
ratings ... offensive, inaccurate, or in general not valuable”)
Reviews of TRUSTed users are more visible
Reviews of DISTRUSTed users are hidden
5. Epinions.com
Dr.P profile page
Dr.P's Web of Trust
(Block List is hidden)
Do you
trust or distrust Dr.P?
Ratings given by Dr.P
6. Real uses of Trust
News sites: Slashdot.org, Kuro5hin.org, ...
Emarketplaces: Ebay.com, Epinions.com, Amazon.com, ...
P2P networks: eDonkey, Gnutella, JXTA
Jobs sites: LinkedIn, Ryze, ...
Friendster, Tribes, Orkut and other “social” sites.
Opensource Developers communities: Advogato.org (Affero.org)
Hospitalityclub.org, couchsurfing.com: hosting in your house unknown people?
Bookcrossing and lending stuff sites.
Network of personal weblogs (the blogroll is your trust list)
Semantic Web: FOAF (FriendOfAFriend) is an RDF format that allows to express social
relationships (~10 millions files) and XFN microformat
PageRank (Google) ... MyWeb2.0 (Yahoo!)
7. Trust networks (are graphs)
Aggregate all the trust statements to produce a
trust network. A node is a user (example: Dr.P).
A direct edge is a trust statement
0
Mena Ben In Epinions,
0.2 Properties of Trust: just 1 and 0!
0.9 weighted (0=distrust, 1=max trust)
0.6
subjective
1
Dr.P Doc asymmetric contextdependent
Trust Metric (TM):
? Uses existing edges for predicting values
of trust for nonexisting edges,
thanks to trust propagation (if you trust
someone, then you have some degree of
trust in anyone that person trusts).
8. TM perspective: Local or Global
1 1
Mary Mena Bill
How much Bill can be trusted?
0 On average (by the community)?
By Mary?
ME 1
Doc And by ME?
Global Trust Metrics:
“Reputation” of user is based on number and quality of incoming edges. Bill has
just one predicted trust value (0.5).
PageRank (Google), eBay, Slashdot, ... Works badly for controversial people
Local Trust Metrics
Trust is subjective > consider personal views (trust “Bill”?)
Local can be more effective if people are not standardized.
9. Controversial Users
Intuitively: a Controversial User is
TRUSTED by some users
DISTRUSTED by some users
Do you want an example?
10. Controversial Users: an example
1 0
1 0
1 0
1 0
1 0
(....) (....)
1 0
100M people 100M people
If you don't know Bush, should you trust Bush?
T(Bush)=0.5? Make sense? Here global metrics don't.
11. Controversial Users: an example
1 0
1 0
1 0
1
1 1 0
R 1 0
D
1 1
(....) (....)
1 0
100M people 100Mpeople
Local Metric makes more sense. Your trust in Bush
depends on your trusted users!
T(R,Bush)=1 T(D,Bush)=0
12. Controversial Users on Epinions
Controversial users are normal in societies
How many controversial users on Epinions.com?
But first, two definitions of Controversiality:
Controversiality Level of A: number of users that
disagree with the majority = Min(#trust, #distrust)
Contr. “Percentage” of A = (TD) / (T+D) in [1, 1]
CP(A)=1 if A is trusted by everyone (loved!)
CP(A)=1 if A is distrusted by everyone (hated!)
CP(A)=0 if A is trusted by n users and distrusted by n users
13. Experiment
Epinions.com dataset
Real Users: ~150K
Edges (Trust / Distrust): 841K (717K / 124K)
~85K received at least one judgement (trust or distrust)
17.090 (>20%) are at least 1controversial (at least 1 user
disagrees with the majority) > Non negligible portion!
1.247 are at least 10controversial
144 are at least 40controversial
1 user is 212controversial! (~400 trust her, 212 distrust her)
14. Experiment
Comparing 2 metrics about accuracy in trust/distrust
prediction.
Global: ebaylike. Trust(A)=#trust/(#trust+#distrust)
Local: MoleTrust, based on Trust Propagation from current
user (simple and fast)
Cycles are a problem > Order peers
based on distance from source user
Trust of users at level k is based only
on trust of users at level k1 (and k)
Trust propagation horizon & decay
15. Experiment
How do we compare metrics?
Leaveoneout: Remove an edge in Trust Network and
try to predict it. Then compute error as absolute
difference between Real and Predicted value.
Also differentiating over trust or distrust statements
16. Exp. on Controversiality Level
y=error made by TM predicting
edges on users with x
Error
controversiality level.
Predicting Distrust is more
Ebay Controversiality level
difficult.
Ebay error on Distrust ~ 0.6
Mole2 error on Distrust ~ 0.4
Error
Error on Trust is similar because
(#trust >> #distrust)
MoleTrust2 Controversiality level
17. Exp. on Controversiality Percentage
CP~0 = Controversial User
Error Ebay = 0.5 on Contr.Us
Error
Error MoleTrust2 smaller
but not as small as we
Ebay Controversiality percentage would like: can we reach 0?
Other experiments in paper:
Error on Trust Edges.
Error
Error on Distrust Edges (very
important to correctly predict
these ones!)
MoleTrust2 Controversiality percentage
18. Other experiments
MoleTrust with different propagation horizons
2, 3, 4
Computing Coverage.
19. Conclusions
In complex societies, it is normal that someone likes
you and someone dislikes you.
Most Papers make assumption of unique quality
value for a peer (and propose an algo for predicting it)
This is IRREALISTIC! (I know this is intuitive but
still ...)
20. Conclusions (2)
As a consequence, we need Local Trust Metrics.
Local TMs are computationally much more expensive than
Global TMs! > Possibly, you run it locally for yourself on
your mobile or on your browser (should be fast!)
Local TMs exploits less information > reduced coverage.
Global Metrics fine in noncontroversial domains: possibly
ok on Ebay, surely not ok on sites about (political?) opinions
Trust networks are everywhere!
More research is needed: Yahoo! (with MyWeb2.0)
and Google are there. More real testbeds, more
proposals of Local TMs, more comparisons, ...
21. Licence of this slides
These slides are released under
Creative Commons
AttributionShareAlike 2.5
You are free:
* to copy, distribute, display, and perform the work
* to make derivative works
* to make commercial use of the work
Under the following conditions:
Attribution. You must attribute the work in the manner specified by the author or licensor.
Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to
this one.
* For any reuse or distribution, you must make clear to others the license terms of this work.
* Any of these conditions can be waived if you get permission from the copyright holder.
Your fair use and other rights are in no way affected by the above.
More info at http://creativecommons.org/licenses/bysa/2.5/
22. The End
The End.
Thanks for your attention!
Questions?