How is organized data used by some web players having not the best intentions? How can tools that try to help individual authors be subverted by spammers? Also, how does Zemanta work and why are we interested in this topic.

1. Quality to Quantity to Quality
on the Web
Andraž Tori, CTO at Zemanta
@andraz

2. Topics
- a bit about Zemanta
- how advanced “data tools” and spammers
interact

3. We are all trying to organize the web

4. Making it right,
making it useful
and linked

7. Not so long time ago, in a city not far away...

8. some other people

9. are trying to do the opposite

10. trying to disorganize it,
make it confusing,
and to profit from that

12. using the tools we have built!

14. Their motives are not sinster
(mostly)

15. it is about profit

16. Profit
- publish as much content as possible
- quality is not (that) important
- get traffic or high page ranking for certain terms
- sell clicks, links or whole “fully built” sites to the
highest bidder
- users and search engines are necessary evil to
be tricked as cheaply as possible

18. So, why do I care?

19. Job opening
You will get a spreadsheet with 180 blog url’s and
logins. You will log into each blog and schedule 2
posts per week ...
You will spice up every post with images and/or
related links within the content, using a Wordpress
plugin called Zemanta
https://www.odesk.com/jobs/Wordpress-Blog-Poster_~~c8c04549b8e6b600

20. And why might you care?
- the organized information is great tool for those
that try to disorganize it
- they are poisoning “our web”, including twitter,
facebook
- and it's hard to see in the fog they are causing
- it is just matter of time when they start poisioning
linked data too

22. What do we do at

23. - is a “personal writing assistant”
- suggesting content while you write (your blog)
- analyzing your text
- connecting it with background knowledge, other
stories on the web, images
- you choose what suggestions to include
- to make your writing more informative, vivid and
useful

27. Opening up the hood

30. the reality

32. How it works
Content
suggestions
Plain text Semantic
Analysis
(article) search
Linked data RSS feeds

33. Main design goals
- Input is meaningful chunk of text (not a keyword
or a phrase)
- Input is (semi) English language
- Has to work across all domains in the open
world
- music, celebrities, finance, entertainment, politics,
gardening, parenting, …

34. Analysis pipeline
Known phrases
Named Entity
extraction
Extraction
(aho-corasick)
Triple store
Surface form features evaluation
Statistical comparison to
background knowledge
Semantic coherence
and hand-tuned
heuristics
etc.
Disambiguated entities

35. Analysis pipeline
Known phrases
Named Entity
extraction
Extraction
(aho-corasick)
Categorization to Dmoz
Triple store
Surface form features evaluation
Statistical comparison to
background knowledge
Semantic coherence
and hand-tuned
heuristics
etc.
Categories Ambigious named entities Disambiguated entities

36. Background knowledge
- Data from Wikipedia, MusicBrainz, Freebase…
and world wild web
- Includes linguistical and semantical properties
+ unstructured data
- Present in two forms:
- in “original” custom built triple store on top of MySQL
(150 GB)
- processed into 7 GB optimized “memory mapped
dump”

37. Background knowledge
- 7M mined and linked up entities and
concepts
Triple store
- 30M aliases
- Refreshed about once a month
- want to make it real-time
- Input data quality is really important
etc.

38. After analysis
Text
SOLR Related articles
articles
SOLR Images
images

39. Example SOLR query

40. boost((( wiki_entities:Health insurance
wiki_entities:Medical underwriting wiki_entities:United States
wiki_entities:Affordable Care Act wiki_entities:Barack Obama
wiki_entities:Lifetime (TV network) wiki_entities:Insurance
wiki_entities:Preventive medicine wiki_entities:Child
wiki_entities:Patient Protection and Affordable Care Act ) ^3.0)
(text:zemhealthinsurq^0.68 text:health^0.62 text:premium^0.36
text:zeminsurcompaniq^0.56 text:increas^0.29 text:rate^0.27
text:zemhealthinsurcompaniq^0.35 text:zempreventcareq^0.26
text:medic^0.26 text:compani^0.23 text:obamacar^0.21
text:todai^0.21 text:polici^0.21 text:care^0.19 ) ^105.0
((dmoz_categories:
Top/Business/Financial_Services/Insurance/Agents_and_Marketers/Health
dmoz_categories:
Top/Business/Financial_Services/Insurance/Agents_and_Marketers/Health/United_States
dmoz_categories:
Top/Business/Financial_Services/Insurance/Agents_and_Marketers/Health/United_States
/California) ^0.1),
(1 - 0.2) * sqrt(1.0/(1.15E-8*float(1285185600000 - date(published_datetime)
ms)+1.0)) + 0.2)

41. Solr
- We adapted Solr for “query by document”
- 52% precision (at 10) on internal evaluations
- plain Lucene MLT comes to 44%
- difference is from “bag of terms” approach over “bag
of words” (terms coming from analysis step)
- Our live index is 5M articles
- Solr is really not optimized to handle 50 terms in
a single query

42. Lucene plain “More Like This”

43. Metrics & tests
- Every part of the system is being constantly
evaluted
- Precision/recall at 5 different points in the
system
- Mostly bi-weekly releases of new datasets and
the engine

44. Overview
- We do pretty deep processing to deliver simple
user experience of “personal authoring
assistant”
- And everything is available over the web API
- tagging
- named entity recognition and disambiguation to
Linked Open Data URIs

45. What API offers?
• Tags
Most used
• Categories
• Concepts and entities Most interesting
• Related articles
• Related images

47. So mash-ups happen...

48. Some API users

49. We are just one of the many people offering
services based on large amounts of web data
each spending man-years trying to organize their
data, trying to offer best possible service

50. now back to the bad guys

52. Job opening
You will get a spreadsheet with 180 blog url’s and
logins. You will log into each blog and schedule 2
posts per week ...
You will spice up every post with images and/or
related links within the content, using a Wordpress
plugin called Zemanta
https://www.odesk.com/jobs/Wordpress-Blog-Poster_~~c8c04549b8e6b600

53. There's more than meets the eye

54. Gather search terms Analyze → Find / create
(extensions, logs, guess) what people search for? such content
Pull additional content Use Zemanta or OpenCalais
Cover your tracks
from Freebase to add tags, images, links
Publish
Amazon Mechanical Turk
Use Zemanta to find
to post comments Profit?
similar blogs
and links back to your site

55. Warnings
- I've seen no single system using the whole
pipeline as described, however all parts were
found in the wild
- Examples used are from all kinds of sites –
good, bad and ugly
- I am not trying to imply that all of the steps in the
diagram are bad, but they can be used by bad
guys efficiently

56. Gather search terms Analyze → Find / create
(extensions, logs, guess) what people search for? such content
Pull additional content Use Zemanta or OpenCalais
Cover your tracks
from Freebase to add tags, images, links
Publish
Amazon Mechanical Turk
Use Zemanta to find
to post comments Profit?
similar blogs
and links back to your site

57. Finding their keywords, niches
- Domain expertise
- Users like to install extensions and say “yes”
- You observe referrers on sites you control
- You buy the data on the black market

59. The sophisticated part of the market
“Demand Media relies on a proprietary algorithm
to help editors best determine what subjects
their writers should tackle.”
Factors:
- Keyword competition
- Revenue
- Driving traffic to/from existing conent
http://emediavitals.com/article/16/demand-media-s-content-assembly-line

60. Gather search terms Analyze → Find / create
(extensions, logs, guess) what people search for? such content
Pull additional content Use Zemanta or OpenCalais
Cover your tracks
from Freebase to add tags, images, links
Publish
Amazon Mechanical Turk
Use Zemanta to find
to post comments Profit?
similar blogs
and links back to your site

61. Find / create content
- Steal
- Take from “open article directories”
- Have your own “content assembly line” like
Demand Media

62. Open article directories

63. Gather search terms Analyze → Find / create
(extensions, logs, guess) what people search for? such content
Pull additional content Use Zemanta or OpenCalais
Cover your tracks
from Freebase to add tags, images, links
Publish
Amazon Mechanical Turk
Use Zemanta to find
to post comments Profit?
similar blogs
and links back to your site

64. Tһiѕ iѕ nοt the text you аre lookinɡ for.

65. Tһiѕ iѕ nοt the text you аre lookinɡ for.

66. Translate it to random language and back to English
Übersetzen sie zufällig Sprache und wieder auf Englisch
Language and translate it happen again in English
Μεταφράστε αυτό σε δειγματοληπτικούς γλώσσα και
πίσω στην αγγλική γλώσσα
Translate this random language back to English
Traduisez au langage aléatoire et revenir à l'anglais
Translate to random language to English and back
它翻译成随机的语言和回英文
Translate it back into the English language and random

67. Covering their tracks
- Trying to fool search engines or people?
- Search engines are catching up
- Google Translate API is being closed due to
“abuse”?
- The trend is “rewriting” by human editors,
procured on the global market

69. Gather search terms Analyze → Find / create
(extensions, logs, guess) what people search for? such content
Pull additional content Use Zemanta, OpenCalais
Cover your tracks
from Freebase to add tags, images, links
Publish
Amazon Mechanical Turk
Use Zemanta to find
to post comments Profit?
similar blogs
and links back to your site

70. Spammers say darndest things

72. Gather search terms Analyze → Find / create
(extensions, logs, guess) what people search for? such content
Pull additional content Use Zemanta or OpenCalais
Cover your tracks
from Freebase to add tags, images, links
Publish
Amazon Mechanical Turk
Use Zemanta to find
to post comments Profit?
similar blogs
and links back to your site

74. Remixing linked data and spam
- Currently mostly the good guys are using Linked
Data
- However, it's just too tempting to be left alone
- Fully synthetic articles using factual information
from linked data?
– Using advanced tools to form proper natural
language sentences and maybe even storyline?

75. Gather search terms Analyze → Find / create
(extensions, logs, guess) what people search for? such content
Pull additional content Use Zemanta or OpenCalais
Cover your tracks
from Freebase to add tags, images, links
Publish
Amazon Mechanical Turk
Use Zemanta to find
to post comments Profit?
similar blogs
and links back to your site

76. Publish
- On hosted third party platforms
- eating their resources
- Platforms have hard time killing spammers
- Smaller ones don't necessarily have the incentive
- If they remove spammer too fast, it is easier for
spammer to probe the limits
- Platforms use “kill with delay”
- Spam detection is resource intensive

77. Gather search terms Analyze → Find / create
(extensions, logs, guess) what people search for? such content
Pull additional content Use Zemanta or OpenCalais
Cover your tracks
from Freebase to add tags, images, links
Publish
Amazon Mechanical Turk
Use Zemanta to find
to post comments Profit?
similar blogs
and links back to your site

78. Valuable comments
As I write this post, Zemanta is showing me
5 different articles that are related to my
post. I could visit each one of these sites
and reach out to the owner to see if they
would be interested in linking to my post,
or I could leave a valuable comment on the
page and include a link back to my post.
http://www.mainelyseo.com/zemanta-review-seo-link-building-with-the-zemanta-plugin/

79. - Guy in previous slide is honest and well-
meaning
- But what if you automate that via Amazon
Mechanical Turk or oDesk?

80. Gather search terms Analyze → Find / create
(extensions, logs, guess) what people search for? such content
Pull additional content Use Zemanta or OpenCalais
Cover your tracks
from Freebase to add tags, images, links
Publish
Amazon Mechanical Turk
Use Zemanta to find
to post comments Profit?
similar blogs
and links back to your site

81. Profit?
- sell ads
- sell links
- sell “fully developed site”
- to the highest bidder

82. Search engines to the rescue?
- Mahalo cut 10% of the staff the day after
Google announced ranking changes
- Demand Media's stock isn't doing that well
anymore
- However this is a never-ending story, we'll have
co-evolution for foreseeable future

83. Ecosystem
- Very sophisticated, large players
- moving to more high quality content, video?
- Small time operations
- using more and more sophisticated tools available
on the market cheaply (modern asymmetric
warfare?)
- Dark industry specifically building tools to poison
the web and sell them to small time operators

84. Food for thought

85. Can we make spammers (and others) work for us,
making linked data better?
(think reCAPTCHA)

86. Could article directories be fruitfully used?
eZineArticles.com, GoArticles.com, etc...

87. Find rewritten articles and use them as parallel
corpus?

88. Could we use global workforce market more
efficiently to get more linked data?

89. Thesis, antithesis, synthesis?
http://xkcd.com/810/

90. Thank you!
Questions?

91. Image sources

http://www.flickr.com/photos/dzingeek/4587871752/

http://www.flickr.com/photos/25101572@N02/4393474025/

http://www.flickr.com/photos/billward/4740384434/

http://www.flickr.com/photos/jurvetson/542500748

http://www.flickr.com/photos/legofenris/4288913574

http://www.flickr.com/photos/ekilby/3733627940

http://www.flickr.com/photos/ekilby/3732799269/

http://www.flickr.com/photos/cipherswarm/38354452

http://xkcd.com/810/