hackU 2012 IIT Delhi YQL

1. Hacking on steroids
with YQL
Anshu
Prateek
Search/YDN
@anshprat

3. Hacking together
systems in 24 hours is
lot of fun.

4. Data manipulation
Hacks = +
Data visualization

5. The web has lot of data around

6. ProgrammableWeb.Com – 6831 APIs

7. Yahoo! has opened
up its data

8. http://developer.yahoo.com/everything.html

9. The
trouble
with
data
• You need to find data API
• Get Access – Signup for key
• Find data endpoint
• Read docs to learn what parameters you
have
• Get data in obscure format
• Use data after converting and filtering
• More APIs you use, more is your
annoyance

10. To make data access
easy on the web,
Yahoo! created YQL

11. YQL turns web
services and data on
the web into
databases.

12. select {what} from {where}
where {conditions}

13. You can select, filter,
sort and limit

14. You can even insert,
update and delete from
it.

15. Finding
Videos
about
Bangalore
• SELECT
*
FROM
youtube.search
where
query='bangalore'

16. Selec@ng
Photos
of
HackdaY
• SELECT
*
FROM
flickr.photos.search
where
text="hackday”
and
api_key=“b5a60b2a…”

17. Inser@ng
data
• INSERT
INTO
bitly.shorten
(login,
apiKey,
longUrl)
•
• VALUES
('ME',
'API_KEY',
'hYp://yahoo.com')

18. upda@ng
data
• UPDATE
social.profile.status
• SET
status="Using
YQL
UPDATE”
• WHERE
guid="NJFIDHVPVVISDX7UKED2WHU"

19. Retrieving
my
Contacts
• SELECT
*
FROM
social.contacts
WHERE
guid=me

20. Accessing
Private
Data
• hYp://query.yahooapis.com/v1/yql
Uses OAuth 1.0 for authorization
OAuth is complicated – use one of our SDKs at
https://github.com/yahoo

21. You can also mix and
match several web
services using the in()
command.

22. select * from search.termextract
where context in (select description
from rss where url='http://
rss.news.yahoo.com/rss/topstories')

23. Almost all the top
APIs on web are
accessible from YQL

24. 1255 APIs

25. Some
of
them
amazon foursquare peerindex
apple geo salesforce
bbc github slideshare
bible google themovideb
boss hackernews tumblr
campfire ign twitter
contentanalysis intuit vimeo
craigslist kiva weather
delicious klout yahoo
dopplr lastfm youtube
etsy netflix zillow
facebook paypal

26. You want even more?

27. Alright, how about this?
atom json
csv microformats
feed rss
html xml

28. The easiest way to start
with YQL is to use the
console
http://developer.yahoo.com/yql/console

29. YQL: http://developer.yahoo.com/yql/console

33. How to get this data in
your app?

34. YQL is a REST API in
itself and has two
endpoints

35. The public endpoint does not need any
authentication.
http://query.yahooapis.com/v1/public/yql?
q={query} &format={format}

36. The private endpoint needs oauth
authentication.
http://query.yahooapis.com/v1/yql?
q={query}&format={format}

37. Output formats are XML
or JSON

38. Let’s
See
It

39. Query
Examples
select
*
from
yahoo.finance.quotes
where
symbol
in
("^IXIC","^DJI","YHOO
","AAPL")

40. Query
Examples
select
*
from
weather.byloca@on
where
loca@on
in
("bangalore,
in",
”new
york,
us")

41. •
Query
Examples
Find
hackday
tweets:
• SELECT
*
FROM
twi]er.search
where
q='hackday’
• Search
Yahoo!
Answers
for
resolved
ques@ons
about
cars:
• select
*
from
answers.search
where
query="cars"
and
type="resolved”
• Find
distance
between
Bangalore
and
Mumbai:
• select
*
from
geo.distance
where
place1="bangalore"
and
place2="mumbai”
• Extract
important
terms
from
top
stories
on
Yahoo!
news:
• select
*
from
search.termextract
where
context
in
(select
descrip@on
from
rss
where
url='h]p://rss.news.yahoo.com/rss/topstories')

42. Query
Examples
• Get
Olympic
medal
list
• select
*
from
html
where
url='h]p://sports.yahoo.com/olympics/
medals.html'
and
xpath='//*[@id='mediasportsoverallmedalcount']/
div[2]/table/tbody/tr/td/a'
• Shorten
a
URL:
• insert
into
yahoo.y.ahoo.it
(url,
keysize)
values
('h]p://
www.javarants.com',
5)
• Search
apartments
in
criagslist:
• select
*
from
craigslist.search
where
loca@on="bangalore"
and
type="apa"
and
query="indiranagar”

43. Query
Examples
• Scrape
news
from
Yahoo!
Finance:
• select
*
from
html
where
url="h]p://finance.yahoo.com/q?
s=yhoo"
and
xpath='//div[@id="yfi_headlines"]/div[2]/ul/li/
a’
• Select,
filter
data
from
google
spreadsheets:
• select
*
from
csv
where
url="h]ps://
spreadsheets.google.com/pub?key=0ArYndzim-­‐
lbrdF8wc3A5QWl1ZGRpdkxRZk80SU9zUXc&output=csv"
and
col5
like
'Bangalore%’
;

44. There are lot of things
you can do with YQL.

45. Play yourself
http://
developer.yahoo.com/yql/

46. One more thing

47. Don’t look for features.
Solve one problem

48. Resources
• All
Yahoo!
APIs
and
Services
• hYp://developer.yahoo.com/everything.html
• YQL
Documentahon
• hYp://developer.yahoo.com/yql
• YQL
Console
• hYp://developer.yahoo.com/yql/console
• YQL
Github
Account
(Contribute
Tables)

49. Thanks!
http://www.slideshare.net/
anshprat
Anshu Prateek
Twitter: @anshprat
Github: http://github.com/anshprat
Web: http://hackalyst.info