Craig Bradford, SMX Munich The future of structured data isn’t about understanding what a thing is, it’s about understanding what a thing can do. It is allowing us to move past "strings to things" and into actions and anticipatory search. In this presentation I cover: How structured data has changed (strings to things) How to get apps indexed by Google Using structured data to say what a thing can do (things to actions) Email markup for events and more The future of Google Now (actions to anticipation) Some predictions and trends about what comes next

1. brought to you by…
March 2015
SEMANTIC OPTIMIZATION WITH
STRUCTURED DATA

2. @CraigBradford
Craig
Bradford
VP,
London
Distilled

3. 2011...

4. We had Panda
Image
Credit

5. But not Penguin
Image
Credit

6. (Not Provided)

7. Schema.org

8. Strings to Things
The goal in 2011

9. source: Tesco
Who is this?
Image
Credit

10. source: Tesco
Snoop Dog

11. source: Tesco
Snoop Dog Snoop Doggy Dogg

12. source: Tesco
Snoop Dog Snoop Doggy Dogg
Snoopzilla

13. source: Tesco
Snoop Dog Snoop Doggy Dogg
Snoopzilla
Bigg Snoop Dogg

14. source: Tesco
Snoop Dog Snoop Doggy Dogg
DJ Snoopadelic
Snoopzilla
Bigg Snoop Dogg

15. source: Tesco
Snoop Dog Snoop Doggy Dogg
DJ Snoopadelic
Snoopzilla
Bigg Snoop Dogg
Snoop Lion

16. source: Tesco
<div
itemscope
itemtype="http://schema.org/Person">
<span
itemprop=“name">Snoop
Dog</span>
</div>
Microdata

17. source: Tesco
<div
vocab="http://schema.org/"
typeof="Person">
<span
property="name">Jane
Doe</span>
</div>
RDFa
<script
type="application/ld+json">
{
"@context":
"http://schema.org",
"@type":
"Person",
"name":
"Jane
Doe"
}
</script>
JSON-LD

18. source: Tesco
<div
itemscope
itemtype="http://schema.org/Person">
<span
itemprop="name">Snoop
Dog</span>
<span
itemprop="sameAs"
content="http://
en.wikipedia.org/wiki/Snoop_Dogg"></span>
</div>
SameAs tag introduced 2013

19. source: Tesco
Success
The reward

20. source: Tesco

21. source: Tesco

22. I won’t talk about any of
that today

23. Google’s mission is to organize the
world’s information and make it
universally accessible and useful.

24. No mention of
webpages or search
engines

25. source: Tesco
The old “Google it”
Search with words on a computer

26. source: Tesco
The old “Google it”
Get a list of webpages

27. source: Tesco
The new “Google it”
Search anywhere

28. source: Tesco
The new “Google it”
Get anything

29. Organise the world’s information

30. Universally accessible?

31. Project Loon
Image
Credit

32. Useful?

33. Useful?
That depends…

34. Useful data helps you get more done ✔️

35. How does Google help us get more done?

36. Understanding user intent

37. query
“london tube stations”

38. explicit aspect of query
implicit aspect of query
iPhone user, on street in London
“london tube stations”

39. explicit aspect of query
implicit aspect of query
iPhone user, on street in London
“london tube stations”
Critical
to
understanding
user
intent

40. 1. Action (Do)
2. Informational (Know)
3. Navigational (Go)

41. “Download Adobe reader”
Likely user intent: Download software
“Do”

42. Standard set of results

43. “Online personality test”
Likely user intent: Take an online personality test
“Do”

44. Standard set of results

45. “Know”
“Germany”
Likely user intent: Find travel and tourism
information for planning a vacation or holiday, or
find information about the German geography,
languages, economy, etc.
Much harder to know what you want

46. Results reflect that

47. “How to remove candle wax
from carpet”
Likely user intent: Find information on how to
remove candle wax from carpet
“Know”

48. No need to visit a page

49. “Go”
“YouTube”
Likely user intent: Go to the homepage of
YouTube

50. Hard to say what you want from YouTube

51. “Sony customer services”
Likely user intent: Go to the customer support
page on official Sony website
“Go”

52. UK
IP
Address
Information you want in search results

53. US
IP
Address
Implicit location information

54. KnowDo Go
Regular search
Knowledge graph
Answers

55. Strings to things
Things to actions

56. Strings to things
Things to actions
✔️
?

57. The future of structured data
isn’t about understanding what a
thing is, it’s about understanding
what a thing can do

58. How can you help people
get more done?

59. source: Tesco
More ways to achieve the same goal

60. Sometimes the best place to
do an action isn’t on a
webpage

61. “The LinkedIn Problem”

62. Connection request
on mobile (I have
app installed)

63. Directed to a login
page

64. Directed to a login
page

65. source: Tesco
This gives Google a problem

66. source: Tesco
They can’t crawl apps
This gives Google a problem

67. They need our help

68. 1. What action can the app take?
2. Where should the action be done?
They need answers

69. 1. What action can the app take?
2. Where should the action be done?
Structured data
They need answers

70. The reward

71. App ChangesWebsite Changes
1. Add
deep
link
annotations
2. Verify
your
website
in
Play
store
1. Update
the
app
manifest
file
How

72. App Changes

73. Intent-filter
(What can this thing do?)

74. <intent-filter android:label="@string/filter_title_viewgizmos">
<action android:name="android.intent.action.VIEW" />
<category android:name="android.intent.category.DEFAULT" />
<category android:name="android.intent.category.BROWSABLE" />
<data android:scheme="http"
android:host="www.example.com"
android:pathPrefix="/gizmos" />
Update Manifest File

75. Adding Deep Links
(Where should the action be done?)

76. android-app://{package_name}/{scheme}/{host_path}
Anatomy of a deep link
Package
IDProtocol • Scheme
• http
• custom
Host
Path

77. android-app://{package_name}/{scheme}/{host_path}
Example - Rick Roll - Get the URL
Package
IDProtocol • Scheme
• http
• custom
Host
Path
www.youtube.com/watch?v=dQw4w9WgXcQ

78. android-app://{package_name}/{scheme}/{host_path}
Find the YouTube App on Google Play
Package
IDProtocol • Scheme
• http
• custom
Host
Path
…store/apps/details?id=com.google.android.youtube

79. Combine
Protocol android-­‐app://
Package
ID com.google.android.youtube/
Scheme http/
Host
Path www.youtube.com/watch?v=dQw4w9WgXcQ
+
+
+
=
android-­‐app://com.google.android.youtube/http/www.youtube.com/watch?v=dQw4w9WgXcQ

80. https://twitter.com/craigbradford
Example - Follow me on Twitter
(Regular Link)

81. Taken to the mobile
version of the site

82. (Deep link)
bit.ly/craigsmx
Example - Follow me on Twitter

83. Native App

84. Add Deep Links to Your Website
<link>
element Schema.org
ActionsSitemap
<url>

85. <link>
element
<html>
<head>
...
<link
rel="alternate"
href="android-­‐app://
com.google.android.youtube/http/www.youtube.com/watch?
v=dQw4w9WgXcQ"
/>
...
</head>
<body>
…
</body>

86. Sitemap
<url>
<?xml
version="1.0"
encoding="UTF-­‐8"
?>
<urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:xhtml="http://www.w3.org/1999/xhtml">
<url>
<loc>http://www.youtube.com/watch?v=dQw4w9WgXcQ</loc>
<xhtml:link
rel="alternate"
href="android-­‐app://com.google.android.youtube/http/
www.youtube.com/watch?v=dQw4w9WgXcQ"
/>
</url>
...
</urlset>

87. Schema.org
Actions
<script
type="application/ld+json">
{
"@context":
"http://schema.org",
"@type":
"WebPage",
"@id":
"www.youtube.com/watch?v=dQw4w9WgXcQ",
"potentialAction":
{
"@type":
"ViewAction",
"target":
"android-­‐app://com.google.android.youtube/http/www.youtube.com/watch?
v=dQw4w9WgXcQ"
}
}
</script>

88. Verify your website and App
http://bit.ly/smxverify

89. *Terms and Conditions do
Apply
Android only (at the moment)

92. No Website?

93. App indexing API
Web and App App Only
Deep linking and/or
indexing API
App indexing API

94. App Indexing API

95. Page
1

96. Page
1 Page
2

97. Page
1 Page
3Page
2

98. Flats for sale
Flats for rent
Flats in London
Flat prices
Flat
Zoopla Flats in London
Personalised
Autocomplete

99. Flats for sale
Flats for rent
Flats in London
Flat prices
Flat
Zoopla Flats in London
Flats for sale
Flats for rent
Flats in London
Flat prices
Flat
Zoopla Flats in London

100. Acquisition Behaviour Outcomes
• Reviews
• In-depth articles
• Social profiles
• Events
• Recipes
• etc.
Old structured data

101. Acquisition Behaviour Outcomes
• Deep linking • App indexing API
New structured data

102. What about Email?

103. No MaybeYes
How many emails could you answer with …

104. 104
Accept Invite?

105. Yes,
subscribe
me
to
this
list
Confirm your Subscription?

106. Yes No Maybe
RSVP

107. Gmail Actions
1. In-App Actions
2. Go-to App Actions

108. One-Click Action
Save Confirm

109. RSVP

110. Reviews

111. Go-to Actions

112. 112
In-App - RSVP
<script
type="application/ld+json">
{
"@context":
"http://schema.org",
"@type":
"Event",
"name":
"John's
Birthday
Party",
...
information
about
the
event
...
"action":
{
"@type":
"RsvpAction",
"actionHandler":
{
"@type":
"HttpActionHandler",
"url":
"https://events-­‐organizer.com/rsvp?
eventId=123",
"method":
"POST",
"requiredProperty":
"rsvpStatus",
}
}
}
</script>

113. 113
Go-To - Watch Movie
<script
type="application/ld+json">
{
"@context":
"http://schema.org",
"@type":
"EmailMessage",
"action":
{
"@type":
"ViewAction",
"url":
"https://watch-­‐movies.com/watch?movieId=abc123",
"name":
"Watch
movie"
},
"description":
"Watch
the
'Avengers'
movie
online"
}
</script>

114. Approval guidelines
bit.ly/smxemailactions

115. How it looks in
Google inbox
It’s easy - here’s a fake event I created

116. How it looks in
Google inbox
Example script: bit.ly/1O8trsE

117. 117
Can’t believe
Google I/O is at my
mum’s house…

118. 118
Can’t believe
Google I/O is at my
mum’s house…
Example script: bit.ly/1O8trsE

119. More benefits to email markup

121. Natural language search

122. This doesn’t work in gmail

123. Google is very good at identifying things

125. Funny looking dog

128. Combine things beer+Seattle

130. Why is that useful?

131. source: Tesco
The new “Google it”

132. Actions to Anticipation
(Anticipatory search)

133. “When
we
started
Google
15
years
ago,
my
vision
was
that
information
would
come
to
you
as
you
need
it.
You
wouldn’t
have
a
search
query
at
all.”
Sergey
Brin
Image
Credit

134. Action Status

135. 1. Potential - A description of an action that is
supported
2. Active - An in-progress action (e.g. while
watching the movie, or driving to a location)
3. Completed - An action that has already
taken place

136. source: Tesco
<script
type="application/ld+json">
{
"@context":
"http://schema.org",
"@type":
"MusicGroup",
"url":
[
"http://www.spotify.com/artist/Katy-­‐Perry-­‐35141/",
"android-­‐app://com.spotify.music/http/../listen"
],
"name":
"Katy
Perry",
"potentialAction":
{
"@type":
"ListenAction",
"target":
[
"http://www.spotify.com/artist/Katy-­‐Perry-­‐35141/?
autoplay=true&ext=1&playedFrom=197",
"android-­‐app://com.spotify.music/http/../listen"
]
}
}
</script>
Music Play Actions
Match intent to potential actions

137. Google Now is getting better at anticipation

138. source: Tesco
The App Web
But it’s limited to Google data

139. Imagine Google had access to all your apps

140. 1. Music you’ve listen to
2. Movies you’ve watched
3. Things you’ve eaten
4. Your heart rate
5. Where you are now
6. Where you might be going
7. Who you’re meeting
8. What you’re reading now
9. What you might read soon

141. Done Doing Might
Do

142. It’s not a ridiculous possibility

143. + +
Where you’re flying to
When you’re flying
When you’ve arrived What hotel you’re
staying in

144. + +
=
Get to your hotel for €30
Nearest driver is 5 minutes away
Order Car Now
You recently took a Lyft from your home

145. + +
=
Get to your hotel for €30
Nearest driver is 5 minutes away
Order Car Now
You recently took a Lyft from your home
They can probably do this now just using only
structured data from flight and hotel confirmation
emails.

146. It’s
amazing
to
the
extent
to
what
software
developers
kind
of
get
lazy,
and
they’re
okay
with
things
taking
a
while,”
Page
said.
“But
it’s
really
not
okay.
Obsessed by speed Image
Credit

147. Apps are slow

148. Extensions are helping but still not great

149. http://bit.ly/smxcardsRead this
The trend of cards

150. Google Now

151. Google Inbox

152. Twitter Cards

153. Image
Credit

154. So that’s anticipation V1…

155. How far can Google go?

156. How far can Google go?
What about diagnosis?

157. Meet Tom

158. Meet Tom
We made a
bet

159. Meet Tom
We made a
bet
Get under 10%
body fat in 6
weeks

160. 81.1kg
Feb
13th
87.2
kg
April
13th
+
6.1kg
I had been going to the gym for a while

161. 81.1kg
Feb
13th
+
6.1kg
Start
Bet started early May
87.2
kg
April
13th

162. I took this very seriously

163. Measured at University of Westminster

164. Measured at University of Westminster
Not me

165. Starting
Numbers
17.2% Body
fat

166. I exercised 6/7 days, quit alcohol and
tracked every meal for 6 weeks!

167. 81.1kg
Feb
13th
+
6.1kg
Start
I lost a lot of weight quickly
87.2
kg
April
13th

168. Finishing
Numbers…

169. Finishing
Numbers
11.2% Body
fat

170. My Reaction
to the
news…

171. But something interesting
happened

172. I started to notice what food made me feel
really good and what made me feel bad

173. Around the same time I read about “The
pulse test” by Dr. Arthur Coca
Image
Credit

174. It is based on the fact that allergens
speed up the pulse. It consists
essentially of testing isolated foods in
order to tell which ones accelerate the
pulse.

175. It is based on the fact that allergens
speed up the pulse. It consists
essentially of testing isolated foods in
order to tell which ones accelerate the
pulse.
Could our smartphones and smart watches
do all this for us?

176. +
What you’ve eaten Your heart rate throughout
the day

177. +
What you’ve eaten Your heart rate throughout
the day
=
You might be lactose intolerant
Visit your GP
Book an appointment
We’ve detected irregularities in your
heartbeat when you consume lactose.

178. The building blocks behind
this is structured data

179. Key points
1. The future of structured data isn’t about understanding
what a thing is, it’s about understanding what a thing
can do (Strings - Things - Action - Anticipation)
2. Use structured data to help people get more done
3. Use structured data to influence all parts of your
marketing (Acquisition - Behaviour - Outcomes)
4. Start planning for a world of APIs and cards

180. Thank You
Craig Bradford
@CraigBradford

181. I’m speaking at SearchLove Boston