How Did We Get to Sesame Street? Google's Search for NLP.

@schachin #StateOfSearchKristine Schachinger
How Did We Get To Sesame Street?
Google’s Search for NLP
Kristine@SitesWithoutWalls.com

In the beginning, there was a …
Large-Scale Hypertextual Web Search Engine

What?

Link Profiles
http://infolab.stanford.edu/pub/papers/google.pdf

The Web 1998

Today in 2019 …
Roughly half of the world's population or
3.8 billion people use the internet every day.

Google
processes
TRILLIONS
of queries a
year & has
indexed
BILLIONS
of Websites.

TODAY, THERE WERE ALREADY OVER
3,567,977,254 Google searches.
http://www.internetlivestats.com/google-search-statistics/

In ONE SECOND today, there were

Dealing With The Data.

Google Search was
founded on
unstructured data.

Unstructured data (or unstructured information) is information that
either does not have a pre-defined data model or is not organized in a
pre-defined manner. Unstructured information is typically text-heavy,
but may contain data such as dates, numbers, and facts as well.
https://www.google.co.uk/search?q=definition+unstructured+data&oq=definition+unstructured+data&aqs=chrome..69i57j0l5.5175j0j7&sourceid=chrome&ie=UTF-8

Unstructured data (or unstructured information) is information that
either does not have a pre-defined data model or is not organized in a
pre-defined manner. Unstructured information is typically text-heavy,
but may contain data such as dates, numbers, and facts as well.
https://www.google.co.uk/search?q=definition+unstructured+data&oq=definition+unstructured+data&aqs=chrome..69i57j0l5.5175j0j7&sourceid=chrome&ie=UTF-8
This is known as the “Bag of Words” approach.

Otherwise Known as Keywords.

Unstructured Data
uses keywords.
https://moz.com/blog/7-advanced-seo-concepts

TF-IDF
Term Frequency Inverse
Document Frequency
ie the frequency of keywords
compared to other documents in
the set

But as queries number in the trillions unstructured
data becomes inefficient.
Data needed structure.

Semantics: Keywords to Queries.

So Google moved from
Relational Databases to
Knowledge Graphs.
Knowledge Graphs

NOTE
Knowledge Graphs
DO NOT EQUAL
THE KNOWLEDGE GRAPH
Knowledge Graphs

“Graph-based knowledge representation has been
researched for decades and the term knowledge graph
does not constitute a new technology.
Rather, it is a buzzword reinvented by Google and
adopted by other companies and academia to describe
different knowledge representation applications.”
Knowledge Graphs
http://ceur-ws.org/Vol-1695/paper4.pdf

Enter Semantic Search
https://web.archive.org/web/20090516213508/http://blog.searchenginewatch.com/090512-201139

What is Semantic Search?

Semantic Search
=
Trying to Understand Intent

Why?

The Holy Grail of
Search?
NLP
(Natural Language Processing)

Welcome THE Knowledge Graph 2012.

Knowledge Graphs are based on known relationships.
THE Knowledge Graph is Google’s graph database.
THE Knowledge Graph

https://twitter.com/manishrjain
https://blog.dgraph.io/post/why-google-needed-graph-serving-system/

https://blog.dgraph.io/post/why-google-needed-graph-serving-system/

https://searchengineland.com/up-close-google-squared-19313
Before the Knowledge Graph – Wonder Wheel

(Knowledge Graphs)
”…quite possibly ...
one of Google's significant achievements”
Nathania Johnson of Search Engine Watch
Knowledge Graphs

Enter Machine Learning

https://www.tensorflow.org/guide/
MACHINE LEARNING

The Knowledge Graph enables you to search for things, people or places
that Google knows about—landmarks, celebrities, cities, sports teams,
buildings, geographical features, movies, celestial objects, works of art
and more—and instantly get information that’s relevant to your query
THE Knowledge Graph

In other words
NOUNS
THE Knowledge Graph

NOUNS=ENTITIES
THE Knowledge Graph

Google moves to
ENTITY SEARCH
THE Knowledge Graph

Knowledge Graph entities
The Knowledge Graph has millions of entries that describe real-world entities like people, places, and things. These
entities form the nodes of the graph.
The following are some of the types of entities found in the Knowledge Graph:
Book
BookSeries
EducationalOrganization
Event
GovernmentOrganization
LocalBusiness
Movie
MovieSeries
MusicAlbum
MusicGroup
MusicRecording
Organization
Periodical
Person
Place
SportsTeam
TVEpisode
TVSeries
VideoGame
VideoGameSeries
WebSite
THE Knowledge Graph ENTITIES

Entities (nouns) + Relationships (nodes) =
THE Knowledge Graph
THE Knowledge Graph

Knowledge Graph
= the Answer Engine
THE Knowledge Graph

Google as an Answer Engine
https://www.google.com/search/howsearchworks/responses/#?modal_active=none

Hummingbird 
“Strings to Things”.
The name was derived from
the speed and accuracy of the hummingbird.

Hummingbird Arrives 2013
Google moves from matching keyword terms to
Google trying to process Natural Language Queries
using the Knowledge Graph.
“Strings to Things"

But Google doesn’t process
Natural Language very well.

Hummingbird was the first step in Google moving
search from strings (unstructured data)
or the “bag of words” approach
 to “things” (structured data)

“Things” are known objects
with known (or learned) relationships.

The Knowledge Graph (Google) is seeded by things known.
Instead of just text without meaning, The KG is a relational
graph with known objects and mapped relationships.
THE Knowledge Graph

Why the Knowledge Graph?
To help better match user intent.
To understand what users want.
THE Knowledge Graph

https://ai.google/research/teams/language/
Why?
Google doesn’t truly process
Natural Language (NLP), but Hummingbird and
The Knowledge Graph were the first steps
towards language understanding in subset of
NLP processes called NLU (Natural Language
Understanding).

NLU is a
subset of
NLP.
https://medium.com/@fahrettinf/natural-language-understanding-f50cc3229991

Things Need Context.
Moving to Vector Relationships.

KEY FACTOR word2vec:
Vector space models (VSMs) represent (embed) words in
a continuous vector space where semantically similar
words are mapped to nearby points ('are embedded
nearby each other').
Hummingbird
https://www.tensorflow.org/tutorials/representation/word2vec

Embedded Word Model
Hummingbird

“…words that appear in the same contexts share semantic meaning. The different
approaches that leverage this principle can be divided into two categories: count-
based methods (e.g. Latent Semantic Analysis), and predictive
methods (e.g. neural probabilistic language models).”
Hummingbird

Hummingbird

Hummingbird added a semantic layer
to the search algorithms.

Not Just Keywords Anymore …
Semantic Interpretations.

Hummingbird adds a
semantic layer to the
search algorithms like
synonyms and close
variants.

Hummingbird adds a
search algorithms that
uses “semantic distance
and term relationships”.

Hummingbird adds a
search algorithms that
uses “phrase based
Indexing and co-
occurrence.”

Page Segmentation.
This part of the
algorithm determines
meaning through
placement.

Entity Salience.
This part of the
algorithm determines
meaning through known
relationships.

Entity Salience.
This part of the algorithm
determines meaning
through known
relationships.
+
2018-19 Google adds the
“topic layer” to the
knowledge graph
(categorical classification)

So Hummingbird moves from
strict word count based modeling
(ie keyword counts) to
probabilistic modeling
(ie predictive interpretation)
via known word vectors+nodes (relationships).
Hummingbird

BUT …..
Google Search still
doesn’t process
Natural Language in
the true sense.

This means we must add an “interpreter”.

Do You Schema?

What is Structured Data?
https://developers.google.com/search/docs/guides/intro-structured-data

Structured data for SEO purposes is on-page markup that
enables search engines to better understand the information
currently on your site’s web pages, and then use this information
to improve search results listing by better matching user intent.

This structured data is defined by using schema to act as the
interpreter. This is the definition we add to the page using
schema code.
Google allows 3 types.
• RDFa
• Microdata
• JSON-LD PREFERS

Schema
JSON-LD is the recommended schema code.
JSON-LD stands for JavaScript Object Notation for Linked Data
This is just a way to implement schema outside the HTML mark-up
structure. RDFa and Microformats required the code to be implemented
via HTML.

Schema
Benefit is it can be removed from the HTML structure, which
makes it easier to write, implement, and maintain.
Resources.
For a good breakdown on what JSON is at the code level.
Portent’s JSON Implementation Guide is very helpful.
https://www.portent.com/blog/seo/json-ld-implementation-guide.htm
And Google has a section in the Developer Guides
https://developers.google.com/search/docs/guides/intro-structured-data

Why Does Schema Matter?

We can act as the interpreter and help “teach”
Google what our site is about.

Adding semantic mark-up
(structured data via schema) allows us to tell
Google what WE SAY our site is about and WHAT
RELATIONSHIPS we define within it.

We can act as the interpreter and help “teach”
Google the context of our content.

We can help give Google a clearer understanding.
That helps us help Google better answer
the questions users ask
and to better surface our content for those users
We give our data meaning
Google Understands

Ranking Without Links

Rank Brain

Rank Brain is used for Unknown Queries where entity
meanings/relationships are unclear or unknown.

Rank Brain.
One of two algorithms that uses AI
on the live results
Rank Brain.

Presence of Rank Brain means
Google is confused …

Why?
Google does not use true NLP in Search
(Natural Language Processing)
Rank Brain.

• Words go in.
• Words get assigned a mathematical address in a vector.
• Similar and related words sit close to each other in the vector space.
• Words are retrieved based on your query and the words it locates in the “best fit” vector.
• These word “interpretations” are used to return results.
• If the relationships are weak or unknown, enter Rank Brain.
• Behind the scenes, data is continually fed into the machine learning process, so as to make
those results more relevant the next time.
Rank Brain – Known Relationships.

Rank Brain Also Uses Users Queries & Clicks
to Help It Understand Query Intent.

Remember this result?

A year later …
(yellow = poor intent match)

Google + Rank Brain Also Uses Users Queries & Clicks
to Help It Understand Query Intent.
+ GEO LOCATION

A year later in London …

Why location?
Semantic Relevancy
Sweets has no definitive entity in the US.

Neural Matching
SEP 2018: Rank Brain has a friend now.

Rank Brain.
What is the difference between
Rank Brain and Neural Matching?
Rank Brain.

Rank Brain vs Neural Matching.
Rank Brain = concepts
Neural Matching = linking words to concepts
“…neural matching, – AI method to better connect words to concepts.” - Google

Rank Brain vs Neural Matching.
A Google patent related to Rank Brain and Neural Matching
describes a system that uses traditional ranking factors to decide
what is relevant, but NOT what is in the top 10.
Which may be re-ordered post retrieval according to
“ad hoc retrieval” methods and ”dynamic relevancy”
https://www.searchenginejournal.com/google-neural-matching/271125/

AND IT DOES
NOT USE LINKS
https://www.oncrawl.com/technical-seo/neural-matching-seo-content-creation-rules/

So should you optimize for Rank Brain and
Neural Matching?
Rank Brain and Neural Matching.

Why would you optimize to rank with AI?
AI Is ever-changing and unfixed.
Don’t waste the time and resources.
Rank Brain.

Google Does Not Even Understand What
Rank Brain (and neural matching) is Actually Doing.
Rank Brain and Neural Matching.

(Gary Illyes)
Rank Brain.

ENTER BERT, ERNIE, AND ELMO.

Takeaways.
• BERT + ELMo are Google
• Ernie is BAIDU’s BERT
Takeaways.

Sesame Street and Seach
• BERT + ELMo are Google
• Ernie is BAIDU’s BERT

Sesame Street and Search: BERT
https://bensen.ai/elmo-meet-bert-recent-advances-in-natural-language-embeddings/

Sesame Street and Search

Sesame Street and Search: Why is BERT Special?
• Language modeling is an effective task for using unlabeled data to pretrain neural networks in NLP
• Traditional language models take the previous n tokens and predict the next one. In contrast, BERT trains a
language model that takes both the previous and next tokens into account when predicting.
• BERT is also trained on a next sentence prediction task to better handle tasks that require reasoning about
the relationship between two sentences (e.g. question answering)
• BERT uses the Transformer architecture for encoding sentences.
• BERT performs better when given more parameters, even on small datasets.

Because BERT can go forward and backwards.

READ THIS
https://mlexplained.com/2019/01/07/paper-dissected-bert-pre-training-of-deep-bidirectional-transformers-for-language-understanding-explained/

Confused yet?

Simply put BERT or language modeling is
“Language modeling – although it sounds formidable –
is essentially just predicting words in a blank.”

Simple answer to a very complex issue?
Do your normal query research then just write in
natural and conversational language.
Create holistic content.

Holistic Content?

Write holistic content.
Use terms that are semantically related.
For a detailed explanation Google explains here > https://www.youtube.com/watch?v=vzoe2G5g-w4&feature=youtu.be&t=32m19s

Write holistic content.
DOES YOUR CONTENT HAVE DEPTH AND WIDTH?
For a detailed explanation Google explains here > https://www.youtube.com/watch?v=vzoe2G5g-w4&feature=youtu.be&t=32m19s

And Use Well Formed Text.

Well Formed Text.
http://www.kurzweilai.net/google-open-sources-natural-language-understanding-tools
Ray Kurzweil on Google NLU

Questions = Well Formed Text
https://ai.google/research/pubs/pub47323

SO ALWAYS REMEMBER &
NEVER FORGET
Google is an Answer Engine.

THINK in Intent, Query Terms, & Context.

And Questions!

Takeaways.
• Think Search Queries NOT Simple Keywords
• Write in natural, conversational language
• Write using holistic content
• Focus on depth and breadth with related terms
• Add Structured Data
• Use well formed text (ie questions) when you can.
Takeaways.

Takeaways.
Helpful Tools
• Entities: InLinks
https://www.inlinks.net
• Schema MarkUp & Monitoring: Schema App
https://www.schemaapp.com/
• Single Page: Google Schema Markup Tool
https://www.google.com/webmasters/markup-helper/?hl=en
TOOLS

• Learn how to code tensorflow
https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist/#0
• NLU vs NLP: What’s the Difference?
https://www.bmc.com/blogs/nlu-vs-nlp-natural-language-understanding-
processing/
• BERT State of Art Pre Learning AI
https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html
• GITHUB or BERT
https://github.com/google-research/bert
• GOOGEL PAPER: BERT: Pre-training of Deep Bidirectional Transformers for
Language Understanding
https://arxiv.org/abs/1810.04805
• Google Brings in BERT to Improve its Search Results
https://techcrunch.com/2019/10/25/google-brings-in-bert-to-improve-its-
search-results/
• Google Blog on BERT
https://www.blog.google/products/search/search-language-understanding-bert/
• Answering Questions Using the Knowledge Graph
https://gofishdigital.com/answering-questions-using-knowledge-graphs/
More About BERT and NLP.

How We Got To Sesame Street
Google’s Search for NLP
Kristine@SitesWithoutWalls.com

How Did We Get to Sesame Street? Google's Search for NLP.

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to How Did We Get to Sesame Street? Google's Search for NLP.

Similar to How Did We Get to Sesame Street? Google's Search for NLP. (20)

More from Kristine Schachinger SEO and Online Marketing

More from Kristine Schachinger SEO and Online Marketing (9)

Recently uploaded

Recently uploaded (20)

How Did We Get to Sesame Street? Google's Search for NLP.