This document discusses various ways to ground the symbols used in knowledge graphs. It describes the traditional "symbol grounding problem" where symbols are defined based only on other symbols. It then outlines several approaches to grounding symbols in non-symbolic ways, such as by linking them to perceptual modalities like images, audio, and simulation. It also discusses grounding symbols via embeddings, relationships to physical entities, and operational semantics. The document argues that richer grounding could help integrate these notions and enhance interoperability, exchange, identity, and reasoning over knowledge graphs.

1. More ways of symbol grounding
for knowledge graphs?
Paul Groth (@pgroth)
Elsevier Labs
pgroth.com
Dagstuhl Seminar 18371
Knowledge Graphs: New Directions for Knowledge Representation on the Semantic Web

2. "How can you ever get off the symbol/symbol merry-go-round? How is
symbol meaning to be grounded in something other than just more
meaningless symbols? This is the symbol grounding problem.”
(Harnard, 1990)
Harnad, S. (1990) The Symbol Grounding Problem.
Physica D 42: 335-346. http://cogprints.org/3106/
What does http://dbpedia.org/resource/Netherlands mean?

3. Symbol Grounding & the Semantic Web
Key notion: Social commitment
(Cregan, 2007)
• designation - what is being referred to
• entailment - what are the (logical)consequences of something
Good enough?
Cregan A.M. (2007) Symbol Grounding for the Semantic Web. In: Franconi E., Kifer M., May W.
(eds) The Semantic Web: Research and Applications. ESWC 2007. Lecture Notes in Computer
Science, vol 4519. Springer, Berlin, Heidelberg

4. Designation & Dereferenceablity
Looking definitions up – Natural Language and Programmatic

5. WIKIDATA VOCABULARY

6. schema:dateModified a rdf:Property ;
rdfs:label "dateModified" ;
schema:domainIncludes schema:CreativeWork,
schema:DataFeedItem ;
schema:rangeIncludes schema:Date,
schema:DateTime ;
rdfs:comment "The date on which the CreativeWork was
most recently modified or when the item's entry was
modified within a DataFeed." .
schema:datePublished a rdf:Property ;
rdfs:label "datePublished" ;
schema:domainIncludes schema:CreativeWork ;
schema:rangeIncludes schema:Date ;
rdfs:comment "Date of first broadcast/publication." .
schema:disambiguatingDescription a rdf:Property ;
rdfs:label "disambiguatingDescription" ;
schema:domainIncludes schema:Thing ;
schema:rangeIncludes schema:Text ;
rdfs:comment "A sub property of description. A short
description of the item used to disambiguate from other,
similar items. Information from other properties (in
particular, name) may be necessary for the description to
be useful for disambiguation." ;
rdfs:subPropertyOf schema:description .
https://www.w3.org/TR/rdf11-mt/
Entailment – logics

7. Are relations good enough to describe entities?
A knowledge graph is "graph structured knowledge bases (KBs) which store factual
information in form of relationships between entities" (Nickel et al. 2015).
Nickel, M., Murphy, K., Tresp, V., & Gabrilovich, E. (2015). A Review
of Relational Machine Learning for Knowledge Graphs, 1–18.

8. Other ways of grounding symbols

9. Sub-symbolic representations (aka embeddings)
Yang, Fan, Zhilin Yang, and William W. Cohen. "Differentiable learning
of logical rules for knowledge base reasoning." Advances in Neural
Information Processing Systems. 2017.
Rocktäschel, T., & Riedel, S. (2017). End-to-end differentiable
proving. In Advances in Neural Information Processing
Systems (pp. 3791-3803).

10. Grounding in physical reality
http://cynthia.matuszek.org/Icra2014GestureLanguage.html
https://www.csee.umbc.edu/~cmat/
“Grounded Language Acquisition: Learning models of
language using data from the noisy, probabilistic physical
world in which robots and humans both reside. This
makes language learning easier (how do you learn the
meaning of "green" without a camera?) and makes
robots more able to understand instructions and
descriptions.”
Wiriyathammabhum, P., Summers-Stay, D., Fermüller, C., &
Aloimonos, Y. (2017). Computer vision and natural language
processing: recent approaches in multimedia and robotics.
ACM Computing Surveys (CSUR), 49(4), 71.

11. Grounding in Perception – Audio / Images
Kiela, Douwe, and Stephen Clark. "Learning neural
audio embeddings for grounding semantics in
auditory perception." Journal of Artificial
Intelligence Research 60 (2017): 1003-1030.
Kiela, Douwe. Deep embodiment: grounding semantics in perceptual modalities.
No. UCAM-CL-TR-899. University of Cambridge, Computer Laboratory, 2017.
Kiela, D., Conneau, A., Jabri, A., & Nickel, M. (2017). Learning visually
grounded sentence representations. arXiv preprint arXiv:1707.06320.

12. Image and Video Grounding Datasets
visualgenome.org
Visual Genome: Connecting Language and Vision Using
Crowdsourced Dense Image Annotations
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji
Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li Jia-
Li, David Ayman Shamma, Michael Bernstein, Li Fei-Fei
Gella, Spandana, and Frank Keller. "An Analysis of Action Recognition Datasets for Language and
Vision Tasks." Proceedings of the 55th Annual Meeting of the Association for Computational
Linguistics (Volume 2: Short Papers). Vol. 2. 2017
Xu, J., Mei, T., Yao, T., & Rui, Y. (2016). Msr-vtt: A large video description dataset for
bridging video and language. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition (pp. 5288-5296).
Miech, A., Laptev, I., & Sivic, J. (2018). Learning a Text-Video Embedding from
Incomplete and Heterogeneous Data. CoRR, abs/1804.02516.
https://www.di.ens.fr/willow/research/mee/

13. Grounding in Simulation
https://ai2thor.allenai.org

14. Operational Semantics (or actually just Javascript)
https://mixedreality.mozilla.org

15. Thoughts
• Potential richer ways to ground the symbols within a knowledge
graph.
• How do we integrate with these notions?
• Things that can be brought to this work
• Interoperability
• Exchange
• Identity
• Reasoning
• Things not mentioned but in the same boat:
• Abstract Meaning Representation
• Universal Dependencies