Presentation in Science Coffee of the Advanced Concepts Team of the European Space Agency. Date: 22.03.2024 Speaker: Mike Heddes (University of California, Irvine) Topic: Introduction to Hyperdimensional Computing Abstract: Hyperdimensional computing (HD), also known as vector symbolic architectures (VSA), is a computing framework capable of forming compositional distributed representations. HD/VSA forms a "concept space" by exploiting the geometry and algebra of high-dimensional spaces. The central idea is to represent information with randomly generated vectors, called hypervectors. Together with a set of operations on these hypervectors, HD/VSA can represent compositional structures, which, in turn, enables features such as reasoning by analogy and cognitive computing. In this introductory talk, I will introduce the high-dimensional spaces and the fundamental operations on hypervectors. I will then cover applications of HD/VSA such as reasoning by analogy and graph classification.

1. An Introduction to
Hyperdimensional Computing
Mike Heddes
University of California, Irvine
Department of Computer Science
March 2024

2. Outline
1. Alphabet
2. Operations
3. Composition
4. Example: The dollar of the Netherlands
5. Application: Graph classification
6. Torchhd: PyTorch library for HD/VSA
1

3. Origin
Hyperdimensional computing (HD), or vector symbolic architectures (VSA)
Observation
The cerebellum cortex operates on high-dimensional representations [1].
• Forming compositional distributed representations [2, 3]
• The intersection of symbolic and connectionist artificial intelligence
• Symbols → vectors, creating a “concept space”
• Distance metric: angular distance
2

4. Flavors of HD/VSA
Many variations of HD/VSA [4]:
• Binary Spatter Codes [5]
• Multiply-Add-Permute
• Holographic Reduced Representations [6]
• and more...
3

5. Alphabet
Within the Multiply-Add-Permute HD/VSA model [7]:
• Symbols x, y ∼ U{−1, 1}d
• Rademacher distribution
〈x, x〉
d
= 1 E
〈x, y〉
d
= 0 Var
〈x, y〉
d
=
1
d
In high-dimensional spaces:
• Inner products are highly concentrated: quasi-orthogonal
• Only d orthogonal symbols
• O(exp(d)) quasi-orthogonal symbols [4]
4

6. Operations: bundling
• Bundling of symbols: component-wise addition
• Useful for representing a set of symbols
• Result can be normalized to {−1, 1}d
E[〈x, x + y〉] = 〈x, x〉 + E[〈x, y〉] = d
5

7. Operations: binding
• Binding of symbols: component-wise multiplication (Hadamard product)
• Distributes over bundling
• Self-inverse property
x ◦ y ∈ {−1, 1}d
x ◦ x = 1
• Useful for the association of symbols
E[〈x, x ◦ y〉] = E[〈1, x ◦ x ◦ y〉] = E[〈1, y〉] = 0
E[〈x, (x ◦ y) ◦ y〉] = E[〈x, x〉] = d
6

8. Operations: permute
• Permute of a symbol: permutation of its components
• Distributes over binding and bundling
• Useful for the assignment of an ordering
〈x, ρ(x)〉 ≈ 0
• We denote ρ(ρ(x)) as ρ2(x)
• Also has an inverse permutation: ρ−1(x)
7

9. Operations
Operation Notation Domain Codomain
Bundling x + y H × H H or Zd
Binding x ◦ y H × H H
Permute ρ(x) H H
• Where H = {−1, 1}d
• Theses operations enable composition
8

10. Composition: multiset
• Representation for a set of symbols {x1, x2, . . . , xn}:
s = x1 + x2 + · · · + xn =
n
X
i=1
xi
• Query the frequency of symbol xj:
E
〈xj, s〉
d
=
1
d
n
X
i=1
E[〈xj, xi〉] =
n
X
i=1
1[xj = xi]
• No free lunch: bundling capacity O(d) [8]
9

11. Composition: sequence
• Representation for a sequence of symbols (x1, x2, . . . , xn):
s = x1 + ρ(x2) + ρ2
(x3) + · · · + ρn−1
(xn) =
n
X
i=1
ρi−1
(xi)
• Let X = [x1, x2, . . . , xn] be the codebook
• Retrieve the (noisy) k-th symbol:
x̃k = ρ1−k
(s) = xk + noise with XT
noise ≈ 0
= ρ1−k
(x1 + ρ(x2) + ρ2
(x3) + · · · + ρn−1
(xn))
= ρ1−k
(x1) + ρ2−k
(x2) + ρ3−k
(x3) + · · · + ρn−k
(xn)
10

12. Composition: key-value store
• Representation for a set of mappings {(k1 ↔ v1), (k2 ↔ v2), . . . , (kn ↔ vn)}:
s = k1 ◦ v1 + k2 ◦ v2 + · · · + kn ◦ vn =
n
X
i=1
ki ◦ vi
• Retrieve the (noisy) value of the j-th key:
ṽj = s ◦ kj = vj + noise with XT
noise ≈ 0
= (k1 ◦ v1 + k2 ◦ v2 + · · · + kn ◦ vn) ◦ kj
= k1 ◦ kj ◦ v1 + k2 ◦ kj ◦ v2 + · · · + kn ◦ kj ◦ vn
11

13. Associative memory
• Clean up of a noisy symbol x̃:
cleanup(x̃, X) = lookup(x̃, X, X) with lookup(q, K, V) = V σ
‚
KTq
d
Œ
• σ is a nonlinear function, e.g., ReLU, Top-k, Softmax
• Variations include:
• Hopfield network [9]
• sparse distributed memory [10]
• attention mechanism [11]
12

14. The dollar of the Netherlands
• Classical example of reasoning by analogy [12] (originally the dollar of Mexico)
• Key-value stores containing country (k1), capitol (k2), and currency (k3):
u = k1 ◦ USA + k2 ◦ WDC + k3 ◦ USD
n = k1 ◦ NLD + k2 ◦ AMS + k3 ◦ EUR
What is the dollar of the Netherlands?
ß
EUR = u ◦ n ◦ USD = EUR + noise with XT
noise ≈ 0
= (k1 ◦ k1 ◦ USA ◦ NLD + k2 ◦ k2 ◦ WDC ◦ AMS + k3 ◦ k3 ◦ USD ◦ EUR
+ k1 ◦ k2 ◦ USA ◦ AMS + k2 ◦ k3 ◦ WDC ◦ EUR + k3 ◦ k1 ◦ USD ◦ NLD
+ k1 ◦ k3 ◦ USA ◦ EUR + k2 ◦ k1 ◦ WDC ◦ NLD + k3 ◦ k2 ◦ USD ◦ AMS) ◦ USD
= USA ◦ NLD ◦ USD + WDC ◦ AMS ◦ USD + EUR + . . .
13

15. Graph classification
Problem
Given a set of labeled graphs, predict the label of a given unlabeled graph.
Approach
1. Split the set of labeled graphs by their label.
2. Create a vector representation of each graph.
3. Create a vector representation of each set.
4. Find the set most similar to the unlabeled graph.
14

16. Graph encoding
As proposed in GraphHD [13], given an undirected graph G = (V, E).
1. Sample a (global) set of vertex-rank symbols
2. Rank vertices by their PageRank centrality
3. Assign vertex-rank symbols to vertices
4. Bind vertex symbols to represent an edge
5. Bundle all edge symbols
g =
X
(i,j)∈E
vi ◦ vj
15

17. Classification overview
Test
data
Train
data
Encoder
Training
Model
M1
M2
...
Mk
Query vector
Similarities
16

18. Comparison with Perceptron
Model
M1
M2
...
Mk
Query vector
Similarities
HD/VSA classifier
y =
MTq
Æ
tr(MTM)
Æ
qTq
Perceptron
y = WT
x + b
17

19. Torchhd: PyTorch library for HD/VSA
Enable quick development of performant HD/VSA experiments and applications [14].
• A simple programming experience
• High-performance execution
• Support for a variety of HD/VSA applications and research directions
• Includes many reference implementations
• Open source: https://github.com/hyperdimensional-computing/torchhd/
18

20. References I
1. Kanerva, P. Hyperdimensional computing: An introduction to computing in distributed representation with
high-dimensional random vectors. Cognitive Computation 1, 139–159 (2009).
2. Smolensky, P. Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist
Systems. Artificial Intelligence 46, 159–216 (1990).
3. Plate, T. A. Holographic Reduced Representations: Convolution Algebra for Compositional Distributed
Representations. in International Joint Conference on Artificial Intelligence (IJCAI) (1991), 30–35.
4. Kleyko, D., Rachkovskij, D. A., Osipov, E. Rahimi, A. A survey on hyperdimensional computing aka vector symbolic
architectures, part i: Models and data transformations. ACM Computing Surveys 55, 1–40 (2022).
5. Kanerva, P. Fully distributed representation. in Proceedings of the Real World Computing Symposium (1997),
358–365.
6. Plate, T. A. Holographic reduced representations. IEEE Transactions on Neural networks 6, 623–641 (1995).
7. Gayler, R. Multiplicative binding, representation operators and analogy. Advances in analogy research: integration
of theory and data from the cognitive, computational and neural sciences (1998).
8. Thomas, A., Dasgupta, S. Rosing, T. Theoretical Foundations of Hyperdimensional Computing. Journal of Artificial
Intelligence Research (JAIR) 72, 215–249 (2021).
9. Hopfield, J. J. Neural networks and physical systems with emergent collective computational abilities. Proceedings
of the national academy of sciences 79, 2554–2558 (1982).
19

21. References II
10. Kanerva, P. Sparse distributed memory and related models. Tech. rep. (1992).
11. Vaswani, A. et al. Attention is all you need. Advances in neural information processing systems 30 (2017).
12. Kanerva, P. What we mean when we say” What’s the dollar of Mexico?”: Prototypes and mapping in concept space.
in AAAI fall symposium series (2010).
13. Nunes, I., Heddes, M., Givargis, T., Nicolau, A. Veidenbaum, A. GraphHD: Efficient graph classification using
hyperdimensional computing. in Design, Automation Test in Europe Conference Exhibition (DATE) (2022).
14. Heddes, M. et al. Torchhd: An Open Source Python Library to Support Research on Hyperdimensional Computing
and Vector Symbolic Architectures. Journal of Machine Learning Research 24, 1–10.
http://jmlr.org/papers/v24/23-0300.html (2023).
20