This document proposes GraphTrans, a framework that uses graph neural networks (GNNs) to learn local structure and a modified Transformer to learn global structure from graphs. It introduces challenges in capturing long-range dependencies with GNNs alone. GraphTrans leverages a GNN backbone to learn local structure and adds a Transformer to pool local embeddings and extract global structures. The document evaluates GraphTrans on biomolecular and computer programming benchmark datasets and analyzes the impact of adding a CLS token.

1. Van Thuy Hoang
Dept. of Artificial Intelligence,
The Catholic University of Korea
hoangvanthuy90@gmail.com
Zhanghao Wu, et al.; NIPS 21

2. 2
Graphs
 Graphs are an important representation of nartual structures

3. 3
Graph neural networks
 Graph neural networks aggregate local neighbourhood structure
 By iteratively pooling immediate neighbourhood, GNNs slowly learn to
represent local structure

4. 4
Challenge for Graph neural network
 Long range dependencies are not represented in GNNs

5. 5
Graph classification
 Long range dependencies are important to graph classification
 Graph classification considers pooling embeddings into single prediction
vector
 Small interactions in modelules may results
in large changes in function

6. 6
Learning global interactions with GraphTrans
 Global attention with Transformers to identify long range dependencies
 A standard GNN submodule learns local, short-range structure, then a global
Transformer submodule learns global, long-range relationships.

7. 7
Graph Transformer
 Transformer alone can not be able to learn graph structure
 Local and global structure
 Capture the structural relation between nodes
 Graph isomorphism problem

8. 8
GraphTrans
 Learning global interactions with graphtrans
Leverage a SOTA GNN
backbone to learn local
structure
Adding a modified
transformer to pool local
embeddings to extract
global structures

9. 9
GraphTrans: GNN module
 Can be used with a 4 variety of GNNs.
 A generic GNN layer stack can be expressed as:

10. 10
GraphTrans: Transformer module
 Can be used in concert with a 4 variety of GNNs.
 A generic GNN layer stack can be expressed as:

11. 11
GraphTrans: example
 Graph from OBG-code2 dataset
 Node 8 and 17 are related, but they are far in graphs

12. 12
Learning global information with CLS token
 Adding of a single global CLS token aggregates global information into a
single vector

13. 13
Evaluation
 NCI biomolecular datasets

14. 14
Evaluation
 OpenGraphBenchmark Molpcba dataset

15. 15
Evaluation
 Computer programming benchmark
OpenGraphBenchmark Code2 dataset

16. 16
Evaluation
 Ablation of <CLS> token

17. 17
Conclusion
 GraphTrans, a simple powerful framework for learning long-range
relationships with GNNs.
 Augment standard GNN layer stacks with a subsequent permutation-
invariant Transformer module
 Limitations:
 Permutation problem
 Isomorphism problem