To ensure that publications are assigned to clusters in a meaningful way, we introduce the notion of stable clusters. Essentially, a cluster is stable if it is insensitive to small changes in the underlying data. Bootstrapping is used to make small changes in the data.

1. Cluster stability
Nees Jan van Eck and Ludo Waltman
Centre for Science and Technology Studies (CWTS), Leiden University
Workshop “Comparison of Algorithms”, Amsterdam
April 20, 2015

2. Problem statement
• A clustering technique can be used to obtain highly
detailed clustering results (i.e., a large number of
clusters)
• A clustering technique can be used to force each
publication to be assigned to a cluster
• However, in a highly detailed clustering, is the
assignment of publications to clusters still
meaningful?
• The assignment of a publication to a cluster may be
based on very little information (e.g., a single
citation relation)
1

3. Example: Waltman and Van Eck (2012)
2

4. Cluster stability
• To ensure that publications are assigned to clusters
in a meaningful way, we introduce the notion of
stable clusters
• Essentially, a cluster is stable if it is insensitive to
small changes in the underlying data
• Bootstrapping is used to make small changes in the
data
• There is no formal statistical framework
• To some extent, this resembles the stability
intervals in the CWTS Leiden Ranking
3

5. Identification of stable clusters:
Step 1
• Collect the citation network of publications
• Create a large number (e.g., 100) of bootstrap
citation networks
• In each bootstrap citation network, perform
clustering:
– Clustering technique of Waltman and Van Eck (2012)
– User-defined resolution parameter
– Smart local moving algorithm of Waltman and Van Eck (2013)
• For each pair of publications, calculate the
proportion of the bootstrap clustering results in
which the publications are in the same cluster
4

6. Identification of stable clusters:
Step 2
• Create a network of publications with an edge
between two publications if the publications are in
the same cluster in at least a certain proportion
(e.g., 0.9) of the bootstrap clustering results
• Identify connected components in the newly
created network
• Each connected component represents a stable
cluster
5

7. Non-parametric bootstrapping
• Sample with replacement from the set of all citation
relations between publications
• Make sure to obtain a sample that is of the same
size as the original set of citation relations
• Some citation relations will occur multiple times in
the sample, others won’t occur in it at all
• Based on the sampled citation relations, create a
bootstrap citation network
• Edges have integer weights in this network
6

8. Parametric bootstrapping
• A bootstrap citation network is a weighted variant
of the original citation network, with each edge
having an integer weight drawn from a Poisson
distribution with mean 1 (cf. Rosvall & Bergstrom,
2009)
• Total edge weight in the bootstrap citation network
will be approximately equal to the number of edges
in the original network
• For large networks, parametric and non-parametric
bootstrapping coincide
• We use parametric bootstrapping
7

9. Data
• Library & Information Sciences (LIS):
– Time period: 1996-2013
– Publications: 31,534
– Citation links: 131,266
• Astrophysics (Berlin dataset):
– Time period: 2003-2010
– Publications: 101,828
– Citation links: 924,171
8

10. Cluster stability LIS
9

11. Stable clusters LIS (resolution 2)
10

12. Stable clusters LIS (resolution 2)
11

13. Cluster stability Berlin
12

14. Cluster stability
13
LIS Berlin

15. Conclusions
• What is a good clustering of publication?
– High accuracy: Publications in the same cluster are topically
related
– High level of detail: It is possible to have a large number of
clusters
– Comprehensiveness: The clustering includes all publications
– Uniformity in cluster size: Clusters are of roughly the same size
• It seems impossible to obtain a clustering that has
all properties listed above
• At least one property needs to be given up
14

16. Conclusions
• Why cannot we have an accurate and detailed
clustering that includes all publications?
– Consider the field of scientometrics
– We would expect an accurate and detailed clustering to have
clusters dealing with topics such as indicators, science mapping,
collaboration, patents, etc.
– However, many publications in scientometrics (e.g., case studies)
do not neatly belong to one of these topics and therefore cannot
be accurately assigned to a cluster
• If we want to have an accurate and detailed
clustering, we need to be satisfied with a clustering
that doesn’t comprehensively cover all publications
• The clustering covers only publications related to
the main topics in the fields 15

17. Conclusions
• Analysis of cluster stability offers an approach to
distinguish between meaningful and non-
meaningful assignments of publications to clusters
• Clustering based on direct citations is
computationally attractive but ignores relevant
information (e.g., bibliographic coupling)
• A post processing procedure can be developed to
try to assign ‘isolated publications’ to stable
clusters based on additional information
• Cluster stability is a general idea that can be
applied also to other clustering approaches
16

18. References
Rosvall, M., & Bergstrom, C.T. (2009). Mapping change
in large networks. PLoS ONE, 5(1), e8694.
http://dx.doi.org/10.1371/journal.pone.0008694
Waltman, L., & Van Eck, N.J. (2012). A new methodology
for constructing a publication-level classification
system of science. JASIST, 63(12), 2378-2392.
http://dx.doi.org/10.1002/asi.22748
Waltman, L., & Van Eck, N.J. (2013). A smart local moving
algorithm for large-scale modularity-based community
detection. European Physical Journal B, 86(11), 471.
http://dx.doi.org/10.1140/epjb/e2013-40829-0
17