Apesar de seu uso generalizado, os métodos bibliométricos têm várias limitações, especialmente quando se trata da análise dos países em desenvolvimento. Esta apresentação irá recuperar estas limitações (cobertura parcial da literatura nacional em países de língua não inglesa, baixa cobertura de livros, etc.), bem como fornecer dados sobre a sua extensão nas diferentes disciplinas. Também serão detalhados as limitações do Fator de Impacto e os argumentos contra a sua utilização em avaliação da pesquisa. Despite their widespread use, bibliometric methods have several limitations, especially when it comes to the analysis of developing countries. This presentation will recall these limitations (partial coverage of national literature for non-English speaking countries, weak coverage of books, etc.) as well as provide data on their extent in the different disciplines. It will also detail the drawbacks of the Impact Factor and argue against its use in research evaluation. A pesar de su uso generalizado, los métodos bibliométricos tienen varias limitaciones, especialmente cuando se trata del análisis de los países en desarrollo. Esta presentación recordará estas limitaciones (cobertura parcial de la literatura nacional de los países de habla no inglesa, la debilidad de la cobertura de libros, etc), así como proporcionar datos sobre su extensión en las distintas disciplinas. También describirá pormenorizadamente los inconvenientes del Factor de Impacto y argumentar en contra de su uso en la evaluación de la investigación.

1. Faculté des arts et des sciences
École de bibliothéconomie et des sciences de l'information
On the Limits of
Bibliometrics
Vincent Larivière
Canada Research Chair on the transformations
of scholarly communication
Université de Montréal
SciELO 15 anos
24 Outubro 2013
SciELO
15 anos
1/15

2. Faculté des arts et des sciences
SciELO
15 anos
2/15
Outline of the talk
- Technical limits
- Coverage
- Limits of indicators
- Impact Factor and prediction of citations
- Citation window
- Adverse consequences

3. Faculté des arts et des sciences
SciELO
15 anos
Technical limits
- No database is covers all of the published
literature.
- Particularly true in the social sciences and
humanities.
- Local journals are not indexed by major
databases and hence, local topics are not well
covered
- No books were indexed (until recently).
3/15

4. Faculté des arts et des sciences
SciELO
15 anos
Percentage of references made to journal articles in the natural and
medical sciences (Larivière et al. 2006)
95%
90%
85%
80%
75%
70%
65%
60%
55%
1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000
Year
Average SCI
Physics
Chemistry
Engineering
Biomedical Research
Biology
Psychology
Clinical Medicine
Earth and Space Sciences
Mathematics
4/15

5. Faculté des arts et des sciences
SciELO
15 anos
Percentage of references made to journal articles in the social
sciences and humanities (Larivière et al. 2006)
5/15

6. Faculté des arts et des sciences
SciELO
15 anos
Proportion of references made to conference proceedings in
the natural and medical sciences (Lisée et al. 2008)
6/15
Computers
Electrical Engineering & Electronics
Civil Engineering
Nuclear Technology
Miscellaneous Engineering & Technology
Aerospace Technology
Industrial Engineering
Mechanical Engineering
Acoustics
Human Factors
Library & Information Science
Chemical Engineering
Optics
General Engineering
Metals & Metallurgy
Applied Physics
0%
5%
10%
15%
20%

7. Faculté des arts et des sciences
SciELO
15 anos
Coverage of journals as a function of the language of
publication (Archambault et al. 2006)
NSE
Language
SSH
Thomson ISI
Ulrich
Difference
Thomson ISI
Ulrich
Difference
English
89%
78%
13%
90%
75%
20%
Czech
0.04%
0.3%
-85%
0.2%
0.2%
8%
Russian
0.5%
0.9%
-48%
0.3%
0.4%
-24%
French
3.3%
3.4%
-3%
3.2%
4.4%
-26%
Multiple languages
0.2%
0.2%
-14%
0.3%
0.5%
-45%
Dutch
2.2%
2.2%
0%
1.3%
2.6%
-48%
German
3.2%
3.9%
-18%
3.0%
5.8%
-50%
Japanese
0.4%
1.7%
-74%
0.2%
0.6%
-64%
Swedish
-
0.1%
-100%
0.1%
0.4%
-69%
Spanish
0.6%
2.6%
-75%
0.9%
3.0%
-69%
Italian
0.1%
0.8%
-83%
0.2%
1.1%
-80%
Danish
0.04%
0.1%
-50%
0.1%
0.3%
-83%
Portuguese
0.1%
0.7%
-85%
0.1%
1.0%
-86%
Chinese
0.3%
2.4%
-88%
0.04%
1.2%
-96%
Polish
0.05%
0.7%
-92%
-
0.9%
-100%
Arabic
-
0.1%
-100%
-
0.3%
-100%
Turkish
0.01%
0.2%
-95%
-
0.1%
-100%
Other
0.3%
1.6%
-80%
0.3%
2.4%
-87%
7/15

8. Faculté des arts et des sciences
SciELO
15 anos
Median age of cited literature
(Archambault et Larivière, 2010)
16
8/15
Arts et
humanités
14
12
Sciences
sociales
10
8
Sciences
naturelles
et génie
6
4
c
u
r
t
l
n
a
i
d
é
m
e
g
Â
2
Médecine
0
1980
1985
1990
Année
1995
2000

9. Faculté des arts et des sciences
SciELO
15 anos
How to correct (some of) these limits
- Creation of book databases
- Book Citation Index (but it still has an angloamerican bias!)
- Triangulate data sources
- Scopus vs. Thomson vs. Google Scholar
- Creation of local databases
- SciELO in Brazil
- Érudit au Québec
9/15

10. Faculté des arts et des sciences
SciELO
15 anos
10/15
Impact factor and its limits

Cannot be compared across disciplines

Skewness of citation distributions

Assymetry between numerator and denominator

Short citation window – especially for SSH

Includes journal self-citations

Is less and less correlated with citations received by papers

11. Faculté des arts et des sciences
SciELO
15 anos
11/15
Different citation practices:
mean citation rates of papers after 2 and 10 years, 1900-2005
Mean - 2 year citation window
7
6
5
24
22
MED
MED
20
18
NSE
16
4
Mean - 10 year citation window
14
SS
12
3
NSE
SS
10
8
2
6
4
1
2
0
1900
1920
1940
1960
1980
2000
0
1900
1920
1940
1960
1980
2000

12. Faculté des arts et des sciences
SciELO
15 anos
12/15
Weakening relationship between Impact
Factor and citations (Lozano et al. 2012)
0,35
y = 0.003x - 5.7759
r2 = 0.579
0,30
0,25
0,20
0,15
y = -0.0049x + 9.9863
r2 = 0.4788
)
2
(
m
r
d
it
f
e
o
lc
a
u
n
A
0,10
0,05
0,00
1900
y = 0.0011x - 1.9418
r2 = 0.4666
1920
1940
1960
1980
2000
2020

13. Faculté des arts et des sciences
SciELO
15 anos
13/15
Decline of top journals (Larivière et al. 2013)

14. Faculté des arts et des sciences
SciELO
15 anos
Adverse effect
- Change of research topics
Type of journal
Brazilian Journals
USA Journals
Other Journals
Brazil in abstract
31%
15%
22%
14/15

15. Faculté des arts et des sciences
SciELO
15 anos
15/15
Bibliography
Archambault, É., Vignola-Gagné, É., Côté., Larivière, V. and Y. Gingras (2006). Benchmarking
scientific output in the social sciences and humanities: The limits of existing databases,
Scientometrics, 68(3): 329-342.
Larivière, V., Archambault, É, Gingras, Y. and É. Vignola-Gagné (2006). The place of serials in
referencing practices: Comparing natural sciences and engineering with social sciences and
humanities, Journal of the American Society for Information Science and Technology, 57(8): 9971004.
Larivière, V., Archambault, É., Gingras, Y. (2008) Long-term variations in the aging of scientific
literature: from exponential growth to steady-state science (1900-2004). Journal of the American
Society for Information Science and Technology, 59(2): 288–296.
Larivière, V., Gingras, Y. and É. Archambault (2006). Canadian Collaboration Networks: A
Comparative Analysis of the Natural Sciences, Social Sciences and the Humanities,
Scientometrics, 68(3): 519-533.
Larivière, V. , Lozano, G.A., Gingras, Y. (2013). Are elite journals declining? Forthcoming in the
Journal of the American Society for Information Science and Technology.
Lozano, G.A., Larivière, V., Gingras, Y. (2012) The weakening relationship between the Impact
Factor and papers’ citations in the digital age. Journal of the American Society for Information
Science and Technology. 63(11): 2140-2145.
Lisée, C., Larivière, V., Archambault, É. (2008) Conference proceedings as a source of scientific
information: A bibliometric analysis. Journal of the American Society for Information Science and
Technology, 59(11): 1776–1784.