Berlin10 beyond the impact factor

Beyond the Impact Factor:
Why the Thomson-Reuters impact factor has to be replaced

Tom Olijhoek

SURF NL

Acknowledgements
Paul Wouters Leiden University
Jelle Wicherts Tilburg University
Björn Brembs Regensburg

This work is licensed under a Creative
Commons Attribution 3.0 Unported License

You are free to:
Copy, share, adapt, or re-mix;

Blog, live-blog, or post video of;

This presentation. Provided that:

You attribute the work to its author and respect the
rights and licenses associated with its components.

Slide Concept by Cameron Neylon, who has waived all copyright and related or neighbouring rights. This slide only ccZero.
Social Media Icons adapted with permission from originals by Christopher Ross. Original images are available under GPL at;

Beyond the impact factor
Tom Olijhoek SURF-NL

The Thomson Reuters impact factor is used to assess the quality of a journal

The TR impact factor CORRELATES VERY WELL
with the perceived quality of a journal

SO WHAT IS WRONG WITH
IT?


MANY THINGS ARE WRONG WITH THE IMPACT
FACTOR

 Fortunately Open Access enables other methods for
Quality Assessment

 But…..the Impact Factor is an obstacle for Open Access

 To get Open Access we need to get rid of the Impact
Factor

 For that we need an attitude change

 For that we need commitment of scientist communities

 In all parts of the world


 Especially in the developing world

 Participation of scientists in the developing word will
make the difference

 To participate on an equal basis in the making of science

 To profit on an equal basis of the fruits of science

 Science is the motor of economic development

 OPEN ACCESS IS THE KEY TO
EDUCATION, INNOVATION, ECONOMIC
DEVELOPMENT AND PROSPERITY EVERYWHERE
ALSO IN THE DEVELOPING WORLD


Lars Bjørnshauge Quote from talk at the PKP conference
September 2011, Berlin:

The push for researchers from [developing
countries] and continents to publish in high
impact factor journals has decisive influence
on the subject of their research and much
more so is a big obstacle for open access
publishing.


The Impact Factor
Introduced in 1960‟s by Eugene Garfield: ISI

citations articles

2008 2006 and 2007

IF=5
Articles published in 06/07
were cited an average of 5 times in 08.


The impact factor

Nothing wrong with using citations quality criteria
for

Using average citations for the average article as
quality indicator for a
That is where things go wrong

You can NOT draw conclusions on INDIVIDUAL
article qualities based on the AVERAGE quality of
ALL ARTICLES in a journal


Our management discourages us from supporting new open access
journals due to their low, or unassigned, impact factor MalariaWorld survey 2012

The USE of the impact factor for
ASSESSING SCIENTISTS is
obstructing the move towards Open
Access

Weak correlation of individual article
citation rate with journal IF

But scientists are judged on NUMBER of
publications in HIGH IMPACT
JOURNALS

Most scientists do not publish in OA
journals for one reason: because it
could hamper their careers

Weak correlation of individual article citation
rate with journal IF

Seglen PO (1997): Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314(7079):497 (15 February)


PLoS Medicine, IF 2-11 (8.4)
(The PLoS Medicine Editors (2006) The
Negotiable Impact Factor Game. PLoS Med 3(6): e291.

Irreproducible Rossner M, van Epps H, Hill E (2007): Show me the
data. The Journal of Cell Biology, Vol. 179, No.
6, 1091-1092

Can be gamed Journals exert pressure to get cited themselves often
Scientists „ask“ to be cited
Fake authors

Björn Brembs
http://www.slideshare.net/brembs/limited-
access-is-a-symptom-not-the-disease


NEGOTIABLE: number of articles which
are cited can be adjusted

(The PLoS Medicine Editors (2006) The Impact Factor Game. PLoS Med 3(6): e291.

Current Biology IF from 7 to 11 in
2003
Bought by Cell Press (Elsevier) in 2001…


NEGOTIABLE

What is obvious from this equation is that the impact factor
depends crucially on which article types Thomson Scientific
deems as “citable”—the fewer, the better (i.e., the lower the
denominator, the higher the impact factor).
(The PLoS Medicine Editors (2006) The Impact Factor Game. PLoS Med 3(6): e291.


Methodological problems with the Impact
Factor

Correlation of IF with:
•Retraction rates: high
•Subjective journal rank: very
high
•Quality of individual articles: low
•Citations: low
•Expert opinion: low
•Methodological standards: low
•Replicability: none

Björn Brembs and Marcus Munafò
http://bit.ly/WNzA1Z


Two More Problems

The Impact Factor is commercially produced
The TR-Impact Factor underscores
research topics from the South by design

50,000 employees
US$600million profit/quarter
Thomson family owns 53%
€30,000-130,000/year subscription rates

€30,000-130,000/year subscription rates
Covers ~11,500 journals (Scopus covers ~16,500)


OPEN ACCESS ENABLES THE
DEVELOPMENT OF NEW WAYS OF QUALITY
ASSESSMENT

While Open Access is hindered by the TR
Impact Factor new methods made possible by
Open Access can replace it!

Measure Impact beyond mere citation
analysis

Measure Impact beyond scientific impact


„New‟ Definition of Scientific Impact


“When I light my candle from yours, I gain from you without subtracting from you. That’s what
sharing knowledge is like”. Peter Suber

Open Access Toll Access
getting new ideas by
fear of losing ideas
sharing

Collaboration Competition

Publish for impact Publish or perish

Focus on quality Focus on quantity

We shouldn‟t be counting the beans but instead taste them
The proof is in the pudding

We need commitment
Open Access is crucial for scientists in the global south
conference The Hague 25 oct 2012
as long as scientific output remains behind walls of paid content, no possibility for a dialogue will exist

Science is the
Open Access opens motor for
Science for All economic
development

Participation of
scientists from
Research is the
Africa, Asia and
key to fighting
Latin America
disease
is necessary for
success


Specs for a global
system
Open Access can contribute to that, provided that the existing dominant
features of the existing system:
Citation counts and the JIF measures of impact (inadequate, insufficient
and subject to gaming)
will be replaced by
measures that much better reflects the impact of research not only on
research itself, but on innovation, health, wealth and societies
(altmetrics).
Luckily these are as well requirements of a successful breakthrough of
OA in the North.

Lars Bjørnshauge
SPARC Europe – www.sparceurope.org

The Hague Oct 25th 2012


ultimately scientists need to
realize that they hold the
power in their own hands

TAKE ACTION


When did scientists start agreeing with this slave-type of
agreement with publishing houses? How could this
nonsense have started? We inherited this sick system, but
that does not mean we should allow it to continue

MalariaWorld Survey on Open Access 2012


A new Journal Impact Factor?
Do we really need it?
In an Open Access world
We only need:
• article level metrics to assess the impact of
articles AND scientists
• a quality indicator (seal?) but NOT impact
factor to assess the quality of journals
In an Open Access world
It does not matter much where something is
published,
more important is the quality of the individual
articles
HOW GOOD VERSUS HOW MUCH AND WHERE


The Relevance Index

Use of new metrics to assess the impact
of scientific works in all areas not only
science but including
innovation, health, wealth and societies


The Reputation Index

Use of the Relevance Index to assess
the reputation of authors / scientists


We need a Quality Indicator
for (0pen access) journals

WHY?


New quality assessment methods in science

Articles Journals Authors

• Citations • Impact Factor • Citation index
• H-Index
• NEW • Citation based • NEW
• Relevance Index • Reputation
• Multi-level tools • NEW Index
• Total Impact • A-Vector • Subjective
• Altmetric • Based on quality • Based on more
explorer of peer review & than citations
quality of
editorial board


A-VECTOR: INTRODUCING A NEW FORM OF
JOURNAL LEVEL METRICS
Results of the Rotterdam colloquium held on 22-23 October 2012

Quality of editorial board Quality of peer-review

• Citation index • “Transparency” indicators
• Reputation • Criteria used by reviewers
• Collaboration • Duration of review process
• Reference density • Post-publication comments
• Openness about
• More indicators • submission and Rejection rates
• potential conflicts of interest
• Aims, scopes and expected
readership
• Reviewer‟s comments and editorial
correspondence ( published
alongside papers
• More indicators


PREDICTION OF JOURNAL QUALITY BY A-VECTOR USING
EDITORIAL BOARD MEMBER QUALITIES AS INDICATORS

 CONCLUSION
• Prediction of Impact Factor based on TCS (total citation  Quality of editorial
score), MCS (mean citation score), H-index of Editorial board members can be
Board Members used to judge the
• Performance test: prediction of IF for established journals (potential) quality of
with error of 1.5 points (95% confidence) Open Access journals

• Requires some refinement, but can be applied to young OA
journals


PREDICTION OF JOURNAL QUALITY BY A-VECTOR USING
PEER-REVIEW TRANSPARENCY FACTORS AS INDICATORS
100 journals / 221 authors

17 established journals: score = 45.4, SD =11.9 CONCLUSION:
12 OA established journals: score = 54.7, SD = 7.7
13 OA Predatory journals: score = 34.1, SD = 6.7 authors‟ assessments of
the quality of the peer-
review of accepted papers
can be predicted by using
a set of 15 indicators of
transparency for the peer-
review process of journals

EXAMPLES:
Recent Scientific Research score 24
PLoSONE score 67.7
Malaria Journal score 50
MalariaWorld Journal score 47


Item/criterion [authors] Converg Item-
ence (η2) rest
correl.

1. Aims, scope, and expected readership of the
.866 .457
journal are clearly specified on the journal’s website

2. Types of submissions that are deemed
appropriate for the journal are explicated on the .899 .587
website

3. Criteria used by reviewers to rate submissions are
.692 .699
specified on the website
4. The website indicates whether all submissions
are sent out for review and who will make final
.821 .660
decisions about them (e.g., editor, associate/action
editor)
5. The website provides timely updates of the status
of submissions during the peer-review process (e.g., .749 .723
under review)


Reliability

Across raters: Convergence
Across items: Reliability

Journal q1 q2 q3 q4 q5 q6
1 5 5 5 5 5 5
1 5 5 1 4 3 2
1 5 4 3 1 5 5
1 4 2 1 4 4 2
2 4 4 5 3 5 3
2 5 5 5 5 5 3
2 5 5 4 3 4 3
2 5 5 3 3 5 3
3 5 5 4 4 4 2

Convergence across raters (13 journals): .838
Reliability across items .900


YOU CAN NOW USE A-VECTOR YOURSELVES
TO PREDICT JOURNAL QUALITY USING THE
NIEW TRANSPARENCY FACTORS

Go to the link :

http://tinyurl.com/8br9m8w

And complete the survey


YOU CAN NOW USE A-VECTOR YOURSELVES
TO PREDICT JOURNAL QUALITY USING THE
NIEW TRANSPARENCY FACTORS

Applying the indicator

The journals we would like you to assess are as follows:

Journal no. 40

Journal title: Regional Studies

Journal URL http://www.tandfonline.com/toc/cres20/current

Journal no. 46

Journal title: The Internet Journal of Psychiatry

Journal URL http://www.ispub.com/journal/the-internet-journal-of-psychiatry/

Journal no. 8

Journal title: BMC Neuroscience

Journal URL http://www.biomedcentral.com/bmcneurosci/

The indicator can be found here: tinyurl.com/8br9m8w


Berlin10 beyond the impact factor

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Empfohlen

Empfohlen (20)

Berlin10 beyond the impact factor