1. Rethinking Metrics of International
Growth and Impact of Open Access
Leslie Chan
Bioline International
University of Toronto Scarborough
Symposium on Innovations in AAAS 2012, “Flattening the World: Building a
Reducing International Global Knowledge Society”
Knowledge Isolation Vancouver, Canada
2. Key points
• Open Access as an enabler
• “Journal” no longer serves the needs of
networked scholarship
• From Wealth of Nations to Wealth of Networks
• Need to rethink measurements of “impact” and
values, especially for development
• Innovations are happening in the “peripheries”
but there are gatekeepers
• Aligning funding and reward policies with new
metrics
3. The problem
• Access to literature from the North
• Dissemination of research from the South
• The barriers are structural and institutional
4. The World of Journal Publishing According to Thomson’s ISI
Science Citation Index
Data from 2002
http://www.worldmapper.org/display.php?selected=205
5.
6.
7.
8. “… at a recent editorial team meeting, we discussed a research
paper from a LMIC author. The science was well done and with a
little editing for English, the paper was potentially publishable. But
should we send it out for review? The question we were wrestling
with was whether its findings were sufficiently new to make it
worthy of page space in the journal. This is always a consideration
for all manuscripts, since competition for space is intense and a
priority is to publish interesting research that adds something new
to the field, rather than too many replications of studies already
done. So the initial response when deciding whether to send the
paper out for peer review was: Reject. We already know this, don't
we?”
9. “No journal can afford to devote all or even most
of its precious page space to studies essentially
finding again what others already found, with only
the places changing. And this may be a good place
to remind authors that we almost never publish
prevalence studies, unless they are truly the first
ever done (and sometimes not even then), since
they tend to be of interest primarily in the
countries within which they were conducted.”
10. So who decide on what is “new” and
legitimate knowledge?
And
Who have access to that knowledge?
11. “We editors seek a global status for our
journals, but we shut out the experiences and
practices of those living in poverty by our
(unconscious) neglect. One group is
advantaged, while the other is marginalised.”Richard Horton,
THE LANCET • Vol
361 • March 1,
2003
12. “Research or reviews that cover diseases unlikely
to be encountered in the western world will not
gather the citations that some editors seek.
But if this commercial environment does
seriously skew content away from what matters
to those people the journal claims to serve, as it
surely does at some journals, the culture of
medicine is distorted, even harmed.”
Richard Horton (2003)
14. “Is the scientific paper a fraud?”
“I mean the scientific paper may be a fraud because
it misrepresents the processes of thought that
accompanied or give rise to the work that is
described in the paper. That is the question and I will
say right away that my answer to it is ‘yes’. The
scientific paper in its orthodox form does embody a
totally mistaken conception, even a travesty, of the
nature of scientific though”.
http://contanatura-
hemeroteca.weblog.com.pt/arq
Sir Peter Medawar uivo/medawar_paper_fraud.pdf
(From a BBC talk, 1964)
23. The Centro de Referência em Informação
Ambiental, CRIA (Reference Center on
Environmental Information) http://www.cria.org.br/
24. Active Bioline Journals graphed with Google Maps.
Retrieved April 26, 2010 from:
http://maps.google.com/maps/ms?ie=UTF8&hl=en&msa=0&msid=117328677437228595850.00044ec490b69019e64a3&z=2
24
25.
26.
27. Bioline users March 27 – April 26, 2010
Retrieved April 26, 2010 from: http://analytics.google.com
27
30. The World of Journal Publishing According to Thomson’s ISI
Science Citation Index
Data from 2002
http://www.worldmapper.org/display.php?selected=205
37. The IF is negotiable and doesn’t reflect
actual citation counts
http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0030291
38. The IF cannot be
reproduced, even if it
reflected actual citations
http://jcb.rupress.org/content
/179/6/1091.full
39. The IF is not statistically sound, even if
it were reproducible and reflected
actual citations
http://www.mathunion.org/fileadmin/IMU/Report/CitationStatistics.pdf
40. The IF are more eff
http://iai.asm.org/content/early/201
1/08/08/IAI.05661-
11.full.pdf+html?view=long&pmid=2
1825063
44. Need for policy alignment
and institutional redesign
Governance of
Knowledge Commons
Rethink the values
and reward system
Social Accounting and
Expanded Values
45. Broadening the definition of “success”,
“impact”, “value” and “capital”
Business value monetary return, financial capital,
efficiency, competiveness
Scholarly value Reputation and citation; trust; symbolic
capital
Institutional value Public mission, community outreach,
intellectual capital
Social value Equity, participation, diversity, social
capital
Political value Evidence based policy, transparency,
accountability, civic capital
46. Institutional
Design
Sustainability as a set
of institutional
structures and
processes that build
and protect the
knowledge commons
(after Sumner 2005,
Mook and Sumner 2010)
47. Conclusions
• Open Access is just the substrate, but an
essential one
• Metrics drive behaviour, but we have been
using the wrong metrics
Impact of Open Access and Alternative Metrics of International GrowthLeslie Chan, University of Toronto Scarborough It is generally acknowledged that researchers and institutions in the Global South suffer from knowledge isolation because of poor infrastructure and lack of access to key resources, including the current literature. The remedy is therefore capacity building and the transfer of not only knowledge, but also the institutional framework of knowledge creation from the North to the South. In this context, Open Access to the scholarly literature is seen as a means of bridging the global knowledge gap. In this presentation, I argue that a key contributor to the continual knowledge divide and the invisibility of knowledge from the Global South is the persistence and dominance of Northern frameworks of research evaluation and quality metrics, coupled with outmoded national and international innovation policies based on exclusion and competitiveness. These narrow measures have tended to skew international research agenda and undermine locally relevant research. A great opportunity that Open Access provides is the means to develop alternative metrics of research uptake and impact that are more inclusive of knowledge from the South, particularly those with development outcomes. In particular, it is important to re-conceptualize and re-design the metrics of research impact to reflect new scholarly practices and the diverse means of engagement enabled by OA and the new wave of social media tools. At the same time, appropriate policies need to be developed to reward open scholarship and to encourage research sharing — issues of particular importance for ending knowledge isolation. Examples of the new kinds of “invisible college” enabled by networking tools and OA will be presented, and particular attention will be paid to innovations emanating from the periphery.
metrics of total publications and citations.Top 15 countries account for 82% of total publicationsAuthor with African institutional affiliation account for less than 1% of global output, and S. Africa has the highest output. The rest are “invisible”Consequence of trying to publish in “International” journal results in neglect of important local problems and solutions that are appropriate for local conditions.
information, research, and publication capacities are intimately linked. Investigators, publishers, editors, and editorial organisations all have important parts to play in solving this global information poverty.Horton R. North and South: bridging the information gap. Lancet 2000; 355: 2331–36.““institutional racism” has a very precise meaning. According to the UK’s Commission for Racial Equality, institutional racism “occurs when the policies and practices of an organisation result in different outcomes for people from different racial groups”. The term, if one accepts that it is appropriate for medical journals, does not mean that individual editors are racist. It does mean that the scientific, medical, and public-health priorities of the rich world are presented as the norm.”
metrics of total publications and citations.Top 15 countries account for 82% of total publicationsAuthor with African institutional affiliation account for less than 1% of global output, and S. Africa has the highest output. The rest are “invisible”Consequence of trying to publish in “International” journal results in neglect of important local problems and solutions that are appropriate for local conditions.
Consequences of publishing in “internatioanlly” indexed journals
he New Invisible College, Caroline Wagner combines quantitative data and extensive interviews to map the emergence of global science networks and trace the dynamics driving their growth. She argues that the shift from big science to global networks creates unprecedented opportunities for developing countries to tap science's potential. Rather than squander resources in vain efforts to mimic the scientific establishments of the twentieth century, developing country governments can leverage networks by creating incentives for top-notch scientists to focus on research that addresses their concerns and by finding ways to tie knowledge to local problem solving. T
The report concludes that web based academic publishing is producing a variety of novel information filters. These allow the researcher to make some sort of limited self-assessment with respect to the response to his/her work. However, this does not mean that these technologies and databases can also legitimately be used in research assessments. For this application, they need to adhere to a far stricter protocol of data quality and indicator reliability and validity. Most new tools do not (yet) comply with these more strict quality criteria.