With increasing number of science journals and articles how can we build a post-publication evaluation system ?

1. Dealing with Noise How to build a post-publication evaluation system

2. PDF downloads Nature Genetics ( N = 168) for 2005. R = 0.71 From: Turning web traffic into citations Action Potential blog

3. Connotea tag:Evolution November 2006 - Papers in Connotea with tag "evolution” published 2003 or 2004 Early 2008 - Papers in Connotea with tag "evolution” published 2005 or 2006 Journal Articles Citations MBE (2003/2004) 480 6771 Nature (2003/2004) 5193 121932 Science (2003/2004) 5306 124377 Connotea Evolution (2003-2004) 86 3243 Connotea Evolution (2005-2006) 149 5059

4. Bloggers talk about interesting papers There are 519 papers published in 170 journals in the period of 2003-2004 that were mentioned in blogs tracked by Postgenomic.

5. Quotes “ (…)what is obvious to everyone but bone-headed bean counters. The only way to assess the merit of a paper is to ask a selection of experts in the field. “ David Colquhoun Improbable Science “ In order to shortcut the work of looking up actual (real) citation counts for investigators the journal impact factor is used as a surrogate to estimate the count. I have always warned against this use”. Eugene Garfield (1998) “ Any measurement system will produce gamesmanship to increase success. Lawrence yearns for the one system that obviates it: face-to-face conversations between peers that allows a truly deep and well focused estimate of scientific merit. “ Philip Campbell

6. Dealing with Noise Peer-review Nature, Science, Cell Lab notebooks, unpublished manuscripts Journal impact factor Perceived impact

7. Dealing with Noise Perceived impact Peer-review Highly recommended papers by your peers Online lab notebooks Pre-print servers Aggregated usage data

8. Post-publication journals (…) nobody wants to have to wade through a morass of papers of hugely mixed quality, so how will the more interesting papers in such an archive get noticed as such? Philip Campbell

9. Post-publication journals Automatic Algorithms Editors Hubs/Portals