SlideShare a Scribd company logo
1 of 8
Download to read offline
PeerJudge: Praise and
Criticism Detection in
F1000Research reviews
Mike Thelwall
University of Wolverhampton
PeerJudge Overview
• Based on a dictionary of review sentiment terms
and phrases from F1000Research reviews
• Each dictionary term or phrase has a praise or
criticism score
• Well written: +2
• Flawed: -4
• Reviews given the maximum positive and negative
scores of words or phrases found in each sentence.
• -1: no criticism … -5: very strong criticism
• 1: no praise …. 5: very strong praise
• Also 12 linguistic rules to cope with negation,
booster words (very, slightly)
PeerJudge Example
• The paper is well written but the study is poorly
designed.
• Praise: 2; Criticism: -4.
• Try: online
• http://sentistrength.wlv.ac.uk/PeerJudge.html
Part of the
dictionary
acceptabl* 3
accurate 2
adequat* 3
appropriate 3
arbitrary -2
balanced 2
bewilder* -3
but 1
careful* 3
clarify -2
clear 4
clearer -3
compelling 3
Technical details
• Java jar program
• portable
• Dictionaries are external plain text files
• Easily customizable
• Fast
• 14,000 reviews per second
• Explains its judgement
• So is transparent and the owner can adjust the dictionary for
recurrent problems
• Agrees above random chance with reviewer scores
• Because based on a dictionary, does not “cheat” by
identifying hot topics, fields, affiliations or jargon
Where is the dictionary from?
• Human evaluation of a development dataset of
F1000Research reviews
• Machine learning to suggest extra terms and
different weights
Limitations
• Designed for F1000Research decisions – needs
dictionary modification for good performance on other
review datasets.
• F1000Research reviews are unbalanced – few negative
decisions
• F1000Research reviews have standard concluding text
that had to be removed – so referees might not
conclude
• Referees often give judgements in field-specialist
languages, avoiding general conclusions
• More substantial modifications may be needed for
technical domains.
• Difficult to do this in advance because very few outlets
publish reviews and scores
Applications
• Warning reviewers if their judgements are
apparently out of line with their scores?
• Warning reviewers if they have not given any
praise.
• As above for editors
• On a larger scale, allow publishers to check for
anomalies in the reviewer process, such as by
identifying journals with uncritical referees (low
average criticism scores).

More Related Content

Similar to Peer judge: Praise and Criticism Detection in F1000Research reviews

FUSD Rubrics C & I - 5th grade
FUSD Rubrics C & I - 5th gradeFUSD Rubrics C & I - 5th grade
FUSD Rubrics C & I - 5th gradeFUSDTechCoach
 
Klaus-MSKCC-Feb-8-2010.ppt
Klaus-MSKCC-Feb-8-2010.pptKlaus-MSKCC-Feb-8-2010.ppt
Klaus-MSKCC-Feb-8-2010.pptJamesBon18
 
Presentación 1.
 Presentación 1. Presentación 1.
Presentación 1.malufa3
 
Dorothy Faulkner - Thesis & viva student version june2012
Dorothy Faulkner - Thesis & viva student version june2012Dorothy Faulkner - Thesis & viva student version june2012
Dorothy Faulkner - Thesis & viva student version june2012OUmethods
 
Be ch 7 assessing students progress
Be ch 7   assessing students  progressBe ch 7   assessing students  progress
Be ch 7 assessing students progressAbdelaziz Aittaleb
 
eMba ii rm unit-3.2 questionnaire design a
eMba ii rm unit-3.2 questionnaire design aeMba ii rm unit-3.2 questionnaire design a
eMba ii rm unit-3.2 questionnaire design aRai University
 
Hci evaluationa frame work lec 14
Hci evaluationa frame work lec 14Hci evaluationa frame work lec 14
Hci evaluationa frame work lec 14Anwal Mirza
 
Laos Session 7: Developing Quality Assessment Items - Rubrics
Laos Session 7: Developing Quality Assessment Items - RubricsLaos Session 7: Developing Quality Assessment Items - Rubrics
Laos Session 7: Developing Quality Assessment Items - RubricsNEQMAP
 
Language testing and the use of the common european framework of reference fo...
Language testing and the use of the common european framework of reference fo...Language testing and the use of the common european framework of reference fo...
Language testing and the use of the common european framework of reference fo...M B
 
English Proficiency Test
English Proficiency TestEnglish Proficiency Test
English Proficiency TestRoselle Reonal
 
Thesis & viva student version 2013 [compatibility mode]
Thesis & viva student version 2013 [compatibility mode]Thesis & viva student version 2013 [compatibility mode]
Thesis & viva student version 2013 [compatibility mode]VreckaScott
 
Testing a Test: Evaluating Our Assessment Tools
Testing a Test: Evaluating Our Assessment ToolsTesting a Test: Evaluating Our Assessment Tools
Testing a Test: Evaluating Our Assessment ToolsEddy White, Ph.D.
 
Cite It Right! Scoring and Teaching GED Reasoning Through Language Arts Test ...
Cite It Right! Scoring and Teaching GED Reasoning Through Language Arts Test ...Cite It Right! Scoring and Teaching GED Reasoning Through Language Arts Test ...
Cite It Right! Scoring and Teaching GED Reasoning Through Language Arts Test ...Meagen Farrell
 

Similar to Peer judge: Praise and Criticism Detection in F1000Research reviews (20)

BUS 301 Week 6
BUS 301 Week 6BUS 301 Week 6
BUS 301 Week 6
 
FUSD Rubrics C & I - 5th grade
FUSD Rubrics C & I - 5th gradeFUSD Rubrics C & I - 5th grade
FUSD Rubrics C & I - 5th grade
 
Klaus-MSKCC-Feb-8-2010.ppt
Klaus-MSKCC-Feb-8-2010.pptKlaus-MSKCC-Feb-8-2010.ppt
Klaus-MSKCC-Feb-8-2010.ppt
 
Presentación 1.
 Presentación 1. Presentación 1.
Presentación 1.
 
FAVIO
FAVIOFAVIO
FAVIO
 
Dorothy Faulkner - Thesis & viva student version june2012
Dorothy Faulkner - Thesis & viva student version june2012Dorothy Faulkner - Thesis & viva student version june2012
Dorothy Faulkner - Thesis & viva student version june2012
 
Be ch 7 assessing students progress
Be ch 7   assessing students  progressBe ch 7   assessing students  progress
Be ch 7 assessing students progress
 
eMba ii rm unit-3.2 questionnaire design a
eMba ii rm unit-3.2 questionnaire design aeMba ii rm unit-3.2 questionnaire design a
eMba ii rm unit-3.2 questionnaire design a
 
Rubric
Rubric Rubric
Rubric
 
Thesis evaluation criteria
Thesis evaluation criteriaThesis evaluation criteria
Thesis evaluation criteria
 
Workshop 9 issue essay 2014
Workshop 9 issue essay 2014Workshop 9 issue essay 2014
Workshop 9 issue essay 2014
 
Hci evaluationa frame work lec 14
Hci evaluationa frame work lec 14Hci evaluationa frame work lec 14
Hci evaluationa frame work lec 14
 
Laos Session 7: Developing Quality Assessment Items - Rubrics
Laos Session 7: Developing Quality Assessment Items - RubricsLaos Session 7: Developing Quality Assessment Items - Rubrics
Laos Session 7: Developing Quality Assessment Items - Rubrics
 
Language testing and the use of the common european framework of reference fo...
Language testing and the use of the common european framework of reference fo...Language testing and the use of the common european framework of reference fo...
Language testing and the use of the common european framework of reference fo...
 
English Proficiency Test
English Proficiency TestEnglish Proficiency Test
English Proficiency Test
 
Thesis & viva student version 2013 [compatibility mode]
Thesis & viva student version 2013 [compatibility mode]Thesis & viva student version 2013 [compatibility mode]
Thesis & viva student version 2013 [compatibility mode]
 
Question Paper Setting
Question Paper SettingQuestion Paper Setting
Question Paper Setting
 
Testing a Test: Evaluating Our Assessment Tools
Testing a Test: Evaluating Our Assessment ToolsTesting a Test: Evaluating Our Assessment Tools
Testing a Test: Evaluating Our Assessment Tools
 
Scopus Journals
Scopus JournalsScopus Journals
Scopus Journals
 
Cite It Right! Scoring and Teaching GED Reasoning Through Language Arts Test ...
Cite It Right! Scoring and Teaching GED Reasoning Through Language Arts Test ...Cite It Right! Scoring and Teaching GED Reasoning Through Language Arts Test ...
Cite It Right! Scoring and Teaching GED Reasoning Through Language Arts Test ...
 

More from Verena139

GWAS and DAS
GWAS and DASGWAS and DAS
GWAS and DASVerena139
 
Tracking data
Tracking dataTracking data
Tracking dataVerena139
 
Data availability and feasibility of validation – A genomics case study
Data availability and feasibility of validation – A genomics case studyData availability and feasibility of validation – A genomics case study
Data availability and feasibility of validation – A genomics case studyVerena139
 
Metrics for oa monographs - introduction
Metrics for oa monographs - introductionMetrics for oa monographs - introduction
Metrics for oa monographs - introductionVerena139
 
Thoughts on metrics for OA monographs
Thoughts on metrics for OA monographsThoughts on metrics for OA monographs
Thoughts on metrics for OA monographsVerena139
 
Operas Metrics Service
Operas Metrics Service Operas Metrics Service
Operas Metrics Service Verena139
 
Reproducibility Analytics Lab
Reproducibility Analytics Lab Reproducibility Analytics Lab
Reproducibility Analytics Lab Verena139
 
Prediction markets
Prediction markets  Prediction markets
Prediction markets Verena139
 
Data availability Study
Data availability Study Data availability Study
Data availability Study Verena139
 
Jisc R&D work in Research Analytics
Jisc R&D work in Research AnalyticsJisc R&D work in Research Analytics
Jisc R&D work in Research AnalyticsVerena139
 
ORCID: Jisc&ARMA final meeting update by Josh Brown
ORCID: Jisc&ARMA final meeting update by Josh BrownORCID: Jisc&ARMA final meeting update by Josh Brown
ORCID: Jisc&ARMA final meeting update by Josh BrownVerena139
 
Orcid implementation in uk 29092014
Orcid implementation in uk 29092014Orcid implementation in uk 29092014
Orcid implementation in uk 29092014Verena139
 
ORCID: Jisc&ARMA progress meeting update by Josh Brown
ORCID: Jisc&ARMA progress meeting update by Josh Brown ORCID: Jisc&ARMA progress meeting update by Josh Brown
ORCID: Jisc&ARMA progress meeting update by Josh Brown Verena139
 
Jisc-ARMA ORCID pilot start-up meeting - presentation by Laure Haak (ORCID)
Jisc-ARMA ORCID pilot start-up meeting - presentation by Laure Haak (ORCID)Jisc-ARMA ORCID pilot start-up meeting - presentation by Laure Haak (ORCID)
Jisc-ARMA ORCID pilot start-up meeting - presentation by Laure Haak (ORCID)Verena139
 
Thunderbolts and lightning outputs
Thunderbolts and lightning outputsThunderbolts and lightning outputs
Thunderbolts and lightning outputsVerena139
 
Weathering the storm outputs
Weathering the storm outputsWeathering the storm outputs
Weathering the storm outputsVerena139
 

More from Verena139 (16)

GWAS and DAS
GWAS and DASGWAS and DAS
GWAS and DAS
 
Tracking data
Tracking dataTracking data
Tracking data
 
Data availability and feasibility of validation – A genomics case study
Data availability and feasibility of validation – A genomics case studyData availability and feasibility of validation – A genomics case study
Data availability and feasibility of validation – A genomics case study
 
Metrics for oa monographs - introduction
Metrics for oa monographs - introductionMetrics for oa monographs - introduction
Metrics for oa monographs - introduction
 
Thoughts on metrics for OA monographs
Thoughts on metrics for OA monographsThoughts on metrics for OA monographs
Thoughts on metrics for OA monographs
 
Operas Metrics Service
Operas Metrics Service Operas Metrics Service
Operas Metrics Service
 
Reproducibility Analytics Lab
Reproducibility Analytics Lab Reproducibility Analytics Lab
Reproducibility Analytics Lab
 
Prediction markets
Prediction markets  Prediction markets
Prediction markets
 
Data availability Study
Data availability Study Data availability Study
Data availability Study
 
Jisc R&D work in Research Analytics
Jisc R&D work in Research AnalyticsJisc R&D work in Research Analytics
Jisc R&D work in Research Analytics
 
ORCID: Jisc&ARMA final meeting update by Josh Brown
ORCID: Jisc&ARMA final meeting update by Josh BrownORCID: Jisc&ARMA final meeting update by Josh Brown
ORCID: Jisc&ARMA final meeting update by Josh Brown
 
Orcid implementation in uk 29092014
Orcid implementation in uk 29092014Orcid implementation in uk 29092014
Orcid implementation in uk 29092014
 
ORCID: Jisc&ARMA progress meeting update by Josh Brown
ORCID: Jisc&ARMA progress meeting update by Josh Brown ORCID: Jisc&ARMA progress meeting update by Josh Brown
ORCID: Jisc&ARMA progress meeting update by Josh Brown
 
Jisc-ARMA ORCID pilot start-up meeting - presentation by Laure Haak (ORCID)
Jisc-ARMA ORCID pilot start-up meeting - presentation by Laure Haak (ORCID)Jisc-ARMA ORCID pilot start-up meeting - presentation by Laure Haak (ORCID)
Jisc-ARMA ORCID pilot start-up meeting - presentation by Laure Haak (ORCID)
 
Thunderbolts and lightning outputs
Thunderbolts and lightning outputsThunderbolts and lightning outputs
Thunderbolts and lightning outputs
 
Weathering the storm outputs
Weathering the storm outputsWeathering the storm outputs
Weathering the storm outputs
 

Recently uploaded

Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityAggregage
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructuresonikadigital1
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Guido X Jansen
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxDwiAyuSitiHartinah
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...PrithaVashisht1
 
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptxCCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptxdhiyaneswaranv1
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?sonikadigital1
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Vladislav Solodkiy
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionajayrajaganeshkayala
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationGiorgio Carbone
 
Optimal Decision Making - Cost Reduction in Logistics
Optimal Decision Making - Cost Reduction in LogisticsOptimal Decision Making - Cost Reduction in Logistics
Optimal Decision Making - Cost Reduction in LogisticsThinkInnovation
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best PracticesDataArchiva
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introductionsanjaymuralee1
 
Rock Songs common codes and conventions.pptx
Rock Songs common codes and conventions.pptxRock Songs common codes and conventions.pptx
Rock Songs common codes and conventions.pptxFinatron037
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxVenkatasubramani13
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerPavel Šabatka
 

Recently uploaded (16)

Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructure
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...
 
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptxCCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual intervention
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - Presentation
 
Optimal Decision Making - Cost Reduction in Logistics
Optimal Decision Making - Cost Reduction in LogisticsOptimal Decision Making - Cost Reduction in Logistics
Optimal Decision Making - Cost Reduction in Logistics
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introduction
 
Rock Songs common codes and conventions.pptx
Rock Songs common codes and conventions.pptxRock Songs common codes and conventions.pptx
Rock Songs common codes and conventions.pptx
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptx
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayer
 

Peer judge: Praise and Criticism Detection in F1000Research reviews

  • 1. PeerJudge: Praise and Criticism Detection in F1000Research reviews Mike Thelwall University of Wolverhampton
  • 2. PeerJudge Overview • Based on a dictionary of review sentiment terms and phrases from F1000Research reviews • Each dictionary term or phrase has a praise or criticism score • Well written: +2 • Flawed: -4 • Reviews given the maximum positive and negative scores of words or phrases found in each sentence. • -1: no criticism … -5: very strong criticism • 1: no praise …. 5: very strong praise • Also 12 linguistic rules to cope with negation, booster words (very, slightly)
  • 3. PeerJudge Example • The paper is well written but the study is poorly designed. • Praise: 2; Criticism: -4. • Try: online • http://sentistrength.wlv.ac.uk/PeerJudge.html
  • 4. Part of the dictionary acceptabl* 3 accurate 2 adequat* 3 appropriate 3 arbitrary -2 balanced 2 bewilder* -3 but 1 careful* 3 clarify -2 clear 4 clearer -3 compelling 3
  • 5. Technical details • Java jar program • portable • Dictionaries are external plain text files • Easily customizable • Fast • 14,000 reviews per second • Explains its judgement • So is transparent and the owner can adjust the dictionary for recurrent problems • Agrees above random chance with reviewer scores • Because based on a dictionary, does not “cheat” by identifying hot topics, fields, affiliations or jargon
  • 6. Where is the dictionary from? • Human evaluation of a development dataset of F1000Research reviews • Machine learning to suggest extra terms and different weights
  • 7. Limitations • Designed for F1000Research decisions – needs dictionary modification for good performance on other review datasets. • F1000Research reviews are unbalanced – few negative decisions • F1000Research reviews have standard concluding text that had to be removed – so referees might not conclude • Referees often give judgements in field-specialist languages, avoiding general conclusions • More substantial modifications may be needed for technical domains. • Difficult to do this in advance because very few outlets publish reviews and scores
  • 8. Applications • Warning reviewers if their judgements are apparently out of line with their scores? • Warning reviewers if they have not given any praise. • As above for editors • On a larger scale, allow publishers to check for anomalies in the reviewer process, such as by identifying journals with uncritical referees (low average criticism scores).