Presentation made on October 21st at ICDIM 2015 Conference, Jeju, South Korea

1. COMPARISON OF TECHNIQUES FOR
MEASURING RESEARCH COVERAGE OF
SCIENTIFIC PAPERS: A CASE STUDY
Aravind Sesagiri Raamkumar, Schubert Foo & Natalie Pang
Wee Kim Wee School of Communcation and Information
Nanyang Technological University, Singapore
Presentation for ICDIM’15
October 21 2015

2. What are we concerned about?
“What is the best technique for measuring research coverage of scientific papers in
order to build initital reading lists for literature review?”
Why is this needed? => To help rookie researchers
=> To help researchers venturing into new resesearch areas
What type of papers? => Papers that provide a good overview and lead to other
important papers
Relative Importance of papers based on? => Diversity
=> Recency
=> Survey papers
=> Popularity
Recommendation Task

3. RELATED WORK
 Traditional methods for ranking research papers
• Citation Count => Leads to Matthew Effect [1]
• Search Keywords Matching => Just a simple search
 Graph Ranking Techniques
• PageRank [2]
• HITS [3]
• Customized Random Walk with Restart [8]
• Hybrid versions (incl. textual [4] and temporal data [5])

4. RELATED WORK
 Recommender System Techniques
• Generic Collaborative Filtering [6]
• Collaborative Filtering with HITS, PageRank and SALSA [7]
So What’s Missing?
• Computation based mainly on shared co-citations and co-references
• Not applicable for interdisciplinary topics
• Emphasis only on paper seminality and popularity

5. STATE-OF-THE-ART TECHNIQUE
 HITS Approach
• Computation based on in-links and out-links of graph nodes
• Provides better results than PageRank [7]
 Where has HITS been used?
• Generic digital library context
Supervised HITS with important documents as training set [9]
• Scientific paper recommender systems context
Precomputed hub score applied as normalized unit vector over
collaborative filtering [7]
 Representation of HITS in Current Domain
• Hub node cites other papers
• Authority node is cited by other papers

6. PROPOSED NEW TECHNIQUES
1. Topical and Peripheral Coverage (TPC)
2. Topical Coverage (TC)
 Key Differentiating Factor
• Usage of Author-specified keywords (AsK) metadata in research
papers
Authors by right have to include keywords in papers
Direct reference to the key themes represented in the research
paper
Applicability in identifying interdisciplinary papers
Availability of both narrow and broad topics

7. TOPICAL AND PERIPHERAL COVERAGE
(TPC)
 Key Differentiator
• Usage of all keywords provided in the research paper
 Measurement Steps
1. Identify the keywords K provided for a paper Pi
2. Extract all the papers in the corpus which have the keywords in K. The
extracted set of papers becomes the base set Pk
3. Extract the references list reflisti and citations list citelisti of Pi
4. Coverage score is measured by counting the number of papers from
reflisti and citelisti that are present in Pk
5. Altenatively, score can be also calculated as ratio score by dividing the
combined count of reflisti and citelisti by count of papers in Pk

8. TOPICAL COVERAGE (TC)
 Key Differentiator
• Usage of specific keyword(s) (pertaining to a research topic) provided in the
research paper
 Measurement Steps
1. Identify a research topic T
2. Extract all the papers in the corpus which have the keyword T. The extracted
set of papers becomes the base set PT
3. Extract the references list reflisti and citations list citelisti of Pi which have T
as keyword
4. Coverage score is measured by counting the number of papers from reflisti
and citelisti that are present in PT
5. Altenatively, score can be also calculated as ratio score by dividing the
combined count of reflist and citelist by count of papers in PT

9. VISUALIZATION OF NETWORK SPACE

10. EVALUATION CRITERIA FOR RESEARCH
COVERAGE
 Why are criteria needed?
• For usage in the recommendation task ‘building a reading list of
research papers for literature review’
• Expectations on the reading list might vary as per the experience level
of the researcher
• Four Criteria for measuring inclusivity extent of:
Diverse papers
Literature Survey/Review papers
Popular papers
Recent papers

11. INCLUSIVITY CRITERIA
 Diverse Papers Inclusivity
• Rationale: A reading list is supposed to contain papers that cover most
sub-topics of the main research topic
• Diversity calculation based on number of interconnections (edges)
between nodes in the sub-graph (recommendation list of papers)
• G(V,E) -> graph built with references and citations of papers about a
particular topic
• G1(V1,E1) -> subgraph built with just the recommended papers from
the coverage measurement technique
• Number of edges E1 is an indication of level of diversity in the
recommended papers
More edges => less diversity
Less edges => more diversity

12. INCLUSIVITY CRITERIA
 Survey Papers Inclusivity
• Rationale: Literature survey/review papers provide an overview of the
existing research performed in the particular research area
• Measured by counting the number of survey papers in the final
recommendation list
 Popular Papers Inclusivity
• Rationale: Popular papers have seminal status in the particular
research area
• These papers have high coverage since they lead to many other papers
by following citations
• Measured by counting the number of papers in the final recommendation
list above the citation count threshold

13. INCLUSIVITY CRITERIA
 Recent Papers Inclusivity
• Rationale: Recent papers that cite many other important papers should
also be considered as papers with decent level of coverage
• Measured by counting the number of papers in the final recommendation
list above the threshold value
E.g. Papers published in the recent two years

14. EXPERIMENT DETAILS
• Corpus: Extract of papers from ACM DL
• Period: published between 1951 and 2010
• Topic for study: Information Retrieval
• Papers count: 976 papers with 21,243 references and citations
• Software and tools used: JUNG java library, MySQL
• Citation count threshold: 50
• Recent papers threshold: published between 2008 and 2010
• Techniques compared: HITS (HC), TPC and TC
• Measurement: Top 20 papers were shortlisted for each technique based on the
corresponding scores

15. SAMPLE RESULTS
Rank Title (year)
References
Count
Citations
Count
1 Information retrieval on the semantic web (2002) 28 225
2 Methods and metrics for cold-start recommendations (2002) 31 1024
3 Stuff I've seen:a system for personal information retrieval and re-use (2003) 32 813
4 Designing a digital library for young children (2001) 20 156
5 Information retrieval on the web (2000) 231 735
6 A cluster-based resampling method for pseudo-relevance feedback (2008) 35 142
7
A case for interaction: a study of interactive information retrieval behavior and effectiveness
(1996) 13 398
8 Query dependent pseudo-relevance feedback based on wikipedia (2009) 34 135
9
Scatter/gather browsing communicates the topic structure of a very large text collection
(1996) 11 245
10 Answer Garden 2: merging organizational memory with collaborative help (1996) 25 399
11 Using information scent to model user information needs and actions and the Web (2001) 26 401
12 A knowledge-based search engine powered by wikipedia (2007) 21 153
13 Evaluation over thousands of queries (2008) 21 79
14 Sources of evidence for vertical selection (2009) 20 126
15
Interactive textbook and interactive Venn diagram: natural and intuitive interfaces on
augmented desk system (2000) 21 117
16 Categorizing web queries according to geographical locality (2003) 21 171
17 Automatic query generation for patent search (2009) 12 67
18 Probabilistic query expansion using query logs (2002) 18 415
19 Extending average precision to graded relevance judgments (2010) 27 51
20 Learning in a pairwise term-term proximity framework for information retrieval (2009) 19 47

16. RESULTS
Criteria Count HC TPC TC
No. of Edges in Subgraph (Diversity) 14 5 18
No. of Literature Survey Papers 6 1 3
No. of Papers with Citation Count above 100
9 16 16
No. of Recent Papers 3 7 5
Rank HC TPC TC
No. of Edges in Subgraph (Diversity) 2 1 3
No. of Literature Survey Papers 1 3 2
No. of Papers with Citation Count above 100
3 1 1
No. of Recent Papers 3 1 2

17. KEY FINDINGS
 Diverse Papers Inclusivity
• TPC had the least number of edges (n=5) => More diverse set of papers
• Why?
Inclusion of non-IR papers in the list since the computations is
based on all keywords
• TC had the most number of edges (n=18) as most papers had shared
relationships since only IR keyword is considered
 Survey Papers Inclusivity
• HC retrieved most papers (n=6) as its scoring logic gives priority to
papers with more connections
• TC (n=3) retrieved more papers than TPC (n=1) as its scope is related
to only topically-relevant papers

18. KEY FINDINGS
 Popular Papers Inclusivity
• Both TPC and TC performed good (n=16) => Ideal for identifying key
seminal papers as these techniques aggregate both citations and
references
• HC retrieved less such papers (n=9) since it gave preference to papers
with high references count
 Recent Papers Inclusivity
• TPC (n=7) had the most number of recent papers followed by TC (n=5)
and HC (n=3)
• Results need to be validated with more research topics as temporal
preferences were not explicitly factored in the conceptual model

19. CONCLUSION
 Overall, Topical and Peripheral Coverage (TPC) technique provided
best results
Caters for inter-disciplinary topics
Identifies recent papers
Covers a wide array of sub-topics in the main research topic
 Topical Coverage (TC) provides feasibility for implemention in the
task for finding similar papers
 As a key to improve results for survey papers inclusivity, TPC and
HITS (HC) are to be combined
We got encouraging results in a subsequent study during offline
evaluation of our system
Results will be reported in a separate paper

20. CLOSING REMARKS
 The integrated TPC and HC technique to be used for the task of building
initial reading list for literature review
 We cordially invite to participate in the user evaluation experiment of the
Rec4LRW system, please provide your details by clicking the below link
URL: https://goo.gl/AuIasT
THANK YOU