It has been suggested that bibliometric analysis of different document types may reveal new aspects of research performance. In medical research a number of study types play different roles in the research process and it has been shown, that the evidence-level of study types is associated with varying citation rates. This study focuses on clinical practice guidelines, which are supposed to gather the highest evidence on a given topic to give the best possible recommendation for practitioners. The quality of clinical practice guidelines, measured using the AGREE score, is compared to the citations given to the references used in these guidelines, as it is hypothesised, that better guidelines are based on higher cited references. AGREE scores are gathered from reviews of clinical practice guidelines on a number of diseases and treatments. Their references are collected from Web of Science and citation counts are normalised using the item-oriented z-score and the PPtop-10% indicators. A positive correlation between both citation indicators and the AGREE score of clinical practice guidelines is found. Some potential confounding factors are identified. While confounding cannot be excluded, results indicate low likelihood for the identified confounders. The results provide a new perspective to and application of citation analysis.

1. ASSOCIATION BETWEEN
QUALITY OF CLINICAL
PRACTICE GUIDELINES
AND CITATIONS GIVEN
TO THEIR REFERENCES
JENS PETER ANDERSEN
MEDICAL LIBRARY, AALBORG UNIVERSITY HOSPITAL
ROYAL SCHOOL OF LIBRARY AND INFORMATION SCIENCE,
UNIVERSITY OF COPENHAGEN

2. ”In some ways bibliometrics is at the stage of European
navigation in the middle ages. The familiar territory is
well, even obsessively, charted but beyond te known
world there are only unknown dragons on the map:”
Lewison, 2002, p. 191

3. SETTING SAIL…
Clinical practice guidelines (CPGs)
as study object:
- Recommendations about best
treatment and diagnosis for
specific diseases.
- Varying quality of CPGs. They are
supposed to be based on the best
research evidence – not all are.

4. LET’S AGREE
The AGREE instrument assesses six groups of variables
pertaining to the quality of CPGs.
The most elaborate group, A3, is labelled ”Rigour of
development” and pertains to the ways in which evidence
was sought, assessed and included in and from the
literature.

5. RESEARCH QUESTION
Is there a connection between the A3-score of CPGs and the
citations given to their references?

6. METHODS
A3 scores were collected from reviews of CPGs containing
AGREE-scores.
CPGs were extracted from Web of Science where possible.
All references from CPGs were extracted.
All citation scores of articles published in the same year and
journal as CPG references were extracted as control group.

7. METHODS II

8. MATERIALS
CPG Reviews
14
CPGs
80
References
5,970
Control group
672,819

9. MATERIALS II
A1 A2 A3 A4 A5 A6 Acum
0.00.20.40.60.81.0
AGREE categories
AGREEscores
Included Excluded
0.00.20.40.60.81.0
Guideline category
A3score

10. RESULTS – CITATION
DISTRIBUTION
1 5 10 50 100 500 5000
110100100010000
log(rank)
log(citationcount)
-5 0 5 10
0.00.20.40.60.81.0 Standard deviations from mean
Density

11. RESULTS II – MAIN
FINDINGS
0.0 0.2 0.4 0.6 0.8
-1.0-0.50.00.51.01.52.0
A3
Z
cor = 0.471, r^2 = 0.222
0.0 0.2 0.4 0.6 0.8
0.00.20.40.60.81.0
A3
PPtop-10%
cor = 0.457, r^2 = 0.209

12. RESULTS III –
CONFOUNDERS?
0 100 200 300 400 500 600
-1.0-0.50.00.51.01.52.0
Number of references
Z
cor = 0.0305, r^2 = 0.000933
0 20 40 60 80
-1.0-0.50.00.51.01.52.0
citations / year
Z
cor = 0.239, r^2 = 0.0571
0 20 40 60 80
0.00.20.40.60.8
citations / year
A3
cor = 0.371, r^2 = 0.00036
0 100 200 300 400 500 600
0.00.20.40.60.81.0
Number of references
PPtop-10%
cor = 0.019, r^2 = 0.0429
0 20 40 60 80
0.00.20.40.60.81.0
citations / year
PPtop-10%
cor = 0.207, r^2 = 0.137
0 20 40 60 80
0.20.30.40.50.60.70.8
citations / year
Acum
cor = 0.317, r^2 = 0.101

13. RESULTS IV
A medium-strength correlation between A3 score and
citedness of references.
No apparent confounding from reference list length or
citedness of CPGs.
More data required.
Results are indicative of connections between citations,
clinical evidence and health impact – but there is no evidence
of causative mechanisms here.

14. DISCUSSION
Can references from other document types than journal
articles broaden the impact concept?
If a study is cited by a CPG, is this a clinical impact, or policy
impact – different from academic citation impact?
CPGs as mega-citations in specific contexts?

15. THANK YOU
FOR YOUR
ATTENTION