Dr. Andreas Oskar Kempf, Ute Sondergeld: Indicator-Based Monitoring of an Interdisciplinary Field of Science. The Example of Educational Research - Presentation at IASSIST 2013

1. Indicator-Based Monitoring of an
Interdisciplinary Field of Science
The Example of Educational Research
Dr. Andreas Oskar Kempf, M.A.(LIS)
Leibniz-Institute for the Social Sciences (GESIS)
Ute Sondergeld, M.A., Wiss. Dok.
German Institute for International Educational Research (DIPF)
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research

2. Contents
1. Project Overview
2. Project Context and Objective
 „Empirical Turn“ in Educational Research
 New Governance Model in Research Funding
 Data Sources
3. Research Indicators
4. Findings
4.1 Analysis of Project Data
4.2 Analysis of Publication Data
5. Summary
6. Outlook
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page

3. 1. Project Overview
 Title:
Development and Changing Dynamics of a Heterogeneous
Social Science Field. Using the Example of Educational Research
 Duration:
05/2011 - 04/2014
 Funding:
Leibniz Association (SAW funding measure)
 Project Lead:
German Institute for International Educational Research (DIPF)
 Project cooperation between:
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page 3

4. 2. Project Context and Objective
„Empirical Turn“ in Educational Research
 DFG Research Units / Priority Programmes for Empirical
Education Research (2000/2002)
 Introduction of National Educational Standards (2003)
 Agreement on a National Report on Education (2004)
 Framework Programme for the Promotion of Empirical
Educational Research (BMBF) (2007)
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page

5. Project Context
New Governance Model in Research Funding
Basic and third party funding in higher education institutions in Germany over time
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page 5
Income(inbn€)
Rateofthirdpartyfunding(%)

6. Project Objective
 Analysis of the structure and development of and within educational
research
 Analysis of communication processes within educational research
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page 6
Educational
Research
Research
projects
Indicators
Publication
outputCitations

7. Data Sources
Project Data
 Sources:
- SOFIS (GESIS)
- BMBF Framework Programme
 Coverage: 9139 projects
 Period: 1995-2009 (end of project)
Publications
 Random sample of 270 projects of
the core disciplines: educational
sciences, psychology, sociology;
periods: 1995-97, 2006-08
 Data collection of research output
 Coverage: about 2.000 publications
Analysis of Reception
 Web of Science, Scopus, Google
Scholar
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page 7

8. 3. Types of Research Indicators
Hinze/Glänzel (2012)
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page
Third party funding
Hinze/Glänzel (2012)
based upon Hornbostel
1999: p. 59.

9. 4.1 Analysis of Project Data
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page

10. Cologne, May 28 – 31 May │

11. Research Activity: Core Disciplines Over Time (2/2)
Decline in psychological projects
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page
61,5%
66.4%
12.6% 9.7%
8.2% 10.0%
0%
10%
20%
30%
40%
50%
60%
70%
80%
1995-1997 1998-2000 2001-2003 2004-2006 2007-2009
Share
Period
Educational sciences
Psychology
Sociology
n = 9139

12. Cologne, May 28 – 31 May │

13. Funding Bodies Over Time (2/2)
Results
There, indeed, is an increase in the share of funding by
foundations and the EU. Differences in funding sources
diminish.
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page 13
0%
5%
10%
15%
20%
25%
1995-1997 1998-2000 2001-2003 2004-2006 2007-2009
Share
Period
Fed. states
Fed. gov.
DFG
Foundations
EU
Third Party Funding (n = 5915)
Hypothesis
Increase in the share of funding by foundations and the EU.

14. Research Cooperation Over Time (1/2)
Results
Networking among research institutions increases
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page 14
7.20%
9.40%
13.80%
17.80%
18.60%
0%
5%
10%
15%
20%
1995-1997 1998-2000 2001-2003 2004-2006 2007-2009
Share
Period
Percentage of Cooperation Projects over Time (n = 7260)
Hypothesis
A growth in national and international cooperations

15. Research Cooperation Over Time (2/2)
Results
The share of cooperations is higher in third party
funded than in self-financed projects.
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page
9.90%
17.40%
0.00%
2.00%
4.00%
6.00%
8.00%
10.00%
12.00%
14.00%
16.00%
18.00%
20.00%
Self-Financed Projects Third Party Funded Projects
Share
Funding Type of Projects
Funding Types in Cooperation Projects (n = 7260)
Hypothesis
A higher proportion of cooperation projects in third party
funded than in in-house projects

16. Empirical Research Methods
Results
There is an increase in the use of empirical
methods between the last two time blocks.
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page 16
74.5%
71.5% 69.4%
64.3%
67.1%
25.5% 28.5% 30.6% 35.7%
32.9%
0%
10%
20%
30%
40%
50%
60%
70%
80%
1995-1997 1998-2000 2001-2003 2004-2006 2007-2009
Share
Period
empirical
non-empirical
Use of Research Methods over Time (n = 9139)
Hypothesis
Increase in the use of empirical research methods

17. 4.2 Analysis of Publication Data
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page

18. 4.2 Analysis of Publication Data – First Results
Hinze/Glänzel (2012)
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page
Third party funding
Hinze/Glänzel (2012)
based upon Hornbostel
1999: p. 59.

19. Publication Activity
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page
Results
The number of publications in total declines.
Likewise, discipline-specific differences seem to decline.
Hypothesis
An increase in the number of publications.
412
278
325
327
268
315
1005
920
0
200
400
600
800
1000
1200
1995-1997 2006-2008
Count
Period
Sociology
Educational Sciences
Psychology
In Total
Number of Publications (n = 1925)

20. Discipline-Specific Publication Types (1/3)
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page
Hypothesis
A greater number of journal articles in psychology than in sociology or
in educational sciences.
Results
In psychology there is a preference for publishing in journal
articles. Over time, a significant increase in the number of journal
articles is measurable in educational sciences. In sociology the
number of journal articles decreases significantly.
0.33
0.22
0.23
0.35
0.34 0.39
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
1995-1997 2006-2008
EMM
Period
Sociology
Educational Sciences
Psychology
Estimated Marginal Means (EMM) of Journal Articles
for each Discipline (n = 535)

21. Publication Types (2/3)
Peer-Review Status of Journal Articles
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page 21
Hypothesis
A greater number of publications with peer-review status in psychology
Results
In general psychological projects publish more often in peer-
reviewed journals than the other disciplines do.
Over time we measured a significant decrease in sociological and
a significant increase in educational projects.
45.20%
20.50%
12.30%
31.10%
42.60%
48.30%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
1995-1997 2006-2008
Share
Period
Sociology
Educational Sciences
Psychology
Percentage of Journal Articles with Peer-Review Status (n = 306)

22. Discipline-Specific Publication Types (3/3)
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page 22
0.28
0.33
0.48
0.26
0.3
0.2
0
0.1
0.2
0.3
0.4
0.5
0.6
1995-1997 2006-2008
EMM
Period
Sociology
Educational
Sciences
Psychology
Estimated Marginal Means (EMM) of Monographs for each Discipline (n=558)
Hypothesis
A greater number of monographs in sociology and educational
sciences than in psychology
Results
Monographs are more often published in educational sciences and
in sociology than in psychology. In psychology and in educational
sciences publishing in monographs decreases significantly. In
educational sciences this decrease is most pronounced.

23. Multi-Authorship
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page
Results
Over time, the number of authors in total per publication
grows significantly due to a significant increase in
psychology and in educational sciences.
Hypothesis
An increase in the number of authors per publication
1.8
2.1
1.7
1.7
1.5
2
2.2
2.7
0
0.5
1
1.5
2
2.5
3
1995-1997 2006-2008
Counts
Period
Number of
Authors in Total
Sociology
Educational
Sciences
Psychology
Estimated Marginal Means (EMM) of Authors
per Publication (n = 1925)

24. International Character of Publications
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page
9.90%
14.20%
0.00%
2.00%
4.00%
6.00%
8.00%
10.00%
12.00%
14.00%
16.00%
1995-1997 2006-2008
Share
Period
Percentage of Publications Written in English
in Total (n = 1925)
Hypothesis
Number of publications written in English increases.
Results
The number of publications written in English increases significantly.
Comparing the three core-disciplines, psychological publications have
the highest share of publications written in English.
10.00%
6.40%
20.40%
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
Sociology Educational Sciences Psychology
Share
Discipline
Sociology
Educational
Sciences
Psychology
Discipline-Specific Differences in English Publication
(1995-07 + 2006-08) (n = 1925)

25. 5. Summary
Research projects in educational research are characterized by:
- a growing importance in the Social Sciences,
- an increase in third-party funding,
- a growing diversity in funding sources,
- an increase in networking activities among research institutes,
- a slow increase in the use of empirical research methods since the year
2005.
Publishing in educational research is characterized by:
- an increase in networking of authors,
- an internationalization effect over time,
- disciplinary differences; over time publishing in educational research
seems to align to psychological publication behaviour.
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page 26

26. 6. Outlook
- Further analysis on the interrelation
between research projects and publications,
- Thematic analysis of projects,
- Further analysis of publication data,
- Reception analysis of publications,
- Indicator specification.
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page 27

27. Acknowledgements
Alexander Botte
Ute Sondergeld
Jürgen Wiesenhütter
Dr. Gabriel Schui
Katja Singleton
Veronika Kuhberg-Lasson
Dr. Sybille Hinze
Valerie Aman
Marion Schmidt
Peter Mutschke
Dr. Philipp Mayr
Karima Haddou ou Moussa
Dr. Andreas Oskar Kempf
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page 28

28. Thank you
for your attention!
Contact:
Dr. Andreas Oskar Kempf, M.A.(LIS)
Leibniz-Institute for the Social Sciences (GESIS)
andreas.kempf@gesis.org
Ute Sondergeld, M.A., Wiss. Dok.
German Institute for International Educational Research (DIPF)
sondergeld@dipf.de
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page 29

29. Bibliography
Deutsche Forschungsgemeinschaft (2012) Förderatlas 2012. Kennzahlen zur
öffentlich finanzierten Forschung in Deutschland. Weinheim.
URL: http://www.dfg.de/download/pdf/dfg_im_profil/evaluation_statistik/foerder
atlas/dfg-foerderatlas_2012.pdf (20.05.2013)
Hinze, Sybille/Glänzel, Wolfgang (2012) Scientometric Indicators in Use: An
Overview. Präsentation im Rahmen der European Summer School for
Scientometrics (ESSS), Leven, Juli 2012.
URL: http://www.scientometrics-school.eu/images/2012Hinze.pdf (25.05.2013)
Hornbostel, Stefan (1999): Welche Indikatoren zu welchem Zweck: Input,
Throughput, Output. In: Röbbecke, Martina/Simon, Dagmar (Hrsg.):
Qualitätsförderung durch Evaluation? Ziele, Aufgaben und Verfahren von
Forschungsbewertungen im Wandel. Dokumentation des Workshops vom 20.
und 21. Mai 1999. Wissenschaftszentrum Berlin für Sozialforschung.
URL: http://skylla.wzb.eu/pdf/1999/p99-003.pdf (23.05.2013)
Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page 30