Talk given at: WCC2010 in Brisbane (KCKS2010), Title: "Informatics Systems and Modeling- Case Studies of Expert Interviews"

1. Informatics Systems and Modeling
- Case Studies of Expert Interviews
Johannes Magenheim (presenter)
Leopold Lehner, Wolfgang Nelles,
Thomas Rhode, Niclas Schaper,
Sigrid Schubert and Peer Stechert
University of Paderborn
Computer Science Education Group
University of Siegen
Department of Computer Science and E-Learning
1

2. Outline
Informatics Systems and Modeling - Case Studies of Expert Interviews
• Theoretically derived Competence Model
• Objectives and Research Methodology
• Achieved research Results:
- Empirically refined Competence Model
- Differences in Experts Views on Scenarios and Competence Components
• Further Research Tasks
SES PS JSM TR LL NIS WN
Sigrid Schubert Johannes Magenheim Niclas Schaper
Peer Stechert Leopold Lehner Wolfgang Nelles
Thomas Rhode
Electrical Engineering & Informatics Informatics, CSE Organizational Psychology
University of Siegen University of Paderborn University of Paderborn
2
Johannes Magenheim, University of Paderborn

3. Theoretical Relations
Modelling,
System
Comprehension System
Competences Properties
Informatics
System
CS
Curricula
System
System Development
Application
3
Johannes Magenheim, University of Paderborn

4. Theoretically derived Competence Model
4
Johannes Magenheim, University of Paderborn

5. CSE: Objectives and Research Methodology
1a. Traditional: 1b. New since 2006: 5. Improving learning environments
System Development System Comprehension since 2011
Evaluation of learning
Analysis of international
environments by
syllabi and curricula
competence measurement
Development of competence stimuli
(authentic and complex)
2. Theoretically derived 4. Instruments to
competence model measure competence
4/2008 4/2010
30 expert interviews
(Critical Incident Technique) Development of test items 6. Competence
and observation of learner- Level Model
centred approaches
2011
Qualitative content analysis
(meaning units)
3. Empirically refined 7. Competence
Expert Rating competence Model Development Model
2/2010 2012
5
Johannes Magenheim, University of Paderborn

6. Research Methodology
• 30 Expert Interviews
• 3 Groups of Experts
- Experts of Informatics
- Experts of Didactics of Informatics
- Expert Informatics Teachers
• Interviews on Use Cases
(Critical Incident Technique)
• Content Analysis (Mayring)
6
Johannes Magenheim, University of Paderborn

7. Prof. Dr. Johannes Magenheim
University of Paderborn – Computer Science Education Group
2 Examples for Use Cases (Scenarios)
Two complex hypothetic scenarios were content analyzed:
(1) “Merchandise Management System” which especially deals
with system development requirements and
(2) “Testing of Unknown Software” which deals with
system comprehension requirements in particular
7
Johannes Magenheim, University of Paderborn

8. Scenario “Merchandise Management System”
Scenario “Merchandise Management System”: “You are asked to develop a
software based merchandise management system for a small school kiosk.”
Question 1: “What is your course of action to solve this task? Which
software engineering workflows do you have to process?”
Question 2: “Which graphical models would you apply?”
Question 2.1: “Which informatics views are important for this task?”
Question 2.2: “Which complexity would you assign to this task?”
Question 3: “Which cognitive skills are required to develop such a software
system?”
Question 4: “Could you imagine a potential pupil’s procedure to solve this
problem?”
Question 5: “Which attitudes, social communicative skills and motivational
aspects are necessary to solve this problem?”
8
Johannes Magenheim, University of Paderborn

9. Empirically refined Competence Model
9
Johannes Magenheim, University of Paderborn

10. Example K4 Non-Cognitive Skills
Theoretically derived competence model Empirically refined competence model
10
Johannes Magenheim, University of Paderborn

11. Further Research Questions
 In which respect do the experts differ in their
competence-relevant statements?
 How can these different contributions be explained
with reference to different expert perspectives,
backgrounds and attitudes towards the topic?
11
Johannes Magenheim, University of Paderborn

12. Further Outcomes
 Experts of all groups contributed to the refinement of the
competence model and the appropriateness of the theoretically
derived categories of the competence model of informatics
modelling and comprehension were confirmed
 Especially, the relevant competence dimension K1 (BASIC
COMPETENCIES) with its categories K1.2 (SYSTEM COMPREHENSION)
and K1.3 (SYSTEM DEVELOPMENT) and their sub-categories could be
confirmed by the descriptions of the experts.
 Furthermore, the experts´ answers on questions concerning social
competence requirements provided valuable and confirming clues to
the fourth dimension Non-Cognitive Skills.
 The closer the experts´ relationship to school, the more
differentiated the non-cognitive skills are described.
12
Johannes Magenheim, University of Paderborn

13. Further Outcomes
• Furthermore: especially experts of informatics felt uncomfortable
with scenarios, which covered parts of informatics, that were not in
their research field.
• The expert of informatics expressed not a negative but a positive
attitude towards the appropriateness of the scenario for informatics
secondary education – in contrast to the expert of didactics and the
expert teacher, which were more critical concerning the
appropriateness of the scenario
• We have to be careful to generalize that experts of informatics are
more critical concerning the school-appropriateness of informatics
learning contents. Such appraisals might also depend on the
personal experiences or other background characteristics of an
expert
13
Johannes Magenheim, University of Paderborn

14. Further work to do.........
• It is necessary to conduct additional empirical research steps to
proof the content and criteria validity of the developed competence
model: The evaluation of the content validity of the model should be
accomplished by an expert rating.
• The different informatics experts have to rate the extracted
competence descriptions concerning their relevance, difficulty,
representativeness and degree of differentiation.
• The evaluation of the criteria validity of the competence model
should be accomplished by developing instruments to measure the
different facets of the competence model and the criteria behaviour
• The resulting correlations between both can be interpreted as
indicators for criteria validity of the competence model.
14
Johannes Magenheim, University of Paderborn

15. Further work to do.......
1a. Traditional: 1b. New since 2006: 5. Improving learning environments
System Development System Comprehension since 2011
Evaluation of learning
Analysis of international
environments by
syllabi and curricula
competence measurement
Development of competence stimuli
(authentic and complex)
2. Theoretically derived 4. Instruments to
competence model measure competence
4/2008 4/2010
30 expert interviews
(Critical Incident Technique) Development of test items 6. Competence
and observation of learner- Level Model
centred approaches
2011
Qualitative content analysis
(meaning units)
3. Empirically refined 7. Competence
Expert Rating competence Model Development Model
2/2010 2012
15
Johannes Magenheim, University of Paderborn

16. Thank you
Prof. Dr. J. Magenheim
University of Paderborn
Computer Science Education Group
Fuerstenallee 11
33102 Paderborn (Germany)
jsm@uni-paderborn.de http://ddi.upb.de
16