This presentation goes with the paper 'From gut feeling to a structured, summative assessment of design competencies', presented at the 14th International Conference on Engineering and Product Design Education (E&PDE2012).
bibliographic reference:
Schelling, J.A., Leurs, B.L.F., Best, S.E. and Mulder, I., From Gut Feeling to a Structured, Summative Assessment of Design Competencies. In the 14th International Conference on Engineering and Product Design Education (E&PDE2012), Antwerp, 2012.
Mattingly "AI & Prompt Design" - Introduction to Machine Learning"
From gut feeling to a structured, summative assessment of design competencies
1. Research Center Creating 010
Rotterdam University of Applied Sciences
From Gut Feeling To A
Structured, Summative
Assessment of Design
Competencies
Jasper Schelling, Bas Leurs,
Saskia Best and Ingrid Mulder
Antwerp, September 6 2012
Photography by Roland Pastoor
2. We introduce a summative method for
assessing the whole set of students’ design
competencies demonstrated in their final
design project when graduating an interactive
media design course, and reflect on the use
of this assessment method.
3. A final design project is often used to give
students the opportunity to demonstrate
whether they have obtained the right level
of expertise to graduate.
4. The level of expertise that should be acquired
by the student is often formulated as a
competency, a combination of specific
professional capacities:
Attitude
Skills
Knowledge
(Bakerman, 2005)
5. In our own observations, however, we have
seen many assessors struggling with lists of
ambiguous criteria used to assess the students’
competencies; it seemed that experienced
assessors often rely on their intuition or
‘gut feeling’ to grade the students, and are
not stimulated to make their choices explicit.
6. At the same time, recent developments in
the accreditation of higher education in
The Netherlands demand a more transparent
assessment and grading system.
9. The final design project consists of two elements:
Investigating state of the art
theory and knowledge. Provides
Thesis the foundation and rationale for
the design project.
A real life situation at a
company in which the students
Designproject
have to demonstrate their
competencies.
10. The final assessment is a one hour session
that consists of four specific stages:
Presentation Interview Deliberation Feedback
(10 min) (20 min) (20 min) (10 min)
11. The students are graded on eleven criteria,
and are scored on four levels:
Research
State-of-the-art Knowledge Excellent
Thesis
Argumentation
Writing Skills and Presentation
Concept
Good
Visualization
Product Delivery
Satisfactory
Designrationale & Emotionale
‘Human Centered’ Design
Professionalism Multidisciplinary Working Unsatisfactory
& Presentation Presentation & Reflection
12. The assessment tool consists of four instruments:
Calculation Aid
Assessment Form
Question List
Manual
14. -".$%.',#/012#33%/,&.',#/1415%6',3*$,.17*+,8/ Institute for Communication,
Media and Information Technology
:'%$*/'/%3<*"01 =.3*0 7.'*0 study programme:
Communication and
077234 Jeroen Stappenmaecker June 16, 2011 Multimedia Design
()*+,+ ;D.3>6*1#I1&.6&%6.',#/
G*+*."&) 9 : - ; 2#33*/'+0
@A3"B,+C43D"03)($('(+$"(%"*43&,E"'A3",3%3&,*A"FG3%'(+$"(%"H,+G$030"($"*+$'3DB+,&,-"(%%G3%E"A&%"&"
*43&,")+*G%"&$0"(%",343I&$'E"'A3",3%3&,*A"($'+"4('3,&'G,3"(%"0(I3,%3E"'A3"D3'A+0+4+H-"(%"*43&,4-"
()*+,+
If an item has been graded with
03%*,(C30E"'A3",3%3&,*A"A&%"430"'+"&"*+$*3B'G&4"D+034E"'A3",3%3&,*A"FG3%'(+$"(%"&$%J3,30"*43&,4- ‘unsatisfactory’, the student
9 9/+.',+I.&'#"C does not pass!
:'.'*H#IH')*H."'1F/#J6*$8* 9 : - ;
@A3"%'G03$'"A&%"G%30"%'&'36+)6'A36&,'"K$+J430H3"&$0"03D+$%',&'3%"&$"G$03,%'&$0($H"+)"'A3"
0(%*(B4($3L%"C+0-"+)"K$+J430H3E"A&%"J3446)+G$030"+B($(+$%"*+$*3,$($H"('E"&$0"A&%"*&B('&4(%30"+$"
: :.',+I.&'#"C 2 P1QR11S1 3 0
+BB+,'G$('(3%"&$0"',3$0%M - -##$ 2 P1TU11S1 4 0
L"8%3*/'.',#/ 9 : - ;
@A3"&,HGD3$'&'(+$"A&%"&"4+H(*&4"%',G*'G,3"&$0"%A+J%"B,+)+G$0$3%%E"&$0"%'&$0B+($'%"+)"+'A3,%"&,3" ; ;D&*66*/' P1TR11S1 +
,3*+H$(%30E"32B,3%%30"&$0"*,('(*&44-"32&D($30M
K",',/81+F,66+1./$1>"*+*/'.',#/
.',G*'G,3E"%'-43E"%B344($HE")++'$+'3%94('3,&'G,3",3)3,3$*3%E"FG&4('-E"4&-+G'"&$0")($(%A($HM
9 : - ;
7 0 P1V11S 2 8 0 B
!"#$%&'
!"#$%&' If an item has been graded with
2#/&*>' 9 : - ; 9 9/+.',+I.&'#"C
2#33*/'+0 ‘unsatisfactory’, the student
V3e
does not pass!
@A3"*+$*3B'"(%"+,(H($&4"&$0"($$+I&'(I3E"'A3,3"(%"&"*43&,"*+A3,3$*3"C3'J33$"'A3"'A3%(%"&$0"'A3"
B,+0G*'"'A&'"*&$"C3"',&*30"C&*K"'+",3%3&,*A"($'+"4('3,&'G,39,3)3,3$*3%"&$0"03%(H$",3%3&,*AE"&$0"'A3" : :.',+I.&'#"C 2 P1QT11S1 2 4
BF03.GS-BLS.NP-14jan2012
B,+0G*'",3%B+$0%"'+"'A3"J(%A3%"+)"C+'A"'A3"'&,H3'"H,+GB"&$0"'A3"*4(3$'M
M,+%.6,+.',#/ 9 : - ;
- -##$ 2 P1QW11S1 3 2
@A3"*+$*3B'"(%"D&03"N'&$H(C43OE"),+D"&$"($('(&4",+GHA"%K3'*A"'+"&",3)($30"03%(H$M"P+$I($*($H"
I(%G&4(%&'(+$%E"%K3'*A3%"&$0"+'A3,"'3*A$(FG3%"&,3"G%30"'+",3B,3%3$'"&$0"%A&,3"(03&%E"B,+C43D%"&$0"
; ;D&*66*/' 1 P1TU11S1 2 0 +
,3%G4'%M
7*6,?*"C
@A3"B,+*3%%"A&%"430"'+"'A3"*,3&'(+$"+)"&"B,+'+'-B3"+)"&$"($'3,&*'(I3"B,+0G*'"'A&'"A&%"C33$"'3%'30"($"
9 : - ; 7 6 P1V11S 3 0 4 B
B,&*'(*3M"@3%'",3%G4'%"A&I3"430"'+"&0&B'&'(+$%"&$0",3*+DD3$0&'(+$%M
!"#I*++,#/.6,+3141!"*+*/'.',#/
G.',#/.6141;3#',#/.6 9 : - ; If an item has been graded with
@A3"%'G03$'"*&$"%GC%'&$'(&'3"03%(H$"03*(%(+$%E"*43&,4-"03D+$%',&'3"'A3"*+A3,3$*3"C3'J33$" 9 9/+.',+I.&'#"C ‘unsatisfactory’, the student
,3%3&,*A"&$0"B,+0G*'E"&$0",3I(3J"'A3"*+$%3FG3$*3%"+)"B3,%+$&4"03%(H$"03*(%(+$%M does not pass!
NO%3./12*/'*"*$N1$*+,8/ 9 : - ; : :.',+I.&'#"C P1XU11S1
@A3"%'G03$'"%A+J%"&C(4('-"'+"3DB&'A(Q3E"A&%"BG'"A(D"+,"A3,%34)"($"'A3"B+%('(+$"+)"'A3"'&,H3'"H,+GB"
&$0"+'A3,"%'&K3A+403,%E"A&%"&*'(I34-"($I+4I30"'A3"'&,H3'"H,+GB"&$0"+'A3,"%'&K3A+403,%"($"'A3"03%(H$" - -##$ 1 P1VU11S1 4 0
B,+*3%%E"A&%"+C%3,I30"J344"&$0"0,&J$"*+$*4G%(+$%"),+D"'A(%M
; ;D&*66*/' 1 P1RU11S1 5 0 +
!"#I*++,#/.6,+3141!"*+*/'.',#/
5%6',$,+&,>6,/."C1J#"F,/8 9 : - ; 2#33*/'+0 9 0 P1T11S 1 8 0 B
@A3"%'G03$'"(%"&C43"'+"*+44&C+,&'3E"+,H&$(%3E"*+DDG$(*&'3"&$0"B&,'(*(B&'3E"&$0"03D+$%',&'3%"3))+,'"
)+,"&$0"*+44&C+,&'(+$"J('A"*+443&HG3%E"'A3"*4(3$'"&$0"+'A3,"%'&K3A+403,%M
!"*+*/'.',#/141G*I6*&',#/
@A3"B,3%3$'&'(+$"A&%"&"*43&,"%',G*'G,3"&$0"(%"%GBB+,'30"I(%G&44-E"'A3"FG&4('-"+)"'A3"&$%J3,%"(%"
9 : - ; E,/.613."F 7 6 4 / 100 = 7, 6 4
*+$I($*($HE"'A3"%'G03$'"(%"&C43"'+",3)43*'"+$"A(%"+,"A3,"B3,%+$&4"03I34+BD3$'E"A&%"+B3,&'30"($036
B3$03$'4-E"A&%"(03$'()(30"&$0"G'(4(%30"43&,$($H"+BB+,'G$('(3%E"&$0"A&%"J+,K30"+$"B3,%+$&4"03I34+B6
D3$'"&$0"'A3"&*FG(%('(+$"+)"$3J"K$+J430H3"&$0"G$03,%'&$0($HM E,/.613."F1@"#%/$*$A
:%>*"?,+#"1@&).,"A :%>*"?,+#"1@$*'.&)*$A :%>*"?,+#"1@>./CA ;D'*"/.61*D>*"'
8
E,/.613."F
>565=?
Els Karreman Niels de Worp Bernadette Delfgauw
=.3*1B1:,8/.'%"* =.3*1B1:,8/.'%"* =.3*1B1:,8/.'%"* =.3*1B1:,8/.'%"*
!"#"!$%&'(%)&*'+,-"."#".&'(%)&*'+,-"/"#"/++0"1"#"12*3443$'""""""""567"#"!$%&'(%)&*'+,-"89:"#".&'(%)&*'+,-";"#"/++0"<95="#"12*3443$'
15. During the assessments of the final
projects we observed assessor teams
when they deliberated on the assessment
of competencies and discussed the
resulting final grade.
16. After the assessment, the use of the
instruments was evaluated with assessors from
the teaching staff. They were asked to give
feedback on how the instruments supported
them in grading the graduation work and
what their experiences were using this tool.
17. Observations during the deliberation phase
of the final assessment showed that assessors
were less likely to rely on their general
impression of the work and the presentation.
Instead, they were actively discussing the
students’ results on the criteria set forth in
the assessment form.
18. When the assessors finished their discussion
and asked the student to reappear before the
committee to receive the final result, the
committee was better prepared to link
their feedback to the criteria set forth in the
assessment and seemed to be more confident
motivating the resulting final grade.
19. In general, the assessors mentioned that the
discussion during the grading was about the
criteria set forth in the assessment form and
not about the final grade.
20. Although the assessment instrument was
helpful in assessing the design competencies,
some assessors indicated that the final grade
did not match with their 'gut feeling'.
21. A piece of visualization software was written
to visualize the scores of the assessments for
each student. This visualization was created
for each of the 58 students.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38. These visualizations in turn provide a
fertile ground for continuing discussion on
assessment procedures, for individual students
as well as for the curriculum in general.
39. Being able to visually compare the outcomes
of various assessments allows us to gain
insight into the different assessments and
to find patterns as well as points of interest,
which inform further course development.
40. The current study demonstrated that structured
assessment instruments have dual purposes:
Assessing the quality of the
work of the students in a
Hollistic Assessment manner fitting with competence
based education.
Improving the Supporting the explicit motivation
discussion among of the nuances of the specific
design competencies.
assessors
41. A side effect of revealing the tacit expertise
of experienced assessors was that the junior
assessors felt they improved their
understanding of the competencies and
associated criteria and proficiency level.
42. This seems to be beneficial for the students
as well. Usually students complained about
perceived subjectivity of the final results;
interestingly, the current cohort hardly
discussed their final grades.
43. All in all, it can be concluded that structured
assessment instruments, when properly
designed, help assessors to distinguish
between a ‘gut feeling’ and their intuition,
as well show the relative merit of a specific
competency and matching proficiency level.
44. Questions?
Contact Information Jasper Schelling
Research Center Creating 010 j.a.schelling@hr.nl
Rotterdam University of
Bas Leurs
Applied Sciences
b.l.f.leurs@hr.nl
creating010@hr.nl