User Experience (UX) is often cited as one of the fastest growing occupations, creating opportunities across nearly every sector for individuals skilled in the application of user-centered design principles and methods. Many Information and Library Schools have responded to this demand by introducing more UX coursework into their curriculum, but the proliferation of agile software development and lean product design has incentivized organizations to look for experienced individuals for UX roles, even those that are considered entry-level. As a result, aspiring information professionals face a paradoxical situation in which they are required to have UX experience before they can gain UX experience. This article provides an assessment of one institution's efforts to overcome this experience gap by offering opportunities for students to participate in three types of authentic client-facing UX projects. Through surveys of students and clients served over four academic years, we provide a set of lessons learned and recommended best practices for incorporating project-based learning opportunities into UX courses.

1. Assessing the Implementation of
Authentic, Client-Facing Projects
in UX Education:
Insights from Multiple Stakeholders
Craig M. MacDonald, Ph.D.
Associate Professor, Pratt Institute
Lillian Rozaklis, Ph.D.
Assistant Director of Assessment, Fairleigh Dickinson University
ASIS&T 2017 Annual Meeting
November 1, 2017

2. The UX field has grown
Source: https://blog.validately.com/thank-steve-jobs-rise-ux-designer/

3. The UX field has grown
and is still growing

4. Graduate education is one pathway
Source: Nielsen/Norman Group, User Experience Careers Survey, 2015

5. Degrees in HCI, LIS, and Info Design
Source: Nielsen/Norman Group, User Experience Careers Survey, 2015

6. Great news! Right?

8. The User Experience Experience
Paradox:
You need to have experience before
you can gain experience.

9. How can educators integrate
practical, real world projects into
UX/HCI courses?

10. In this presentation:
(1) Summarize previous research about bridging
the classroom/real-world gap in HCI
(2) Describe our efforts to address experience
paradox using three types of authentic UX
projects at Pratt Institute
(3) Present results of an assessment of these
efforts from the perspective of three
stakeholder groups: students, clients, and
faculty

11. Previous Research
(Spoiler: this is not new)

12. Many ways to make HCI class “real”
Simulating the “studio design critique” in a
classroom setting (Hundhausen, Fairbrother, &
Petre, 2012)
Teaching students how to overcome presentation
challenges using “The Panda Hat of Doom”
(Hunsucker et al., 2016)

13. Project-led Problem-based Learning
HCI educators have incorporated client-facing
projects into their courses as a type of service
learning (e.g., Schneiderman et al., 2016; Lazar, 2011;
Lassere, 2011; Ritter, 2014)
Many helpful benefits have been cited:
• Help students understand trade-offs of real design
problems
• Give students extra motivation
• Provide students with strong portfolio pieces
• Prevent plagiarism

14. Most of the literature focused
on undergraduate students
in computer science
programs
Most studies were limited to
a single type of project in a
single course
Our study focuses on
graduate students in a
library & information
science program
Our study assesses
implementation of multiple
types of client-facing
projects
PLUS: We take a program-
evaluation approach and
consider perspectives of
multiple stakeholders
(students, clients, faculty)
Past Research | Our Research

15. Our Efforts
Three types of projects

16. Course-embedded projects
Projects completed to satisfy a portion of
students’ overall course grade
Course: “Usability Theory & Practice”
6-8 week assignment: design and conduct a
usability study on behalf of an outside client in
teams of 3-5 students
Overall: 17 projects with 61 students
(once per semester since spring 2014)
• Modified in spring 2016 to add more client-student
interaction

17. Partnership with Columbia
University and NYC Mayor’s
Office
Key Question: how usable is the
NYC Open Data portal for the
average NYC resident?
4 students
8-week usability research project
(March-May 2016)
Designed usability study with 16
participants (8 remote; 8 in-
person)
Identified major usability problems
and proposed solutions
Results delivered in a formal report
and live presentation to client
representative
Vignette: NYC Open Data Portal

18. Course-long projects
Projects that lasted the entire duration of a course
and determined the majority of students’ grade
Course: “Projects in Information Experience Design”
Main goal: offer students practical, hands-on
experience applying user-centered methods to help
an outside client
Guided by “design thinking” framework
Overall: 2 projects with 18 students
(9 students per course)

19. Partnership with the Intrepid Sea, Air,
and Space Museum
Key Question: how can Intrepid
improve the museum experience for
international visitors?
9 students + professor
5-week research + design project
(May-June 2016)
Interviews, observations, stakeholder
meetings, competitive analysis, social
media analysis, wayfinding audit,
usability evaluation
Used insights to craft personas + design
proposals for (a) new wayfinding
system, (b) new ticketing kiosk and
website, and (c) guidelines for maps
and other information materials
Results presented on-site to museum
leadership team
Vignette: The Intrepid Experience

20. Extracurricular projects
Projects completed outside of a course
Designed to satisfy the needs of an individual client
rather than serve specific educational objectives
Ranged from 8-12 weeks long
Involved teams of 3-8 students
Incorporated variety of methods
Overall: 6 projects with 24 students
(3 paid projects; 3 pro bono)

21. Partnership with ESPN; Funded by
NYC Media Lab
Key Question: why don’t users
watch videos on ESPN’s mobile
website?
4 students + faculty lead
8-week research + design project
(Oct-Dec 2015)
Log analysis, interviews, diary study,
survey, competitive analysis
Used insights to design high-fidelity
prototype of new ESPN mobile
website that highlighted mobile
videos
Results presented on-site to cross-
departmental audience
Vignette: The Press Play Project
http://nycmedialab.org/projects/espn-press-play

22. Assessment
Students, clients, and faculty perspectives

23. Methodology: Students/Alumni
Questionnaire
Perceived levels of knowledge/behavior/interest before
and after participating in a client-facing UX project
Overall satisfaction with the project experience
Most important thing they learned (open-ended)
Overall value of participating in client-facing projects
Sample: 59 of 87 respondents (68.6%)
44 of 61 (72.1%) course-embedded projects
16 of 18 (88.9%) course-long projects
12 of 24 (50.0%) extracurricular projects
Reported working with 23 of 25 possible clients

24. Retrospective pretest
Compared to
traditional
pretest/posttest:
-more convenient
-more accurate
-reduces possibility of
response-shift bias

25. Retrospective pretest: results
Note: Wilcoxon signed-rank test (non-parametric alternative to t-test)

26. Student satisfaction: results

27. Putting learning into practice

28. Managing messiness

29. Student value: results

30. Methodology: Clients
Questionnaire
Whether student work met expectations
Level of involvement
Overall quality of student work
Explain how (or if) their organization benefitted from
participating in the project
Indicate whether they would participate again
Sample: 14 of 23 respondents (63.6%)
9 of 15 course-embedded projects
2 of 2 course-long projects
3 of 5 extracurricular projects

31. Client expectations: results

32. Client involvement: results

33. Client satisfaction: results

34. Client impact: results
12 of 14 respondents said their organization had
either already used or had concrete plans to use
some ideas or solutions provided by the
students

35. Client benefits: results
Improved quality of their product or service
Helped them set strategic priorities
Seeing the UX process modeled by students

36. Client willingness to return: results
All 14 clients said they would participate again
It was enjoyable/fun
It helps them do UX projects more regularly
Gives them fresh eyes on a problem
Requested changes:
More feedback/interaction with students
More opportunities to challenge students’ ideas

37. Methodology: Instructor
Critical self-reflections and observations
Extend lessons learned from previous research
Sample: Me

38. Multiple project types
Extracurricular projects are the closest simulation
to the “agency model” of UX consulting
But, finding, organizing, and leading these types of
projects is incredibly time-consuming, especially
during the academic year
The results of this study show that other models
that tie projects to course-related educational
outcomes are also very effective.

39. Client recruitment strategies
Establish relationships with local groups
Target outreach to specific communities (start-
ups, non-profits, libraries, museums, etc.)
Use past clients and students as references
Document and publicize completed projects
(Ritter, 2014)

40. Setting project expectations is key
Develop a client orientation protocol that:
Sets overall project scope and expectations
Pre-schedules client-student interactions
Defines project milestones/timelines
Helpful tip: create a simple contract or MOU that
can be re-used each semester (Lazar, 2011)

41. Dual role: teacher/project manager
Merges “sage on the stage” with “guide on the
side” educator roles (Schneiderman, 1998)
This is a learned skill (Ritter, 2014)
Can scaffold projects to ensure students are
prepared to succeed (Lasserre, 2011; Lazar,
2011)
Schedule course-embedded projects later in the
semester
Establish pre-requisites for course-long projects
Recruit qualified students for extracurricular projects

42. Summary & Conclusions

43. Overall Results
The overwhelming majority of students/alumni
and clients found their project experience to be
positive and impactful.
Offered insights from the instructor’s perspective
for developing a sustainable model for client-
facing UX projects.

44. Future Work: More contexts
Just one institution’s perspective; results may not
be entirely applicable to other contexts
A single graduate-level LIS program, located in a major
urban center, with courses offered 100% face-to-face
Is there a more universal set of guidelines/best
practices that are applicable to other
disciplines, in other course formats, and at
other educational levels?

45. Future Work: Finding balance
Many students wished for more opportunities to
reflect on their experiences and to have more client
interactions.
Clients also wished for more time to work directly
with students and challenge their
ideas/perspectives.
What is the appropriate balance between (1)
providing opportunities for student self-reflection
and self-assessment and (2) enhancing project
“authenticity” by adding more student-client
interaction?

46. Final Thoughts
Client-facing projects offer immense value to all
involved stakeholders:
Students had universally positive experiences
Clients were highly satisfied (and would return)
As an instructor, it was personally and professionally
rewarding
But, continued research in this area is needed.
Are there differences between entry-level UX
professionals with and without client-facing project
experience?

47. Thanks for listening!
Q&A
Craig M. MacDonald, Ph.D.
Associate Professor, Pratt Institute
cmacdona@pratt.edu
www.craigmacdonald.com