Presentation at Boston UPA mini-conference

1. MITRE Study 2007
A Multi-Method Study at MITRE
UPA Boston Mini-Conference
28 May 2008
Bryn Dews
Marsha Nunes
© 2001 The MITRE Corporation. All rights reserved.
2
MITRE Background
Three Federally Funded R&D Centers (FFRDCs)

– Air Force (Defense and Intelligence)
– FAA (Aviation System Development)
– IRS (Enterprise Modernization)
Engineering environment

Lots of disciplines/specialties on sponsor projects

Heavy focus on System Engineering Lifecycle

model(s)
© 2003 The MITRE Corporation. All rights reserved.
My Background: BA  HCI
3
Hired as Business Analyst (Compensation &

Benefits) after 9+ years as BA/Manager for HR in
Banking
Team Lead on PeopleSoft Implementation

HRIT Manager

Team Lead for PeopleSoft in IT

Web Applications Project/Team Lead

Re-org opportunity to raise awareness for HCI

2003 – Team official – UCD Team Lead

– 2004 – add to staff
– 2005 – another add to staff
© 2003 The MITRE Corporation. All rights reserved.
1

2. 4
Purpose of 2007 MITRE Study
Investigate and observe how MITRE employees

use Intranet and other Corporate IT tools and
software
Focus Areas:

– Workspace and desktop setup
– Finding information
– Corporate Portal
– Phonebook
– Microsoft Outlook (email management)
– Awareness
Follow-on to Summer 2006 field study

© 2003 The MITRE Corporation. All rights reserved.
5
Methodology
Observations
(n=6)
Interviews (n=36)
Survey (n = 537/1260)
© 2003 The MITRE Corporation. All rights reserved.
6
Our Process: Overview
Create Project Plan

Hire Intern

IT stakeholder interviews (to narrow study

focus)
Structure study parts

Collect data

Summarize data

Present data to stakeholders and larger IT

community
© 2003 The MITRE Corporation. All rights reserved.
2

3. 7
Stakeholder Interviews
Development organizations

Top-Down interviews/meetings

Straw-man study outline

– Prior year study as starting point
– New ideas from usability testing
© 2003 The MITRE Corporation. All rights reserved.
8
Structure Study Parts
Interviews first

– Environment questions/observations
– Structured tasks
– Features used
– Catalog of applications
Observations next

Survey third

– Short and sweet
– Closed questions (corresponding to interviews)
© 2003 The MITRE Corporation. All rights reserved.
9
Collect Data
Interviews – Whole UCD Team

– Cross section (geographic, job level, job type,
organization)
– Template for notes
– Common terminology
– One hour format… loosely followed
Observations – three staff

Survey – reviewed by whole team

– Sr. Mgt. added open response question
© 2003 The MITRE Corporation. All rights reserved.
3

4. 10
Analyze/Summarize Data
Intern dissected quantitative responses in

interviews; put in tables; graphed
Team highlighted qualitative points in

interview/observation transcripts, created sticky-
notes
Affinity diagramming of notes for qualitative data

Survey Pro used to analyze survey data

Manual analysis of survey open response data

Project and Team Leads reviewed/fine-tuned final

report
© 2003 The MITRE Corporation. All rights reserved.
11
Read Out Study Results
Stakeholder organizations

– Sr. Management early
– Some came asking for it!
– Development organizations
– Loop back to other organizations, Chief
Engineers
COI sharing

Mostly small groups -> very interactive

© 2003 The MITRE Corporation. All rights reserved.
12
Lessons Learned
4 hour observations are tough sell

– Try more, shorter sessions next year
Smaller window = less data

Easier to schedule

Less tedious/invasive

Multiple facilitators make it harder to pick up on

trends across all sessions.
– Try more frequent team updates
– Review session notes as they’re written
Analyzing qualitative data in crunch always daunting!

© 2003 The MITRE Corporation. All rights reserved.
4

5. 13
More Lessons Learned
Affinity Diagram

– Worked well for analyzing qualitative data
But…
– Multiple people reading notes, adding to diagram
led to duplication
Streamlined process by reading and highlighting same

printouts
– Consistent session number labels would make it
easier to backtrack
– Good handwriting helps 
Overall well worth effort in terms of data collected

and outreach to MITRE community!
© 2003 The MITRE Corporation. All rights reserved.
14
Backup Slides
© 2003 The MITRE Corporation. All rights reserved.
15
Excerpts of Interview Checklist
PHYSICAL WORKSPACE
 What MITRE handouts & Giveaways are in/on their
workspace (PECE Cube, mugs)?
 Approx. monitor size & # of monitors?
 Screen Resolution?
 Any Assistive Technologies?
 Is SecurId present?
 For Sites, any noticeable work environment
differences?
© 2003 The MITRE Corporation. All rights reserved.
5

6. 16
Excerpts of Interview Checklist
OUTLOOK
 Are they using Tasks/Contacts and/or Notes?
 How many emails are in their inbox?
 How many email folders do they have?
 Are they using Rules to manage their email?
 Are they using flags to manage their email?
© 2003 The MITRE Corporation. All rights reserved.
17
Excerpts of Interview Catalog
FOR EACH SHOW A SCREEN SHOT OF THE APPLICATION OR
COLLECTION AND ASK:
How often do you use it?

Why?

How you get to it?

How did you learn about it?

NOTE: applications/sites included were pilots,
new/recently rolled/updated or additional features
in heavily used applications
© 2003 The MITRE Corporation. All rights reserved.
18
Excerpt of Observation Checklist
Desktop Browser
Monitor Size? Windows maximized?
Screen Resolution? Open/Close new
browser windows?
Windows maximized? Reuse browser
window? # of windows?
Number of apps open? Use Tabs? # of tabs?
Number of desktop Bookmarks? #?
shortcuts? Organization?
© 2003 The MITRE Corporation. All rights reserved.
6

7. 19
Survey Design
20 Questions (no kidding!)

Frequency of use (scale)

Ease of use (scale)

Appeal (scale)

Ever use (yes/no)

Which do you use (checkboxes)

Write in three improvements (short

answer)
Any other suggestions (long response)

© 2003 The MITRE Corporation. All rights reserved.
7