A brief survey on the different metrics that can be used in the field of assistive wheelchair navigation, including usability questionnaires, clinical scales, task and psych metrics, etc. It also includes obstacle courses and a benchmarking proposal. An invited presentation to ICORR'13 workshop CanWheel (http://www.canwheel.ca/events/upcoming/icorr-2013-workshops/)

1. ICORR 2013 - Seattle TEC2011-29106
Cristina Urdiales
Metrics and benchmarking for
assisted wheelchair navigation
Grupo ISIS (UMA)

2. ICORR 2013 - Seattle TEC2011-29106
Wheelchair metrics
There are no universally accepted metrics/procedures to evaluate navigation
There are benchmarking
procedures in established obstacle
courses
BUT they do not provide
information on everyday
navigation at preferred
environments
for example ...
The Valutazione Funzionale Mielolesi (VFM)
Tufts Assessment of Motor Performance (TAMP)
Wheelchair Skills Test (WST)

3. ICORR 2013 - Seattle TEC2011-29106
Wheelchair metrics
There are well known questionnaires that can be applied to any situation;
e.g.:
● TLX (General): overall workloadl task difficulty, time pressure, performance, mental/sensory
effort, physical effort, frustration level, stress level, fatigue, activity type.
● PIADS (Assistive devices): self-esteem, well-being, quality of life, embarrassment, eagerness
to try new things …
● MPT (Assistive technology): people’s condition, ADL, opinion on technology
● FEW (wheelchair users): ergonomics
● QUEST (wheelchair users): usability, weight or dimensions of the device, service delivery,
repair and servicing ...
… but they are usually subjetive and too general i.e. users give opinions
and they do not need to provide information on their specific condition
QUEST PIADS

4. ICORR 2013 - Seattle TEC2011-29106
Condition metrics
Questionnaires can be (and usually are) complemented with clinical scales to
characterize the user ; e.g.:
● (MMSE) or Folstein (Cognitive): quantitative measure of cognitive status in adults
● GDS (Cognitive): constructed to assess depression and general well-being in the elderly
● Barthel Index (functional): measures how a person’s copes with the activities of daily living
● CIRS (physical): cumulative effect of diverse conditions
… but they only provide information about the state of the user at the moment
the scale was obtained and each covers specifics, not the global picture.
These questionnaires can be complemented with biometrics sensors to obtain
information in real time, but wearables are often not comfortable enough to
carry for prolongued times and only a few provide stable readings during activity
(e.g. pulsioximeters)

5. ICORR 2013 - Seattle TEC2011-29106
Course-related metrics
●Wheelchair propulsion, assessed in terms of period of time, a fixed distance or longest
distance possible.
●Transfer from and to the wheelchair, usually examining the performance in different
transfers.
●Negotiation of kerbs, with kerb’s height ranging from 0.025 to 0.15m, and some requiring
ascending and descending the kerb.
●Ascending slopes, defined in terms of inclination (ranging from 1 to 11 degrees) and
length (ranging from 3 to 21m).
●Traversing tracks, i.e. slalom, figure of eight and obstacle course.
●Sprinting over a fixed distance (ranging from 6.5 to 30m).
●Performing a wheelie.
●Managing brakes, negotiating doors or loading the wheelchair into a car.
Typically measured while the user completes a certain preestablished course.
Usually very specific and objetive.

6. ICORR 2013 - Seattle TEC2011-29106
Path-related metrics
● Deviation with respect to the canonical trajectory (often quantified into a few bins as ok,
acceptable and wrong)
●Success/failure
Typically measured when there is a pre-established visual path to track. Very
objetive. Can be combined with task metrics (see next slide)

7. ICORR 2013 - Seattle TEC2011-29106
Task metrics
●Task success (or) degree of success
●Task completion time
●Number of collisions
●Distance travelled
All quantitative measures that can be objectively related to the user’s performance
with the wheelchair while performing a given task
Global (the task needs to be finished to obtain them)
●Smoothness, related to sharp direction changes
●Safety, related to obstacle proximity
●Directness, related to the skill of keeping directions
Local (can be obtained at any given location)
●Time and Distance to Stop (Ts, Ds)
●Time and Distance to Collision (Tc, Dc)
●Obstacle clearance (distance when an obstacle is avoided)
Dynamic (the task needs to be finished to obtain them)

8. ICORR 2013 - Seattle TEC2011-29106
Psych metrics
Psych metrics aim at (indirectly) evaluating the user’s (mental) state. These
metrics are very dependent on interfaces and ergonomics and, consequently, have
been widely used in those fields.
● Interaction number (with the interface)/ amount of information exchanged between
user
and machine (usability)
● Number of error situations (usability)
● Joystick Variation (more than 10%) (mental workload)
● Intervention level, defined as the portion of time that the user moves a joystick (mental
workload)
● Disagreement, defined as the angle between the user command and the wheelchair
motor command (frustration, stress)
● Inconsistency, defined as the variation of the user output when facing similar situations
(cognitive skills)

9. ICORR 2013 - Seattle TEC2011-29106
Benchmarking
●It consists of trying to find how a generic user would respond to every faced
situation
•Generic users do not exist, but they can be modelled from a large enough
number of subjects via data mining
•If situations are defined at high level, there are infite variations. If they are
defined at local level, there are only a few1
1
J. Minguez, J. Osuna, and L. Montanor, “A divide and conquer strategy based on situations to achieve
reactive collision avoidance in troublesome scenarios,” IEEE Trans. on Robotics, 2009.

10. ICORR 2013 - Seattle TEC2011-29106
Data mining
Given a large enough sample space, we split it into classes where users
respond similarly to similar situation. Their prototypes are equal to the
average of all samples within the bin weighted by their respective efficiency,
i.e.the best solution taken by the largest number of users

11. ICORR 2013 - Seattle TEC2011-29106
User profiling and benchmarking
Using the proposed prototype user we can ...
Subject 14 (Guillain-Barre,
MMSE=23, Barthel=28)
… see what skills any given user
lacks with respect to the average
user
… see what efficiency a given user
has in a task in comparison with the
average user
Subject 813 (rep. Brain stroke,
MMSE=20.3, Barthel=29)

12. ICORR 2013 - Seattle TEC2011-29106
Conclusions and future work
(Standard) Benchmarking procedures and metrics are of key importance in
wheelchair navigation
Typically we need to combine usability questionnaires, clinical indexes and
several metrics to see the bigger picture
Profiling helps greatly the analysis of results through personalization
Can we use real time metrics to (indirectly predict potential health issues (alarms)?
Can we use profiles for better personalization and adaptation of assistance?

13. ICORR 2013 - Seattle TEC2011-29106
Cristina Urdiales
Grupo ISIS (UMA)
Metrics and benchmarking for
assisted wheelchair navigation