1. Methodological challenges and solutions in
euroFOT
Samantha Jamson
University of Leeds
September 2009
www.eurofot-ip.eu
2. euroFOT Objectives
• Apply a common European
approach for Field Operational
Tests Test Management Centres
• Perform multiple coordinated
tests of Intelligent Vehicle
Systems with ordinary drivers in
real traffic
• Investigate performance, driver
behaviour and user acceptance
• Assess the impacts on safety,
efficiency and the environment,
www.eurofot-ip.eu
3. Functions
• Longitudinal control functions
FCW Forward Collision Warning
ACC Adaptive Cruise Control
SL Speed Limiter
• Lateral control functions
BLIS Blind Spot Information System
LDW Lane Departure Warning
IW Impairment Warning
• Others
CSW Curve Speed Warning
FEA Fuel Efficiency Advisor
SafeHMI Safe Human Machine Interaction
www.eurofot-ip.eu
4. From Research Questions to
Hypotheses #I
• Research Questions (Example)
ACC: Does ACC improve the drivers’
distance keeping behaviour?
• Research questions are the reason for
conducting the study.
• They are often rather general.
• More detail is required to plan the study well
formulate hypotheses …
www.eurofot-ip.eu
5. From Research Questions to
Hypotheses #2
• Hypotheses (Examples)
ACC: The percentage of TTC(min) decreases
by at least 20 per cent.
ACC: The variance in headway in a
following situation decreases.
• Hypotheses must be testable.
• They are specific, usually directional and identify
the Performance Indicator(s) (PI) in question.
www.eurofot-ip.eu
6. From Research Questions to
Hypotheses #3
• From PIs to Sensors
Percentage of TTC(min): Necessary to collect
speed, distance to vehicle ahead, speed of
vehicle ahead.
Variance in headway: Necessary to collect
distance to vehicle ahead.
• PI determine which measures are necessary.
• The measures have to be logged via sensors.
www.eurofot-ip.eu
7. Procedure for euroFOT
• The research question has been asked.
• The research question has to be broken down into
hypotheses (specific, testable, unidirectional). This is a
vital step!!!
• A hypothesis contains at least one PI.
• The PI is looked up in the PI matrix, which informs about
the measures necessary for the PI.
• The necessary measures are looked up in the measures
matrix, which informs about the sensors that can be used
to log the measures.
• (PI & measures matrix started within FESTA)
www.eurofot-ip.eu
8. Defining the experimental framework
• Top‐down procedure, setting
out various
recommendations &
assumptions.
• Face‐to‐face clinics helped us
to brainstorm and use the
expertise of the consortium
• Produce a design that would
enable the collection and
analysis of data that could be
used to produce statistically
reliable results.
www.eurofot-ip.eu
9. Identifying the constraints
i. Budget – the euroFOT workplan and budget are, as in all research projects, the
major deciding factor when developing the experimental procedures.
ii. Vehicles – the types of vehicles will depend largely on the availability of the
functions on a particular model. Higher class vehicles which inevitably have
more functions available appeal to older individuals with more disposable
income than say a younger driver who may favour the lower insurance
premiums and running costs of a smaller vehicle.
iii. Economic climate – reduced vehicle sales. European new passenger car sales
fell 7.8 percent in 2008, the high‐end markets suffering most as drivers may
opt for smaller, cheaper alternatives. This squeezes the available participants.
iv. Drivers – drivers will mostly be self‐selected (with the exception of the truck
drivers who are selected by the fleet manager). Bias will be noted and
compared to the country specific population statistics
v. Multiple functions – the vehicles are models which are manufactured for the
current market. They are, in most cases, not specific “test” vehicles but are
owned or leased by the driver. In this respect the eight functions under
consideration in euroFOT are often sold in bundles – groups of functions.
www.eurofot-ip.eu
10. Power analysis
Power analyses are undertaken to determine the required sample size in
order to make accurate and reliable statistical conclusions, and to
determine how likely the statistical test will be able to detect effects of
a given size in a particular situation.
When planning a FOT three interrelated questions arise regarding the
experimental design:
• how large is the effect of the system under investigation,
• how many cars have to be equipped to find this effect and
• how long do they have to be driven to obtain a sufficient amount of
data.
www.eurofot-ip.eu
11. Power analysis conclusions
i. Monte Carlo simulations have shown that when at least 120
participants are included, with a mileage of 15k per year, even with the
small effect size that can be expected in a FOT, a sufficient power will
be reached.
ii. Including more cars or more unique participants should take
precedence over measuring for longer periods: e.g. measuring for a
year with 60 participants could fail where measuring for half a year
with 120 participants could provide a significant result.
iii. Reduce the variance measured between participants ‐ choose a
homogenous group of drivers for example male drivers between 30‐40
years of age with similar mileage. This would improve the power, but
at the cost of the generalisability (external validity) of the results.
iv. Baseline and treatment phase should be as near to equal as possible
www.eurofot-ip.eu
14. How has this procedure helped
euroFOT?
The approach taken in this workpackage, has, as far as possible given the
identified constraints, enabled the VMCs within euroFOT to develop
experimental procedures that are both robust and unified
With regards to the experimental designs, we have been able to achieve
a coordinated effort in realising a common experimental with only a
couple of deviations
We have also been successful with regards to developing the
questionnaires that provide the subjective data in a unified format for
use in the impact analysis at the end of the project.
www.eurofot-ip.eu
