Basic Engineering Design (Part 6): Test and Evaluate

The Engineering
Design Cycle
Part 6:
Test & Evaluate
http://labs.ee.washington.edu/community/BasicEngDes/BasicEngDes_TestEvaluate/

Basic Testing & Evaluation:
• Almost always includes
some redesign and
reconstructing!
• Involves multiple steps from
component level to system
level and relies on the block
diagram to be logical and
convincing.
The Engineering Design Cycle
Identify the
Problem
Construct
Prototype
Research the
Need
Select the
Best Solution
Design
Solutions
Redesign/
Iterate
Test and
Evaluate
Present the
Solution

Basic Testing & Evaluation
Verifying Functionality
Test &
Evaluate:
Verifying
Component
Functionality
in the
Engineering
Design
Cycle
Testing a design can be done in many different ways and the exact
testing strategy, amount of data, and analysis of that data required to
verify the goodness and functionality of the design varies widely
with the type and application of the design.
Where does
testing begin?
Image Source: http://www.nanork.org/

Starting from the Block Diagram
Every design, whether a sensing system or otherwise, can and
should be expressed in a block diagram. This block diagram outlines
the major components of the system and how they are
interconnected as well as the inputs and outputs to the system.
Test &
Evaluate:
Verifying
Component
Functionality
in the
Engineering
Design
Cycle

Starting from the Block Diagram
Every design, whether a sensing system or otherwise, can and
should be expressed in a block diagram. This block diagram outlines
the major components of the system and how they are
interconnected as well as the inputs and outputs to the system.
This system has a single type of sensor,
a 3-axis magnetometer used
in two locations (on both wrists)
Test &
Evaluate:
Verifying
Component
Functionality
in the
Engineering
Design
Cycle

In this example, the three axis magnetometer (and the magnet that
delivers a signal to the magnetometer) is the most important part
of the design. It’s functionality is critical to the proper operation
and the overall success of the design. As a result, it requires much
more extensive testing than supporting components.
The sensor requires extensive
component level testing before
moving on to system level
testing
Testing of Main Components
Test &
Evaluate:
Verifying
Component
Functionality
in the
Engineering
Design
Cycle

Phase 1: Test Component(s) in an
• Isolated and
• Controlled environment
.... To identify inherent limitations of
the component
• Controlled environment:
... To identify inherent limitations on
the component imposed by the
environment
Phase 3: Test Component(s) in
• Realistic environment:
... To verify functionality in the
target environment
• Range of realistic environments
... To generalize the functionality of
the component (and establish its
system limits)
Testing of Main Components
Test &
Evaluate:
Verifying
Component
Functionality
in the
Engineering
Design
Cycle

The First Step is Testing the component in an
Isolated & Controlled environment:
Is the sensor isolated from
extraneous variations in the
environment?
Verifying Component Functionality
Testing Critical Components
No
Restructure the testing
strategy

environment?
Is the input to the sensor
controllable?
Yes
No
strategy

environment?
controlled?
Collect sensor data
(output)
Yes
Yes

Is data normal
(Gaussian)?
environment?
controlled?
Collect sensor data
(output)
Yes
Yes
strategy
No

Is data normal
(Gaussian)?
Calculate precision
and accuracy
Yes
environment?
controlled?
Collect sensor data
(output)
Yes
Yes

Is data normal
(Gaussian)?
Calculate precision
and accuracy
Yes
environment?
controlled?
Collect sensor data
(output)
Yes
Yes
Move on to next input
of interest

NOTE:
Requires a statistical test
to establish that the data
indeed represents a
normal distribution
(= goodness of fit)
Just “eyeballing” the histogram for a particular
dataset is useful and gives some idea as to
whether the data are normal, but is not in and
of itself, conclusive!
Is data normal
(Gaussian)?

Requires a statistical test
to establish that the data
indeed represents a
normal distribution
(= goodness of fit)
Common mathematical tests for
univariate normality include:
• D'Agostino's K-squared test,
• Jarque–Bera test*
• Anderson–Darling test
• Cramér–von Mises criterion
• Lilliefors test for normality*
• Shapiro–Wilk test
• Pearson's chi-squared test
• Shapiro–Francia test
Is data normal
(Gaussian)?
NOTE:

The next step is testing the component in a
Controlled Environment:
Is the sensor in a
typical operating
environment? No
strategy

Is the sensor in a
typical operating
environment?
Is the input to the
sensor controlled?
Yes
No
strategy

Is the sensor in a
typical operating
environment?
Is the input to the
sensor controlled?
Yes
Collect sensor data
(output)
Yes

Is the sensor in a
typical operating
environment?
Is the input to the
sensor controlled?
Yes
Collect sensor data
(output)
Yes
Is data normal
(Gaussian)?
strategy
No

Is the sensor in a
typical operating
environment?
Is the input to the
sensor controlled?
Yes
Collect sensor data
(output)
Yes
Is data normal
(Gaussian)?
Calculate precision
and accuracy
Yes
Move on to next input
of interest

When is Testing in
Isolated and Controlled Environments Done?
Have Single Input, Single Output measurements
been collected and verified in isolated and
controlled environments?
Combine and plot characteristic curve with
error bars
Yes
Can differences between expected and
experimental behavior be explained?
Plot Expected and Experimental Behavior on
same Graph
Yes
Done!

When is Testing in
Isolated and Controlled Environments Done?
Have Single Input, Single Output measurements
been collected and verified in isolated and
controlled environments?
Combine and plot characteristic curve with
error bars
Yes
Can differences between expected and
experimental behavior be explained?
Plot Expected and Experimental Behavior on
same Graph
No
Done!
Do differences impact
functionality?
No

Finally:
Does the critical component perform according
to the design specification under typical
operating conditions?
Verify and Document Functionality
Test sensor under these conditions
Does it meet design specification?
Verify and Document
Functionality
Identify extremes of operating conditions most
likely to detrimentally impact the component.
Yes
Yes
No
No
Troubleshoot
Identify Problem
Return to Select Solution,
or Construct Prototype Phase

The Bluetooth modem should also require minimal testing as a
component in the system. Verifying functionality will require:
• Simulating the type of input to the modem coming from the
microcontroller and going into the microcontroller (both range
and frequency).
• Testing the modem in both controlled (laboratory) environments
and in the application environment.
Testing of Supporting Components
Test &
Evaluate:
Verifying
Component
Functionality
in the
Engineering
Design
Cycle

The Bluetooth modem should also require minimal testing as a
component in the system. Verifying functionality will require:
• Simulating the type of input to the modem coming from the
microcontroller and going into the microcontroller (both range
and frequency).
• Testing the modem in both controlled (laboratory) environments
and in the application environment.
Although the Bluetooth
Modem is also an off-the-
shelf component, it
should also be tested to
verify its functionality.
Test &
Evaluate:
Verifying
Component
Functionality
in the
Engineering
Design
Cycle

The microcontroller should require minimal testing as a component
in the system. It should be tested with known and controlled inputs
of the same range and frequency of the accelerometer to ensure that
the signals are properly converted to digital signals and processed
(as programmed and intended) in the microcontroller itself.Test &
Evaluate:
Verifying
Component
Functionality
in the
Engineering
Design
Cycle

The microcontroller should require minimal testing as a component
in the system. It should be tested with known and controlled inputs
of the same range and frequency of the accelerometer to ensure that
the signals are properly converted to digital signals and processed
(as programmed and intended) in the microcontroller itself.
Although the Microcontroller is an
off-the-shelf component, it
will need to be tested as individual
components to ensure they are working
properly as components in this system.
Test &
Evaluate:
Verifying
Component
Functionality
in the
Engineering
Design
Cycle

Once both critical and
supporting components
are tested and evaluated,
it is time to move forward
to:
System Testing
Identify the
Problem
Construct
Prototype
Research the
Need
Select the
Best Solution
Design
Solutions
Redesign/
Iterate
Test and
Evaluate
Present the
Solution

Test &
Evaluate:
System
Testing
in the
Engineering
Design
Cycle
Once the main components are tested at a component-by-
component level, it is time to move on to main subsystem testing
(multiple main components) that operate together to deliver
system functionality.
For example:
• Identification of quality of beer in a major manufacturing
facility using multiple vapor (gas) sensors – the objective here is
for multiple sensors to collectively differentiate "good" beer
from "bad" beer.
• Risk assessment for carpal tunnel syndrome – the objective
here may be to identify one or both wrists at risk for carpal
tunnel syndrome and thus recommend posture/behavior
changes based on whether one or both wrists are improperly
positioned.
• Multiple axis outputs on accelerometer may be needed to
identify both the orientation and the speed of an object.
Testing of Subsystems & Systems

Testing of Main Subsystems/Systems
Have main component measurements
been collected and verified in isolated
and controlled environments?
Have subsystem measurements been
collected and verified against expected
sensor behavior?
Has a “Typical, Representative,
and Realistic” Environment for
System operation been defined in
the design?
Test System under
“Typical,
Representative,
and Realistic”
conditions
Yes
Yes
Is System Functional?
Yes
A functional system
either predicts the
output of interest within
specified accuracy OR it
can discern patterns of
output with a specified
rate of successful
identification of those
patterns.
Test &
Evaluate:
System
Testing
in the
Engineering
Design
Cycle

The three axes of an
accelerometer may
be used to detect the
orientation of a
tennis racquet (to
which they are
attached) within a
certain # of degrees
of accuracy. Whether
or not precise
detection of
orientation is a valid
system test depends
on the application!

Alternatively, the three
axes of an
accelerometer may be
used to detect whether
or not the tennis
racquet is in the proper
posture for a successful
swing. Orientation
may be relative to a
particular user or
situation; system
testing should reflect
the context of the
application!

Another form of system
testing involves
identifying false positives
and false negatives – the
graph on the right shows
comprehensively how
many inputs were
detected correctly (upper
left and lower right
quadrants) and how
many were detected
incorrectly (upper right
and lower left quadrants)
for a wide range of
people.

With functionality tested,
evaluated, and verified for
critical components,
supporting components,
and the system as a whole,
it’s time to move forward
to:
User Level Testing
Identify the
Problem
Construct
Prototype
Research the
Need
Select the
Best Solution
Design
Solutions
Redesign/
Iterate
Test and
Evaluate
Present the
Solution

User Testing and Evaluation
Once a system or design is thoroughly tested and evaluated to verify its functionality,
it is time to show the design to potential users.
Step 1:
Identify Potential Users:
• Who is most likely to use this design, and of
those people, who can afford the design?
• Where can these potential users be found?
Example:
Air Quality monitoring system for bicycle use
Potential Users:
Bike commuters in a major city
Target 3-5 users for each round of tests!
There is no such thing as too many testers, so test as much as possible.

Step 2:
Present your solution to your users/testers in a representative "problem"
environment – where the user would use the system in "real life“.
An Example:
Carpal Tunnel Monitoring System
Testing Environment:
Using a computer at a desk
Texting on a Smartphone

Step 2:
Another Example:
Eyeglass compliance monitoring
Testing Environment:
Driving a vehicle
Using a computer

Step 2:
Sometimes, a storyboard helps
situate the user in the
environment so that the user test
does not go astray!
A storyboard is a series of
illustrations or images that allows
someone to pre-visualize a
situation. In this case, the
situation would be the context in
which the design tester is using
the design.

Step 3:
Watch what your users do with the design:
• How do they interact with the design?
• Do they use the design properly?
• Where do they get confused?
• What information do they glean from the system? Is it what you intend?
• Do they seem interested in the system? Bored? Confused?
Write down what you observe,
while refraining from interpreting
what you observe until later!

Step 4:
Ask your users to complete a short survey
or series of questions about the design:
• Written, or
• Verbal
Use a combination of close-ended, multiple
choice or Likert-scale questions and open-
ended questions that seek to identify both
the "good" and the "bad" in the design.
Examples of Open Ended Questions:
Would you use this design in your daily life? Why or why not?
What do you like most about this design?
What do you like least about this design?
Examples of Close-Ended Questions

Test, Evaluate, Redesign....
A Prototype must be
tested:
• Not only at a
functional level (do the
components and
system work as
intended?), but also:
• At a user level (will a
potential customer
use it as intended?)
Identify the
Problem
Construct
Prototype
Research the
Need
Select the
Best Solution
Design
Solutions
Redesign/
Iterate
Test and
Evaluate
Present the
Solution

Thanks for joining us as we explored the sixth step of the
Engineering Design Cycle: Test & Evaluate.
Complete Educational Materials (including assignments,
quizzes, and recorded lectures) on the topic of Basic
Engineering Design can be found at:
http://labs.ee.washington.edu/community/BasicEngDes/
The Engineering Design Cycle

Basic Engineering Design (Part 6): Test and Evaluate

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Basic Engineering Design (Part 6): Test and Evaluate

Similar to Basic Engineering Design (Part 6): Test and Evaluate (20)

More from Denise Wilson

More from Denise Wilson (17)

Recently uploaded

Recently uploaded (20)

Basic Engineering Design (Part 6): Test and Evaluate