This document summarizes a research project on developing an augmented reality (AR) based human-machine interface (HMI) for hydraulic excavators. The project aims to identify how AR can augment operator cognition and improve HMI designs. Technical solutions proposed include using heads-up displays and haptic feedback devices. Future work involves usability testing of AR-simulated HMIs and ergonomic analysis of head-up display inspired designs. The presenter learned about fluid power systems, AR, ergonomics and time management through this research experience.

1. Research Project Number
Hydraulic Excavator HMI DESIGN
USING Augmented Reality
Georgia Institute of Technology | Milwaukee School of Engineering | North Carolina A&T State University | Purdue
University
University of Illinois, Urbana-Champaign | University of Minnesota | Vanderbilt University
Presenter: Michael Ekundayo
Advisor: Dr. Eui Park
Graduate Supervisor: Joseph Akyeampong
North Carolina A&T State University
CCEFP REU 2012
June 4 – August 10, 2012

2. Outline
1. Background
2. Research Objective
3. Technical Solutions
4. Implementation
5. Lessons Learned
6. Idea
7. Future Work
8. Q&A
Event 2
Event Date(s)

3. Background – Fluid Power Systems
Hydraulics Pneumatics
Car lift Air
Compressor
Jet engine Wind
wiring turbines
Event 3
Event Date(s)

4. Background – Hydraulic
Excavators
• Work Domains include:
• Construction
(commercial and
residential)
• Mining
• Water & Sewer
• Forestry & Agriculture
Event 4
Event Date(s)

5. Background – Excavator
HMI
Issues
– Ease of Use (Cognitive)
• Less Intuitive
• Information deficient
– Safety (Physical)
• Layout of
Displays/Controls
• Repetitive use of
controls
Event 5
Event Date(s)

6. Background – Virtual vs.
Augmented Reality
Virtual Reality Augmented Reality
• A computer generated, • A combination of the a
interactive, three- real scene and a virtual
dimensional environment scene generated by a
in which a person is computer that augments
immersed. the scene with additional
information.
Event 6
Event Date(s)

7. Background - Augmented Reality
Technologies
An AR system consists of:
– a camera for acquiring video
images of the real world.
– a computer for generating virtual
objects.
– display unit (e.g. monitor/head-
mounted/projector) for merging
the virtual and real scenes.
– a tracking unit (e.g.
mechanical/optical/inertia),
trackers) for obtaining real-time
position/orientation of the user.
– fiducial markers for registering
(or overlaying) virtual objects on
the real world
– interaction devices (e.g.
joystick/mouse/haptic etc.) to
manipulate/navigate the
augmented environment.
Event 7
Event Date(s)

8. Examples of AR
Applications
• SkinVaders is the first
game to use Augmented
Reality to physically bring
a person body into a game.
• Google glasses developed
for A.R. application
Event 8
Event Date(s)

9. Examples of AR
Applications
IKEA uses augment
reality to fit a customer
with a product by imaging
the room with the
products like furniture,
refrigerators, dishwasher
and other products
Event 9
Event Date(s)

10. Research Objective
To develop an excavator HMI
utilizing augmented reality (AR)
technology
Event 10
Event Date(s)

11. Role of AR in Hydraulic
Excavator HMI Design
• The role of AR in hydraulic excavator
HMI design is to identify additional
information that can be used to
augment the operators cognition to
enhance his/her task performance.
• AR design strategies will be used to
recommend improved HMI designs
for hydraulic excavators.
Event 11
Event Date(s)

12. Technical Solutions (Ease of
Use)
Heads Up Display • Originated in jet fighters to provide
flight information to pilots without
requiring them to look away from their
usual viewpoints.
• sought to ease information
overload and to make the
instrumentation less complicated
• Common in aircrafts; being
introduced in cars
Hydraulic excavators will provide
great application platform to ease
the operators information
overload
Event 12
Event Date(s)

13. Approach/Technical
Solutions
• Haptic technology is a technology
Haptic Technology which takes advantage of the sense
of touch by applying forces,
vibrations, or motions to the user.
• Haptic devices (or haptic
interfaces) are mechanical devices
that mediate communication
between the user and the computer
• For hydraulic excavators it will
provide an additional modality
(sensory cue) to facilitate the
operators cognition, thereby
leading to enhanced usability.
Event 13
Event Date(s)

14. Implementation
Event 14
Event Date(s)

15. What I Learned
I have learned about
• Fluid Power Systems
• Hydraulic Excavators
• Augmented Reality
• Heads Up Displays
• Haptic Devices
• Ergonomics
• Time-management
• Readiness: to be prepared for everything
and anything
Event 15
Event Date(s)

16. Idea
• Google Earth has been explored for AR/MR related
applications to give remote viewing of geo-spatial
information (Frohlich et al., 2006) and urban planning
(Phan & Choo, 2010).
• Google Earth can be used with AR to guide the design
and implementation of HMIs for hydraulic excavators.
Question to be Addressed:
– How to connect Google earth with A.R. in real time?
Event 16
Event Date(s)

17. Future Work
• Assist with conducting usability testing
with AR-simulated HMIs
• Assist with performing ergonomics
analysis of HUD-inspired HMI in Jack.
Event 17
Event Date(s)

18. Q&A
•?????????
Event 18
Event Date(s)