Portable and Mobile Systems in Assistive Technology - The Crosswatch Traffic Intersection Analyzer: A Roadmap for the Future - Coughlan, James (s)

1. The Crosswatch Traffic Intersection
Analyzer: a Roadmap for the Future
James Coughlan, Ph.D.

2. Traffic intersections: difficult
for many visually impaired
travelers
How to align oneself
properly to crosswalk?
What is the layout of the
intersection?
How to remain inside
crosswalk while
walking?
When is it safe to cross?
How to avoid moving 2
vehicles?

3. Past work
“Crosswatch” system (Ivanchenko et al.,
ICCHP 2010; etc.) – computer vision
smartphone app that performs the
following functions in real time for visually
impaired user:
•find crosswalk markings (e.g., zebra
stripes), determine user’s orientation
relative to crosswalk “corridor”
•find Walk light and announce if it’s on
3

4. Crosswatch (past work)
Two kinds of misalignment relative to
crosswalk corridor: (1) translated to the
left; (2) direction error.
Crosswatch can calculate both kinds of
misalignment.
4

5. Crosswatch (past work)
How Crosswatch finds crosswalk and
calculates alignment 5

6. Crosswatch (past work)
Crosswalk scene Zoomed-in
region showing
Walk light 6

7. Advantages of
Crosswatch approach
Information provided to user in real time
System requires no new infrastructure
since it analyzes the visual features that
are already there
7

8. Related work
Aranda & Mares, ICCHP 2004: computer
vision for Walk light detection
Zebralocalizer (Ahmetovic, Bernareggi &
Mascetti, 2011): smartphone computer
vision system
Bohonos et al., ICCHP 2008: Bluetooth
beaconing at traffic intersections
8

9. Next steps
Travelers need a broad range of
information about intersections:
•What type of intersection, features in it
•Where are the crosswalk and other
important features
•When is it time to cross?
9

10. “What” information
What is the intersection
layout? Four-way, three-
way, etc.?
Are there stop signs?
Signal lights? Walk light?
Is there a pedestrian
walk button?
Is there a median strip
somewhere in the
crosswalk?
10

11. “Where” information
Where are all the crosswalks? How am I
oriented relative to the one I want?
Where is the pedestrian walk button relative to
where I am standing?
How long is the crosswalk? How far is it to the
median strip?
11

12. “When” information
When is it time to cross, according to Walk light
(or traffic lights if there is no Walk light)?
Note: this is not necessarily the same as when
it is safe to cross!
12

13. How to get What, Where,
When information?
Realization: it is impractical to expect computer
vision to figure out all this information on its
own!
Intersections, and the features in them, are
extremely variable, and not always easy to see
(even for people with normal vision)
13

14. Walk buttons come in
many shapes and sizes
14

15. Find the walk button!
15

16. Median strips
Without high-
resolution depth
information to show
the elevated median
strip, it is hard to
find reliably
16

17. Complicated intersections
Where a visually impaired traveler may
need the most guidance… computer
vision may have the most difficulties!
17

18. Solution: GIS
Use computer vision where it is
necessary:
(b)find user’s location and orientation
relative to intersection features
(b) detect signal lights in real time (e.g.,
Walk light)
Augment with GIS (Geographic
Information Systems) 18

19. How this works
GPS tells you which intersection you’re at
Smartphone compass and accelerometer
indicate direction you’re pointing in
Computer vision analyzes visible features
in scene and determines (x,y) GPS
coordinates (to within about 1 meter)
Use GIS to find information relevant to
current location (look up online, or
download all information for a city or
region onto smartphone) 19

20. How this works (continued)
GIS has data for
each intersection,
including layout of
all crosswalks, list
of important
features and
landmarks and their
locations
20

21. Much to do
Where to find this GIS information?
Some available through municipal/transit
agencies, Google Maps
But much of it may require crowd-sourcing:
volunteers will contribute information about
intersections in their neighborhood to
online GIS database 21

22. Much to do (continued)
Develop Crosswatch algorithms for
estimating high-resolution GPS
coordinates
Appropriate Crosswatch user interface: for
instance, how to help user pan left to right
to capture entire intersection scene
User surveys (what should Crosswatch do
22
and not do?) and testing

23. Conclusion
Crosswatch needs to expand its
functionality to provide extensive “what”,
“where” and “when” information
Use computer vision where it is needed
and rely on GIS to provide other
information (data infrastructure)
Future idea: use GIS to help with route
planning that is tailored to needs of traveler
23

24. Thanks to…
Second author: Dr. Huiying Shen (Smith-
Kettlewell)
Collaborators: Dr. Megan Lawrence
(Smith-Kettlewell), Dr. Volodymyr
Ivanchenko (Amazon Lab 126)
Funding from NIH and NIDRR
24