1. The Retinal Sub-surface Imaging
Testbed @ Rensselaer:
3-D Spatial Mapping & Referencing Problems
Faculty: Prof. Badri Roysam (RPI), Prof. Charles V. Stewart (RPI),
Collaborators: Dr. James N. Turner (RPI/Wadsworth), Howard L. Tanenbaum, MD, (Center for Sight)
Graduate Students: Ali Can (RPI), Hong Shen (RPI), Kripa Rajashekhar (RPI)
Undergraduates: Ameesh Makadia (RPI), Jesse Raymond (RPI), DJ Wilsey (RPI)
Goals & Barriers
The goal is to develop spatial mapping and referencing technologies for diverse sub-surface imaging problems. Much
like Global Positioning Satellites (GPS) have enabled novel commercial and military capabilities, these methods will
enable a generation of highly capable “spatially aware” instruments for sub-surface operations. E.g., in the medical
context, this technology can be used to guide surgical tools, monitor treatment dosages, detect and track changes to
tissue, provide safety shutoffs and alarms, and construct virtual environments for surgical planning and training.
To date there are no successfully accepted surgical systems based on real-time computer vision. To break this
barrier, vision systems that can operate accurately, fault tolerantly, and predictably over extended durations in the
context of high scene complexity, varying image quality, and modeling limitations are needed. Notwithstanding these
complexities, the systems must be totally ‘transparent’ and nearly invisible to the users.
Significance
There is a compelling need to reduce the failure rate (≈
50%) in laser retinal surgery. This is the best-available
treatment for the leading causes of blindness affecting
25-30 million in U.S. alone. Broader applications within
ophthalmology include automatic functional mapping of
the retina for glaucoma, change detection, and
automated scoring of clinical trial images. Most sub-
surface images must eventually be related spatially with
surface images for action planning.
Technical Approach
• Integrated instrumentation for diagnosis and
surgery: A multi-spectral imaging system is that can
capture images of multiple layers of the retina. ICG
images from infrared wavelength is used for
diagnosis, while red-free images from visible-
wavelength is used for spatial referencing in real-
time surgery.
• Progressive, Exploratory feature extraction:
Extracts a sequence of partial results that contain
high quality features needed for registration and
referencing without visiting all the pixels.
• Spatial mapping of the curved retina: Robust
hierarchical vision algorithms to mosaic the curved
retina from projections.
• Real-time spatial referencing: Fast indexing
algorithms to identify feature correspondences.
Based on this, the spatial transformation between
the live images that are captured during surgery and
the wide-area retinal map are computed constantly
and in real-time.
Objective Lens
ICG Barrier Filter
(pass > 805nm)
Dioptic
Correction Lens Focusing
Lens
Picture Angle
Lens
QTH
Lamp
Visible-
Spectrum
CCD
Ocular
Lens
Integrating
Sphere
795nm
Excitation
Laser Diode
Laser Line Filter
(795nm, BW 10nm)
Red-Free
Filter
(510-600nm)
Near-
Infrared
CCD
Dichroic
Filter
ON/OFF mirror
Beam Mixer
Real-Time Image Processor
Collimator Lens
Display and
Interface
795nm
Surgical Laser(Diode)
Y-axis Steered,
Dot-Silvered
Glass
Joystick
Servo
> 650nm
< 650nm
Center Stop 1-to-1 Relay
Fiber
Optic
X-axis
Steered Mirror
Model
Eye
x-y stage controlled
by a separate PC.
3-D Rotation
Apparatus for eye model Electric shutter to simulate blinks
Tiltedmirror
with imaging aperture
Figure 1: (Top) Retinal testbed. (Middle) Current setup.
(Bottom) Retinal surface image; Sub-surface image; &
Overlay of the surface image onto the sub-surface image.

2. Relation to NSF ERC
Related spatial mapping and referencing problems occur in all sub-surface imaging
problems. They are particularly relevant when it is desired to image a much larger
region of space than can be acquired by the sensor, and whenever it is desired to plan
and execute specific actions (e.g., tool guidance, navigation, surgery.)
Current Status
Algorithms for mosaicing the curved retina from projections, and 3-D confocal
images are now well developed within this group. Effective 2-D referencing
algorithms have been developed. We’re currently working on full 3-D
reconstruction & indexing based rapid 3-D spatial referencing algorithms. Parts of
the instrumentation have been built.
Plans and Project Evolution
The retinal testbed will be assembled during the first year. The methods will be
generalized progressively over the next three years in the context of other
applications within the ERC. The model eye will be replaced with other models. A
website will disseminate spatial mapping and referencing code. A fully working
clinical trials-ready prototype will be demonstrated in 3 years. Longer term, we
plan to make spatial mapping and referencing a core capability in a variety of
intelligent “spatially aware” instruments in ophthalmology and beyond.
Key References
[1] "Rapid automated tracing and feature extraction from live high-resolution
retinal fundus images using direct exploratory algorithms," IEEE Trans. on IT in
Biomed., vol. 3, no. 2, pp. 125-138, June 1999
[2] “Image Processing Algorithms for Retinal Montage Synthesis, Mapping, and
Real-Time Location Determination," IEEE Trans. BME, vol. 45, no. 1, pp. 105-
118, January 1998. (Reprinted in IMIA Yearbook, 1999).
[3] "Robust Hierarchical Algorithm for Constructing
a Mosaic from Images of the Curved Human
Retina," Proceedings IEEE –CVPR Conf., vol. 2,
pp. 286-292, Fort Collins, Colorado, June
1999.(Best Paper Award)
[4] “Optimal Scheduling of Tracing Computations
for Real-time Vascular Landmark Extraction from
Retinal Fundus Images,” submitted to IEEE Trans.
on IT for Biomedicine.
[5] "A feature-Based Technique for Joint, Linear
Estimation of High-Order Image-to-Mosaic
Transformations: Application to Mosaicing the
Curved Human Retina," submitted to IEEE-CVPR
conf., June 2000.
PI Contact Information
Badrinath Roysam, Associate Professor, ECSE
and Biomedical Engineering Depts., Rensselaer
Polytechnic Institute, Rm JEC 6046, 110, 8
th
Street, Troy, NY 12180; Phone: 518-276-8067;
Fax: 518-276-6261; Email: roysam@ecse.rpi.edu
Other Connections
The Center for Sight provides data, and medical guidance. The Wadsworth Center is involved in instrument
development, especially testing. The Woods Hole Oceanographic Institute will collaborate on oceanographic
applications. The Scheie Eye Institute (Philadelphia) will drive other ophthalmic applications, & clinical trials.
x
y
z
x '
y'
z'
u '
v'
u
v
P
p
p'
Retina
Lens
Iris
Cornea
Reference Camera
Coordinate System
Optic
Disk
Choroid
Vitreous
Humour
Fig. 2a: Imaging geometry.
Fig. 2b: Layered retinal structure.
Fig. 2c: Retinal Landmarks.
Fig. 3: (Left) Image frame. (Right) Retinal mosaic.