During this panel presentation information was shared on a collaborative project between the Yale University Art Gallery and the Yale CS department. Staff established that significant imaging data potentially crucial to the work of restoring damaged paintings, could be improved by leveraging the combined strengths of multiple modalities. we therefore aimed to undertake a collaborative exploratory project, with the assistance of Post-Doc students in the Computing and the Arts department of Computer Science at Yale, to design new software that would allow these modalities to be used together. For further information contact the presenter.

1. My
name
is
John
ffrench
and
I
am
the
Director
of
Visual
Resources
at
the
Yale
University
Art
Gallery.
I
oversee
the
imaging
department
which
photographs
the
collecBon
and
also
events,
exhibiBons,
etc
as
well
as
overseeing
the
Rights
and
ReproducBons
office
who
deal
with
the
disseminaBon
of
images
for
external
requests.
With
the
project
I
am
discussing
today,
I
was
more
of
an
organizing
parBcipant
or
intermediary
if
you
will
between
the
conservaBon
staff
and
the
computer
science
group.
It
is
important
to
know
a
few
things
before
we
begin.
I
am
NOT
a
computer
scienBst
–
though
I
can
claim
to
have
several
as
good
friends.
And
I
am
not
a
conservator,
though
my
group
does
provide
treatment
photography
for
the
conservaBon
department.
Also
I
happen
to
be
married
to
a
conservator
so
that
helps!
Recently,
new
two
and
three
dimensional
imaging
modaliBes
have
been
found
useful
in
gaining
insight
into
the
restoraBon
of
damaged
painBngs.
Non-­‐Invasive
Imaging
is
parBcularly
useful
in
restoraBon
as
it
provides
extensive
informaBon
about
a
work
without
physical
contact.
However,
combining
the
results
of
different
modaliBes
is
extremely
difficult,
and
conservators
generally
use
informaBon
from
each
mode
in
isolaBon.
In
this
project,
soSware
was
developed
to
overcome
the
barriers
in
combining
data
and
to
create
an
intuiBve
interface
for
conservators
to
examine
works.
The
soSware
allows
the
conservator
to
combine
images
in
a
common
view
and
idenBfy
the
same
region
in
the
work
in
mulBple
images
simultaneously.
The
soSware
further
allows
the
conservator
to
combine
data
values
to
idenBfy
materials
1

2. In
an
interdisciplinary
collaboraBon,
art
and
computer
imaging
experts
from
the
Yale
University
Art
Gallery
and
the
Department
of
Computer
Science
began
a
project
in
2011
to
examine
selected
Early
Italian
panel
painBngs
combining
a
variety
of
imaging
techniques
that
include
digital
photography,
3D
scanning,
tomography
and
a
novel
form
of
photography
called
Polynomial
Texture
Mapping
(PTM).
At
that
Bme
no
soSware
existed
that
allowed
for
the
various
imaging
modaliBes
to
share
common
coordinate
systems.
For
example,
it
was
not
possible
to
overlay
the
PTM
data
on
the
correct
corresponding
secBon
of
a
3D
model.
Irma
Passeri,
a
conservator
at
the
Art
Gallery’s
laboratory,
working
with
Holly
Rushmeier,
Professor
of
Computer
Science,
established
that
significant
imaging
data
potenBally
crucial
to
the
work
of
restoring
damaged
painBngs,
could
be
improved
by
leveraging
the
combined
strengths
of
mulBple
modaliBes.
Professor
Rushmeier
therefore
aimed
to
undertake
an
exploratory
project
in
collaboraBon
with
Ms.
Passeri,
with
the
assistance
of
Post-­‐Doc
students
in
the
CompuBng
and
the
Arts
department
of
Computer
Science
at
Yale,
to
design
new
soSware
that
would
allow
these
modaliBes
to
be
used
together.
When
fully
refined
and
tested,
the
soSware
applicaBon
was
be
made
available
as
an
open
source
product.
2

3. In
2011,
while
there
were
means
to
display
informaBon
showing
various
conservaBon
image
types,
there
was
no
way
for
a
conservator
to
do
comparaBve
analysis
of
different
file
types
without
using
several
programs.
There
were
certainly
programs
such
as
the
Ghent
altarpiece
Project,
however
these
were
more
a
means
of
educaBon
for
the
public
than
that
of
a
research
tool.
Also
Yale
developed
West
Campus,
a
off-­‐site
arts
and
sciences
area
which
has,
or
will
have
shared
and
collaboraBve
imaging
labs,
as
well
as
conservaBon
labs.
The
parBcipants
of
those
spaces
were
looking
for
ways
in
which
to
create
new
means
of
Campus
collaboraBon
and
a
project
like
this
was
a
good
test-­‐bed.
3

4. In
extreme
short,
the
project
aimed
to
provide
a
means
for
varying
images
to
be
overlaid
with
each
other
in
a
computer
environment
to
beaer
enable
conservators/
curators
the
ability
to
study
works
of
art
(image
sources
were
tradiBonal
photography
-­‐
historic
images
and
modern,
mulB-­‐spectral
imaging,
UV,
IR,
xray,
CAT
scans,
PTM,
and
3D
laser
scans
to
name
a
few).
This
was
seen
as
an
exciBng
collaboraBon
with
the
Computer
Science
department
and
one
of
the
cultural
insBtuBons
on
campus
as
start
of
many
more
collaboraBon
projects
on
campus.
IniBal
1
year
grant
was
awarded
in
the
amount
of
$80,000
which
would
cover
the
acquisiBon
of
capture,
equipment
for
the
capture
and
research
of
objects
(NextEngine
3D
scanner,
materials
to
build
a
hyper-­‐spectral
camera,
computer
systems
to
store
data
and
build
out
the
soSware
interface),
and
funds
to
cover
50%
of
the
salary
of
a
programmer/imaging
scienBst.
Yale
University
Art
Gallery
would
provide
the
object(s),
studio
space
and
20%
of
staff
Bme
of
a
conservator
and
imaging
specialist.
4

5. I’ll
aaempt
to
run
the
soSware
at
the
end
of
this
talk,
but
for
now
will
step
you
through
a
few
of
the
features
of
the
program.
We
all
know
how
live-­‐demo
scenarios
oSen
go
and
given
that
this
program
is
processor
intensive.
Here
you
can
see
a
screenshot
of
what
the
interface
looks
like.
There
are
four
sample
files
loaded
into
the
system,
a
3D
model
created
from
a
NextEngine
scanner,
a
CAT
scan
of
a
panel
painBng,
a
Hyper-­‐spectral
image,
and
another
view
of
the
CT
scan.
The
rogram
is
Mac/PC
based
–
though
admiaedly
it
performs
best
on
a
PC.
5

6. Here
you
can
see
the
various
menu
opBons
available.
In
the
top
menu
you
can
have
the
opBon
to
open
a
2D
or
3D
file
directory.
Open
The
two
leSmost
icons
allow
easy
access
to
the
dialog
box
for
opening
files.
The
first
opens
2D
and
3D
images,
and
the
second
opens
CT
imaging
data.
Render
The
following
five
icons
are
only
usable
when
working
with
3D
meshes.
•
Lightbulb:
Toggles
the
direcBonal
light.
•
Points:
Displays
the
mesh
as
a
series
of
points
or
verBces.
•
Wireframe:
Displays
the
mesh
in
wireframe
mode.
•
Surface:
Displays
the
surface
polygons
on
the
mesh.
•
Texture:
Toggles
the
texture
display.
View
These
two
icons
alter
what
you
see
in
the
image
panel
when
displaying
an
image.
•
Spiral:
Toggles
the
use
of
interpolaBon
on
the
picture
or
texture.
•
Info:
Toggles
the
display
of
computer,
image,
and
perspecBve
informaBon
in
the
6

7. CT
Image
Control
This
toolbox
is
only
acBve
when
working
with
CT
scanning
data.
You
can
use
it
to
navigate
through
a
series
of
2D
image
slices
or
to
select
display
opBons
when
working
with
3D
models.
2D
Image
Stack
Control
These
opBons
are
only
accessible
when
the
“2D
Rendering”
mode
is
selected.
From
here,
you
can
see
the
current
slice
that
is
displayed
in
the
image
panel,
as
well
as
the
total
number
of
slices
available
to
scroll
through.
You
can
also
change
the
view
in
the
image
panel
to
a
front,
side,
or
top
perspecBve
and
flip
the
image
upside
down.
Perhaps
most
importantly,
you
can
navigate
through
the
full
range
of
slices
by
dragging
the
slider
at
the
boaom
from
beginning
to
end
and
back
again.
If
your
image
has
many
slices
and
the
slider
is
not
precise
enough
to
navigate
to
your
desired
slice,
you
can
also
move
more
slowly
through
the
slices
with
the
leS
and
right
arrow
keys.
7

8. Spectrum
The
Spectrum
toolbox
is
most
useful
when
working
with
hyperspectral
images
and
textures;
it
can
be
used
for
standard
RGB
images,
but
will
only
contain
three
meaningful
data
points—the
red,
green
and
blue
reflectance
values.
The
toolbox
has
no
effect
when
working
with
3D
meshes
with
no
texture
or
medical
imaging
data
in
grayscale.
To
use
the
Spectrum
toolbox,
simply
right-­‐click
on
a
given
pixel
or
cell
to
display
a
graph
of
the
available
wavelengths
ploaed
against
the
normalized
reflectance
values.
The
ultraviolet
spectrum
consists
of
those
wavelengths
to
the
leS
of
the
violet
line
on
the
graph,
while
the
infrared
spectrum
appears
to
the
right
of
the
red
line.
8

9. To
view
movie
navigate
to:
hap://www.youtube.com/watch?v=txlwwlSRIQw
9

10. The
source
code
and
supporBng
documentaBon
is
posted
on
Sourceforge
At
the
recent
VAST
conference
a
collaboraBve
paper
was
wriaen
on
the
project
There
has
been
limited
demonstraBon
of
the
product
to
the
audience
it
was
IniBally
desBned
for.
At
last
years
MCN
it
was
shown
to
a
few
people
aaending
(hence
the
interest
and
request
to
present
our
findings
this
year).
It
was
going
to
be
presented
at
IS&T
however
a
conflict
of
interest
in
reporBng
on
the
same
findings
as
announced
at
the
VAST
conference
required
us
to
pull
out
at
the
last
minute.
10

11. Currently
the
program
is
being
further
developed
with
new
CS
programmers
and
there
is
focus
on
a
new
imaging
project
underway
to
study
medieval
manuscripts
and
the
pigments
used.
Through
that
project,
addiBonal
tools/features
are
being
added.
These
new
tools,
while
potenBally
useful
to
the
iniBal
parBcipants,
are
more
aimed
at
the
needs
of
the
2nd
phase
supporters
of
the
project
than
further
establishing
the
core
needs
of
the
original
group.
Unfortunately
the
iniBal
partners
of
the
project
are
not
using
the
program
as
intended
the
new
direcBon
of
the
program
is
more
in
support
of
a
previously
unknown
need,
11

12. WHAT
NEXT??
Projects
take
Bme
and
money
to
complete,
but
we
don’t
always
plan
into
project
Bme
for
usability
tesBng,
or
promoBon
of
products.
It
is
an
important
step
to
factor
in.
Gewng
this
into
more
test-­‐users
hands,
ideally
conservators
who
have
the
need,
but
can
also
provide
valuable
feedback
and
suggest
further
development.
While
we
are
starBng
to
see
some
level
of
conservators
and
imaging
scienBst
express
interest
in
the
program,
sharing
this
program
through
conservaBon
circles,
such
as
AIC,
would
be
the
next
logical
step
and/or
finding
partners
outside
of
Yale
to
work
with
on
the
collaboraBve
development
of
such
programs.
Ideally
trying
to
work
closer
with
another
group
interested
in,
or
working
towards
similar
ends
would
be
logical.
More
so
in
recent
years,
Yale
is
keen
to
openly
share
informaBon,
resources
and
is
looking
for
wider
collaboraBon.
But
we
as
a
community
need
to
find
beaer
ways
in
which
to
bring
awareness
of
such
programs.
12