The document describes the Lightspeed Automatic Interactive Lighting Preview System. It aims to provide fast feedback for lighting design by precomputing a deep framebuffer cache of scene properties like normals and textures, and reevaluating shading on the GPU based on new lighting parameters. Key components include automatic program analysis to separate static and dynamic shader code, deep framebuffer generation from the preprocessed scene, and a GPU-based relighting engine to interactively preview lighting changes at high quality.

1. The Lightspeed Automatic Interactive
Lighting Preview System
Jonathan Ragan-Kelley / MIT CSAIL
Charlie Kilpatrick / ILM
Brian Smith / ILM
Doug Epps / Tippett Studio
Paul Green / MIT CSAIL
Christophe Héry / ILM
Frédo Durand / MIT CSAIL

2. Lighting Design
■ End of pipeline
fixed geometry, viewpoint, material
■ Slow feedback
10-60 mins to render

3. Lighting Design
another hour later...
1
■ End of pipeline
fixed geometry, viewpoint, material
■ Slow feedback
10-60 mins to render

4. Goal: Fast Lighting Preview
Exploit redundancy between previews
■ geometry
■ view
■ material

5. High-Level Approach
■ Precompute deep-framebuffer cache
e.g. ...
normal position diffuse specular
texture texture
■ Preview dynamically on GPU
as the user specifies new light parameters

6. Prior Work

7. Render Caching
Parameterized Ray Tracing [Séquin & Smyrl 1989]
G-Buffer [Saito & Takahashi 1990]
Fast Relighting Engine [Gershbein & Hanrahan 2000]
■ Precomputed buffer
(normals, texture)
with BRDF*light reevaluated at runtime
■ Shortcomings
■ Simplistic shading
■ No antialiasing, motion blur, transparency

8. Pixar’s Lpics
[Pellacini et al. 2005]
■ Production shaders
& lights (RenderMan)
■ Exploit programmable GPUs
■ No antialiasing, transparency
■ Requires extra shader-writing work
■ (final) RenderMan version
+ extra caching code
■ GPU preview version

9. Specializing Shaders
[Guenter et al. 1995]
■ Given dynamic parameters, automatically
split shader code into:
■ static
■ dynamic
■ Simple language
no control flow
■ Simple shaders
Plate 1: Test image.
Plate 1: Test image. Plate 2: Image obtained by reshading image of Plate 1. This
Plate 2: Image obtained by reshading image of Plate 1. This
took 12 seconds unspecialized, 0.5 seconds specialized.
took 12 seconds unspecialized, 0.5 seconds specialized.
Bibliography
Bibliography
[1] Andersen, Lars Ole. Self-applicable C Program
[1] Andersen, Lars Ole. Self-applicable C Program [9] Jones, Neil D., Carsten K. Gomard, and Peter Sestoft.
[9] Jones, Neil D., Carsten K. Gomard, and Peter Sestoft.
Specialization. Proceedings of the ACM SIGPLAN
Specialization. Proceedings of the ACM SIGPLAN Partial Evaluation and Automatic Program Generation.
Partial Evaluation and Automatic Program Generation.
Workshop on Partial Evaluation and Semantics-Based
Workshop on Partial Evaluation and Semantics-Based Prentice-Hall, 1993.
Prentice-Hall, 1993.
Program Manipulation (San Francisco, California, June 12-
Program Manipulation (San Francisco, California, June 12-
[10] Mogensen, Torben. The Application of Partial Evaluation to
[10] Mogensen, Torben. The Application of Partial Evaluation to
20, 1992). Yale University technical report
20, 1992). Yale University technical report
YALEU/DCS/RR-909, 1992, 54-61. Ray-Tracing. Master's thesis, DIKU, University of
Ray-Tracing. Master's thesis, DIKU, University of
YALEU/DCS/RR-909, 1992, 54-61.
Copenhagen, Denmark, 1986.
Copenhagen, Denmark, 1986.

10. Precomputed Radiance Transfer
e Transfer for Real-Time Rendering
requency Lighting Environments
e.g. Sloan et al. 2002, Ng et al. 2003
Jan Kautz John Snyder
x-Planck-Institut für Informatik
Microsoft Research
jnkautz@mpi-sb.mpg.de johnsny@microsoft.com
use and
hat cap-
process,
over the
equency
s global
ect onto
o actual
ampling
e rigidly
ting and
Figure 1: Precomputed, unshadowed irradiance from [34] (left) vs. our
pherical
precomputed transfer (right). The right model can be rendered at 129Hz
Inappropriate for our application
ntly on with self-shadowing and self-interreflection in any lighting environment.
o a dot
Glossy frequency lighting environments, using a low-order spherical
■ Large precomputation cost
e further harmonic (SH) basis to represent such environments efficiently
lighting without aliasing. The main idea is to represent how an object
g points scatters this light onto itself or its neighboring space.
■ Largely ignore light functions
m rigidly To describe our technique, assume initially we have a convex,
ers. We diffuse object lit by an infinitely distant environment map. The
roach.
■ Limitations on shading models
object’s shaded “response” to its environment can be viewed as a
chniques, transfer function, mapping incoming to outgoing radiance, which
in this case simply performs a cosine-weighted integral. A more
complex integral captures how a concave object shadows itself,
where the integrand is multiplied by an additional transport factor
tions are representing visibility along each direction.
tunately, Our approach is to precompute for a given object the expensive
environ- transport simulation required by complex transfer functions like
25], rad- shadowing. The resulting transfer functions are represented as a
ple point dense set of vectors or matrices over its surface. Meanwhile,
ndering. incident radiance need not be precomputed. The graphics hard-
difficul- ware can dynamically sample incident radiance at a number of
s of real points. Analytic models, such as skylight models [33] or simple

11. Our Design Goals
1. High-performance preview
■ Low-latency feedback
■ Fast initial precomputation
2. Seamless integration
■ Same input: RenderMan scene & shaders
■ Same output: high quality image
3. Ease of implementation & maintenance

12. System Overview

13. System Overview
RenderMan
scene &
shaders

14. System Overview
RenderMan cache &
scene & preview
shaders shaders
automatic
caching

15. System Overview
RenderMan cache &
preview preview image
scene &
shaders shaders
relighting
automatic
engine
caching
(gpu)

16. System Overview
RenderMan cache &
preview preview image
scene &
shaders shaders
relighting
automatic
engine
caching
(gpu)
lighting parameters
one-time precomputation iterative preview

17. System Overview
RenderMan cache &
preview preview image
scene &
shaders shaders
relighting
automatic
engine
caching
(gpu)
lighting parameters
Automatic
for real production scenes
‣ specialization & translation
‣ cache compression

18. System Overview
RenderMan cache &
preview preview image
scene &
shaders shaders
relighting
automatic
engine
caching
(gpu)
lighting parameters
Automatic High fidelity
for real production scenes antialiasing, motion blur,
‣ specialization & translation transparency
‣ cache compression ‣ indirect framebuffer

19. Automatic Caching
RenderMan cache &
preview preview image
scene &
shaders shaders
relighting
automatic
engine
caching
(gpu)
lighting parameters

20. Program Analysis

21. Program Analysis
mySurface(color c) {
albedo = c*Texture(U,V);
shade = Light(L)*BRDF(N,L);
return albedo*shade;
}

22. Program Analysis
mySurface(color c) { dynamic
albedo = c*Texture(U,V);
shade = Light(L)*BRDF(N,L);
return albedo*shade;
}
what part of the code depends on
the lighting parameter L?

23. Program Analysis
mySurface(color c) { dynamic
albedo = c*Texture(U,V);
shade = Light(L)*BRDF(N,L);
static
return albedo*shade;
}
everything else is static

24. Program Analysis
mySurface(color c) { dynamic
albedo = c*Texture(U,V);
shade = Light(L)*BRDF(N,L);
static
return albedo*shade; cache
}
values at the boundary are cached
for reevaluation

25. Program Analysis
mySurface(color c) {
albedo = c*Texture(U,V);
shade = Light(L)*BRDF(N,L);
return albedo*shade;
}

26. Program Analysis
albedo shade
mySurface(color c) {
albedo = c*Texture(U,V);
shade = Light(L)*BRDF(N,L);
return albedo*shade;
}
c Texture Light BRDF
U V L N

27. Deep Framebuffer Generation
mySurface(color c) {
albedo = c*Texture(U,V);
shade = Light(L)*BRDF(N,L);
return albedo*shade;
}

28. Deep Framebuffer Generation
caching
shader
mySurfaceCache(color c) {
albedo = c*Texture(U,V);
cache(N);
cache(albedo);
}

29. Deep Framebuffer Generation
N
albedo
final deep
renderer framebuffer
cache
caching
shader
mySurfaceCache(color c) {
albedo = c*Texture(U,V);
cache(N);
cache(albedo);
}

30. Deep Framebuffer Generation
N
albedo
deep
framebuffer
cache
caching
shader

31. Deep Framebuffer Generation
N
albedo
deep
framebuffer
cache
caching
shader
mySurfaceDynamic(vector L) {
N = lookup();
albedo = lookup();
shade = Light(L)*BRDF(N,L);
preview return albedo*shade;
shader }

32. Deep Framebuffer Generation
N
albedo
deep
framebuffer
cache
caching
shader
mySurfaceDynamic(vector L) {
N = lookup();
albedo = lookup();
shade = Light(L)*BRDF(N,L);
preview return albedo*shade;
shader }

33. Deep Framebuffer Generation
N
albedo
deep
framebuffer
cache
caching
shader
rendered
shade image
preview
shader

34. Automatic Caching

35. Automatic Caching
RenderMan
scene scene preprocessor
RenderMan
shaders

36. Automatic Caching
RenderMan
scene scene preprocessor
program
analysis
RenderMan
shaders

37. Automatic Caching
RenderMan
scene scene preprocessor
caching
shaders
program
analysis
RenderMan
shaders

38. Automatic Caching
deep
RenderMan framebuffer
scene scene preprocessor cache
x2
x1
final N
P
renderer
caching
shaders
program
analysis
RenderMan
shaders

39. Automatic Caching
deep
RenderMan framebuffer
scene scene preprocessor cache
x2
x1
final N
P
renderer
caching
shaders
program
analysis
RenderMan dynamic
shaders preview
shaders

40. Automatic Caching
challenge: cache size
deep
RenderMan framebuffer
scene scene preprocessor cache
x2
program analysis is
final
x1
N
P
conservative
renderer
caching
shaders generates
many cached terms
program
analysis (hundreds)
RenderMan dynamic
shaders preview
shaders
large caches
(gigabytes)

41. Cache Compression
deep
RenderMan framebuffer
scene scene preprocessor cache
x2
x1
final N
P
renderer
caching
shaders
Remove redundant
program
analysis values after caching
RenderMan dynamic
shaders preview
shaders

42. Cache Compression
deep
RenderMan framebuffer
scene scene preprocessor cache
x2
x1 N
final N cache P
P
renderer compression
caching
shaders
Remove redundant
program
analysis values after caching
RenderMan dynamic
shaders preview
shaders

43. Cache Compression
deep
RenderMan framebuffer
scene scene preprocessor cache
x2
x1 N
final N cache P
P
renderer compression
caching
shaders
Remove redundant
program
analysis values after caching
RenderMan
shaders
dynamic
preview
→ 5x compression
shaders

44. System Overview
deep
RenderMan framebuffer
scene scene preprocessor cache
N
P
final cache
renderer compr.
comp.
caching
shaders
program
analysis
RenderMan dynamic
shaders preview
shaders

45. System Overview
deep
RenderMan framebuffer
scene scene preprocessor cache
N
P
final cache
renderer comp.
caching
shaders
program
analysis
RenderMan dynamic
shaders preview
shaders

46. System Overview
deep
RenderMan framebuffer
scene scene preprocessor cache
N
P
final cache
renderer comp.
relighting
caching
shaders engine
(gpu)
program
analysis preview image
RenderMan dynamic
shaders preview
shaders lighting parameters

47. System Overview
antialiasing?
RenderMan
scene scene preprocessor
deep
framebuffer
cache
transparency?
N
P
final cache
renderer comp.
relighting
caching
shaders engine
(gpu)
program
analysis preview image
RenderMan dynamic
shaders preview
shaders lighting parameters

48. Classical Deep Framebuffer
B
A

49. Classical Deep Framebuffer
B
A

50. Classical Deep Framebuffer
x2 deep
x1 framebuffer
N cache
B P
cache
shade
A
Cache one sample per-pixel rendered
Shade cache values to render image image

51. Classical Deep Framebuffer
x2 deep
x1 framebuffer
N cache
B P
cache
shade
A
How can we do antialiasing? rendered
image

52. Classical Deep Framebuffer
supersample image x2
x1
N
B P
cache
shade
A
How can we do antialiasing?
transparency?

53. Classical Deep Framebuffer
x4
supersample x3
cache N
B P
cache
shade
A
supersample
shading
How can we do antialiasing?
transparency?

54. Classical Deep Framebuffer
x4
x3
N
B P
...?
cache
shade
A
How can we do antialiasing? ...?
transparency?

55. Our Indirect Framebuffer
■ Decouple shading from visibility
■ only supersample visibility
‣ Do what RenderMan does
■ Compress samples
based on static visibility

56. RenderMan / REYES
B
show
*filtering*
pixel color?
A

57. RenderMan / REYES
B
b1
show
*filtering*
pixel color?
b2
a1
A
dice into micropolygons

58. RenderMan / REYES
B
b1
show
*filtering*
pixel color?
b2
a1
A
shade micropolygons

59. RenderMan / REYES
B
b1
show
*filtering*
pixel color?
b2
a1
A
rasterize micropolygons

60. RenderMan / REYES
B
b1
show
*filtering*
pixel color?
b2
a1
A

61. RenderMan / REYES
1 2
B a1 b2
b1
show
*filtering*
1 2 pixel color?
b2
3 a 1
4
3 4
A a1 a1
pixel = a1+b2+a1+a1

62. RenderMan / REYES
1 2
B a1 b2
b1
show
*filtering*
1 2 pixel color?
b2
3 a 1
4
3 4
A a1 a1
pixel = a1+b2+a1+a1
4

63. RenderMan / REYES
1 2
B a1 b2
b1
show
*filtering*
1 2 pixel color?
b2
3 a 1
4
3 4
A a1 a1
pixel = a1+b2+a1+a1 = 0.75*a1+0.25*b2
4

64. RenderMan / REYES
1 2
B a1 b2
b1
show
*filtering*
1 2 pixel color?
b2
3 a 1
4
3 4
A a1 a1
pixel = ?

65. RenderMan / REYES
B
b1
b2
a1
A
pixel = ?

66. RenderMan / REYES
1
B a1 b1
α=0.3 α=1.0
b1
0.3*a1+0.7*b1
1
b2
a1
A
pixel = ?

67. RenderMan / REYES
1 2
B a1 b1 b2
α=0.3 α=1.0 α=1.0
b1
2
0.3*a1+0.7*b1 1.0*b2
1
b2
a1
A
pixel = ?

68. RenderMan / REYES
1 2
B a1 b1 b2
α=0.3 α=1.0 α=1.0
b1
2
0.3*a1+0.7*b1 1.0*b2
1
b2
3 a 1
4
3 4
A a1 a1 b2
α=0.3 α=0.3 α=1.0
0.3*a1 0.3*a1+0.7*b1
pixel = ?

69. RenderMan / REYES
1 2
B a1 b1 b2
α=0.3 α=1.0 α=1.0
b1
2
0.3*a1+0.7*b1 1.0*b2
1
b2
3 a 1
4
3 4
A a1 a1 b2
α=0.3 α=0.3 α=1.0
0.3*a1 0.3*a1+0.7*b1
pixel = 0.225*a1+0.35*b1+0.25*b2

70. Indirect Framebuffer

71. Indirect Framebuffer
b1
a1b2

72. Indirect Framebuffer
b1
a1b2
x4 x4 deep
x1 x3 x3 buffer
N x2 x2
P P P
a1 b1 b2
no longer image space
(per-micropolygon)

73. Indirect Framebuffer
b1
a1b2
x4 x4 deep
x1 x3 x3 buffer
N x2 x2
P P P
a1 b1 b2
shading
shade samples
shaded like a conventional deep-framebuffer

74. Indirect Framebuffer
b1
a1b2
indirect framebuffer
x4 x4 deep
x1 x3 x3 buffer
N x2 x2
P P P
a1 b1 b2
shading
shade samples

75. Indirect Framebuffer
b1
a1b2
indirect framebuffer
x4 x4 deep
x1 x3 x3 buffer 0.225 0.35 0.25
N x2 x2
P P P
a1 b1 b2
shading
shade samples
densely stored in vertex array, on GPU

76. Indirect Framebuffer
b1
a1b2
indirect framebuffer
x4 x4 deep
x1 x3 x3 buffer 0.225 0.35 0.25
N x2 x2
P P P
a1 b1 b2
blend
shading final
shade samples pixel
4 subpixel samples, 2 transparent layers
only 3 unique micropolygons

77. Indirect Framebuffer
b1
a1b2
indirect framebuffer
x4 x4 deep
x1 x3 x3 buffer 0.225 0.35 0.25
N x2 x2
P P P
a1 b1 b2
blend
shading final
shade samples pixel
4 subpixel samples, 2 transparent layers
only 3 unique micropolygons
all contributions linearized into a single weight

78. More samples,
b1 Same cost
a1b2
indirect framebuffer
x4 x4 deep
x1 x3 x3 buffer 0.242 0.331 0.245
N x2 x2
P P P
a1 b1 b2
blend
shading final
shade samples pixel

79. More samples,
b1 Same cost
a1b2
indirect framebuffer
x4 x4 deep
x1 x3 x3 buffer 0.242 0.331 0.245
N x2 x2
P P P
a1 b1 b2
blend
shading final
shade samples pixel

80. More samples,
b1 Same cost
a1b2
indirect framebuffer
x4 x4 deep
x1 x3 x3 buffer 0.242 0.331 0.245
N x2 x2
P P P
a1 b1 b2
blend
shading final
shade samples pixel

81. Indirect Framebuffer Results
antialiasing
motion blur
transparency
identical to RenderMan

82. Indirect Framebuffer Results
antialiasing
motion blur
transparency
identical to RenderMan
RenderMan
45M micropolygons
60M subpixel samples
brute-force
>100M samples
Indirect framebuffer
10M deep fb
+15M indirect fb
720x389, 64x supersampling

83. System Architecture

84. System Architecture
RenderMan
scene scene preprocessor
RenderMan
shaders

85. System Architecture
RenderMan
scene scene preprocessor
program
analysis
RenderMan
shaders

86. System Architecture
RenderMan
scene scene preprocessor
caching
shaders
program preview
analysis shaders
RenderMan
shaders

87. System Architecture
RenderMan
scene scene preprocessor
final
renderer
caching
shaders
program preview
analysis shaders
RenderMan
shaders

88. System Architecture
RenderMan
scene scene preprocessor
deep fb indirect fb
final cache x2
x1
renderer compr. N
P
caching
shaders
program preview
analysis shaders
RenderMan
shaders

89. System Architecture
RenderMan
scene scene preprocessor relighting engine (GPU)
deep fb indirect fb
final cache x2
x1
renderer compr. N
P
caching
shaders
program preview
analysis shaders
RenderMan
shaders

90. System Architecture
RenderMan
scene scene preprocessor relighting engine (GPU)
deep fb indirect fb
final cache x2
x1
renderer compr. N
P
caching
shaders shade
program preview
analysis shaders
RenderMan
shaders

91. System Architecture
RenderMan
scene scene preprocessor relighting engine (GPU)
deep fb indirect fb rendered image
final cache x2
x1
renderer compr. N
P
caching
shaders shade blend
program preview
analysis shaders
RenderMan
shaders

92. System Architecture
RenderMan
scene scene preprocessor relighting engine (GPU)
deep fb indirect fb rendered image
final cache x2
x1
renderer compr. N
P
caching
shaders shade blend
program preview
analysis shaders
lighting
parameters
RenderMan
shaders
one-time precomputation iterative preview

93. Additional Features
■ Shadows
■ Subsurface Scattering
■ Performance
enhancement
■ Progressive Refinement
■ Light Caching …
■ Tiling

94. Results
initial feedback: 0.05 sec
full refinement: 0.7 sec
Lightspeed
720x389 offline render: 5 mins

95. Results
initial feedback: 0.05 sec
full refinement: 0.7 sec
RenderMan
Lightspeed
720x389 offline render: 5 mins

96. Results

97. Results
caching: 16 mins
initial feedback: 0.11 sec
full refinement: 2.7 sec
914x389 offline render: 59 mins
13x13 antialiasing
42 lights

98. Results
caching: 16 mins
initial feedback: 0.11 sec
full refinement: 2.7 sec
Lightspeed
RenderMan
914x389 offline render: 59 mins
13x13 antialiasing
42 lights

99. Results
caching: 16 mins
initial feedback: 0.11 sec
full refinement: 2.7 sec
Lightspeed
RenderMan
914x389 offline render: 59 mins
13x13 antialiasing
42 lights

100. Results

101. Results

102. Results

103. Results

104. Limitations
■ GPU programming model
■ Dynamic calls to external C code
■ Complex data structures
■ GPU limits (bandwidth, memory, registers)
■ Fully-accurate dynamic ray tracing
■ Unbounded dynamic loops
■ Additional features not yet implemented
■ Indirect diffuse
■ Deep shadows
■ Non-linear lights

105. Discussion
■ Automatic and manual specialization
(Lpics) both have advantages
■ Manual specialization allows hand optimization
■ Compiler requires up-front R&D, never perfect
(especially on the GPU)
■ Saves significant time in production
■ Some material parameters are editable
■ Cache compression is key to practical
automatic specialization
much simpler than fancy static analysis
■ Indirect Framebuffer is powerful, scalable

106. Summary
■ Interactive lighting preview
milliseconds instead of hours
■ Automatic caching for our production scenes
■ Program analysis
■ Cache compression
■ Indirect framebuffer
■ Efficient antialiasing, motion blur, transparency
■ Progressive refinement
■ In use on current productions

107. Acknowledgments
Inception: Pat Hanrahan, Ujval Kapasi
Compilers: Alex Aiken, John Kodumal
Tippett: Aaron Luk, Davey Wentworth
ILM: Alan Trombla, Ed Hanway, Dan Goldman,
Steve Sullivan, Paul Churchill
Images: Michael Bay, Dan Piponi
Money: NSF, NVIDIA, MSR, Sloan & Ford
fellowships

108. Summary
■ Interactive lighting preview
milliseconds instead of hours
■ Automatic caching for our production scenes
■ Program analysis
■ Cache compression
■ Indirect framebuffer
■ Efficient antialiasing, motion blur, transparency
■ Progressive refinement
■ In use on current productions

109. EXTRAS

110. Cache Compression
deep framebuffer cache
Remove redundant
values after caching
■ non-varying
same for many pixels
■ non-unique
same within a pixel
unique
varying
→ 4-5x compression

111. Cache Compression
shader dynamic varying unique
(static analysis) (compressed)
generic surface 402 145 97
metallic paint 450 150 97

112. Visibility Compression
Indirect
RenderMan Framebuffer
scene res. samples shade subpix shade indirect
robot 914x389 13x13 2.1M 32M 633k 1.6M
robot
720x306 13x13 1.5M 21M 467k 3.8M
(blur)
pirate 640x376 4x4 2.5M 2.3M 327k 716k
hairs 720x389 8x8 43M 58M 11M 17M