1. Temporally Coherent Stylization of 3D Animations based on Textures Pierre Bénard Supervised by François Sillion & Joëlle Thollot 07/07/2011

2. Introduction Expressiverendering An alternative to photorealism Tools for visual communication Interactive techniques Direct user feedback Dynamic visualization, video games… [Herz98] [GTDS04] [CAS+97] 2

3. Video Games Team Forteress 2 Okami Prince of Persia Jet Set Radio 3

4. “Borderlands”, Gearbox software (2009) Texture painting McGuire et al., Stylized Rendering in Games, SIGGRAPH 2010 Course 4

5. “Borderlands”, Gearbox software (2009) Improved Sobel filter McGuire et al., Stylized Rendering in Games, SIGGRAPH 2010 Course 5

6. “Borderlands”, Gearbox software Final compositing (textures + edges + fog) McGuire et al., Stylized Rendering in Games, SIGGRAPH 2010 Course 6

7. “Borderlands”, Gearbox software (2009) Original Stylized McGuire et al., Stylized Rendering in Games, SIGGRAPH 2010 Course 7

9. Stylization as depiction systems [Willats 97, Durand 02] © Patrick Martin 3Dscene Spatial Primitives Marks 2D image perspective projection 2D lines and regions pastel strokes Attributes color, shading,curvature… 9

10. Stylization as depiction systems Marksmedium and pattern Marks Brush strokes Pencil strokes Paper Watercolor pigments 10

11. Hand–made Animation « Il pleut bergère », Jérémy Depuydt (2005) Random changes Popping / Flickering vs.Temporal continuity 11

12. Naïve CG solutions Static marks  Shower-door effect 12 vs. Coherent Motion

13. Naïve CG solutions Texture mapping 3D appearance 13 vs. Flatness

14. 3 goals Flatness Motion coherence Temporalcontinuity 14

15. 3 goals Flatness Randomchanges Static marks Motion coherence Temporalcontinuity Texture mapping Ideal solution Conflicting goals compromise 15

16. Contributions Regions Dynamic Solid TexturesP. Bénard, A. Bousseau, J. Thollot,I3D 2009 NPR Gabor NoiseP. Bénard, A. Lagae, P. Vangorp, S. Lefebvre, G. Drettakis, J. Thollot,EGSR 2010 Lines Self-Similar Line ArtmapP. Bénard, F. Cole, A. Golovinskiy, A. Finkelstein,NPAR 2010 Active Strokeswork in progress 16 Temporally coherent bricks for stylized animations

18. Region Stylization

20. 2 medium representations Distribution of marks Textures [CTP+03] [VBTS07] 20

21. 2 medium representations Distribution of marks Few-marks Many-marks Textures [CTP+03] [VBTS07] 21

22. Few-marks Flatness Few-marks[Mei96,Dan99,HE04,VBTS07,LSF10] [VBTS07] [Meier96] Painterly rendering 3D distribution of anchor points 2D stroketexture Temporalcontinuity Motion coherence 22

24. Clutter / holes

25. Popping / flickeringor [Meier96] [Daniels99] [HE04] [LSF10] 23

26. Many-marks Many-marks[KC05,BKTS06] 24 Flatness Temporalcontinuity Motion coherence

28. Strong 2D appearance

29. Local motion and/or popping

30. Specific patterns(canvas fiber, watercolor pigments)[KC05] [BKTS06] 25

31. 2 medium representations Distribution of marks Textures In image space: local or global In objet space [CTP+03] [VBTS07] 26

32. Image space: local [BNTS07] Flatness Local image space[BNTS07] 27 Temporalcontinuity Motion coherence

34. Regeneration artifacts

35. Post-production[BNTS07] 28

36. Image space: global [CTP+03] Global image space[CTP+03,CDH06,BSM+07] + 29 Flatness Temporalcontinuity Motion coherence

38. Infinite zoom mechanism

39. 3D motion approximatedsliding[BSM+06] [CDH+06] 30

40. Object space [PHWF01] Flatness Object space [KLK+00,PHWF01,FMS01] 31 Temporalcontinuity Motion coherence “Tonal Artmaps”

42. MIP-mapping

43. Perspective distortion

44. Limited styles[KLK+00] [FMS01] 32

45. Previous Work: Summary 33 Flatness Many-marks Flatness Flatness Flatness Global image space Temporalcontinuity Motion coherence Few-marks Object space Flatness Local image space Temporalcontinuity Temporalcontinuity Temporalcontinuity Motion coherence Motion coherence Motion coherence Temporalcontinuity Motion coherence

47. Goals Real-time texture-based approach Accurate 3D motion object space approach Infinite zoom Dynamic Canvas [CTP+03] Easy parameterization solid textures 35

48. Dynamic Solid Textures P. Bénard, A. Bousseau, J. Thollot, Dynamic Solid Textures for Real-Time Coherent Stylization. I3D 2009 36

49. Infinite Zoom Mechanism Shaded Region Stylization Texture “Fractalization”  1 solid texture : 4 octaves freq. x2 freq. x2 freq. x2 octave 2 octave 1 octave 3 octave 4 weightedsum 37

50. Infinite Zoom Mechanism Shaded Region Stylization Zoom cycle octave 2 octave 1 octave 3 octave 4 weighted sum 38

51. Infinite Zoom Mechanism 39

52. Implementation details One solid texture + 3D coordinates Procedural textures [Per85, Ola05] or synthesized from 2D exemplars[KFCO+07] Ogre3D rendering engine Additional cost: 10% compared to Gouraudshading 40

54. Temporally Coherent Stylization 42 Shot in Ogre3D

55. Limitation: Fractalization Trade-off: Medium fidelity vs. Temporal continuity Linear blending new frequencies contrast loss 43

56. Statistical Analysis Regularpatterns Irregularpatterns Cross-hatching Grid Dots Hatching Paper Noise Paint Pigments S1 Near-regularpatterns Irregularpatterns Cross-hatching Grid Dots Hatching Paper Noise Paint Pigments Z-Scores -1.5 -1.0 -0.5 0.0 0.5 1.0 S2 P. Bénard, J. Thollot, F. Sillion, Quality Assessment of Fractalized NPR Textures. APGV 2009 Z-Scores -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 Fractalization Evaluation 44

58. NPR Gabor Noise P. Bénard, A. Lagae, P. Vangorp, S. Lefebvre, G. Drettakis, J. Thollot,A Dynamic Noise Primitive for Coherent Stylization. EGSR 2010 46

59. Procedural noises Sparse convolution [Lewis 84,89] Spot Noise [van Wijk 91] Gabor Noise [LLDD09] Offers significant spectral control Support anisotropy  Our solution: NPR Gabor Noise 47

60. Gabor Noise [LLDD09] Definition Sum of randomly positioned and weighted kernels Gabor kernel random positionsand weights noise 48 anisotropic isotropic 𝐺𝐾,𝑎 ×cos(𝐹0,𝜔0)

61. NPR Gabor Noise Flatness Noise parameters in image space Evaluation in image space 49

62. NPR Gabor Noise Flatness Coherent motion Point distribution on the surface of the 3D model 50

63. NPR Gabor Noise Flatness Coherent motion 51

64. NPR Gabor Noise Flatness Coherent motion Temporal continuity Smooth LOD mechanism 52

67. more points53

68. GPU Splatting Algorithm Generate 2D point sprites Point distribution Texture sprites 54

69. LOD Mechanism Blending scheme using statistical properties Reduce popping Preserve noise appearance visibility = 0 visibility = 1 Far Close 55

70. Style Design 56

71. Style Design 57

72. Isotropic patterns 58

73. Variations according to shading 59

74. Orientation guided by curvature 60

77. Stylization: Perceptual evaluation Evaluate success of various solutions according to: Flatness Motion coherence Temporalcontinuity P. Bénard, A. Lagae, P. Vangorp, S. Lefebvre, G. Drettakis, J. Thollot,A Dynamic Noise Primitive for Coherent Stylization. EGSR 2010 63

78. Methodology 15 naïve subjects, ~ 20-30 minutes Ranking tasks “Rank the images/videos according to … ” User Study: Setup 64

79. Compared Methods Shaded Region Stylization Local image space Global image space Object space Adv DST D2D nprGN SD TM Extreme cases Many-marks 65

80. Complex stimuli Shaded Region Stylization Adv DST D2D SD TM nprGN 66

81. Results: Flatness Image space methods more flat for simple scene Many 3D cues  flatness hard to perceive in complex scene 67

82. Results: Coherent motion Object space methods more coherent 68

83. Results: Temporal continuity High variance Advection and NPR Gabor Noise produce more changes: organic motion 69

86. Conclusion on Region Stylization Two new solutions Object space: Dynamic Solid Texture Many-marks: NPR Gabor Noise 72 Flatness Flatness NPR Gabor Noise Dynamic Solid Textures Motion coherence Motion coherence Temporalcontinuity Temporalcontinuity

87. Line Stylization

90. Mapping Policies Line Stylization Stretching Tiling 76

91. Mapping Policies 77

93. Stretch or compress the textureTiling Flatness Motion coherence Stretching Temporalcontinuity 78

94. Mapping Policies 79

97. Incoherent slidingFading / Artmap Flatness Tiling Motion coherence Temporalcontinuity 80

98. Mapping Policies 81

102. Blending artifactsFlatness Fading Motion coherence Temporalcontinuity 82

104. Self-Similar Line Artmap (SLAM) P. Bénard, F. Cole, A. Golovinskiy, A. FinkelsteinSelf-Similar Texture for Coherent Line Stylization. NPAR 2010 84

105. SLAM Definition Lmax / 2 Lmax 85

106. SLAM Properties 1) Density 2) Smooth variation 86

107. SLAM Properties 3) Self-Similarity 87

108. SLAM Properties 3) Self-Similarity 88

109. SLAM 89

110. SLAM Synthesis by Example Parametric texture synthesis[PS00] Property:(empirical observation, no formal proof) small change in noise seed  small change in output texture [PS00] Portilla and Simoncelli, A Parametric Texture Model based on Joint Statistics of Complex Wavelet Coefficients, 2000 90 Exemplar [PS00] SynthesizedTexture White Noise Seed

111. Seed Pyramid Construction Lmax =  Wsource Wf Seed Pyramid Lmax/ 2 0.5 Wf αWf+ (1-α) [WfWf] [WfWf] 1 Lmax 91

112. SLAM Synthesis by Example 92

113. Results: Synthesis Synthesized textures Examples 93

114. Results: 2D animation 94

115. Results: Fixed 3D lines 95

116. Results: Fixed 3D lines 96

119. Coherent Parameterization Flatness Remain linear in screen-space Motion coherence Evolve according to the motion of the objet Temporal continuity Adapt to the topological events 99

121. Multiple brush strokes per line

122. Dependent on the input connectivityCoherent Stylized Silhouettes, Kalninset al., SIGGRAPH 2003 100

123. Previous Work Kalninset al., Coherent Stylized Silhouettes, SIGGRAPH 2003 101

124. Pipeline 3D scene Feature samples Lines extraction Line drawing Snakes Brush Paths Tracking Parameterization Cleaning Vectorization Stylization 102

126. Local tangent

127. VelocityLines extraction Line drawing Line drawing Snakes Brush Paths Tracking Parameterization Cleaning Vectorization Stylization 103

129. In image space

130. PersistentLine drawing Snakes Brush Paths Tracking Parameterization Cleaning Vectorization Stylization 104

131. Snakes: Tracking Advection Relaxation 105 Frame f Frame f+1

132. Snakes: Tracking Advection Relaxation 106

133. Cleaning - Vectorization 107

134. Snakes: Tracking + Cleaning 108

135. Pipeline 3D scene Feature samples Lines extraction Line drawing Snakes Brush Paths Tracking Parameterization Cleaning Vectorization Stylization 109

136. Brush Paths Linear screen space parameterization 110

137. Shape abstraction Segments fitting 111

138. Shape abstraction Arcs fitting 112

139. Offsets 113

140. Offsets 114

141. Deformable models 115

142. Deformable models 116

144. Conclusions & Perspectives

145. Contributions 4 new bricks for time coherent stylization Dynamic Solid Textures NPR Gabor Noise Self-Similar Line Artmap Active Strokes Formal evaluation Formulation into 3 goals 2 perceptual studies 1 objective quality metric 119

146. Future Work 2D Animation Ongoing work with Antoine Boellinger(Master 1) 120

147. Future Work 2D Animation Ongoing work with Antoine Boellinger(Master 1) Evaluation & Objective metrics Optical flow analysis Stylization by optimization Versatile system to explore the space of trade-offs 121

148. Acknowledgments 122

149. Thank you for your attention. 123