ANYFACE*: Create Film Industry-Quality Facial Rendering & Animation Using Mainstream PCs | SIGGRAPH 2019 Technical Sessions

AnyFace
Weibo Xie (NEXT)
Kai Wang (Intel)
SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
A Solution for Creating Film Industry Quality Facial
Rendering and Animation on Mainstream Platforms

Introduction

Agenda
“Siren” for Game
• Ultra Detail with Face Painting
• Bone Based High Fidelity Facial Animation
From “Siren” to AnyFace
• Lossless Face Customization
• Animation Retarget
• Object Space Deformer
Platform Optimizations
Conclusion
Q&A

Digital Human
• Just high quality feeling
• Limited texture size
• Limited mesh complexity
Digital Human vs Game (Face Quality)
• 100% match
• Up to16k texture
• Unlimited mesh complexity
Game

Material
4K Base, Normal, Roughness, Wrinkle Maps

Rock Painting
Painting face as a rock!

Face Painting

Face Masks

4 layers, 128 width detail normal

4 layers, 128 width detail normal, roughness

4 layers, 128 width detail normal, roughness, base cavity

4K normal, roughness

Line
(Vertical)
Line
(Horizontal)
Dot_1 Dot_2
Different Face Masks

Shared Face Mask

Blend Shape Based Facial Animation

Digital Human
• Just Neutral Face
• 170+ Bones
• Bone Animation Data
• How to Paint Bone Weight?
• Achieve Same Quality as BS?
• 3-4 Months for Artist!!!
Digital Human vs Game (Facial Animation)
• 40+ Full Face Blend Shape
• 700+ Spit Blend Shape
• Rig-Logic Data Control Animation
Game

BlendShape 170 Bones
AnimationSet
Bone Indices Bone Weights
Siren with SSDR

Bones Facial AnimationSiren with Bone Based Facial Animation

Prove the Fidelity
UE4 Runtime Compare
• Unwrap Face Position to Texture
• Compare Textures
MeshCompare
• https://github.com/dgovil/MeshCompare
• Same Topology
• Maya Plugin
BlendShape 70 Bones 170 Bones

Unwrapping Face Material

Unwrapping Position
BlendShape Unwrap Position

5000 Tris Unwrap Position
Unwrapping Position

Mesh Error
Error

Mesh Diff Matrix

Face Part Split & Align
100% to the target is our target!

https://www.russian3dscanner.com/wrap3-tutorials/
Replace In Wrap3

Wrap Graph

Face BS Split & Align (Wrap Replace)

More Variance
https://www.3dscanstore.com/3d-body-parts-models

N Heads N Hats

N Heads One Hat

• Unwrapping Face in Object Space
• Delta Position Related to Neutral Face
• Projecting Mesh Vertex to Face
Object Space Deformer

Unwrapping Neutral Face

Delta Position

2nd UV for Unwrapping Face

Projecting Mesh Vertex to Face

Seam

Better UV For Unwrapping Face

Better UV for Hat

Fixing the Seam

Hat/Mask

Glass Deformer

Hair Deformer

Hair UV

Hair/Fuzz Deformer

Platform Optimizations
• Face Painting Optimization
• Mesh LOD
• Bone LOD
• Deformer Optimization

Face Painting Optimization
• Detail Texture Packing
• RG (Normal)
• B (Roughness)
• A (Diffuse Cavity)

Detail Texture Packing

Detail Texture Packing
• Without Packing
• 3 sampling for Base layer
• 4 detail layers for each face
• 3 sampling for each detail layer
• 15 times!
• With Packing
• 3 sampling for Base layer
• 4 packed detail layers for each
face
• Reduced to 7 times!

Mesh LOD 40,000 Tris

Mesh LOD 5,000 Tris

170 Bones 40,000 Tris

Compute Shader Time for Skinning (5 Faces)
0
50
100
150
200
250
MESH LOD0 MESH LOD1 MESH LOD2 MESH LOD3
MICROSECOND Bone_LOD0 Bone_LOD1 Bone_LOD2
Mesh LOD0 Mesh LOD1 Mesh LOD2 Mesh LOD3
Bone_LOD0 184 104 80 44
Bone_LOD1 194 99 68 40
Bone_LOD2 193 111 84 43SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
GPU Duration for 5 faces on Iris Pro 580 - Skinning

Base Pass Time Analysis (5 Faces)
0
500
1,000
1,500
2,000
2,500
MESH LOD0 MESH LOD1 MESH LOD2 MESH LOD3
MICROSECOND
Bone_LOD0 Bone_LOD1 Bone_LOD2
Mesh LOD0 Mesh LOD1 Mesh LOD2 Mesh LOD3
Bone_LOD0 2092 1582 1408 1254
Bone_LOD1 2008 1467 1439 1151
Bone_LOD2 2015 1582 1347 1287SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
GPU Duration for 5 faces on Iris Pro 580 – Base Pass with material simplified

CPU Time (ms) for Animation (5 Faces)
Animation Time on Game Thread Animation Time on Worker Threads
1.33
1.11
1.07
0.86
1
0.79
4-core/8-thread
8-core/16-thread
0 0.2 0.4 0.6 0.8 1 1.2 1.4
LOD3 LOD2 LOD1
1.3
0.45
0.36
1.16
0.43
0.34
1.08
0.41
0.33
0 0.2 0.4 0.6 0.8 1 1.2 1.4
single-thread
4-core/8-thread
8-core/16-thread
LOD3 LOD2 LOD1

Bone & Mesh Optimizations
• Vertex number is a deciding factor to skinning time.
• Proportional to computation cost (each vertex has 4 bone weights)
• Bone number is basically not relevant.
• Vertex number can largely affect to Base-pass time on some
platforms.
• Current bone matrices computation is on CPU side.
• Greater impact for mesh complexity on GPU.
• Greater impact for bone complexity on CPU.
• Better performance on CPUs with more cores and further
optimizations can be done.

Deformer Optimizations
• LOD Meshes for Deformer
• Buffer Size Policy
• 128 buffer for near shot
• 32 for 5 meters away
• Static delta texture for 15 meters away
• RGBA32f->RGBA16f

128x128 Buffer

32x32 Buffer

2x2 Buffer

GPU Time of Deformer (1 Face)
317
237
164
130
RGBA16F 32x32 MESH LOD0
RGBA16F 32x32 MESH_LOD1
Deformer time for 1 face on Iris Pro 580• Mesh LOD is a deciding factor and
lower mesh complexity largely
improves the perf.
• RT formats and buffer size have
less impact on perf.
• For RGBA16F 32x32, ~130 us per
face is good for our current game
titles.

Conclusion
• Bring Siren to Game
• Ultra Detail
• Bone Based High Fidelity Facial Animation
• One Hair/Attachment for Any Face
• Customization
• Different Platforms

Future Work
• Face Library Based on Our Tech
• Further Work on Animation Retarget
• Introducing More AI Elements
• AI Face Painting
• One Photo Avatar
• Different Styles (e.g. Cartoon)
• All Our Tech In A Game Production

References
• [unreal] “Digital Humans”, unreal
• [Adelle] “Achieving High-Quality, Low-Cost Skin: An Environment Approach”, Adelle Bueno,
Eduardo Mosena , GDC2017
• [Binh] “Smooth Skinning Decomposition with Rigid Bones”, Binh Huy Le , Zhigang Deng,
SIGGRAPH Asia 2012
• [R3dS] “Wrap3 Tutorials”, R3dS
• [Ryan] “Automated and Improved UV Dilation”, Ryan Brucks, 2016
• [dgovil] ”github.com/dgovil/MeshCompare”, dgovil, 2018
• [Andrew]” Photogrammetry and Star Wars Battlefront”, Andrew Hamilton, Kenneth Brown
• [IDZ] https://software.intel.com/en-us

• NEXT
• Li Shen
• Yu Gu
• Vladislav Perge
• Jun bai Li
• Qiang Fu
• Jiayang Yang
• Li Jin
• Xiaojun Ding
• Jinjian Xu
• Bo Yu
• Shengjun Tang
• Guofeng Chang
• Intel
• Cage Lu
• Kai Wang
Acknowledgements
• Family
• Epic

Question?

Legal Disclaimer
Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service
activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with
your system manufacturer or retailer or learn more at [intel.com].
Results have been estimated or simulated using internal Intel analysis or architectural simulation or modeling and provided to
you for information purposes. Any differences in your system hardware, software or configuration may affect your actual
performance.
Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors and
Intel Integrated GPU. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems,
components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should
consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the
performance of that product when combined with other products. For more information go to www.intel.com/benchmarks.
Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for
optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets
and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on
microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with
Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please
refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered
by this notice.
Configurations used for the perf data:
• Macbook Pro with Intel Iris Graphics 550, 530, 515 some with fixed 850Mhz frequency and some with dynamic frequency
• Latest Windows OS
• macOS 10.15 Beta, macOS 10.14.5
• Intel® Gen11 processor graphics
• All testing was performed at Intel
Intel, the Intel logo, are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries.
*Other names and brands may be claimed as the property of others.
© Intel Corporation.

ANYFACE*: Create Film Industry-Quality Facial Rendering & Animation Using Mainstream PCs | SIGGRAPH 2019 Technical Sessions

Recommended

Recommended

More Related Content

More from Intel® Software

More from Intel® Software (20)

Recently uploaded

Recently uploaded (20)

ANYFACE*: Create Film Industry-Quality Facial Rendering & Animation Using Mainstream PCs | SIGGRAPH 2019 Technical Sessions