This document describes the development of SelfieCap, an app that uses computer vision and facial tracking to allow users to control 3D animations in real-time. The developers originally wanted to create an app for automating comic drawing but faced challenges with human pose estimation and drawing abilities. They pivoted to focus on facial motion capture without markers by using constrained local models and a point distribution model to track facial keypoints. Significant effort went into integrating computer vision, 3D graphics engines, and video streaming on iOS. The result allows video streams to be enhanced with real-time 3D graphics controlled by a user's facial movements.

1. SelfieCap
By Bin Shen, Jennifer Rubinovitz, Michał Wolski

2. DRAW
1. People love comics!
2. Most people can’t draw
3. We could try to use
computer vision to
automate the process
Originally wanted to draw comics

3. Proof
Momentcam
10,000,000 - 50,000,000 installs
Cartoon Camera
10,000,000 - 50,000,000 installs
Otaku Camera
1,000,000 - 5,000,000 installs

4. Ran into some issues
1. State of the art in human pose estimation:
http://www.vision.ee.ethz.ch/~hpedemo/

5. More issues
2. We couldn’t find a
single API to do the
drawing for us
3. So we tried to do the
drawing ourselves

6. Turns out we can’t draw either
4. We used Cocos2D to
draw and animate this
guy =>

7. Pivot
Motion capture without the markers

8. Face Tracking

9. How it works
● Learn a model for each
keypoint on the face
○ considering small patches in
the image
● Then constrain their
arrangement using a point
distribution model

10. Constrained Local Models

11. Point Distribution Model
image
point scaling
parameter
rotation
matrix
translationbasis of
variationsmean
location
of
object
point
non-rigid
motion
parameter

12. Global shape prior
● The mean point locations are
computed by taking the
average of their locations in a
labeled dataset
● The basis of variation can be
found by running PCA on the
covariance matrix of the points
PCA Components Demo

13. Graphics
Turns out to be pretty hard:
● Had to find a good graphics
engine for iOS
● Then learn how to use it
(ended up picking Cocos3D)

14. Model & Rigging
More things to learn:
● Using Maya
● Rigging Models
Then we had some
issues porting them to
iOS

15. Model & Rigging
● Maya => .pod => iOS
using PVRGeoPOD
converter
● Most of the advanced
maya functionality
didn’t work on iOS
○ could only use joints

16. Model & Rigging
● Normal directions
● Smooth meshes
● Texturing
● Texture mapping
● Skinning(weight
painting)

17. iOS Application
● Facetracker
○ iOS integration
○ OpenCV
● Graphics
○ Cocos3d
○ Model display
○ Model tweak
● Video streaming
○ AVFoundation

18. Graphics on iOS
Video Stream
+ Face Tracking
+ 3D Graphics =>

19. Possible Improvements
● Use animation blending instead of key
points
○ smoother control
● Integrate more things into 3D
○ Interaction
● Performance tuning

20. Demo!