a reading note of "Video stylization for digital ambient displays of home movies". I like this paper and try to understand it. enjoy !

1. STUDY OF Video Stylization for Digital Ambient Displays of Home MoviesNPAR 2010 TinghuaiWang and John Collomosse University of Surrey, UK David Slatter, Phil Cheatle and Darryl Greig Hewlett-Packard Labs, Bristol UK. Video clips are stylized into cartoons or paintings, and sequenced according to semantic and visual similarity

2. Abstract “Digital Ambient Display” Video  cartoon of home movie Video segmentation based on multi-label graph cut Video  temporal coherent region maps (tracking regions) enhance cartoon painting System Algorithm

3. Outline Introduction System Overview View Stylization Multi-label Graph cut, region propagation, refining region label, smoothing and filtering, stroke placement and shading Video Sequencing Stochastic composition, rendering transitions Results and Discussion Conclusion Video  temporal coherence region Video  Home movie

4. Introduction

5. Digital Ambient Display A genre of content consumption experience which we call ambient experience Displaying still images in an ambient way Digital picture frame Displaying video content inan ambient way ?  Digital Ambient Displays(DAD)

6. Digital Ambient Displays (DADs) Video  “mid-level” scene abstract using Color region segmentation Video  temporal coherence region regionpropagation,multi-labelmraphiccut Video  Home Movie Video selection,compositionandtransition

7. Related work Stochastic selection of video clips Stochastic transitions between video frames [Schodl et al. 00] Single video and based on visual similarity Composition of photos for abstract [Collomosse and Hall 03] Video artwork [Slatter et al. 10; Bizzocchi 08] Little work of the use of artistic video stylizationin ambient displays

9. Video Segmentation Labeling regions and tracking regions in temporal 4 1 1 1 4 5 5 4 2 2 2 3 3 3 t

10. System overview

11. Video Stylization Multi-label Graph cut, region propagation, refining region label, smoothing and filtering, stroke placement and shading Video  temporal coherence region

12. Video Segmentation A novel coherent video segmentation Multi-label graph cut on successive video frames … color distribution built Gaussian Mixture Model (GMM) of each region past frames fn-3 fn-2 previous frame fn-1 Multi-label graph cut propagated by motion Current frame fn label

13. Video Segmentation Assign region labels existing in frame It-1 to each pixel p in frame It(p) Find the best mappingl : P L where L = { l(1), …, l(p), … l(|P|) } , P is an 8-connected lattice of pixels To minimize the global energy function to encourage Spatial homogeneity of contrast within each frame Temporal consistency of color distribution between frames labeling

14. Minimize global energy E U : temporal consistency of color distribution between frames V : spatial homogeneity of contrast within each frame where 1) L is label set of the previous frame 2) P is connected pixels in belong to labels 3) Θ is the colorhistory model

15. Minimize energy of V V : spatial homogeneity of contrast within each frame 2 1 ? 3 Punish pair points (8-connected neighbor) where they have different label but have high color homogeneity !

16. Minimize energy of V V : spatial homogeneity of contrast within each frame 2 1 ? 3 Punish pair points (8-connected neighbor) where they have different label but have high color homogeneity !

17. Minimize energy of U U : temporal consistency of color distribution between frames Color histogram at pixel p– label/color at each frame color distributions of different label assignment pixel p the color distribution at pixel pwith label L1 the color distribution at pixel pwith label L4 255 255 color color

18. Minimize energy of U U : temporal consistency of color distribution between frames color distributions of different pixel pixel n the color distribution at pixel n with label l(pn) the color distribution at pixel m with label l(pm) pixel m color color

19. Minimize energy of U U : temporal consistency of color distribution between frames N:Normal distribution (μ, Σ) Σ N:Mixture of Gaussians (GMM) Θ:parameters of all GMMs, Θ = {ωik, μik, Σi,k; i = 1, …, L; k = 1, …, Ki}

20. Minimize energy of U U : temporal consistency of color distribution between frames Motion propagation O : the prior labeling of pixels

21. Multi-label Graph Cut Minimize E is a NP-hard problem Multi-label graph cut α-expansion iterationfor each label until E can not decrease [Boykov and Kolmogorov 2004].

23. Multiple-Label Graphic Cut Hong Chen, “Introduction to Min-Cut/Max-Flow Algorithms”

24. Multi-label graph cut on Binary Label [PAMI04] Boykov and Kolmogorov, “An Experimental Comparison of Min-Cur/Max-Flow Algorithms for Energy Minimization in Vision”

25. Multi-label graph cut on Binary Label Mini-Cut problem Max-Flow problem of each pixel

26. Multi-label graph cut on Binary Label Mini-Cut problem on boundary Max-Flow problem

27. Region propagation Estimate the motion of It-1 using RANSAC search based on SIFT features [Lowe 04] rigid motion + deformation I’t-1 Propagation labeling per pixel from I’t-1 It Incorrect motion estimation ? Use thinned skeleton to mitigate imprecise motion estimation

28. Region/Skeleton ≡ regions skeleton pruning skeleton robust region region propagation with motion error motion estimation ?

29. Skeleton to robust motion estimation use only the skeletons whose distance to the boundary exceeds apre-set confidence

30. Region propagation It-1 Labeled It-1 skeleton I’t-1 It region label warped according to per-pixel motion estimation replace regions with skeleton to robust motion estimation

31. GMM Build a GMM color model for each region li Sampling historical colors of labelled pixels over recent frames How to sample historical colors? contribution weight More recent color contributes more importance

32. Refining region labels How about new objects appear in It?

33. D Refining region labels Keeptwo color models for each label l in frame It (1) Historical color model (2) an update color model If |Mh – Mu| > threshold, new objects are deemed present

34. Build color model for the new objects Extract the dominant colors Mean-shift to cluster the spatial-color modes (XY+RGB) CMM and skeleton on the new region Re-applying Graphic cut optimization locally within the region Using new labels

35. Smoothing and filtering Spatio-temporal smoothing Gaussian filter of 3x3x3 (x-y-t) Filtering Remove false segmentation and short-lived object smoothing filtering

36. D Filtering - remove short-lived object D l : duration of label l K disconnected objects (e.g. c1, c2, c3…, cK) with the same label dl,k : duration of kth object with label l τr : threshold. in this paper, 6 frames If the duration of any of these disconnected video object within this time window is shorter than threshold, this video object is removed

37. Stroke Placement and Shading β-spline stroke Face detection Painterly Rendering Painterly Rendering with Curved Brush Strokes of Multiple Sizes [Hertzmann 1998] Interpolate an orientation field from the shape of the region in this paper

39. Video Sequencing Stochastic composition, rendering transitions Video  Home movie

40. Stochastic composition

41. Stochastic composition Video sequencing depends on ds (Va, Vb) : semantic distance (tag) between tags of video A and B dv (i, j) : the similarity of videos V1 V2 V3 V4 Vi

42. Rendering Transitions Smoothing video transition similar similar Region Morphing similar similar

43. Rendering Transitions Smoothing video transition 0.5 0.4 0.1 C(:) indicates mean color similarity; A(:) indicates relative area; S(:) indicates shape similarity interms of region compactness

44. Results 23 videos, manually tagged collection

46. This paper

47. ‘synergistic’ mean-shift + edge (Comaniciu, 2002)

49. Result - KITE background detail may (optionally) be abstracted by modifying the initial frame segmentation to merge unwanted detailed regions

50. Result - DRAMA correct handling of regions that disappear and appear within sequences

51. Conclusion Digital Ambient Display (DAD) Select, stylized and transitions between clips automatically A novel algorithm for coherent video segmentation based on multi-label graph cut Parse scene structures to enable shading and painterly effects Create interesting transition effects between clips using region correspondence

52. Future work Backward propagation of region labels to improve coherence of segmentations Improve painterly renderingby region motion caused by occlusion vs. object deformation Graph optimization algorithm similar to [Kovar et al. 02] to plane routes through a subset of clips e.g. to encompass a theme such as “family vacations” rather than traversing the whole database Automatic meta-data annotation on user video collection, e.g. photo categorization [Ruiz et al. 03]

53. END