1. ICCV祭り2011.11.27
1
CV勉強会＠関東（第18回）
Content-Based Photo Quality Assessment
shirasy
紹介する論文テーマ：
本資料は、以下の学会発表論文を引用しております。
Content-Based Photo Quality Assessment
W. Luo, X. Wang and X. Tang
in Proceedings of IEEE International Conference on Computer Vision (ICCV) 2011.
http://www.ee.cuhk.edu.hk/~xgwang/papers/luoWTiccv11.pdf

2. 2
自己紹介
最近、参加している活動
CV関連
デザイン関連
写真関連

3. 3
はじめに
視覚的な美的観点からの写真の自動品質評価の関心が高まる

4. These features were not specially designed for photo quality assessment.
4
関連研究
写真の自動品質評価の先行研究
○ブースティング
Tong et al. [18] used boosting to combine global low-level features for the
classification of professional and amateurish photos
○写真撮影の経験則
Ke et al. [11] designed a set of high-level semantic features based on rules
of thumb of photography. They measured the global distributions of
edges, blurriness, hue.

5. 関連研究
○着目領域
Some approaches employed regional features by detecting subject areas,
since human beings percept subject areas differently from the background.
• Datta et al. [5] divided a photo into 3 × 3 blocks and assumed that the
central block is the subject area.
• Luo et al. [12] assumed that in a high quality photo the subject area has
a higher clarity than the background.
5
写真の自動品質評価の先行研究

6. 6
高品質の指針
様々なタイプの写真を撮る時、プロカメラマンは、異なる美的基準を持ち、
異なる写真技術を用いることが知られている[2,19]

7. 既存手法の大きな問題は、コンテンツの多様性を考慮せずに
全ての写真を同じように扱うこと
Photos are manually divided into seven categories based on
photo content:
7
本論文のポイント
風景植物動物夜景人物静止物構造物

8. プロやアマチュアカメラマン
の写真：17,613枚
8
本論文のポイント
ポイント
• CCCCoooonnnntttteeeennnntttt bbbbaaaasssseeeedddd pppphhhhoooottttoooo qqqquuuuaaaalllliiiittttyyyy aaaasssssssseeeessssssssmmmmeeeennnntttt
• new subject area detection methods
• new global and regional features
• TTTThhhhiiiissss wwwwoooorrrrkkkk iiiissss ccccoooonnnnssssttttrrrruuuucccctttt aaaa llllaaaarrrrggggeeee aaaannnndddd ddddiiiivvvveeeerrrrssssiiiiffffiiiieeeedddd bbbbeeeennnncccchhhhmmmmaaaarrrrkkkk
ddddaaaattttaaaabbbbaaaasssseeee ffffoooorrrr tttthhhheeee rrrreeeesssseeeeaaaarrrrcccchhhh ooooffff pppphhhhoooottttoooo qqqquuuuaaaalllliiiittttyyyy aaaasssssssseeeessssssssmmmmeeeennnntttt

9. 9
写真の品質評価方法
A photo is classified as high or low quality only if eight out
of the ten viewers agree on its assessment.
写真は評価者10名の内8名が、その評価に同意した場合にのみ、
高または低品質として分類される。※評価者の主観

10. 風景植物動物夜景人物静止物構造物
10
高品質写真/低品質写真例
高品質
低品質
風景植物動物夜景人物静止物構造物
所感：低品質の写真も、「白飛び」、「黒潰れ」、「ピントが甘い」と言うことはない様子。
「プロならば、このように撮るだろうという推測」が（評価者の）判断基準に
なっていると思われる。

11. 時間
11
写真の品質
（補足） 品質とは・・・
User Experience（UX） ：品質は、実用的品質と快楽的品質に大別
実用的品質： 使いやすさ、便利さ、機能性など
快楽的品質： 魅力さ、満足感など
実用的品質○
○(本論文で
の観点と思
われる品質）
快楽的品質
エピソード的累積的UX
UX
予測的UX 一時的UX
本論文では、職業写真家以外の利用が対象と思われる。また、
撮影物との関係性は無視。

12. 12
提案手法
• CCCCoooonnnntttteeeennnntttt bbbbaaaasssseeeedddd pppphhhhoooottttoooo qqqquuuuaaaalllliiiittttyyyy aaaasssssssseeeessssssssmmmmeeeennnntttt
3. Global Features
Professionals follow certain rules of color composition and scene
composition to produce aesthetically pleasing photographs.
・3.1. Hue Composition Feature
4. Subject Area Extraction Methods
The way to detect subject areas in photos depends on photo content.
・4.1. Clarity based region detection
・4.2. Layout based region detection
・4.3. Human based region detection
5. Regional Features
We propose a new dark channel feature to measure both the clarity
and the colorfulness of the subject areas.
・5.1. Dark Channel Feature
・5.2. Human based Feature
・5.3. Complexity Feature

13. 13
提案手法
• CCCCoooonnnntttteeeennnntttt bbbbaaaasssseeeedddd pppphhhhoooottttoooo qqqquuuuaaaalllliiiittttyyyy aaaasssssssseeeessssssssmmmmeeeennnntttt
3. Global Features
Professionals follow certain rules of color composition and scene
composition to produce aesthetically pleasing photographs.
・3.1. Hue Composition Feature
写真を攻勢する色の組み合わせに着目
⇒高品質のパターンと低品質のパターンを混合ガウス分布にて学習

14. 14
提案手法
• CCCCoooonnnntttteeeennnntttt bbbbaaaasssseeeedddd pppphhhhoooottttoooo qqqquuuuaaaalllliiiittttyyyy aaaasssssssseeeessssssssmmmmeeeennnntttt
3. Global Features
Professionals follow certain rules of color composition and scene
composition to produce aesthetically pleasing photographs.
・3.1. Hue Composition Feature

15. 15
提案手法
• CCCCoooonnnntttteeeennnntttt bbbbaaaasssseeeedddd pppphhhhoooottttoooo qqqquuuuaaaalllliiiittttyyyy aaaasssssssseeeessssssssmmmmeeeennnntttt
4. Subject Area Extraction Methods
The way to detect subject areas in photos depends on photo content.
・4.1. Clarity based region detection
・4.2. Layout based region detection
・4.3. Human based region detection
A clarity based subject area detection method was proposed in [12].
Since it used a rectangle to represent the subject area and fitted
it to pixels with high clarity, the detection results were not
accurate.
This paper improves the accuracy by oversegmentation.

16. 16
提案手法
• CCCCoooonnnntttteeeennnntttt bbbbaaaasssseeeedddd pppphhhhoooottttoooo qqqquuuuaaaalllliiiittttyyyy aaaasssssssseeeessssssssmmmmeeeennnntttt
4. Subject Area Extraction Methods
The way to detect subject areas in photos depends on photo content.
・4.1. Clarity based region detection
・4.2. Layout based region detection
・4.3. Human based region detection

17. 17
提案手法
• CCCCoooonnnntttteeeennnntttt bbbbaaaasssseeeedddd pppphhhhoooottttoooo qqqquuuuaaaalllliiiittttyyyy aaaasssssssseeeessssssssmmmmeeeennnntttt
4. Subject Area Extraction Methods
The way to detect subject areas in photos depends on photo content.
・4.1. Clarity based region detection
・4.2. Layout based region detection
・4.3. Human based region detection
Hoiem et al. [9] proposed a method to recover the surface layout
from an outdoor image.
The scene is segmented into sky regions, ground regions, and
vertical standing objects as shown in Figure 6.
This Paper take vertical standing objects as subject areas.

18. 18
提案手法
• CCCCoooonnnntttteeeennnntttt bbbbaaaasssseeeedddd pppphhhhoooottttoooo qqqquuuuaaaalllliiiittttyyyy aaaasssssssseeeessssssssmmmmeeeennnntttt
4. Subject Area Extraction Methods
The way to detect subject areas in photos depends on photo content.
・4.1. Clarity based region detection
・4.2. Layout based region detection
・4.3. Human based region detection
Face detection [21] to extract faces from human photos.
For images where face detection fails, this paper uses
human detection [4] to roughly estimate the locations of
faces.

19. 19
提案手法
• CCCCoooonnnntttteeeennnntttt bbbbaaaasssseeeedddd pppphhhhoooottttoooo qqqquuuuaaaalllliiiittttyyyy aaaasssssssseeeessssssssmmmmeeeennnntttt
5. Regional Features
We propose a new dark channel feature to measure both the clarity
and the colorfulness of the subject areas.
・5.1. Dark Channel Feature
・5.2. Human based Feature
・5.3. Complexity Feature
• 特徴量Dark channelは、透明性、
彩度、色相の成分が複合。
• Dark channelの値は、画像の
ボケ度合いが高い程、Dark
channelの値も高くなる。

20. 20
提案手法
• CCCCoooonnnntttteeeennnntttt bbbbaaaasssseeeedddd pppphhhhoooottttoooo qqqquuuuaaaalllliiiittttyyyy aaaasssssssseeeessssssssmmmmeeeennnntttt
5. Regional Features
We propose a new dark channel feature to measure both the clarity
and the colorfulness of the subject areas.
・5.1. Dark Channel Feature
・5.2. Human based Feature
・5.3. Complexity Feature
写真撮影時、ライティングは品質に影響するため、ハイライト、陰影、明瞭度
（周波数成分を分析）を特徴として着目

21. 21
提案手法
• CCCCoooonnnntttteeeennnntttt bbbbaaaasssseeeedddd pppphhhhoooottttoooo qqqquuuuaaaalllliiiittttyyyy aaaasssssssseeeessssssssmmmmeeeennnntttt
5. Regional Features
We propose a new dark channel feature to measure both the clarity
and the colorfulness of the subject areas.
・5.1. Dark Channel Feature
・5.2. Human based Feature
・5.3. Complexity Feature
空間的な複雑さを測定するためセグメンテーション結果を利用
Let Ns and Nb be the numbers of super-pixels in
the subject area and the background,
||S|| and ||B|| be the areas of the subject area and
the background.

22. 22
実験結果
RRRReeeeggggiiiioooonnnnaaaallll ffffeeeeaaaattttuuuurrrreeeessss

23. 23
実験結果
GGGGlllloooobbbbaaaallll ffffeeeeaaaattttuuuurrrreeeessss

24. 24
実験結果
特徴量の組み合わせ

25. 25
実験結果
特徴量の組み合わせ場合のROC曲線

26. 従来法と比較し、提案手法は品質の評価判断結果の一致度が高い
26
まとめ
• 新たな対象領域の検出法、全体および局所の特徴量に着目した
コンテンツベースの写真の品質評価を提案
• ベンチマーク用データベースを用意
今後の課題
The integration of automatic photo categorization and quality
assessment.

27. 27
参考文献（1/2）
[1] S. Bhattacharya, R. Sukthankar, and M. Shah. A Framework for Photo-Quality Assessment and
Enhancement based on Visual Aesthetics. In Proc. ACM MM, 2010. 4, 6, 7
http://www.cs.ucf.edu/~subh/docs/pubs/mmfat05123-bhattacharya.pdf
[2] J. Carucci. Capturing the Night with Your Camera: How to Take Great Photographs After Dark. Amphoto,
1995. 1
http://www.amazon.com/Capturing-Night-Your-Camera-Photographs/dp/0817436618
[3] D. Cohen-Or, O. Sorkine, R. Gal, T. Leyvand, and Y. Xu. Color harmonization. In Proc. ACM SIGGRAPH,
2006. 3, 4
[4] N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In Proc. CVPR, 2005. 5
http://lear.inrialpes.fr/people/triggs/pubs/Dalal-cvpr05.pdf
[5] R. Datta, D. Joshi, J. Li, and J. Wang. Studying aesthetics in photographic images using a computational
approach. In Proc. ECCV, 2006. 1, 3, 6, 7
http://infolab.stanford.edu/~wangz/project/imsearch/Aesthetics/ECCV06/datta.pdf
[6] C. Grey. Master Lighting Guide for Portrait Photographers. Amherst Media, Inc., 2004. 1
http://www.amazon.co.jp/Master-Lighting-Guide-Portrait-Photographers/dp/1584281251
[7] K. He, J. Sun, and X. Tang. Single image haze removal using dark channel prior. In Proc. CVPR, 2009. 5
http://research.microsoft.com/en-us/um/people/jiansun/papers/dehaze_cvpr2009.pdf
[8] K. He, J. Sun, and X. Tang. Single image haze removal using dark channel prior. IEEE Trans. on PAMI,
2010. 5
http://research.microsoft.com/en-us/um/people/jiansun/papers/dehaze_cvpr2009.pdf
[9] D. Hoiem, A. Efros, and M. Hebert. Recovering surface layout from an image. Int’l Journal of Computer
Vision, 2007. 4, 5
http://cvcl.mit.edu/SUNSeminar/hoiem_efros_scenelayout_IJCV06.pdf

28. 28
参考文献（2/2）
[10] X. Jin, M. Zhao, X. Chen, Q. Zhao, and S. Zhu. Learning Artistic Lighting Template from Portrait
Photographs. In Proc. ECCV, 2010. 1
http://blog.scse.buaa.edu.cn/%E9%87%91%E9%91%AB.pdf
[11] Y. Ke, X. Tang, and F. Jing. The design of high-level features for photo quality assessment. In Proc. CVPR,
2006. 1, 2, 6,7
http://www.yanke.org/papers/cvpr06photo.pdf
[12] Y. Luo and X. Tang. Photo and video quality evaluation: Focusing on the subject. In Proc. ECCV, 2008. 1,
3, 4, 5, 6,7
http://www.csd.uwo.ca/courses/CS9840a/StudentPapers/ECCV2008LuoTang.pdf
[13] H. Mante and E. Linssen. Color design in photography. Focal Press, 1972. 3
[14] M. Nishiyama, T. Okabe, Y. Sato, and I. Sato. Sensationbased photo cropping. In Proc. ACM MM, 2009. 3
[15] X. Ren and J. Malik. Learning a classfication model for segmentation. In Proc. ICCV, 2003. 5
[16] A. Savakis and S. Etz. Method for automatic assessment of emphasis and appeal in consumer images, Dec.
30 2003. US Patent 6,671,405. 1
[17] M. Tokumaru, N. Muranaka, and S. Imanishi. Color design support system considering color harmony. In
Proc.IEEE International Conference on Fuzzy Systems, 2002. 3
[18] H. Tong, M. Li, H. Zhang, J. He, and C. Zhang. Classfication of digital photos taken by photographers or
home users. In Proc. PCM, 2004. 1, 2
[19] L. White. Infrared Photography Handbook. Amherst Media, Inc., 1995. 1
[20] L. Wong and K. Low. Saliency-enhanced image aesthetics class prediction. In Proc. ICIP, 2009. 1, 3
http://www.comp.nus.edu.sg/~lowkl/publications/wonglk_icip2009.pdf
[21] R. Xiao, H. Zhu, H. Sun, and X. Tang. Dynamic cascades for face detection. In Proc. ICCV, 2007.