以視覺為基礎之智慧型環境研究

本著作採用創用CC 「姓名標示」授權條款台灣3.0版

Intelligent
Video Surveillance and
Sousveillance

Wang, Yuan-Kai(王元凱)
Electrical Engineering Department,
Fu Jen Univ., Taiwan (輔仁大學電機工程系)
Email: ykwang@mail.fju.edu.tw
URL: http://www.ykwang.tw
2010/11/06

Wang, Yuan-Kai(王元凱) Video Surveillance and Sousveillance p. 2

Contents
1. Image, Vision and Intelligence
2. Intelligent Video Surveillance
3. Intelligent Video Sousveillance
4. Conclusions


1. Image, Vision, and
Intelligence
 Human's vision system


Image, Vision, and
Intelligence
in the Digital World
 Digital vision system

+
Robust Computer Vision Algorithms


Camera Based Environment
Person : Camera = 1 : N

Outside-In : Video Surveillance, Human Computer Interface
Inside-Out : Video Sousveillance, Egocentric Vision


2. Video Surveillance

 Video surveillance
 Use video camera to monitor an
area for crime investigation
 From video surveillance
to visual surveillance
 Impose video analytics
by computer vision algorithm


Video Surveillance Market
• CCTV has been a mass-product
market
• Since the 911 event, the market
is continuously increasing
（百萬美元）

Source: JP Freeman


Who’s Watching You?
 UK has the most CCTV cameras
in Europe
 4.2 million cameras which is
 20% of the world's CCTV
 1 camera for every 14 people in UK
 On average a person can be
caught on camera 200-300 times a
day


Crimes Breaking by CCTV
92年度 93年度前3季統計
案類／年度
件數人數件數人數件數人數
總計 610 689 720 796 1330 1485
竊盜 364 412 425 447 789 859
搶奪 91 69 104 104 195 173
強盜 44 53 39 65 83 118
殺人 18 36 13 20 31 56
擄人勒贖 5 15 1 2 6 17
重傷害 4 8 3 3 7 11
恐嚇取財 5 6 1 2 6 8
強制性交 5 5 3 3 8 8
其他 74 85 131 150 205 235


CCTV v.s. Crime Breaking
 監視系統對破獲刑案的助益
120000 7000
監視器數量
100000 6000
因監視器破獲件數
5000
80000
4000
60000
3000
40000
2000
20000 1000

0 0
92 93 94 95 96 97
年度

 監視器數量和監視器破獲件數兩者間呈現
正向關係


Important Crime Cases
 近年來運用路口監視器
偵破社會矚目重大案件
 白米炸彈客
 汐止市殺警奪槍案
 蠻牛千面人案
 台南國道襲警奪槍案
 新莊襲警奪槍案
 英國倫敦地鐵爆炸案
 台中角頭槍殺案


Case Study
 94年5月17日台中市蠻牛千面人案，
造成全省恐慌
 破案關鍵在於
 幾個放置毒蠻牛
的超商監視器
錄到千面人身影
 歹徒車號被
提款機監視器
清楚拍下
 動員500警員
觀看6000小時的錄影資料


CCTV Video Surveillance
Video Display & Record

VCR / DVR

Analog
Multiplexer components
Centralized
Monitoring
Video Capture

analogue analogue analogue analogue


Digital Video Surveillance
High scalibility
IPCam + analog camera
Network transmission
Remote control
Digital storgage

digital
Network
digital

digital

analogue analogue analogue
analogue


Technology Trend of
Video Surveillance

資料來源：拓璞產業研究所，2008年5月


Surveillance over IP
Paradigm shift of video surveillance
 Role from security monitoring to the personalized video contents
 Advent of the intelligent surveillance

Changes in technology & desire
1. Network
2. Video compression
3. Live images Intelligent
Surveillance
IP Surveillance
CCTV (DVR)
CCTV (VCR)
1G
2G 3G


Why Intelligent Surveillance

 Too many cameras, too few human guards
 “After only 20 minutes, human attention to
video monitors degenerates to an
unacceptable level.” (Sandia National Laboratories)


Applications of
Visual Surveillance


Visual Surveillance
Visual Surveillance = Digital CCTV + Video Analytics
Smart/Intelligent Surveillance


Video Analytics
影像擷取相機異常偵測人臉辨識查詢、過濾、聯防

Video Image Object Object Object Behavior
Capture Enhance /Event Tracking
/Event
Analysis Retrieval
Detection Recognition

強光抑制警戒線、
跌倒、人潮行為分析
路徑追蹤、流浪漢監控


IBM S3

Exploratory Computer Vision Group in IBM T.J. Watson Research Center.
http://www.research.ibm.com/ecvg/


ObjectVideo


ITRI

王元凱以視覺為基礎之智慧型環境(Vision Based Intelligent Environment) p. 24

經濟部學界科專計畫 (1/3)
 MOEA’s Technology Development
Program for Academia
 Name: Construction of Vision-
Based Intelligent Environment
 Phase I : 2004/5 ~ 2008/4
 Phase II: 2008/11 ~ 2012/10
 Involved people
 29 professors from 18 universities
 110 staffs

Fu Jen University Department of Electronic Engineering Yuan-Kai Wang


 智慧型建築
 目標：開發智慧型建築內部空間不可或缺的全
方位、主動式、機動性的智慧性視訊監控系統
A1 日夜活動式廣域安全監視
系統

A2 視訊監控中央管理系統

A3 室內突發事件分析系統
 攝影機網路  感測網路


 智慧型社區與城市
 目標：開發戶外社區及城市大範圍區域之穩定、成熟而
具產品面向的智慧性視覺監控系統

B1 人車偵測與辨識系統

B2 都會區人物追蹤系統

B3 室外事件分析與搜尋系統


Activities in the VBIE Project
 參加國際展覽
 研究技術需
 展示化：技術需能常駐展示
 系統化：大型整合展示
 指標化：技術有量化指標
 市場化：建立產業鏈地圖、政策規劃
 專利化：專利佈局、專利地圖分析
 商品化：網路行銷百餘項技術
 參與國際標準制訂(ONVIF)
 引導業界投資
 與警政機關合作


A Proposed Architecture for
Police Office (1/2)

智慧型視訊監控技術在警政治安上之可行性研究，
詹毓青，中央警察大學資訊管理所碩士論文，2009


A Proposed Architecture for
Police Office (2/2)

智慧型視訊監控技術在警政治安上之可行性研究，
詹毓青，中央警察大學資訊管理所碩士論文，2009


Demos from ISLab@FJU
 2.1 Mobile Video Surveillance
 2.2 Night Vision
 2.3 Reconfigurable Hardware for
Moving Object Detection
 2.4 Super-resolution
 2.5 Camera Tampering Detection
 2.6 System Integration


2.1 Mobil Video Surveillance

Event-driven Instant Messaging


Current Mobile Surveillance
 Mobile phone directly connects to
the camera

Mobile networks
(circuit-switched
& packet-switched) Remote monitor
Camera client

Drawbacks:
1. Not event-driven
2. High communication cost


Instant Alarm Messaging
Is Important


Our System Architecture

Keyframe Web
Selection Server

PC
Object
1. Object Info. Video
Detection
2. Keyframe Streaming
&Tracking
3. Video Clip Mobile
Phone

Video SMS
Transcoding Messaging


Our System: Web-based
Browsing Interface


Our System –
Event, Key Frame, Video Clip


Mobile Interface
SMS Message for Mobile
Event Notification video streaming


Protocol

1.監控 1.監控
新竹視訊台北
視訊
分析分析

3.
Video Analytic 確認 Video Analytic
2 3. 1
確認
2.5G
5. 2.5G 5. 3GP
3GP 確認確認
2. SMS 2. SMS

3G
4.關鍵
畫面 4.關鍵 3G
畫面
6.監控
視訊 6.監控
串流視訊
串流
輔仁大學


Performance
 Instant messaging
 PC: Within 1 seconds
 Mobile phone: ~ 5 seconds
上午 12:52 - 上午 12:00
Average time of an event
3.9 sec
上午 12:00
Face detection
Skin detection
3.4 sec
1.7 sec

下午 11:45 - 上午 12:00
間隔描述

上午 11:54 - 下午上午 11:31 - 下午 2:24
2:02 上午 11:40 - 下午 2:15
Detection & Key frame MMS
tracking Selection 3~5 sec
5.7 sec 5.1 sec

Mobile Streaming
(RTSP)
上午 12:52 - 上午 12:00
Transcoding
4.1 sec


2.2 Night Vision
 Night vision means to detect
moving objects from night images
 < 1 lux
 High-level noises
 Night vision should be important
because crimes
usually happens
at night


Moving Object Detection
 Background subtraction is the
most important method
 However, it
can not work
at night
 Bad image
quality
 Bad detection
quality


An Example Environment
 Entry of elevators in the hall
 Two challenges
 Low-light-level:
Low contrast, strong noise
 Drastic light change:
When elevators open, strong light emits
from the elevators 緊急出口警示燈 (微弱)
現場唯一光源(門外屋簷) IPCam架設位置

1 2 3 4
真實場景 by RICOH RX200 (Defaults Setting)


2.3 Reconfigurable Hardware
for Moving Object Detection

Background subtraction, ...
• 2.8 GHz Intel CPU
• Software: C/C++ FPGA
• Frame rate: 10 fps for 1 channel


Background Subtraction
Current
Frame B k + 1

Background
M k +1 ( x, y ) P k+1 Image Update
Background Image
= Pk +1 ( x, y ) − Bk ( x, y ) -
Bk
M
k + 1
Bk +1 ( x, y )
= αBk ( x, y ) + (1 − α ) Pk +1 ( x, y )
Post Processing

Motion Object Image

Speed up by (1) Circuit design, (2) Parallelization


Parallelization by Hardware
 Parallelism: 7-level pipeline
 SIMD with stream processing


Design & Results
 Hardware: Altera Cyclone II 2C35
 Design: Verilog HDL with Quartus II

Background New Frame Result
Frame rate
• Background module : 368 fps
• Whole system : 51 fps


Experimental Comparison
 PC: 2.8GHz CPU, C implementation
 FPGA can speed up 500 times
2.8G
51

CPU
FPGA
25M 10

Clock(Hz) FPS


2.4 Super-resolution
 Object images are too small
 Super-resolution is to
 Construct a high-resolution image
from low-resolution images
 Challenges: Ill-posed problem

Low-resolution High-resolution
image image


The Proposed Method
 Pervious methods
 Interpolation, reconstruction
 Feature-based learning
 Dimension Expansion
Y X
Image Space
Dimension • Maximum A Posteriori
Reduction
(MAP) Estimator
Feature Space Dimension
• Gibbs Prior
Expansion
x • 2D2 Locality
High-resolution Preserving Projection
Features


Experiments
 Four-time scale
 Low-resolution is 70×50
 High-resolution is 140×100

Ground Truth Low-resolution Bicubic Our Method
NN interpolation interpolation


Experimental Results


2.5 Camera Tampering
Detection
 Possible tampering
 Spray-painting
 Replacement
 Hit/collision
 Defocus
 Blocking
 ...


Tampering Examples (1/2)
Before After

Spray
painting

Defocus


Tampering Examples (2/2)
Before After

Replacement
(Day)

Replacement
(Night)


The System
 Sabotage
detection
before visual
surveillance
algorithms
 Server-based
solution for
large-scale
surveillance
systems

王元凱以視覺為基礎之智慧型環境(Vision Based Intelligent Environment) p. 57

2.6 System Integration
in the VBIE Project
 Smart Building
 Integrated monitoring within building
 Tracking target: person
 Smart Campus
 Long range tracking within campus
 Tracking target: car and person

Fu Jen University Department of Electronic Engineering Yuan-Kai Wang


Heterogeneous Cameras
 We use various kinds of cameras
環場攝影機

PTZ攝影機

紅外線熱像攝影機
固定式攝影機

活動攝影機
活動攝影機畫面


Smart Building (1/2)
 Characteristics
 11 techniques are integrated by top-
down design
 It follows the CMMI software
engineering method
Documents of spec requirement,
testing, ...
 A long-term test site is built


Smart Building (2/2)

NTSC一般攝影機 PTZ網路攝影機魚眼攝影機

Smart Campus (1/2)
13.機器人導引訪客導引
6.智慧型車牌偵測
12.快速人臉辨識
7.行人交錯偵測與追蹤
11.人員連續追蹤
8.特定人員追蹤與特寫
10.軌跡式人員追蹤與檢索

3.循園多攝影機轉場監控 9.多頻道視訊平滑轉場

2.多攝影機整合式視訊監控

4.多重解析度顯示及轉換
5.手勢指定物件
1.串聯式車牌辨識

14.整合平台及信號傳輸
61


Smart Campus (2/2)


3. Video Sousveillance
 Environment cameras
 Outside-in images
 Body-worn cameras
 Inside-out images


Paradigm Shift
 Evolution of computing paradigm
 Desktop computing
 Mobile computing
 Wearable computing
 Evolution of camera technology
 Desktop vision
 Mobile vision
 Wearable vision⇒ Egocentric vision


Desktop Computing
No interaction with
the environment

User is focused
on the system

stationary


Jurassic Mobile Computing

Goal: Stay in touch for mobile usage


Current Mobile Computing

 Still has drawbacks
 Hands are not free
 Screen is too small
 Need keyboard
 Not proactive


Wearable Computing
 Wear the computer on the body

Smart glasses
Smart clothing


The Hardware
 Computer: Embedded system
 Input
 No keyboard
& mouse Gestures

 But sensors
 Output Text Audio Special Purpose


Input Sensor - Glove
 Hand Gesture Interaction


Output Display
 Optical see-through HMD (Head
Mounted Display)


Add Vision Sensor
 Video see-through HMD
Wearable Vision


Why Wearable Vision (1/2)
 Capture of casual events


Why Wearable Vision (2/2)
 Game assistant


Where to Wear an Extra Eye

Mayol-Cuevas, Tordoff, Murray, "On the Choice and Placement of
Wearable Vision Sensors," IEEE Trans. SMC A, March 2009.


Where to Wear an Extra Eye


Steve Mann’s WearComp

S. Mann, "Humanistic Computing: WearComp as a New Framework
and Application for Intelligent Signal Processing", Proc. of IEEE, vol.
86, no. 11, 1998.


US Army's Land Warrior
 Helmet with OLED
display to show
map & troop
locations

http://en.wikipedia.org/wiki/Land_Warrior


Demo of ISLab@FJU

The X-Eye


The X-Eye
 Live photo management
 with more comfortable display Demo


X-Eye Components

觸控面板

顯示器移動電源
Camera
自製
USB
外殼 USB 筆電
連接線
Hub
BeagleBoard
微投影機 SD卡 USB-WIFI卡
鍵盤

USB-RS232
讀卡機
控制線滑鼠


1st Generation Prototype


Embedded Computer
USER RESET

OMAP3530 Processor
Peripheral I/O •600MHz Cortex-A8
•NEON+VFPv3
•USB Host •16KB/16KB L1
•256KB L2
•JTAG •430MHz C64x+ DSP
•32K/32K L1
•DVI-D video out •48K L1D
•32K L2
•S-Video out •Power VR SGX GPU
•64K on-chip RAM
•SD/MMC+ POP Memory
•256MB LPDDR RAM
•Stereo in/out •256MB NAND flash

•RS-232 serial1

•Alternate power

•USB 2.0 HS OTG
7.6 cm
83


Output - Pico Projector
 Very small projector
 Large screen for mobile usage
 Screen: 15”~30”
 Resolution: 480 x 320
 Brightness: 7 lumens
 Contrast ratio: 1000:1

2009~2018年微型投影機市場預估 (單位：百萬台)
（來源：DisplaySearch Pocket Projector Technology and Market Forecast Report）


Input - Camera


User Interface
 Gesture Recognition with
Bare Hand
 No keyboard & mouse

Capture egocentric images


Photo Mode

Capture Command: capture images

Switch Command: Mode switch


Manage Mode
Original Photos

Next Command

Previous Command

Switch Command 2(to photo)
88


X-Eye Hardware Architecture

2010.04.25
89


X-Eye Software Architecture


X-Eye Software Stack
Module Version Function
Gesture Command
1.0 Gesture Recognition
Module
OpenCV 1.0 Image Processing
FFMpeg 0.5.1 Audio/Video Codec
QT 4.6.2 Windows Interface
VMWare 6.5.3 Virtual OS
Ubuntu (host) 9.04 Development OS
Ubuntu (client) 9.10 Filesystem
Kernel (client) 2.6.29 Linux Kernel
91


Algorithm (1/2)
 Gaussian Mixture Model (GMM)
 Model four colors
1
N
1 ( − ( x − mi )T Covi−1 ( x − mi ))
p ( x | c) = ∑ ωi e 2

i =1 2π || Covi ||1/2

 Expectation Maximization (EM)
 Parameter estimation of GMM
E Step M Step
N
1
∑ E(z
N
1
ω p( x j | m , C ) ωit +1 = t +1
) m = ∑ E(z
t t t
)x j
E ( zij ) = Nω t +1
i i i ij i ij
i N j =1 i j =1

∑ ω tp p( x j | mtp , C tp )
=C t +1 1
t +1 ∑
N
E ( zij )[( x j − mit +1 )( x j − mit +1 )T ]
p =1
N ωi
i
j =1

92


Algorithm (2/2)
 Color Identification
 c
ˆ =
arg max P( x | c j ), j 1 ~ k
cj

 Performance optimization by Look
Up Table (LUT) for real-time
 Gesture Recognition
 Four gestures: capture, switch,
next, previous,

93


Smart Camera
 Egocentric vision needs
small and smart cameras

FJUCam1
FJUCam2

Image Image Image
Capturing Processing Recognition


FJUCam1 - Hardware
• Weight: 35gm
• Power sources: •Size:
• 5V DC current 6 x 4.5 x 5 (cm)
• 5V Cell Battery (W x H x D)
• Power
consumption:
1W

Three Modules
1. Main board, 2. Lens module
3. Storage module


FJUCam1 - Software
 Development environment
 C Language
 PC Windows + Cygwin + GCC
 cc3 library (open source developed
by CMU)


FJUCam1 - Face Detection
 The Adaboost algorithm
 Proposed by Viola and Jones in 2001
 Cascaded weak classifiers(21 cascades)
 Algorithm refinement
 Reduced to 5 cascades
 Fixed-point arithmetic
 Stream processing for only 64KB
memory utilization

Image
FJUCam Display
Face Detection


FJUCam2 (1/5)
 Cons of FJUCam1
 Low speed, small memory
 Low image resolution
 Next-generation FJUCam
 Adopts multicore technology
CPU + DSP (MPSoC chip)
99/12 Image
RAM
128MB Resolution
FJUCam2 VGA

98/2
64KB FJUCam1 CIF
Processor
60MHz 600MHz
Speed


FJUCam2 (2/5)
 TI OMAP3530@ Cortex-A8 600MHz
@ DSP C64x+ 412MHz
@ PowerVR SGX 530
 Integrated L1 memory for ARM (16kB I-
Cache, 16kB D-Cache, 256kB L2)
 256MByte low power mobile DDR
 512MByte NAND Flash
 Mini HDMI interface
 Serial port(UART/RS232) x 2
 PCB dimensions
 USB port (Host)  55mm × 55mm
 USB port (Client)  Outside dimensions
 Micro SD card slot (L×W×H)
 USB power Back side – Expansion  63mm 59mm×13 mm
 GPIO socket for extra peripherals
 Weight ： 20 grams


FJUCam2 (3/5)
Software (on board)
 Angstrom with Linux
kernel 2.6.32 Camera Input using ARM
 VCam Image/Video OpenCV+FFMpeg
DSP
processing demo Image/Video
 Using DSP and the Processing
integrated vision
libraries GUI Output using QT
 Other supported
embedded OS: easy interface &
powerful computing
Android

 We are compiling “VCam Laboratory Manual”
which will be made publicly at the end of 2010


FJUCam2 (4/5)
 Algorithms going to be
developed for FJUCam2
 Color tracking
 Gesture recognition
 Face tracking and recognition
 Human/hand/finger detection
 Event detection
 Video summarization
 Distributed vision processing


FJUCam2 (5/5)
 Possible applications
 Wearable computing
 Augmented reality
 Robotic vision
 Visual surveillance
 Medical applications for
patient monitoring


4. Conclusions
 Image processing
and computer vision
 is funny, is cool
 Has been realistic,
comes into real life
 It is time to study it
 Hardware is small, cheap, and
wearable
 More robust algorithms


How to Study
 Basic courses
 Programming skills
 Engineering Mathematics
 Signal and Systems
 Digital Signal Processing
 Advanced courses
 Digital Image Processing
 Computer Vision
 Pattern Recognition
 Artificial Intelligence


Activities
 中華民國影像處理與圖形識別學會(IPPR)
 電腦視覺、圖學暨影像處理研討會(CVGIP)
 IEEE (國際電機電子學會)
 Transactions on
 Image Processing
 Pattern Analysis and Machine
Intelligence
 Medical Imaging
 Conferences
 ICIP, ICME, ICPR, ...


The End

Free for Any Questions

Wang, Yuan-Kai(王元凱) Video Surveillance and Sousveillance p.

本簡報授權聲明
 此簡報內容採用 Creative Commons 「姓名標示 - 非商
業性台灣 3.0 版」授權條款
 歡迎非商業目的的重製、散布或修改本簡報的內容，但
請標明： (1)原作者姓名：王元凱； (2)圖標示：
 簡報中所取用的部份圖形創作乃截取自網際網路，僅供
演講者於自由軟體推廣演講時主張合理使用，請讀者不
得對其再行取用，除非您本身自忖亦符合主張合理使用
之情狀，且自負相關法律責任。

以視覺為基礎之智慧型環境研究

Recommended

Recommended

More Related Content

Similar to 以視覺為基礎之智慧型環境研究

Similar to 以視覺為基礎之智慧型環境研究 (18)

More from IEEE International Conference on Intelligent Information Hiding and Multimedia Signal Processing

More from IEEE International Conference on Intelligent Information Hiding and Multimedia Signal Processing (16)

Recently uploaded

Recently uploaded (20)

以視覺為基礎之智慧型環境研究