Intelligent Video Surveillance with Cloud Computing

智慧型視訊監控
與雲端運算技術
Wang, Yuan-Kai (王元凱)
Electrical Engineering Department,
Fu Jen Univ. Taiwan (輔仁大學電機工程系)
Email: ykwang@mail.fju.edu.tw
URL: http://www.ykwang.tw
2010/11/24

本著作採用創用CC 「姓名標示」授權條款台灣3.0版

Wang, Yuan-Kai (王元凱) Cloud Vision p. 2

What about this Talk
 Intelligent video surveillance for
 Public safety
 Home security
 High performance cloud computing
 GPGPU
 Algorithm parallelization
 VSaaS: video surveillance as a service
 Mobile cloud computing
 All demo examples are done by our
ISLab(www.islab.tw)


Who Am I
 輔仁大學
 電機工程系副教授，資訊中心前主任
 中央大學網路學習科技研究所兼任副教授
 中華民國影像處理與圖形辨識學會
理事、監事、秘書長
 中華民國大專校院資訊服務協會
理事
 Who Is Who in the World
Who Is Who in Science and Engineering
Who Is Who in Asia


What Did I Do
 經濟部學界科專計畫(2004~2012)
 以視覺為基礎之智慧型環境建構
 國科會計畫(2010~2011)
 以視覺為基礎之睡眠障礙分析
 產學合作
 智慧型行動視訊監控系統
 智慧型嵌入式攝影機之研究(I)(II)
 不受限環境人臉辨識系統於機器人之應用
 電腦視覺於家庭照護之應用
 智慧型運輸系統之研究


Contents
1. Intelligent Video Surveillance
2. Cloud Computing and GPGPU
3. VSaaS: Video Surveillance as a
Service
4. Conclusions and Discussions

6

1. Intelligent
Video Surveillance

 1.1 Video Surveillance
 1.2 Intelligent Video Surveillance
 1.3 Demos


1.1 Video Surveillance

 Video surveillance
 Use video camera to monitor an area for
crime investigation
 Two applications
 Police and public safety
 Home seciruty


Video Surveillance Market
• CCTV has been a mass-product
market
• Since the 911 event, the market
is continuously increasing
（百萬美元）

Source: JP Freeman


Who’s Watching You?
 UK has the most CCTV cameras in
Europe
 4.2 million cameras which is
 20% of the world's CCTV
 1 camera for every 14 people in UK
 On average a person can be caught on
camera 200-300 times a day


Crimes Breaking by CCTV
92年度 93年度前3季統計
案類／年度
件數人數件數人數件數人數
總計 610 689 720 796 1330 1485

竊盜 364 412 425 447 789 859

搶奪 91 69 104 104 195 173

強盜 44 53 39 65 83 118

殺人 18 36 13 20 31 56

擄人勒贖 5 15 1 2 6 17

重傷害 4 8 3 3 7 11

恐嚇取財 5 6 1 2 6 8

強制性交 5 5 3 3 8 8

其他 74 85 131 150 205 235


CCTV v.s. Crime Breaking
 監視系統對破獲刑案的助益
120000 7000
監視器數量
100000 6000
因監視器破獲件數
5000
80000
4000
60000
3000
40000
2000
20000 1000

0 0
92 93 94 95 96 97
年度

 監視器數量和監視器破獲件數兩者間呈現
正向關係


Important Crime Cases
 近年來運用路口監視器
偵破社會矚目重大案件
 白米炸彈客
 汐止市殺警奪槍案
 蠻牛千面人案
 台南國道襲警奪槍案
 新莊襲警奪槍案
 英國倫敦地鐵爆炸案
 台中角頭槍殺案


Case Study
 94年5月17日台中市蠻牛千面人案，
造成全省恐慌
 破案關鍵在於
 幾個放置毒蠻牛
的超商監視器
錄到千面人身影
 歹徒車號被
提款機監視器
清楚拍下
 動員500警員
觀看6000小時的錄影資料


Home Security
 Video surveillance for homecare
 Use CCTV/IP cameras to monitor
homes
 Like千里眼@中華電信
 Increasing demands for
 Burglar care
 Child care
 Pet care
 Elder care
 Community/neighborhood care
 Health care
 Sleep care


A Survey @ Taiwan 2009
 台灣經濟部通訊產業推動小組委託
資策會進行調查
 台北、台中、高雄等
3大都會區18歲以上
，對社區大樓或居家
服務提供多媒體娛樂
與商務或智慧生活
應用有興趣的民眾
 668份有效問卷


Summary of the Survey
 安全監控是3群年齡層都最重視的應
用
 第二重視的應用：隨年齡有所不同


Burglar Care
 Care about burglar events when
You are at home
Nobody is at home (Vacation home)


Child Care - Baby Monitor
 Care about baby's
 Wake-up
 Crying Audio/video
 Suffocation Wireless Device


Child Care – Nanny Monitor
 Nanny may not appropriately
take care of the baby
 Use a hidden camera for monitoring


Elder Care
 Home : aging in place
 Monitoring the elder by cameras
 Fall detection by cameras

 Institutional and nursing homes


Pet Care

• While the owner is away from home
• A surveillance camera helps assure
• Pet's well-being
• Dogs and cats don't cause damage


Community Care
 Parking lot, fence  Dangerous
public place
ex., swimming pool

 Access control


Health Care
 Video conferencing at home
with doctors for diagnostics


Sleep Care
 Feel not well for sleeping?
 Go to sleep at hospital's sleep center
 Bio-signal and video are recorded
 OSA: Obstructive Sleep Apnea
 Why not sleep at home?

CPAP


Video Surveillance
Generations
Paradigm shift of video surveillance
 Role from security monitoring to the personalized video contents
 Advent of the intelligent surveillance

Changes in technology & desire
1. Network
2. Video compression
3. Live images Intelligent
Surveillance
IP Surveillance
CCTV (DVR)
CCTV (VCR)
1G
2G 3G


CCTV Video Surveillance
Video Display & Record

VCR / DVR

Analog
Multiplexer components
Centralized
Monitoring
Video Capture

analogue analogue analogue analogue


Digital Video Surveillance
High scalibility
IPCam + analog camera
Network transmission
Remote control
Digital storgage

digital
Network
digital

digital

analogue analogue analogue
analogue


Visual Surveillance
Visual Surveillance = Digital CCTV + Video Analytics
Smart/Intelligent Surveillance


1.2 Video Analytics
 Intelligent video surveillance
 Use video camera to monitor an area for
 Crime prevention
 Intelligent ICT service
 From video surveillance
to visual surveillance
 Impose video analytics
by computer vision algorithm


Why Visual Surveillance

 Too many cameras, too few human guards
 “After only 20 minutes, human attention to video
monitors degenerates to an unacceptable level.”
(Sandia National Laboratories)


Applications of
Visual Surveillance


Video Analytics
影像擷取相機異常偵測人臉辨識查詢、過濾、聯防

Video Image Object Object Object Behavior
Capture Enhance /Event Tracking
/Event
Analysis Retrieval
Detection Recognition

影像強化警戒線、路徑追蹤
流浪漢監控跌倒、人潮行為分析


IBM S3

Exploratory Computer Vision Group in IBM T.J. Watson Research Center.
http://www.research.ibm.com/ecvg/


ObjectVideo


ITRI


VBIE科專計畫 (1/3)
 經濟部學界科專計畫
 以視覺為基礎之智慧型環境建構
Construction of Vision-Based Intelligent
Environment (VBIE)
 第一期4年計畫: 2004/5 ~ 2008/4
 第二期4年計畫: 2008/11 ~ 2012/10
 參與人力
 29 位教授，來自18 所大學與研究機構
 110 位研究人員


經濟部學界科專計畫 (2/3)
 智慧型建築
 目標：開發智慧型建築內部空間不可或缺的全
方位、主動式、機動性的智慧性視訊監控系統
A1 日夜活動式廣域安全監視
系統

A2 視訊監控中央管理系統

A3 室內突發事件分析系統
 攝影機網路  感測網路


經濟部學界科專計畫 (3/3)
 智慧型社區與城市
 目標：開發戶外社區及城市大範圍區域之穩定、成熟而
具產品面向的智慧性視覺監控系統

B1 人車偵測與辨識系統

B2 都會區人物追蹤系統

B3 室外事件分析與搜尋系統


Activities in the VBIE Project
 參加國際展覽
 研究技術需
 展示化：技術需能常駐展示
 系統化：大型整合展示
 指標化：技術有量化指標
 市場化：建立產業鏈地圖、政策規劃
 專利化：專利佈局、專利地圖分析
 商品化：網路行銷百餘項技術
 參與國際標準制訂(ONVIF)
 引導業界投資
 與警政機關合作


Current Solutions of Police


Equipments in the Pole


Cameras on the Poles

固定式攝影機及密閉式雙層鋁製防護罩
固定式攝影機及密閉式雙層鋁製防護罩


DVR in Closed Case

監視錄影設備


A Proposed Architecture for
Police Office (1/2)

鄧紹華、詹毓青，智慧型視訊監控技術在警政治安上之可行性研究，
中央警察大學資訊管理所碩士論文，2009


A Proposed Architecture for
Police Office (2/2)

鄧紹華、詹毓青，智慧型視訊監控技術在警政治安上之可行性研究，
中央警察大學資訊管理所碩士論文，2009


Current Solutions
for Home Security
 Standalone system with three
components
 Camera: CCTV, IPCam, SmartCam
 Storage: DVR
(Digital Video Recorder)
2. DVR
 View: PC,
Smart Phone
3. Viewer

1.Camera


Problems of Current Solution
 Purchase, maintenance, update of
 Facility & Storage
 Cabling

Rental service could be better
than buying a home security system
Like: electricity, gas, cable TV, ...

⇒ Cloud Computing


Problems of Current Solution
 Time to watch is unknown
 Real-time event alert is helpful
 Lack of smart sensors

Instead of video recording in the cloud,
How about video analytics?

⇒ Intelligent Video Surveillance
by Cloud Computing


Evolution to Cloud
監控儲存管理分析虛擬化

智慧分析
IP網路
監控
IP視訊監控

DVR
類比CCTV
(DVR儲存)
Time Lapse 數位
VCR
類比CCTV
(VCR儲存)
CCTV camera
VSaaS
類比
類比CCTV
時間
1950~ 1980 1990 2000 2010 2011
部份資料來源：拓璞產業研究所，2008年5


1.3 DEMOS
 1. Camera Tampering Detection
 2. Tripwire
 3. Face Recognition
 4. Smart Building


DEMO 1:
Camera Tampering Detection
 Possible tampering
 Spray-painting
 Replacement
 Hit/collision
 Defocus
 Blocking
 ...


Motivation
 For a large video surveillance
installation
 How to ensure every camera is OK?
 A case study
 A police IDC with more than 600
cameras
 A person is responsible for checking
 One Day One check in morning
 Takes about 40 minutes
 Then you never know it functions well
until tomorrow


Replacement
 Intentionally by human

 False alarm: Earthquake

5
3


Spray Painting

5
4


Defocusing

 False alarm:
water drops

5
5


Blocking – Full Occlusion

 False alarm: passing of large objects

556
6


Blocking – Partial Occlusion
貼紙廣告傳單寶特瓶

包裝盒雨傘塑膠玩具

557
7


The System
 Sabotage detection before visual
surveillance algorithms
 Server-based solution for large-scale
surveillance in a police IDC
 > 400 cameras
 Advantages
 No more daily check
with 40 minutes
 Alert functions
24 hours


Architecture (Current)

Our System
...

攝影機異常合成影像
偵測系統偵測紀錄
...

影像分配器控制中心
...

整段影像或
DVR
紀錄影像
5
9


Demo 2: Tripwire
 Tripwire detection is an important
application for proactive crime
prevention
 Restricted ingress and egress
 Unidirectional or bidirectional
 Precondition：
Precise Moving
Object Detection d(Position)

and Tracking


Moving Object Detection in
Day-and-Night
 Background subtraction is the most
important method
 However, it can not work at night
 Image processing techniques must be
added
 Night vision should be important
because crimes
usually happens
at night


Can't do tripwire detection in these two cases


Example with GUI
 Various Input
Interface:
 Webcam
 IP Camera
 Analog Camera
 Customize Multi
Tripwire
 Self-defining
direction
 Active alarm &
log(xml, video)


Applications
 Traffic control

警戒線：對由下而上的車
輛進行警報與計數


DEMO 3: Face Recognition
 Video-based method in
unconstrained environment
Training image Test video


Applications
 門禁管理
 生物特徵認證
 相簿管理
 人員計數

http://picasa-readme.blogspot.com/2009/09/picasa-35-now-with-name-tags-build-7967.html


DEMO 4: Smart Building
 Integrated monitoring within building
 Tracking target: person
 11 techniques are integrated by top-
down design
 A long-term test site is built


Heterogeneous Cameras
 We use various kinds of cameras
環場攝影機

PTZ攝影機

紅外線熱像攝影機
固定式攝影機

活動攝影機
活動攝影機畫面


The Scenario

NTSC一般攝影機 PTZ網路攝影機魚眼攝影機

2. Cloud Computing
and GPGPU
 2.1 Cloud Computing
 2.2 High Performance Computing
 2.3 HPC Cloud by GPGPU
 2.4 CUDA
 2.5 Parallel Computing
 2.6 Demos


2.1 Cloud Computing
 Put computation and data in the cloud
1st Cloud Nth Cloud
Computing Storage Application Computing Storage Application

Web Server Web Server

Provider 1
..... Provider N
Internet

End-User PC Notebook Phone/PDA


Technologies for
Computing Cloud
 It needs a data center in the cloud
with high computing resources
 CPU clusters
(Server Farm)
 Storage: SAN
 Fiber
networking
 Also it needs
 Virtualization
 Web Service


Services in Cloud Computing
 AaaS Architecture as a Service
 BaaS Business as a Service


CaaS
DaaS
Computing as a Service
Data as a Service
XaaS
 DBaaS Database as a Service
 EaaS Ethernet as a Service
 FaaS Frameworks as a Service
 GaaS Globalization or Governance as a Service
 HaaS Hardware as a Service
 IMaaS Information as a Service
 IaaS Infrastructure or Integration as a Service
 IDaaS Identity as a Service
 LaaS Lending as a Service
 MaaS Mashups as a Service
 OaaS Organization or Operations as a Service
 SaaS Software or Storage as a Service
 PaaS Platform as a Service
 TaaS Technology or Testing as a Service
 VaaS Voice as a Service
 VSaaS Video Surveillance as a Service


Cloud Architecture
User-Level
應用
Social Computing, Enterprise, ISV,…

User-Level 程式語言
Web 2.0 介面, Mashups, Workflows, …
S
Middleware

a P 控制
a a I Core
Qos Neqotiation, Ddmission Control,
Pricing, SLA Management, Metering…
S a a Middleware
虛擬化
S a VM, VM management and Deployment
S
硬體設施
System Level Infrastructure: Computer, Storage,
Network


Service Hierarchy
More about
SaaS for
SaaS
multimedia

PaaS

IaaS

Hardware
Data Storage
Clusters Others
Centers s


SaaS for Multimedia
 Challenges
 Can IaaS afford multimedia computing,
such as intelligent video surveillance?
 Digital signal is continuously fed into a
massive data set
 Intensive computing power is necessary
for ISP(Image and Signal Processing)

We need High Performance Computing (HPC)


2.2 High Performance
Computing (HPC)
 Classically it is called
supercomputing
 Supercomputing has been combined
with networking technology and
evolved into HPC
Virtualization Utility Cluster
Service Parallel
Model Distributed
High
Cloud Web 2.0 Grid Performance
Computing Computing


Why HPC?
 It is used for
 Massive data processing
 Intensive computation
 Example application: video processing
 Intelligent video surveillance,
Computer vision, ...
 Complexity of video processing
 One video: 1 Megapixels, 30fps, 100 flops
per pixel
 ⇒ 3 Gigaflops per video


New Approaches for HPC
 Cluster/distributed computing
 MAP-REDUCE (Google)
 MPI
 Multi-processing computing
 Multicore CPU
 GPGPU
 FPGA/DSP
 CellBE
 Combination
 GPU cluster


HPC Architecture of IaaS
The end user sees a
finished application
Compute
Power
End user Load
Firewall
Balancer
Improved
servers by
Virtual Machine is
deployed and started co-processing:
Virtual
Machine From multicore
Automation to manycore
Software Virtual The virtual machine is
Owner Machine uploaded to storage
and configures to use
storage.
Storage

IaaS
Vendor


Two Approaches to HPC
 Symmetric multiprocessing (SMP)
 Multicore CPU,
GPGPU,
multicore DSP
 Homogeneous computing
 Asymmetric multiprocessing (AMP)
 CPU+GPGPU,
CPU+FPGA,
CPU+DSP
 Heterogeneous computing


SMP: Multicore CPU
 Two or more CPUs on a chip
Core i7: Quad cores

Intel Core 2 Duo

One
Processor

With multiple
execution Cores


SMP: Multicore CPU
 Increased compute density
 Increase performance with
energy saving


SMP: GPGPU
 GPGPU has more homogeneous cores
than CPU
 120 ~ 512 cores
 GPGPU is more powerful than
multicore CPU
 For DSP processing
 Vendors: nVidia, ATI, Intel


AMP: Co-processing
 Multicore CPU + Stream processor

CellBE ISP
IO FPGA (Image and
Signal
GPGPU Processing)


CellBE (1/2)
 By IBM, Sony, Toshiba alliance from
2000
 One PowerPC core, 8 compute
cores (SPEs)
 Sony
PS3


CellBE (2/2)
 Toshiba:
SpursEngine
 IBM:
 Blade server QS22
 RoadRunner


FPGA+CPU (1/2)
 Ex.: XtremeData Inc. XD1000/2000


FPGA+CPU (2/2)


GPGPU+CPU
 GPU has been a co-processor of CPU
 GPU helps improve graphics processing
 CPU focuses SMP + AMP
on IO
control


GPU Cluster (1/2)
 GPU cluster is a brand new
architecture for HPC
 GPU+CPU as a compute node
 Many compute nodes are clustered as a
unified HPC computer
 Early experimented from 2010
 UIUC NCSA: AC cluster
 Maryland CPU-GPU cluster
 CSIRO GPU cluster
 Taiwan: NCHC, NTU
 More researches are actively studied


GPU Cluster (2/2)
 Example:

Maryland
CPU-GPU cluster

• If utility, load balancing, and virtualization can
be incorporated into the GPU cluster
architecture
• Cloud computing can be more powerful
•Especially for intelligent video surveillance


Multi-Everything
 Parallelism everywhere with SW &
HW assistance
Virtual •New Parallel Languages
Container OS OS OS •New Threading tools
VMM: •Thread Management &
Abstraction layers
Operating •Transactional memory
App App App •Auto-threading compilers
System:
•Auto-threading hardware

Application: Thread Thread Thread

Code/Data Code/Data Code/Data
Thread: Segment Segment Segment

Chip: CPU GPU DSP


2.3 HPC Cloud by GPGPU
 HPC cloud is
 Cloud computing with
HPC architecture and performance
 HPC As A Service (HPCaaS):
HPC in the cloud


Ex.: Amazon HPC
 Five families of Amazon EC2
 Standard, Micro, High-CPU, High-
Memory, Cluster Compute
 Cluster Compute
 Large computational problems for
business and researchers
 Provide Amazon Elastic MapReduce
service (Hadoop)
 Cooperated with researchers at the
Lawrence Berkeley National
Laboratory to develop the HPC cloud
offering


HPC Cloud by GPGPU
 In my viewpoint
 GPGPU can be an enabling technology
for HPC cloud
 For single computer/one compute node
 GPU can be used to greatly
enhance performance
 Personal supercomputer (nVidia)
 If one compute node is not enough
 Clustering many
compute nodes


Advantages of GPGPU for
Cloud Computing
 Compared with PC-based cloud
computing
 Improve space density
 Improve compute performance
 Compared with supercomputer
 Application/software portability
 Power saving
 Similar teraflops performance


Challenges in HPC Cloud
 Virtualization
 Software development
 No more free lunch for software
 Software has to be re-written
 Multi-threading
 Programming model
 Multi-threading
 SPMD(Single Program Multiple Data)

Let's take GPGPU/CUDA model as an example


2.4 CUDA
 ex.: nVidia Fermi/GT200
 Tesla 20-series/10-series

Multicore Many-core


Memory Hierarchy
 Good for
locality
principle
 Temporal
locality
 Spatial
locality
 Good for
ISP


CUDA
 Parallel programming for nVidia's
GPGPU
 Use C/C++ language
 Java, Fortran, Matlab, ...
 When executing CUDA programs,
the GPU operates as coprocessor to
the main CPU


CUDA Hardware
Environment: CPU+GPU
 GPU
PCI-E
 Organizes, interprets, and CPU GPU
communicates information
 GPU
 Handles the core processing on large quantities
of parallel information
 Compute-intensive portions of applications
that are executed many times, but on different
data, are extracted from the main application
and compiled to execute in parallel on the GPU


Processing Flow on CUDA
Main
2 Memory
CPU 3
Copy processing 5 Instruct the
data Copy the processing
result
4
1 Memory
for GPU Execute
Allocate parallel in
device memory each core

6
Release
device memory


GPUs for Multimedia

3.5X 10 X 10 X
PowerDirector7 Ultra CUDA JPEG Decoder DivideFrame GPU Decoder

26 X 10 X
Hyperspectral Image GPU Decoder Motion Estimation for
Compression on (Vegas/Premiere) - H.264/AVC on
NVIDIA GPUs Using the Power of Multiple GPUs
NVIDIA Graphic Card to Using NVIDIA CUDA
Decode H.264 Video Files


GPUs for Computer Vision(1/2)

87 X 26 X 200 X 100 X
CUDA SURF – A Real-time Leukocyte Tracking: Real-time Spatiotemporal Image Denoising with
Implementation for SURF ImageJ Plugin Stereo Matching Using the Bilateral Filter
TU Darmstadt University of Virginia Dual-Cross-Bilateral Grid Wlroclaw University
of Technology

85 X 100 X 8X 13 X
Digital Breast Fast Optical Flow on GPU A Framework for Efficient Accelerating Advanced MRI
Tomosynthesis At Video Rate for Full HD and Scalable Execution of Reconstructions
Reconstruction Resolution Domain-specific Templates University of Illinois
Massachusetts General Onera On GPU
Hospital NEC Labs, Berkeley, Purdue


GPUs for Computer Vision(2/2)

20 X 13 X 109 X 263 X
GPU for Surveillance Fast Human Detection with Fast Sliding-Window GPU Acceleration of Object
Cascaded Ensembles Object Detection Classification Algorithm
Using NVIDIA CUDA

300 X 10 X 45 X 3X
Audience Measurement – Real-time A GPU Accelerated Canny Edge Detection
Real-time Video Analysis Visual Tracker by Evolutionary
for Counting People, Face Stream Processing Computer Vision System
Detection and Tracking


Programming Challenges of
CUDA
 We have to manually parallelize the
algorithm
 We need expertise both in
 Algorithms of image and signal processing
 Filtering, frequency analysis, compression,
feature extraction, recognition, ...
 Theory, tools and methodology of parallel
computing
 Communication, synchronization, resource
management, load balancing, debugging, ...


2.5 Parallel Computing
 Serial
Computing

 Parallel
Computing


What is Parallel Computing?
 Solving a problem simultaneously with
multiple processing elements (PE’s),
or multiple cores
 Dividing the problem into independent parts
 Each PE executes its part concurrently with
the other PE’s


Parallelization
 Multicore/Multi-threading
 Data Parallelization
 Data distribution
 Parallel convolution
 Reduction algorithm
 Amdahl’s law
 Memory Hierarchy Management
 Locality principle
 Program accesses a relatively small portion of the address space at any instant of time


Four Steps to Create
a Parallel Program
Partitioning

D O
e A r
c s c
o s p p h p p M
m i 0 1
e 0 1
a p0 p1
p g s p
o n t p
s
i m p p r
p p i p2 p3
e 2 3
a n
t t 2 3 g
i n i
o t o
n n
Sequential Tasks Parallel Processors
Processes program
computation


Methods to Decompose a
Problem
 Two basic approaches to partition
computational work
 Domain decomposition GPGPU
 Partition the data used
Cooperate
in solving the problem
 Function decomposition CPU
 Partition the jobs (functions)
from the overall work (problem)


Domain Decomposition (1/3)
 An image example
 It is 2D data
 Three popular partition ways


 Domain data are usually processed
by loop
 for (i=0; i<height; i++)
for (j=0; j<width; j++)
img2[i][j] = RemoveNoise(img1[i][j]);
j
i

The X-ray image
of a circuit board
Original image(img1) Enhanced image(img2)


j
i A three-block partition
example OpenMP
 // Thread 1 CUDA(SPMD)
for (i=0; i<height/3; i++)
img2[i][j] = RemoveNoise(img1[i][j]);
 // Thread 2
for (i=height/3; i<height*2/3; i++)
fork(threads)
subdomain 1 subdomain 2 subdomain 3
i=0 i=4 i=8 img2[i][j] = RemoveNoise(img1[i][j]);
i=1 i=5 i=9  // Thread 3
i=2 i=6 i=10
i=3 i=7 i=11
for (i=height*2/3; i<height; i++)
join(barrier) img2[i][j] = RemoveNoise(img1[i][j]);


Function Decomposition (1/2)
 Functional decomposition
is a way to exploit task
parallelism
 Task parallelism by threading
 Fork operation creates multiple
threads
 Each thread runs on a PE
 A thread may perform the same
or different operations on the
same or different data


Function Decomposition (2/2)
 Ex.: Image Streaming
 An image capture & streaming program
 Partitioned into 3 tasks (functions)
 Each function is set to be a thread,
running in a PE, or CPU core

Compress
Capture Images, Stream the Internet
Images Process Images
Images
Input Output
Computer


Issues with Parallelization
 Good parallel programs
 Execute correctly
 with good speedup
 Ideal speedup by Amdahl's law
 Speedup = N if you has N cores
 However, no ideal speedup exists
 Because parallel overhead, such as
 Data communication
 Data dependencies and synchronization
 Other issues: design overhead
 No free lunch for software development


2.6 DEMOS
 1. FPGA for Moving Object Detection
 2. GPGPU for Image Restoration
 3. GPGPU for Feature Extraction


DEMO 1: FPGA for
Moving Object Detection

Background subtraction, ...
• 2.8 GHz Intel CPU
• Software: C/C++ FPGA
• Frame rate: 10 fps for 1 channel


Background Subtraction
Current
Frame B
k + 1

Background
M k +1 ( x, y ) P k+1 Image Update
Background Image
= Pk +1 ( x, y ) − Bk ( x, y ) -
Bk
M
k + 1
Bk +1 ( x, y )
= αBk ( x, y ) + (1 − α ) Pk +1 ( x, y )
Post Processing

Motion Object Image

Speed up by (1) Circuit design, (2) Parallelization


Parallelization by Hardware
 Parallelism: 7-level pipeline
 SIMD with stream processing


Experimental Comparison
 PC: 2.8GHz CPU, C implementation
 FPGA: 25MHz, Verilog HDL
2.8G
51 Speedup
≈ 100 * 5
= 500
CPU
FPGA
25M 10

Clock(Hz) FPS


DEMO 2: GPGPU for
Image Restoration
 Restore and enhance an image
 Its complexity is high for large images

Original Complexity: Restored
O(N2)


The Method
CPU GPGPU
Copy Data to Illumination
GPGPU Estimation

Illumination
Compensation

Reduction
Algorithm

Copy Data Contrast
from GPGPU Enhancement

Intel Core 2 - 2 cores Tesla C1060 - 240 SPs
(3.0GHZ) (1.296GHZ)


Parallelization by GPGPU
 Multicore/Multi-threading
 Tesla C1060 : 240 SP (Stream Processor)
 CUDA: , Thread , Block , Grid
 Data Parallelization
M pixels PE data time
1 pixels pixels 1 pixels t0 t1 t2 t3 t4 t5
A(0) A(0)+A(1) A(0)+A(1)+A(2)+A(3) sum
0
1 A(1)

{
M PE i PE i 2 A(2) A(2)+A(3)
pixels pixels pixels pixels A(3)
3
{

4 A(4) A(4)+A(5) A(4)+A(5)+A(6)+A(7)
pixels 5 A(5)
1 pixels 1 pixels 6 A(6) A(6)+A(7)
pixels 7 A(7)


Experimental Results (1/2)

Original images CPU results GPGPU results


Experimental Results (2/2)

Original images CPU results GPGPU results


Execution Time
18000

16000

14000

12000
CPU-SSR
ms

10000 CUDA-SSR
CPU-MSR
8000
CUDA-MSR
6000 CPU-MSRCR
CUDA-MSRCR
4000

2000

0
256 x 256 512 x 512 1024 x 1024 2048 x 2048

Image Sizes


GPGPU Speedup over CPU
35

30
30x
speedup
Speedup

25

20
CUDA-SSR

15 CUDA-MSR
CUDA-MSRCR
10

5

0 Image
256 x 256 512 x 512 1024 x 1024 2048 x 2048 Sizes


DEMO 3: GPGPU for
SIFT Feature Extraction
 SIFT
 Scale Invariant Feature Transform
 Invariance of feature points
 Translation
 Rotation
 Scale


Applications of SIFT
Object recognition/tracking
Image retrieval
Autostitch


Parallelize SIFT by GPGPU

Intel Q9400 Geforce GTS 250
Quad cores 128 SPs
(2.66GHz) (1.836GHz)


Experimental Results
CPU GPU


Execution Time

CPU:
10 seconds
in average
ms

GPGPU:
0.8 seconds
in average


Speedup

13x speedup in average

3. VSaaS
 Computing Cloud
 Mobile Cloud
 Demo


VSaaS
 Video Surveillance as a Service
 Pay-as-you-go (Rental) Service
 Bringing SaaS to physical security,
hosted & managed video
 Potential to make video surveillance
cheaper and easier to deploy and use
 Re-shape the home and small business
market segments for video surveillance
 Only few systems started from 2010


Advantages of VSaaS
 Economics
 Reduced TCO
(Total Cost of Ownership)
 Eliminated time and effort
 Efficiency
 Fast and flexible deployment
 Easy scalability and upgrade
 Access anytime anywhere


Architecture
DVR by SaaS

 SaaS
Camera

 VSaaS
 Camera should
be place in local
places
 DVR in clouds


A VSaaS Example
 Securitystation in UK
 Functions
 Live viewing, recording, storage,
playback and system management
 Web interface
 Mobile access with iPhone or iPad
 Plug and Play (PnP) support of IP
cameras
 No video analytics currently
 Rental service: £5/camera/month


A VSaaS Exmaple
 Securitystation in UK

Currently,
1.VSaaS can
be seen only
in industry
2.Video analytics
has not yet been
put into VSaaS


Cloud Family
 Impact of Cloud Computing
 Death of the desktop
 Transfer to thin client
1. 雲端的「雲」 : Computing cloud
Public cloud
Private cloud
Enterprise cloud
Hybrid cloud

2. 雲端的「端」 : Mobile Cloud


Mobile Cloud
 Accessing the “cloud” with mobile
devices
 Mobile devices becoming part of a
larger cloud construct
 Advantages
 Offloading computation and save
energy


Mobile Cloud Computing
 ABI research think it can be next
disruptive force, because
 More phones as connected nodes
 Feature phones, but not smart phones
 Applications are not related to carrier
ABI Research in 2009
1. By 2008, 42.8 million users
(1.1% of mobile cellular subscriber)
2. By 2015, 240 million users, (19%)
$5.2 billion revenues


Future Architecture of
Mobile VSaaS

Producer Consumer


Problems of VSaaS
•Video is transmitted and stored in a remote site
Capture Record/Analysis View

xDSL
FTTH/Fiber

•Transmission by ADSL is limited
• 640x480 videos with H.264
• 2~4 cameras at a frame rate of 10 fps
may be supported
Front-end video analytics is necessary
More research study is needed


DEMO 1: Mobil Video
Surveillance

Event-driven Instant Messaging


Current Mobile Surveillance
 Mobile phone directly connects to the
camera

Mobile networks
(circuit-switched
& packet-switched) Remote monitor
Camera client

Drawbacks:
1. Not event-driven
2. High communication cost


Instant Alarm Messaging
Is Important
【連絡人】
【客戶端】【保全監控中心】 3
1
異常訊息發報通知連絡人

低速專線
或電話線

歹徒入侵
3
【勤務中心】通知警察
保全主機

各式感知器 2
服務派遣處理


Our System Architecture

Keyframe Web
Selection Server

PC
Object
1. Object Info. Video
Detection
2. Keyframe Streaming
&Tracking
3. Video Clip Mobile
Phone

Video SMS
Transcoding Messaging


Our System: Web-based
Browsing Interface


Our System –
Event, Key Frame, Video Clip


Our Mobile Interface
SMS Message for Mobile
Event Notification video streaming


Our Protocol

1.監控 1.監控
新竹視訊台北
視訊
分析分析

3.
Video Analytic 確認 Video Analytic
2 3. 1
確認
2.5G
5. 2.5G 5. 3GP
3GP 確認確認
2. SMS 2. SMS

3G
4.關鍵
畫面 4.關鍵 3G
畫面
6.監控
視訊 6.監控
串流視訊
串流
輔仁大學

4. Conclusions and
Suggestions


Conclusions
 Video surveillance can be widely
applied to
 Home security
 Building/Campus security
 Public safety
 Video analytics, or intelligent video
surveillance, can benefit from
 Cloud computing
 High performance computing


Suggestions
 HiCloud with Video Analytics
 CaaS for Academic Research


Hicloud with Video Analytics
影像擷取相機異常偵測人臉辨識查詢、過濾、聯防

Video Image Object Object Object Behavior
Capture Enhance /Event Tracking
/Event
Analysis Retrieval
Detection Recognition

影像強化警戒線、路徑追蹤
流浪漢監控跌倒、人潮行為分析


CaaS for Academic Research
 The academy needs CaaS
 For data intensive computing
 Models
 The academy as users
 Rent CaaS
 The academy as technology sources
 Technology transfer
 Delegated research project
 The academy as partners
 A target: smart campus
 Long range tracking within campus
 Tracking target: car and person


Smart Campus (1/2)
13.機器人導引訪客導引
6.智慧型車牌偵測
12.快速人臉辨識
7.行人交錯偵測與追蹤
11.人員連續追蹤
8.特定人員追蹤與特寫
10.軌跡式人員追蹤與檢索

3.循園多攝影機轉場監控 9.多頻道視訊平滑轉場

2.多攝影機整合式視訊監控

4.多重解析度顯示及轉換
5.手勢指定物件
1.串聯式車牌辨識

14.整合平台及信號傳輸
161


Smart Campus (2/2)

163

The End
Free for Questions

本簡報授權聲明
 此簡報內容採用 Creative Commons 「姓名標示 - 非商
業性台灣 3.0 版」授權條款
 歡迎非商業目的的重製、散布或修改本簡報的內容，但
請標明： (1)原作者姓名：王元凱； (2)圖標示：
 簡報中所取用的部份圖形創作乃截取自網際網路，僅供
演講者於自由軟體推廣演講時主張合理使用，請讀者不
得對其再行取用，除非您本身自忖亦符合主張合理使用
之情狀，且自負相關法律責任。

Intelligent Video Surveillance with Cloud Computing

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie Intelligent Video Surveillance with Cloud Computing

Ähnlich wie Intelligent Video Surveillance with Cloud Computing (20)

Mehr von IEEE International Conference on Intelligent Information Hiding and Multimedia Signal Processing

Mehr von IEEE International Conference on Intelligent Information Hiding and Multimedia Signal Processing (16)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Intelligent Video Surveillance with Cloud Computing