This document discusses cyber-physical systems and the Internet of Things. It outlines Tata Consultancy Services' research programs in areas like mobile phone sensing, camera sensing, signal and image processing, and human activity detection using sensors. The goals are to develop an IoT platform for affordable healthcare and wellness solutions using mobile phones to detect physiological parameters. Research is also described on indoor localization, cognitive load detection using EEG, and emotion recognition using cameras. TCS has several innovation labs conducting exploratory research on mobile interactive remote sensing applications.

1. 1
Copyright © 2014 Tata Consultancy Services Limited
Internet-of-Things and Cyber-physical Systems
- Exploratory Research in Signal Processing, Communication and Computing
Dr. Arpan Pal
Principal Scientist and Research Head
Innovation Lab, Kolkata
TCS
17-May-15

2. 2
Cyber-physical Systems – Internet-of-Everything
Humans
Physical
Objects and
Infrastructure
Computing
Infrastructure
Physical
Context
Discovery
INTERNET OF EVERYTHING
Physical Context
Discovery
What is happening,
where and when
People Context
Discovery
Who is doing what,
where and when, who is
thinking what
Internet
of
Digital
Internet
of
Things
Internet
of
Humans
ABI Research. May 7, 2014
"In the next century, planet earth will don an electronic skin. It will use the
Internet as a scaffold to support and transmit its sensations.“ - Neil Gross 1999

3. 3
It’s a Connected Universe
NEVER FORGET YOUR PILLS MONITOR YOUR ACTIVITYMONITOR THE AGED
source: http://postscapes.com/internet-of-things-examples/
HEAT YOUR HOME
EFFICIENTLY
MAKE SURE THE OVEN IS
OFF
TRACK DOWN THOSE LOST
KEYS
KEEP THE CITY CLEAN RECEIVE POLLUTION WARNINGS USE ELECTRICITY MORE
EFFICIENTLY
Connected
Individual
Connected
Home
Connected
City

4. 4
Research Programs Outline
Mobile Phone
Sensing
Camera Sensing Other SensorsBio-Sensing
Signal and Image Processing
Protocols and Networking
Parallel and Distributed Computing
Data Analytics (Computational and Semantic) and Modeling
E
D
G
E
C
L
O
U
D
Personal Context
Discovery
(Location, Activity,
Psychology)
• Mobiles
• Cameras
• Bio-sensing
• Manage Scale
•Reduce Network Load
•Increase Compute
Capacity
•Reduce Storage
Requirement
• Handle Interoperability
• Easy-to-use Analytics
Physiological
Sensing using
Mobile
Phones
Mobile
phone
and
Robot
based
sensing
Human Activity
Detection and
Behavior ModelingIoT Platform Solutions
Affordable
Wellness
&
Healthcare
Mobile
Interactive
Remote
Sensing
PROGRAMS

5. 5
Click to edit Master title styleProgram: IoT Platform Solutions

6. 6
Integrated Platform for Intelligent Enterprise
People Feedback & Emotions
Social Media
Integrated Services
Sensors & IoT
Platform
Legacy Monitoring & Control Systems Enterprise Data
Smart Integration Platform
Transportation Human Resources Energy
OperationsSafety Asset Tracking
Smart Integrated Services
Sense
Analyze
Extract
Respond
Intelligence
Smart Domain Services
Supply Chain
Security and
Surveillance
Sense: People Context, Appliances, Building, Plant, Utility Infrastructure
Sync Transportation with Remote Operations
Link Asset Tracking and Safety
with Surveillance
Employee Wellness and
Energy Preservation as
Community Initiatives
Intelligent Integration Platform
Integrated Intelligent Services

7. 7
Requirements and Challenges for IoT – Need for a Platform
Applications need support for
Visibility
Capture & store data
from sensors
Insights
Patterns, relationships
and models
Control Optimize and actuate
TCUP – TCS Connected Universe Platform
A horizontal platform for addressing the IoT Software and Services market
Model-driven Development
Model the Domain Knowledge
Model the Infrastructure – Network, Storage, Compute
Model the Analytics – map to Domain Requirements
Model the Architecture – Device and Cloud
TCUP Platform
Model the Sensor – Semantics, Phenomenon

8. 8
TCUP Design and Architectural Highlights
18 patents filed, Standard Body Contribution - IETF and Singapore ITSC
• Fog Computing – Utilize unused compute power of edge devices
Distributed Computing on Edge Devices
• To reduce network congestion
• Adequate Security and Reliability
Adaptive, Lightweight yet Secure Communication Protocols
• For economical scaling of sensor data store
Efficient Compression
• Statistical and Information-theoretic measure to find out potential privacy-
breaching content
Sensitivity Measurement and Privacy Preservation
• Semantic annotation of sensors
• Sensor Search Engine
Semantic Enabled Sensor Explorer
• Algopedia – Algorithm Repository, Search and Recommendation
• Semantic Sensor Web
Model-driven Development
Manage
Scale,
Reduce
Cost
Handle
Privacy
Manage
Diversity
and Inter-op
Ease of
Development

9. 9
Horizontal operators
(semantic integration) operates on data from heterogeneous sources to created integrated data streams.
Semantic Sensor Web - From Data to Wisdom
temperature
humidity
odor
image
high temperature
gaseous odor
light
concentrated light
high temperature
indicates fire
gaseous odor indicates
gas discharge
Fire from
Gas Leak,
evacuate
immediately,
send fire fighting team
equipped with gas leakage
data
information
knowledge
wisdom
Vertical operators
(semantic abstraction) operates on
artifacts at each level and
transcends them to the next level
F PCS(Data, KB*) → Information
F PCS(Knowledge, KB) → Wisdom
F PCS(Information, KB) → Knowledge
KB: Knowledge base
Adopted from: Physical-Cyber-Social Computing: An early 21st Century Approach, Amit Sheth et. al.

10. 10
Research Outcomes – Some of the Results
Publications in ACM Sensys, Ubicomp, Infocomm, Middleware

11. 11
Program: Human Behavior Modeling and Data
Collection

12. 12
Human Data Collection and Behaviour Modelling - Program
Overview
Research Goals - Given a context, predict behaviour. Given behaviour, find out context.
Focus Domains – Organization Behavior, Consumer Behavior
Current Available Models – Statistical, Need of the Day – Models based on physical data
Meetings –
Group or
One-on-one
Digital
Communication
Individual
@Work or
Leisure
Individual –
Day in the
Life Of
• Strength and
Polarity of
Relationships
• Meeting Flow
• Emotive State
• Outcome
• Formal / Informal,
Business / Social
• Strength and
Polarity of
Relationships
• Tenor of
Communication
• Nature of
Communication
• Location and
Time spent
• Mood and
Physical State
• Engagement
Level
• Cognitive Load
• Location and
Time spent
• Activity
• Behavioral
Routine
• Social Interaction
• Moment of
decision making
Microphone
Email / Knome
Mobile Phone,
Desktop
Mounted
Cameras,
EEG/GSR
Mobile Phone,
Kinect, EEG/GSR,
Smart Meter
Surveillance
Camera

13. 13
Human Identification and Activity Detection using Kinect
Human Identification
– Skeleton Model Based / Depth based
– 20 joints of skeleton data
• 2D Camera with IR
depth sensor
• Excitation by IR light
pattern
Human Identification
• Gait cycle detection
• Feature extraction from skeleton
joints
• Training
• Recognition
Papers in IEEE Fuzz, CEC, IEEE SMC, UbiComp and ECCV
Activity
• Sitting
• Standing
• Walking
Human activity recognition using RGB-D
Accuracy is above 90%

14. 14
Activity Detection using Mobile Phone Inertial Sensors
Activity Detection
– Uses Accelerometer Data
– Gyroscope and Magnetometer for orientation correction
– Step Count, Stride Length Estimation
– Walking, Brisk Walking, Running, Sitting, Falling
Classification
Continuous Data Stream
Windowed Data
Zero Normalization
Linear Interpolation
Low Pass Filtration
Frequency Spectrum
Identifying non-activity window
using frequency spectrum
Peak Detection and Step
Validation using IPA;
calculating step cycle lengths for
all valid steps in the window
Classification of window
activity using step frequencies
derived from step cycle lengths
Continuous Data Stream
Windowed Data
Zero Normalization
Linear Interpolation
Low Pass Filtration
Frequency Spectrum
Identifying non-activity window
using frequency spectrum
Peak Detection and Step
Validation using IPA;
calculating step cycle lengths for
all valid steps in the window
Classification of window
activity using step frequencies
derived from step cycle lengths
Peak Detection and Step Validation using IPA;
Calculating Step cycle lengths for all valid
steps in the window
Classification of window activity using step
frequencies derived from step cycle lengths
Noise Cancellation and pre-processing
Calorie Count from Step Count and Type of Activity
Papers in UbiComp
~90% Accuracy
~80% Accuracy

15. 15
 Geo-fencing
– Using Magentometer
 Proximity Detection
– Using Bluetooth RSSI
 Inertial Navigation
– Step Count + Stride Length (personalized model)
– Gyroscope and Magnetometer-corrected Inertial
Navigation
 Wi-Fi based Zoning and Triangulation
– Based on RSSI of known location of 3 or more
access points
– Attenuation modeling of the building
– Unsupervised Learning through physical
modeling
 Fusion, Tracking and Correction
– Kalman Filter based Tracking
– Particle Filter based Correction
Mobile Phone based Indoor Localization – Inertial and Wi-Fi
• Colleague
Finder in Large
Offices
• Shopper
Localization in
Retail Stores
• Emergency
Evacuation in
Large Buildings
Papers in Mobiquitous, UbiComp
Selected for Indoor Localization
Competition in IPSN 2014

16. 16
Cognitive Load on Human Brain
Cognitive Load
 23+45=?
 1846890129 + 2374609823=?
Use Cases
 Personalized education
 User interface design
EEG GSR
Papers in IEEE SMC, IEEE Fuzz, ACM BIBE

17. 17
Emotion and Engagement level using Camera
Why Camera
− Unobtrusive Sensing @Work
Purpose
− Identifying the mood a person at work
− How a person is engaged at work
Scope
− Facial emotion analysis on unconstrained environment using camera on
desktop/laptop
− Mood and human engagement at work place using desktop camera
 Partial occlusion for facial expression recognition
 Engagement analysis
 Micro emotions / expressions
http://www.ecse.rpi.edu/~cvrl/tongy/aurecognition.html
Irfan Essa (1994), “Analysis, interpretation and synthesis of facial
expressions“, PhD Thesis, MIT, Cambridge, MA, USA. (Advisor: Alex
(Sandy) Pentland)

18. 18
Program: Affordable Healthcare and Wellness

19. 19
Affordable Healthcare using Mobile Phones
 Sensing various physiological parameters using
smartphone sensors with minimal attachments
 Low cost solution for initial screening in preventive
healthcare / wellness
 Solutions need to be stable, repeatable and robust
 New disease diagnostics and treatment protocols
– Requires long observation over large set of
patients
 Geriatric care, monitoring chronic patients
 Go into Wearable in future
 Heart condition
– Heart rate, Heart rate variability, Blood Pressure, ECG
– Foetal Heart rate
 Lung condition
– Spirometry, Respiratory Rate
 Pupil condition
– Pupillary dilation response
 Pulse Diagnostics
Purpose
Scope
Robust Solutions using camera, microphone and accelerometer

20. 20
PoC Approach and Expected Novelty
HR, HRV, RRPPG extractionRealtime
Video
Audio from
Mic
Accelerometer
Cardiovascular
Model
SpO2
BP, ECG
Spirometry
Breathing
Rate
Image of eye
Pupillary
Reflex

21. 21
Photo-plethysmography (PPG) using Mobile Phone Camera
Subject1 Subject2 Subject3
Actual Detected Actual Detected Actual Detected
68 66 66 63 85 84
2.9% 4.5% 1.1%
Papers at Mobihoc, IEEE BIBE, SenSys, ICASSP
Data set Pd Ps PP-diff < 15
Standard dataset (14 features) 92.9% 74.7% 77.9%
TCS dataset - add height, weight, age 99.3% 82.7% 85.5%

22. 22
Program: Mobile Interactive Remote Sensing

23. 23
Sensing the Physical World
Mobile phone based crowd
sensing
Robot assisted sensing
www.popularmechanics.com
www.engadget.com
www.allthingssd.com
apollo2.cs.illinois.ed
u
Camera based sensing

24. 24
Intelligent Transportation – Vehicle Model Driven Sensor Data Analysis
KNOWN PARAMETERS EFFECTS TARGET INDUSTRY
Vehicle Type & Driving Behavior Road Condition Monitoring City Municipality
Road Condition & Driving Behavior Car Prognosis Automotive
Road Condition & Vehicle Type Driving Behavior Analysis Insurance
Acceleration a(t) = f (H(t), v(t), R(t), D(t))
H(t)
Papers in ICST, Percom

25. 25
Phone Microphone based Sound Scaping
Solution Overview
Event driven with participatory sensing aided audio surveillance system
• Classification of Traffic Noise (Honk Detection) and Crowd Noise
Papers in CODIS, ISSNIP, ISDA

26. 26
Phone Camera based 3D Reconstruction from 2D images
Input Images
Dense Reconstruction without using mobile
inertial sensors
- 20 images, compute time (4 core, 1GPU) ~ 20 min
- 120 images, compute time (16 core, 1GPU) ~ 30 min
- Bandwidth saving ~ 8 times, if done on mobile
Sparse Reconstruction using Mobile Inertial
Sensors for Camera Position Estimation
• 20 images, compute time (4 core, 1GPU) ~ 3 min
(without using inertial sensors)
• 20 images – compute time (4 core, 1GPU) ~10
sec. (with inertial sensors)
• Bandwidth saving ~ 200 times, if done on mobile
• Sparse good enough for many applications
• Mobile Sensing and ACCV (submitted)
• Dense Reconstruction with mobile inertial
sensors under progress with more number of
images
 target < 1min
Dense Reconstruction
-120 images
Dense Reconstruction
- 20 images
• Low cost solution for 3D reconstruction from multiple 2D images captured from mobile device.
• Motion information from the inbuilt inertial sensors – for camera position estimation
• Applications in Agro-advisory service, Remote Diagnostics, Remote Healthcare

27. 27
Multi-sensor Fusion for Robot-assisted Sensing
Application in remote sensing in hazard-prone areas
• Robot carries 2D camera and heat / chemical sensors on a rotating arm
• 3D reconstruction from the 2D vision
• Estimation of Heat / gas leak / sound Source (direction and range) through passive directional
signal processing
• Fusion of heat / gas / sound map on reconstructed 3D vision map
www.ese.wustl.edu
Ongoing Work
 Possible reuse from 2D-3D reconstruction and sound classification
Cloud point
from 3D
vision
Possible
gas / heat
source
(ROI)
Source
direction
and intensity

28. 28
Innovation @TCS

29. 29
TCS - Pioneering IT Innovation in India
1970s
Offshore Model
Alliances with Major IT
Players
University Alliances,
Systems Engineering
1970s
-1980s
New Labs, Bio Suite
Dhruvam
Silicon Valley
Ecosystem
Mastercraft, Revine,
Quartz Program,
Tools Foundry, Rice
Husk Ash
2000 -
2005
2007-
2011
IPR Focus and Policy
Talent Management
TCS COIN TM
Software Tools -
Casepac Migration
Re-engineering
Industry & Services
Practices Set Up
Foundation of TRDDC-
India’s first Industrial R&D
Center in IT, S/W Engg
CTO/R&D structure
Innovation Framework
More Domain Labs
2006
1981 1987
1991
1998
2012
4E Model
Scaled
Invention,
Co-Innovation
The journey continues…

30. 30
Innovation@TCS - Innovation Labs
Bangalore, India1
TCS Innovation Labs - Bangalore
Chennai, India2
TCS Innovation Labs - Chennai
TCS Innovation Labs - Retail
TCS Innovation Labs - Travel & Hospitality
TCS Innovation Labs - Insurance
TCS Innovation Labs - Web 2.0
TCS Innovation Labs - Telecom
Cincinnati, USA3
TCS Innovation Labs - Cincinnati
Delhi, India4
TCS Innovation Labs - Delhi
Hyderabad, India5
TCS Innovation Labs - Hyderabad
TCS Innovation Labs - CMC
Kolkata, India6
TCS Innovation Labs - Kolkata
Mumbai, India7
TCS Innovation Labs - Mumbai
TCS Innovation Labs - Performance Engineering
Peterborough, UK8
TCS Innovation Labs - Peterborough
Pune, India9
TCS Innovation Labs - TRDDC - Process Engineering
TCS Innovation Labs - TRDDC - Software Engineering
TCS Innovation Labs - TRDDC - Systems Research
TCS Innovation Labs - Engineering & Industrial Services
1 2
3
4
5
97
6
8
• Associates in R&D2000+
• Innovation Labs19
• Papers Published in last two years1000+
• Patents filed in last two years350+
• Patents granted till date100+
• TCS RSP funded Research Scholars150+
• Internships in Labs in last two years200+

31. Thank You
IT Services
Business Solutions
Consulting