AI in Media: Guide to Applications and Research

Artificial Intelligence (AI) in media applications and services © IRT 2019
1
Artificial Intelligence (AI) in media
applications and services
Dr.-Ing. Christian Keimel
keimel@irt.de

2
Agenda
Outline of today’s presentation
• AI – what is it?
• AI in Media
• Monitoring as a use case for AI in media
• Research topics for AI in broadcasting
• Challenges and open questions

3

4
Unsere Mission
As a worldwide renowned key
research and competence centre for
audio-visual technologies we
research, evaluate, and develop new
technologies with the aim to strat-
egically adjust broadcast concepts to
new market environments and needs
Representing Broadcasters Standardisation
Maintaining Interoperability
Technology
Evaluation
Technology Scouting
Prototyping and
Pilots
Applied Research
Supporting Technology
Introduction
Knowledge TransferServices

5
60 Years of Milestones in Broadcast Technology

6
Our topics
Next
Generation
Audio
Future
Video
Artificial intelligence
Meta data
All IP/IT
IP distribution
Platforms & services
Accessibility &
Design for all
5G

7
IRT in numbers
• Research and competence centre of the public broadcasting corporations in Germany
(ARD, ZDF, Deutschlandradio), Austria (ORF) and Switzerland (SRG/SSR)
• Location: Munich, Germany (BR TV production facility Freimann)
• Non-profit limited liability company with 14 associates
• Founded in 1956
• Approx. 130 employees
• Annual budget: ~ 25 Mio. €

8
Deep Learning and AI
Artifical Intelligence (AI)
Machine Learning (ML)
Neural Networks (NN)
Deep Learning (DL)
Technologies
Behind the current
trend of „AI“
Recent progress in „AI“
mostly done here

9
What is AI? – Definitions (1)
Definition 1:
„Colloquially, the term "artificial intelligence" is applied when
a machine mimics "cognitive" functions that humans associate
with other human minds, such as "learning" and "problem
solving“ “
[Russel, S., Norvig, P. (2009): Artificial Intelligence: A Modern Approach]

10
Definition 2:
„Artificial intelligence (AI) refers to systems that display
intelligent behaviour by analysing their environment and
taking actions –with some degree of autonomy –to achieve
specific goals.“
[European Commission, COM/2018/237 final]

11
Definition 3:
„Artificial intelligence (AI)—defined as a system’s ability to
correctly interpret external data, to learn from such data, and
to use those learnings to achieve specific goals and tasks
through flexible adaptation“
[Kaplan, A., Haenlein, M. (2019): Siri, Siri, in my hand: Who’s the fairest in the land? On the interpretations,
illustrations, and implications of artificial intelligence]

12
What is AI? – Where are we today?
Strong/General AI Weak/Narrow AI
• Can solve any problem
• Equal to human intelligence
• Can solve specific problems
(e.g. speech recognition)
State-of-the-Art

13
Machine Learning – Basics
System learns/trains “patterns/rules” based on representative data
Training data is often labelled (supervised learning)
Cat
Dog
Data
System
CatLabel
Data
point
Cat-Dog-Classifier
Classes

14
After the completed training, new data from classes contained in the training
can be recognised (inference)
Data from unknown classes not contained in the training data won’t be
recognised or assigned a wrong class
New data,
known class
Machine Learning – New data
Cat
Dog
New data,
unknown class
Cat
Dog
96%
4%
96%
4%
Confidence
Wrong classification

15
Training data – Where to get it?
Existing data
Already labelled data (for supervised learning) in (freely) available training sets
Example: ImageNet
Labelling the data yourself
Manually adding a label to each data point
Can be done inhouse (e.g. in workflow) or using online service provider
(crowdsourcing)

16
Training data – Where to get it?
Indirect labelling by (end-)users in applications
Users perform the labelling task in the context of an
arbitrary application
Users do not explicitly know that they are
performing a labelling task
Example: security checks (Captchas)

17
Neural Networks – Artificial Neurons
Based on the structure of neurons encountered in nature
First concepts and developments in the 1950s
Σ
summation
𝑥1
𝑥2
𝑤1
𝑤2
input weights
activation
function
output
non-linearity
𝑧 = ቊ
0, 𝑦 < 0
𝑥1 𝑤1 + 𝑥2 𝑤2, 𝑦 ≥ 0
𝑦
𝑥1 𝑤1 + 𝑥2 𝑤2
(in this example)

18
Neural Networks – Training
Weights w are iteratively changed until z equals the values of the
labels in the training set (or rather a cost function has been optimised,
usually minimised)
Simple example:
Σ
𝑥1
𝑥2
𝑤1
𝑤2
𝑧𝑦
𝑥1 = 2, 𝑥2 = 4 → 𝑧 = 2
1
0,5
0,5
1
0,5
0,25
6 ≠ 2
3 ≠ 2
𝟐 ≡ 𝟐
𝟐
𝟒
Repeated for all data in the
training set

19
Neural Networks – Architecture
Combination of multiple neurons in a network that consists of
multiple (hidden) layers that are connected
First application in the 1980s (MLP)
Cat
Dog
hidden layer(s)
10 weights15 weights
25 weights
per iteration,
per data point
input layer

20
Deep Learning
Many layers, specific architectures suitable for image/video
processing (CNN) or temporal dependencies (RNN, LTSM)
Became popular since the 2010s…
…
…
…
Cat
Dog
… … …
many hidden layers (>> 2)
millions of parameters,
per iteration,
per data point

21
Deep Learning – Why now?
The current success of deep learning has many reasons:
• Five decades of research in machine learning
• CPUs/GPUs/storage developed for other purposes
• lots of data from the Internet
• resources and efforts from large corporations
• tools and culture of collaborative and reproducible science
Resulting in frameworks e.g. Tensorflow

22
Deep Learning – CPU/GPU power
[Francois Fleuret (2019), Deep Learning]
Flop/USD

23
Deep Learning – Lots of Data
Example: image data sets
[Francois Fleuret (2019), Deep Learning]

24
Deep Learning – Performance
ImageNet data set
• 1000 categories,
• > 1M images

25
Object recognition ImageNet data set
28,2
25,8
16,4
11,7
6,7
3,6
5,1
2 2
8 8
22
152
0
20
40
60
80
100
120
140
160
0
5
10
15
20
25
30
2010 2011 2012 2013 2014 2015 Human
ErrorTop-5[%]
Layers

26
[Krizhevsky et al. (2012), He et al. (2015), Szegedy et al. (2015)]
Deep Learning - Architectures
AlexNet
2012
ResNet
2015
GoogLeNet
2015

27
Deep Learning – Acurracy vs complexity
[Canziani et al. (2017)]

28
Example: object recognition in images with cloud services
November 2017
„a man brushing his teeth“
January 2019
„a sculpture of a man“
[Alberto Messina, RAI (2019)]

29
Deep Learning – Comparison to nature
Number of neurons in network compared to nature
[Goodfellow et al. (2016): Deep Learning]

30
Deep Learning as AI – Products & Services
Similarities
• Same or fairly similar architectures are often used
Differences
• Different parameters in learning process i.e. hyperparameters
• Amount and quality of training data
Training data is essential for accuracy

31
Audio-visual content understanding
Video
sentiment
analysis
semantic
analysis
sentiment key words/
concepts
object
recognition
face
recognition
identity sentimentobjects
context free with contect
audiovisuell
sound
recognition
objects
speaker
recognition
identity
XXXXX
textual
OCR
speech-to-text
(transcription)
text

32
AI in media
Understanding of audio-visual content
• Extracting information from audio and video assets for indexing and use in
further applications:
o Enrichment of archive content for increased „findability“
and reusability
o Enrichment during production via object recognition
o Customised views for different target groups
o Recommendations (editors and consumers)
o Verification

33
AI in media
Automatic content generation
• “Robojournalism” & individualised content
• Teaser/trailer/highlight creation, auto-summarisation
Optimisation in distribution and production
• Streaming & encoding decisions, network routing etc.
News gathering/Monitoring
• Trend detection in/monitoring of audio-visual (news) content

34
Should detect
topic here
Monitoring – Motivation
Life cycle of a news event
Popularity/visibility
„Trending topic“
→ Peak popularity
t
Event is
spreading
Event is
getting stale

35
Role of Social Media
Social Media plays an increasingly important role in the news cyle
t
Original post
& dissemination starts
Dissemination gains
speed through influencers
(hash tag becomes popular)
Popularity/visibility

36
Twitter vs Instagram
Users/Source „offical“ „private“
Demography Older Younger
Content mostly Text Images/Videos
Hash tags in posts Few Lots
Geolocation use Uncommon Very common

37
Is considering only text enough?
• Hashtags and text are chosen by the creator of the message
• Don‘t necessarily describe the complete content of audio-visual assests
News events are potentially not visible in the trends,
especially for regional events with few „influencers“

38
Idea
Analyse audio-visual content from (public, localised) social media posts,
recognise concepts/object with AI, and use this information for trend detection
Image/
Videos Concepts/
Objects Trends

39
Proof of concept – Search by location

40
Proof of concept – Search results

41
Proof of concept – Detection results

42
Proof of concept – Related content

43
Proof of concept – Detection results

44
Proof of concept – Video detection results

45
Proof of concept – Video detection results
Nach unten scrollen für Transkript & mehr…

46
Why use AI in media and broadcasting?
Benefits of using AI in broadcasting
Content – Creator
• Cost savings
• Workload reduction
→ Assign people to more important tasks
• Staying relevant (competition by Social Media etc.)
Content – Consumer
• Increase audience
Technolgy/Distribution
• Cost savings by more efficient use of resourccess

47
Research topics under consideration at IRT
Addressing challenges in broadcast-related applications
• Verification: how to verify audio-visual content for authenticity/source
• Speech-to-text for dialects: “of-the-shelf” models work not very well for
non-standard speech
• Clean Feed: automatic customising for multiple distribution channels
requires clean feeds from “normal” material
• Training diversity: existing large-scale training material are not
representative enough for regional content

48
Verification
Verify content (semi-)automatically
• Automating existing concepts
Automate manual concepts for content verification that are
already used e.g. time/place verification via shadow length
https://www.suncalc.org/#/49.2548,7.0426,16/2019.04.10/15:15/2/1

49
Speech-to-text for dialects
Extend speech-to-text beyond “standard speech”
• Transfer learning for dialects
Leverage large corpus of “standard speech” for (pretrained)
base model, then retrain model with smaller corpus of dialects
corpus model
Standard speech
model
Dialect
corpus
retraining

50
Clean Feed
Automatic customising for multiple distribution channels requires clean feeds
• Detection of salient objects and prioritisation
“Objectification” of audio-visual content for distribution dependent
rearrangement with preservation of semantically most salient objects
„objectification“ &
semantic analysis
source material
1
2
3
prioritisation distribution
formats

51
Training diversity
Creating more representative training material automatically
• Leveraging “hidden” meta data
Using existing information in audio-visual assets to generate
labels for training e.g. information in lower thirds banner text
Label
Data
Labels for
face recognition

52
Training diversity
Alternative to strictly supervised learning approach
• Reinforcement learning
Using implicit user feedback contained in workflow actions as
environment variable(s)/reward e.g. selection of specific result
from search results
model
Selected result
reward

53
AI challenges
Training data
• Diversity: regional content/languages and/or minorities are not
represented sufficiently
• Bias free: all classes are not always represented uniformly; bias can be
problematic (example „fake news“)
• Reproducibility: if the training data is changing, the models change
i.e. predictions may change over time

54
AI Challenges
Architecture/algorithms
• Daten protection/privacy: often not a major priority, especially as the
development of AI technology is driven by companies from the USA and
China (for cloud services)
• Explainibility: Predictions can sometimes not be understood/explained
• Portability: trained models are not necessarily portable between
cloud providers
• Accuracy: even 99% accuracy is not sufficient for fully automated
systems in some use cases

55
AI – Assistive or Autonomous?
Assistive AI
• Assists creators/journalists/operators in content creation and editing
• Final decisions und responsibility stays with a human
Autonomous AI
• Replaces creators/journalists/operators in content creation and editing
• No human in the loop – who will be responsible for the decisions?
Mandatory identification of AI generated content?

56
Data protection & privacy
Is AI even causing any new issues?
• Many problematic applications have already been possible, but until now
the weren‘t done as it wasn‘t effective on a large scale
• Example: complete surveillance with face recognition in public places was
to expensive with human operators, but now „cheap“ with AI
• But easier implementation of such applications lower inhibitions
against using them
Is it sufficient to us existing regulations?

57
AI – What will the future hold?
Hypothesis
• Weak AI based system for audio-
visual content recognition are
standard and will remain so
• Use of the term „AI“ will become less
popular, but the technology behind it
will continue to be used

58
All rights reserved. All text, images, graphics and charts are protected by copyright.
Reproduction or use of the content is not permitted without the express consent of the author.
Experts in audio-visual media
Dr.-Ing. Christian Keimel
Data & Security
Floriansmuehlstraße 60
80939 Munich
Tel +49 89 323 99 – 303
FAX +49 89 323 99 – 351
www.irt.de
keimel@irt.de
Thank you for your attention!

AI in Media: Guide to Applications and Research

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie AI in Media: Guide to Applications and Research

Ähnlich wie AI in Media: Guide to Applications and Research (20)

Mehr von Förderverein Technische Fakultät

Mehr von Förderverein Technische Fakultät (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

AI in Media: Guide to Applications and Research