SAARIKOSKI YLE metadata machine

Identifying use cases
and evaluating ML technology
Case: the Metadata Machine Project at Yle
Lauri Saarikoski, Yle, the Finnish Broadcasting Company
Matt Eaton, GrayMeta

Metadata Machine, FIAT/IFTA 2019
Industry Drivers:
Content
Lorem ipsum
congue
Volume
Lorem
ipsum
Lorem
ipsum
Lorem
ipsum
Lorem ipsum
congue
Lorem
ipsum
Lorem
ipsum
Lorem
ipsum
Lorem
ipsum
Lorem ipsum
congue
Lorem
ipsum
Lorem
ipsum
Lorem
ipsum
Lorem
ipsum
Need for
More
Content
Metadata
Volume
Increasing
Personalised
Content
Expected
New Ways
of
Consuming
Content
More
Distributio
n Channels

Industry Drivers:
Technology
Generating &
Using New
Content
Metadata
Easier
Rapidly
Maturing
Machine
Learning
Services
Growing
Adoption of
Cloud
Services
APIs
Enabling
Easier
Integration

Archives as part of a Media Company, case Yle
Media Production
● inhouse
● production houses
Media Distribution &
Publishing
● Online, TV, Radio
● 3rd party platforms
Yle Archives
[that’s us, 61 people]

Data in and out of the archives
Yle Archives
Production
Distribution
Existing collections
5,6 PB of media,
4,7 M metadata objects,
annually 12k legacy
objects get some
metadata added
Audience analytics,
publishing metadata
Annual hours:
47k audio,
19k linear video
Media,
content descriptions
Annual hours:
Video 6,7k
Audio 27k
+ photos, music

Help from
Automated Metadata?
1. Increasing Volumes of content being archived
2. Demand for Archive content increasing, as
distribution to multiple platforms require content
3. Data entry is compromised by number of
resources available to manually enter data
4. Only a small % of the archive is enriched with
detailed metadata
5. New types of metadata needed e.g. Audio
Classification & facial detection
6. [Historical] Content which lack metadata entirely
7. Compromise of metadata Quality due to time
constraints.
Why look into machine learning
in the first place?

Series of Pilots at Yle Archives
2016
2017
2018
2019
● Music identification
● Automated keyword
extraction market review
● Video recognition
technical pilots
● Image recognition
market review & piloting
● Face recognition
technical pilot
● Speech recognition
technical pilot
● Speech recognition on
demand MVP
● Speech recognition &
automated keyword
extraction MVP● IPR related to
data mining, memad.eu
● Study groups on AI ● Metadata Machine
● Facial recognition data
as UX
in production systems

Pilot projects
reduce complexity
After each pilot,
.. we know more about what to do next
and what questions still need looking
into.
.. our people have a more realistic view
on what to expect and what to discuss.
.. our people are more involved in
designing their work and
[maybe] find new tools more
acceptable than earlier.
People Technology
Organisation
Metadata
Conventions

The Metadata Machine
Pre-study / PoC Spring 2019

All audiovisual content is analysed as early as possible
Vision: The Metadata Machine
Content creation
(raw material)
Procurement
(ready-made
content)
Publishing
(published content)
Archiving
(what do we have?)
Automatic content analysis engine
Speech
recognition
Image
recognition
Person identifier Fingerprinting
Sound
identifying
Video frame
color analysis
Music identifier
Text analysis
Company-wide metadata database on all content items
Language
identifier
...

Approach of this Pre-study Project
Business
Cases
Company
Wide
Solution
Multiple
AME
services
What’s
available
today
Single
Solution
Single
Providers
What
could be
available
Our Focus
Avoided
How, when and why to get in to production?
Technological
Details
How to
build the
road to
future
Yet
another
POC

Business Units Involved and Covered Today
Production
Management
Photo Archive
Video / Audio
Archives
Audience
Insight
OnDemand
(Yle Areena)
News
SportRadio / Audio
Master Control
Room & Media
Logistics
Senior
Leadership
Translation &
Versioning
Team
Architecture &
Technology

Project Timeline and Structure
1. Buy a metadata machine (Graymeta Curio)
2. Involve the whole company to identify potential use cases for automatic metadata
3. Each team tries to solve their use cases with the machine
4. Collect the results from the teams, identify the most prominent business cases
5. Decide on the next step, e.g. investment
test round 1 test round 2 test round 3
analysis &
next steps
February September

©2019 GrayMeta. All rights reserved. PROPRIETARY & CONFIDENTIAL
Curio connects all of your content,
creating a single interface & API to
find and use any file.
Utilize automated metadata to help
find and use your content.

Metadata Machine by the Numbers
637 hours of content
2193 assets22 Yle Testers
7 different ML providers used
23 Use cases analysed
37 audio files
873 images
1281 videos
2 application files5 insight groups
45 Different Automatic Metadata Extractors
100,000+ API calls

Identifying Use Cases for Automated Metadata
100+ ideas
10+ proof of
concepts
1+ to production 10+ to production
Metadata Machine pre-study project 2019

Approaching the Business Value
What kind of metadata
does it require?
Does it improve existing
processes?
Does it enable something
completely new?
Does it save money
or time?
Does it increase customer
satisfaction?
Is the technology solution
available today?
What are the direct and
indirect costs involved?
How to optimize the costs?
How does it affect the
surrounding production
process / way of working?
How to combine human
work with automation?
What are the success
criterias / KPIs … ?
...

[After being refined during and after the project]
Main Archive Use Cases
1. Enriching existing metadata for archive content and adding new types of metadata
• It also can be used to improve search functionality
2. New possibilities to browse and navigate through the Archive collection
• New faceted navigation functionality
• Ability to effectively browse through and filter archive based on enriched content.
3. Enhancing the metadata creation workflows by increasing automation levels.
• Automation allows for focus on metadata quality
• Automation allows for the team to manage the increase in demand and content being
archived.

What Did We Do?
Aim:
To test various Automated Metadata Services as to understand where the metadata creation workflows can be enhanced, by increasing automation levels; as well
as, being able to generate metadata for collections which are currently lacking.
Services Tested:
● OCR - Optical Character Recognition
● Speech to Text
● Tags & Descriptions
Methods Used:
● Variations on Extractor Settings & Thresholds, to refine results and remove false positives.
● Varied content from old Black and White footage to newer content - to understand the value across the entire archive.
Selected test videos from
Archive to test against various
Machine Learning (ML) Services
Configured relevant Machine
Learning (ML) Services to
Process Photos.
Refined and Tested different
extractors and confidence
thresholds.
Compared results and built
conclusions on the validity of
services for the Archive Team
1 2 3 4
● Facial Detection
● Audio Classification
● Logo Detection
● Curio’s User Interface
● Curio’s API

Sample Facial and Logo Recognition
Selected test material from
photo archive to test against
Identified & Trained Persons of
interest within AI Studio.
Identified & Trained Logos of
interest within AI Studio.
Configured relevant Machine
Learning (ML) Services to
Process Images.
Analysed the impact on Search
& Discovery through Facial
Detection
& Discovery through Tags,
Descriptions & OCR
& Discovery through Logo
Detection
Reviewed and reported on the
success & failures of ML for
Images
1
5
2
6 7
3
8
4

Sample Speech to Text (ASR)

Sample OCR

Sample Audio Detection - Radio Content

Sample Audio Detection - Video Content

Findings and conclusions
Learned during this and previous projects

Conclusion 1:
Identified Basic
Analysis Bundles for
Audio and Video
Drafted based on
● User needs
● Technology readiness level
● Availability of technology
● Expected impact vs. costs

Basic Audio analysis bundle:
Speech/music segmentation + ASR +
automated tagging
● later expand with speaker
identification, spoken language
identification etc.
● Can also be applied to video
Basic Video analysis bundle:
Audio analysis bundle +
facial recognition + OCR
● generic video analysis and
natural language descriptions
are not ready enough
Conclusion 1:
Identified Basic
Analysis Bundles for
Audio and Video

2: Importance of
Focusing on Use Cases -
Not Only Technology
Technology Requirement:
“We should use ML on archive content to get metadata”
Define Use Case How Business Will Use Technology
“We should get a better sense of what happens during a
single archive program”
a. As a user I can navigate within a program
based on what is discussed in it.
b. As a user I can navigate within a program
based on topics discussed in it.
c. As a user I can see in the ASR result
who is speaking and
where there is music in the program.

Without a use case, it is hard to know
how well you perform.
What to measure?
● For technical tests:
technical criteria (WER, recall etc.)
● For a use case: added value
“What effect did this have on your
work?”
“Did it help?”
Success criteria depends on the case
you are solving, not on technical
metrics.
2: Importance of
Focusing on
Use Cases -
Not Only Technology

3: Strategies for Integrating
Machine Learning into
Human Processes
1. Use the data as it comes from ML
services
→ When people won’t be looking at the data & the quality is
good enough?
OR
2. Interact with the ML services and the data
→ If the ROI of human effort makes sense?
OR
3. Look at the ML data but enter the
metadata manually
→ When automation helps you grasp the context but end
results need to be high quality?

Humans
Media storage
Human tasks, system roles
Metadata storage - Across Content Supply Chain (not just Archive)
UX for testing and
comparing ML services
Process orchestration
ML services
Metadata
unifier & mapper
ML services
ML services
ML services
ML services
Search & Browse UX
+ Curate the ML models,
metadata mappings etc.
+ Use the metadata and
review it’s quality
Systems
+ Review ML services
and build on them
UX for managing ML
Machine Learning Services

Future Roadmap
Short Term Medium Term Long Term
For Archive Content with No Metadata
- deploy Audio & Visual ML Bundles
Introduce face recognition
in a subset of Archive
Further Investigate
Audio Classification
Speaker Recognition Based
on Audio
Increase Automation of
metadata forms
Process Rest of Archive
Introduce Machine Learning for High Re-Use
Archive Content
Perhaps in
collaboration with
Sports Production
or Analytics
Identify Speaker dependent on
new machine learning services
Use Machine Learning to generate
content metadata on portion of archive
where no metadata exists
Detect Music vs
Speech to speed
up assessing
rights clearance
and focus ASR
use
Augment existing manually
created metadata forms for
archived content with automated
machine learning

● For piloting purposes the content volumes can be kept low
○ Someone typically has to go through the results afterwards
● Involve the wider Archive team to help evaluate the use cases and to increase the sense of involvement
○ This will lead into a better understanding of technology among your staff
○ Expectation management: getting rid of fears and hype
● Distributed work is great for involving people, but needs careful management
○ The project team can support and help with e.g. evaluation criteria and setup & facilitate discussions
● Reserve enough time for setting up and result analysis
○ Setting up the systems and logistics take their own time
○ Running the analyses is fast, making sense of the results is slow.
○ Before you start:
● narrow down the ‘thing’ you are testing / evaluating
● decide on your methodology / approach
● choose suitable content for testing
○ Reporting results and conclusions is easier if the project setting supports this from the beginning
Practical advice for running a project

Final Thoughts
After deciding your first Use Case you can..
- Define success criteria for this case and measure it
- Build a roadmap to realise and enhance your case
- Find right components to realise your case
- Decide on the type of human curation needed for your case
Some types of Machine Learning services can already be put to good use.
Iterative approach seems to work well since everything is changing and you need to start your
journey sooner rather than later.
Please share your ideas for smart types of human participation in the ML assisted work!

Contact information:
Lauri Saarikoski
Development Manager at Yle, the Finnish Broadcasting Company
lauri.saarikoski@yle.fi
Matt Eaton
Managing Director, EMEA at GrayMeta Inc
matt.eaton@graymeta.com
Thank you!

SAARIKOSKI YLE metadata machine

SAARIKOSKI YLE metadata machine

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie SAARIKOSKI YLE metadata machine

Ähnlich wie SAARIKOSKI YLE metadata machine (20)

Mehr von FIAT/IFTA

Mehr von FIAT/IFTA (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

SAARIKOSKI YLE metadata machine

Hinweis der Redaktion