The Metaverse will require more content than we can realistically build manually, and it will need to be dynamically generated, personalised and completely interactive. This talk focuses on two groundbreaking projects: Creating an entirely AI-generated and interactive TV show and our inevitable future of 3D streamed media, specifically interactive streamed volumetric sports broadcasts. Adam will show how both projects work under the hood with live demos of the technology and breakdowns of the key points. Components include procedural cinematography, shot evaluation, ML pose estimation, dialogue and narrative synthesis, touch interactivity, intuitive UX and the challenges of streaming huge amounts of content to mobile devices. You’ll see what will hopefully be the very first, fully AI-generated TV show running 24 hours a day and never before presented versions of the latest Metacast technology, transforming how media is broadcast.

1. Building for the Metaverse
From AI-generated stories to
interactive volumetric content
Adam Myhill, Creative Director @ Unity Technologies
Date: Wednesday October 26th
Time: 11:00 AM - 11:40 AM
Location: Room 1 - Upstairs
@adamjcmyhill

2. The Metaverse has a
content creation
problem

3. Adam Myhill
20 years in video games
5 years in film as a DP
Created Cinemachine, sold to Unity
Created Cinecast, Metacast
7 patents in camera, UX, streaming, volumetric file formats
Technical Emmy
Creative Director of Metacast / Sports @ Unity Technologies
Investor and Advisor to Transitional Forms
Owner of 3 alpacas

4. What even is the metaverse?

10. Adam’s hot take on the Metaverse
A persistent shared environment
On a decentralized open standard
Proof of Position + Spatial Contracts
Dynamic subscription to content layers - modified by time, position,
biometrics
While purely creative sub-worlds will be cool (pick the blue or red door)
The data-overlaid layers of content over IRL will be where it really takes
hold

12. An incredible amount of
content to be made

13. CREATE + PRESENT
How do we make systems that
Compelling, high quality experiences?
(when you’re not there to do it)

14. Transitional Forms

15. Transitional Forms - leaders in AI generated content

16. WE’RE GIVING CONTENT
A MIND OF ITS OWN.

18. 18
WE’VE BUILT A REAL-TIME SHOW
PLATFORM WITH MASSIVE
POTENTIAL.
Our Transforms TV prototype is already proving that
Artificial Intelligence is a viable creative partner for real-
time content through:
★ Live human input
★ Autonomous content generation
★ Dynamic scene assembly
★ Real-time effects
Prototype
Reel (1min)

19. Prompt
Art

20. Prompt
Story

21. Prompts
Show!

22. Text to Experience

24. 1:00, 3:55,, 8:23

28. Learnings
Subtle changes in input can have such huge changes on output
The differences between GPT-2 and GPT-3 are astounding and each one has its
quirks and character
It can be totally dynamic - the show is rendered in real-time - so many
possibilities of how people can interact and modify
Building a completely procedural ‘show’ is a possibility

29. Unity Metacast

31. Metacast
Vision / Mission: Revolutionize how fans
engage with sports
3D volumetric capture of the event
Streamed in 3D to your mobile
Full agency over time and POV
Super hard
We have it working, it’s real

32. Metacast

33. How does it work?

34. Build your own capture studio
Solve volumetric data (partnership)
Run ML pose estimation
Make your own ML pose estimation
Make your own volumetric file format
Make your own compression
Create custom AI camera system
Make custom volumetric NLE
Figure out how to stream giant data
Design app to render 3D quickly
Design 3D interaction UX

35. Build your own capture studio
Solve volumetric data (partnership)
Run ML pose estimation
Make your own ML pose estimation
Make your own volumetric file format
Make your own compression
Create custom AI camera system
Make custom volumetric NLE
Figure out how to stream giant data
Design app to render 3D quickly
Design 3D interaction UX
Difficult

36. Volumetric Pipeline & Platform
End to end delivery of Volumetric data to fans and broadcasters
Raw
Volumetric
Data
First Citizen
Unity Format
Storage &
Streaming
Unity CCD
Unity
Volumetric Editor
Unity App
Active
Distribution
Networks

40. Estimate Depth
Generate Point
Sets
Depth/PC Fusion
(Registration)
Color
Reprojection
Meshing &
Reduction
Algorithm Overview
Type #1 - Calibrated Environment (Metastage etc)

42. Commercial pose
estimation breaks
down…
We had to build our own, which also means training

43. How do you film a
variable scenario
(when you’re not there)

44. Tech partnership with INRIA in France

46. Nobody can be there
to ‘film’ a variable
scenario in real-time

51. New ways to monetize
Virtually try-on and purchase gear from within the app, personalized in-world ads
Drive and measure engagement through integrations
with e-commerce and social media platforms.

52. Be respectful of
the medium

53. “Metatok” gives a
compressed TikTok-
like UX to sports
broadcasts
Swipe left/right for
highlights
Swipe up/down for
different fights or
games
At any time you can
still touch-orbit

54. Where’s the
metaverse?

56. What did we learn?

57. Start with the user
experience first
DO NOT scale up until
you’ve got something
compelling

58. Objectives:
- How can Unity render
point cloud data with
dynamic lighting
- How can Unity power
cinematic sports replays
without keframe
cameras?

59. Focus on the UX
If it doesn’t feel like
magic, you’re not done

60. We’ve been iterating
on this UX for years
It’s hard to make
things feel effortless
Simple is hard
Complex is easy

61. Consumers don’t
really care about
technology
People only care about
how things make them
feel

62. We have been
building content
Bottom up, or
inside-out

65. AI and ML are
beginning to take over
They’re top-down
You start, with the end

66. $$$$
Months of work
Some kid on his gaming PC after dinner

70. THE ENORMOUS SCALE OF DYNAMIC, REAL-TIME
CONTENT IS LITERALLY BEYOND HUMAN.
There simply aren’t enough creators on earth to fulfill the promise of endless,
immersive, hyper-personalized content.

71. Procedural generation
Machine Learning
Artificial Intelligence

72. Thank you!
Questions?
@adamjcmyhill