Usman Shakeel, AWS Tech Lead's presentation to the Media Supply Chain track at the Media & Entertainment Cloud Symposium on November 4, 2016

1. Media Supply Chains in AWS Terms
Usman Shakeel | Amazon Web Services

2. Playout & Distribution
Visual Effects & Editing Analytics
DAM & Archive
Digital Supply Chain
Publishing
OTT
Acquisition
AWS Media Segments

3. Playout & Distribution
Elemental, EC2, S3/Glacier
EFS, CloudFront
Visual Effects & Editing
EC2, S3, EFS, EBS, GPU, DX
Analytics
Kinesis, DynamoDB
AML, RedShift, EMR
DAM & Archive
Snowball, S3, Glacier
EC2, DX, Lambda
DynamoDB, RDS
Digital Supply Chain
Elemental, EC2, S3/Glacier
CloudFront, EFS, Snowball
GPU, DX
Publishing
EC2, S3, Glacier,
ELB, CloudFront
OTT
Elemental
EC2, S3
CloudFront
Acquisition
Elemental, EC2,
S3/Glacier
CloudFront,
Snowball, DX
AWS Media Segments

4. Media Supply Chains
Playout & Distribution
Production/Post Production B2B/Cinematic Delivery
DAM & Archive
Digital Supply Chain
Publishing
OTT
Content Acquisition
Ingestion
Manual QC
Auto-QC
Metadata Extraction
Transcoding
Delivery
Packaging
Clips/Edits
Complex workflow/State management
Bursty sub-workflow tasks (transcoding/auto-QC, Packaging)
Elastic Licensing
Content movement between workflow tasks (a common repository)
Different delivery formats for each affiliate and downstream
Manual tasks – (content movement or interactivity)
CMS/DAM
Normalization

5. Normalization
All-In Cloud-Enabled Domino Effect
Playout & Distribution
Live TV
Production/Post
Production
Editorial
Dailies
Post
Social Media
Marketing/Promo
B2B/Cinematic
Delivery
DPP, IMF, NABA
DAM & Archive
Active Archive
Golden Copy
Preservation
Digital Supply Chain
Publishing
OTT
Live-To-VOD
AVOD, SVOD, TVOD
Catchup TV
Cloud PVR
Content Acquisition
Live
File
Ingestion
Manual QC
Auto-QC
Metadata Extraction
Transcoding
Delivery
Packaging
Clips/Edits
CMS/DAM

6. Ingest Store Edit/Manual Tasks Process
PUSH OR PULL
CONTENT & ARTIFACTS
ACCESS SOURCE FILES
LOCATED IN S3
CREATE, EDIT, QC &
COLLABORATE IN THE CLOUD
SCALE OUT ON ELASTIC
CAPACITY FOR PROCESSING
(Transcoding, QC, Packaging and distribution)
Media production and delivery companies are leveraging AWS to accelerate their media supply chains, ingestion,
storage, editing/manual QC of content and processing with highly scalable cloud computing and storage.
Using Cloud for Media Supply Chains

7. Ingest
PUSH OR PULL
CONTENT & ARTIFACTS
AWS Services
• Amazon S3 Transfer Acceleration: Fast data transfers
• AWS Direct Connect: Dedicated network connectivity
• AWS Snowball: Petabyte scale data transfer
• Amazon Cloudfront: Global content delivery network
• Amazon Storage Gateway: On-prem to cloud storage connectivity
• Amazon Marketplace: One-click partner software deployments
• Elemental: Content contribution from on-premises encoders
Partners
Media Supply Chains

8. AWS Services
• Amazon S3/SIA: Highly scalable, durable object storage
• AWS Storage Gateway: On-prem to cloud storage connectivity
• Amazon Glacier: Secure, durable, petabyte-scale data archival
• Amazon EFS: Simple, elastic, scalable file system
• Amazon EBS: Persistent block storage
Partners
Store
ACCESS SOURCE FILES
LOCATED IN S3
Media Supply Chains

9. Edit/Manual Tasks
CREATE, EDIT, Clips, Manual QC &
COLLABORATE IN THE CLOUD
AWS Services
• Amazon EC2: Scalable CPU/GPU computing capacity
• Amazon Workspaces: Virtual desktops in the cloud
• Amazon Appstream: Remote Application Streaming
• Amazon EFS: Simple, elastic, scalable file system
• Amazon EBS: Persistent block storage
• Amazon S3: Highly scalable, durable object storage
Partners
Media Supply Chains

10. Process
SCALE OUT ON ELASTIC
CAPACITY FOR PROCESSING
AWS Services
• Amazon EC2: Scalable CPU/GPU/Spot computing capacity
• Amazon EFS: Simple, elastic, scalable file system
• Amazon EBS: Persistent block storage
• Amazon S3: Highly scalable, durable object storage
• Elemental Cloud: On-demand, scalable video processing
Partners
Media Supply Chains

11. AWS Services
• Amazon S3/SIA: Highly scalable, durable object storage
• AWS Storage Gateway: On-prem to cloud storage connectivity
• Amazon Glacier: Secure, durable, petabyte-scale data archival
• Amazon EFS: Simple, elastic, scalable file system
• Amazon EBS: Persistent block storage
Partners
Asset Management
Metadata for Assets
and supply chain workflow
Media Supply Chains

12. INFRASTRUCTURE

13. Task Specific Compute Instances
Compute Intensive
Intel ES-2666 v3 (Haswell)
optimized specifically for EC2
Memory Intensive
Lowest price point per GiB of RAM
GPUs
1,536 CUDA cores
4GB of video memory
Enhanced Networking
Higher PPS, Lower network jitter, low latency
IO Intensive
SSD Storage, EBS Optimized
High Storage
24 x 2000 GiB per instance
Amazon S3
SQS Queues (orchestrated by Amazon SWF)
M4
M3
C4
C3 I3G2
R4
P2 D2
R3
X1

14. Reserved
Make a low, one-time
payment and receive
a significant discount
on the hourly charge
For committed
utilization
On-Demand
Pay for compute
capacity by the hour
with no long-term
commitments
For spiky workloads,
or to define needs
Spot
Bid for unused capacity
at a Spot Price -
fluctuates based on
supply & demand
For time-insensitive or
transient workloads
Consumption Models

15. AWS Storage Services Maturity
Amazon EFS
File
Amazon EBS
Amazon EC2
Instance Store
Block
Amazon S3
Amazon
Glacier
Object
Data Transfer (Ingest/Egress)
AWS Direct Connect AWS Snowball ISV Connectors Amazon Kinesis Firehose S3 VPC
EndPoint
AWS Storage
Gateway
S3 – Infrequent
Access
Events
S3 Event
Notifications
S3 Transfer
Acceleration

16. Post Production tasks
NLE/Review & Approve/VFX
(Hybrid, on premises or in cloud)
Raw Assets
Secondary Copy
Amazon Glacier in another Region
Raw Assets
Primary Copy
Amazon Glacier
Finished (Golden Copy) Archive
Primary Copy
Amazon Glacier
Asset Management (DAM/MAM)
(Hybrid, on premises or in cloud)
Active Archive
Amazon S3
(Standard)
Active Archive
Amazon S3
(Infrequent Acccess)
Secondary & Tertiary Copies
Amazon Glacier in another Region
A secondary offsite copy of the active archive – usually cold
Used for DR purposes
Media Supply Chain
(Hybrid, on premises or in cloud)
B2B Delivery over DX or Internet
Amazon Glacier
ingest
Affiliates/Third Party Vendors
ingest
AWS Direct Connect
AWS Snowball
S3 Transfer
Acceleration
(Primary Copy) AWS Region 1
(Secondary & Tertiary Copies – optional)
AWS Region 2 or on-premises
Glacier
S3
Infrequent
Access
S3
Standard
Lifecycle Mgmt
Production Archive

17. Availability Zone A
Private subnet Private subnet
AWS
region
Virtual Private
Gateway
Content Value-add
Service
Content Value-add
Service
Availability Zone B
Amazon
S3
VPC
VPN
connection
VPC Endpoints
No IGW
No NAT
No public IPs
Free
Robust access control
Customer
network
Access Control with S3 VPC Endpoint

18. corporate data center AWS cloud
users
Content
Servers
disk
tape storage
Processing
Layer
Amazon S3
Amazon EBS
Amazon
Glacier
KMS/
HSMClient side
encryption
role
IAM
Encrypted
Content
role
Direct Connect
S3 VPC Endpoint
Hybrid Media Network

19. STORAGE PERFORMANCE

20. EBS Volumes
SSD Based HDD Based
gp2
General Purpose SSD
io1
Provisioned IOPS SSD
st1
Throughput Optimized HDD
sc1
Cold HDD
Baseline: 3IOPS per GB up to 10,000
Burst: 3000 IOPS (for volumes up to 1TB)
Throughput: 160 MB/s
Latency: Single digit ms
Capacity: 1GB to 16TB
Usage
Boot Volumes
Bursty workloads (Rendering)
Databases (Transactional)
Low latency operations (Streaming)
Baseline: 100IOPS up to 20,000
Throughput: 320 MB/s
Latency: Single digit ms
Capacity: 4GB to 16TB
Usage
Sustained IOPS
Critical Applications with IOPS
Guarantee (no burst)
Baseline: 40MB/s per TB up to 500MB/s
Burst: 250MB/s per TB up to 500MB.s
Capacity: 500GB to 16TB
Usage
Large Block, high throughput sequential
Transcoding, QC, Packaging
Rendering
Baseline: 12MB/s per TB up to 192MB/s
Burst: 80MB/s per TB up to 250MB.s
Capacity: 500GB to 16TB
Usage
Sequential
Logging, Active Archive/Backup

21. Object store as Processing tier
Partition
1
Partition 2
Partition
n
An S3 Backend Storage
Auto-scaled/HA Front-end
TCP
Distributed Key-map
Optimizations
Replication across multiple Availability Zones
No FS – everything GET/PUT/DELETE
• Write once – Read many
• Globally unique namespaces (object names unique within a bucket)
Requests Per Second (>300)
• Partitioning
• Random Prefix
Puts vs. Gets
• CDN – GET heavy workloads
Network Throughput
• TCP Window Scaling
• TCP Selective Acknowledgements
• S3 Session Usage
• Multi-part Uploads
• Buffered Writes
• Range GETs
Consistency/Commit Latency
• Read after Write
• Objects replicated across multiple AZs
• Not suitable today for workloads with <MS latency/IO requirement
Client(s)
Client-side bandwidth

22. Elastic File System (EFS)
• Designed to support Peta-Byte scale file systems
• Throughput scales linearly to storage
– Burst: 100MB/sec per TB of storage for 12 hours each day
– Baseline: 50MB/sec per TB of storage continuously
A 10 TB filesystem can burst up to 1GB/s for 12 hours each day
and drive500MB/sec aggregate read / write throughput continuously.
• Same latency spec across each AZ
• Thousands of concurrent NFS connections
• Works great for Large I/O sizes
• Pay for only what you use not what you provision
• Managed with multi-copy durability
Render Farms
Big data analytics
Transcode Farms
Home directories
Content management
Web serving
Metadata-
intensive
jobs
High throughput and parallel I/O
Low latency and serial I/O

23. Workload Specific Storage
Object Store
(Amazon S3)
Block Store
(Amazon EBS)
Shared FS
(Amazon EFS)
Performance
Per Operation Latency Variable Lowest, Consistent Low Consistent
Throughput Scale Multiple GBs per sec Single GBs per sec Multiple GBs per sec
Data
Availability/Durability
Across Multi-AZ Redundant within an AZ Across Multi-AZ
Characteristics
Access 1-1000’s of Clients A single EC2 instance 1-1000’s of Clients
Use Cases
Active Archive and
lifecycle to Cold Archive
Media Processing
(Transcoding, rendering,
QC, Farms) depending on
Application conformity
Databases
Boot Volumes
Media Processing
(Transcoding, rendering,
QC, Farms) Singleton
Streaming
Data Analytics
Media Processing
(Transcoding, rendering,
QC, Farms) Distributed
Content Management
Web Serving/Publishing

24. SOLUTIONS

25. AWS Import/ Export
S3 multi-part API
For Source, Renditions, Metadata Sidecar Files
Ingest
AWS Elastic
Beanstalk
DAM
Web Service
CloudFront
Distribution
Amazon
DynamoDB
Amazon
CloudSearch
DAM Catalog
Rendition Workers
EC2 ASG
Amazon SNS Topic
Amazon SQS Queue
Rendition Jobs
Amazon SQS Queue
Metadata Processing Jobs
S3 Notification
Elastic Transcoder
Proxy /
Thumbnail/Renditions
Generation
Processing
Metadata Workers
EC2 ASG
Simple Media Supply Chain

26. AWS Import/ Export
S3 multi-part API
S3 as backend storage for Content Files
accessible to other processing tasks
Amazon Elastic
Transcoder
S3 Notification
Trigger a Lambda Function
to Start a transcoding job
Ingest
S3 Notification
Lambda function to
generate a signed
URL to share the file
Update CMS or
Metadata
Serverless Media Supply Chain (concept)

27. § Expand compute
capacity of Manager
node(s)
§ 100ms billing & short
run-time
§ Ideal for chunk
storage, pre- & post-
processing
§ Don’t forget binary
encapsulation!
https://github.com/binoculars/aws-lambda-ffmpeg
Lambda Pre and Post Processing

28. A high-level Media Supply Chain in the cloud (concept)
EC2, ETS can request the data-key on behalf of customer
Provide CPK for S3 encryption at rest
AWS Import/ Export
S3 multi-part API
Amazon S3 For Source, Renditions, Metadata, Sidecar Files
Ingest
AWS Elastic
Beanstalk
CloudFront
Distribution
DynamoDB CloudSearch
S3 Notification
Content Processing
Amazon SWF, SQS, SNS and AWS
Lambda can be used to orchestrate
the processing workflow tasks
CloudFront
Distribution
User Logs
Amazon DynamoDB
Amazon Redshift
Users Connected Devices
Analytics and Recommendations Engine
Amazon EMR
Amazon DataPipeline
Elastic Transcoder
EC2 ASG
Rendition Workers
EC2 ASG
Metadata Workers
Metadata Proc queueRenditions Queue
Amazon Elastic
Transcoder
Rights Management
EC2 ASGELB
DRM Wrap
Encrypt
Package
Watermark
QC
EC2 ASG
Amazon SNS Topic
DAM Web Service
Personalization Recommendations
EC2 ASG
RDS
Front End Web Application
DAM Catalog
Content
Archive
Amazon
Glacier
Key Management Service
Content owner provides
the master key
BI & Reporting
Users Web Browser
Content Owner

29. IMF PACKAGING
Stack: s3-ta / snowball / ?
Partners: teradek / nuke (vr/360) / ?

30. Cloud Based Editing
• NVIDIA GPU based EC2 instances
• Teradici PCoIP
• Amazon AppStream/Nice DCV
• Amazon Workspaces
Artist applications
Artist workstation
(Dumb Client)
Remote Application
running on a G2 instance
G2

31. Thank You!