Learn how Amazon Rekognition is using deep learning-based image and video analysis to power data-driven decision making for creative asset production, more targeted influencer marketing and advertising, and visual authentication for a variety of use cases. Learn about Amazon Rekognition image and video features. Hear customer stories from specific vertical use cases, such as media and advertising, and walk through some demonstrations and architectures for common use cases.

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Venkatesh Bagaria
Senior Product Manager, Amazon Rekognition
Zach Schwitsky
Co-Founder and CEO, Limbik
BDA303
Amazon Rekognition
Deep Learning-Based Image and Video Analysis

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Rekognition
VideoImage &

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Rekognition Image
Object and Scene
Detection
Facial
Analysis
Face
Recognition
Text in Image
Deep Learning-Based Image analysis service
Unsafe Image
Detection
Celebrity
Recognition

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Rekognition Video
Deep Learning-Based Video analysis service
Object and Activity
Detection
Pathing Face Detection &
Recognition
Real-time Live
Stream
Unsafe Video
Detection
Celebrity
Recognition

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Rekognition Customers

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Rekognition
Regions
US West (Oregon)
(us-west-2) US East (N. Virginia)
(us-east-1)
EU (Ireland)
(eu-west-1)
AWS GovCloud (US)
(us-gov-west-1)
Asia Pacific (Tokyo)
(ap-northeast-1)
Asia Pacific (Sydney)
(ap-southeast-2)
US East (Ohio)
(us-east-2)

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Console demo

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DetectLabels
{
"Image": {
"Bytes": blob,
"S3Object": {
"Bucket": "string",
"Name": "string",
"Version": "string"
}
},
"MaxLabels": number,
"MinConfidence": number
}
Image API – Request and Response
{
"Labels": [
{
"Confidence": number,
"Name": "string”
}
],
"OrientationCorrection": "string"
}

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
GetLabelDetection
StartLabelDetection
{
“ClientRequestToken": "string",
"JobTag": "string",
"MinConfidence": number,
"NotificationChannel": {
"RoleArn": "string",
"SNSTopicArn": "string”
},
"Video": {
"S3Object": {
"Bucket": "string",
"Name": "string",
"Version": "string”
}
}
}
Video API – Request and Response
{
"JobStatus": string,
"StatusMessage": string,
"VideoMetadata": {
"Format": string,
"Codec": string,
"DurationMillis": number,
"FrameRate": float,
"FrameWidth": number,
"FrameHeight": number
},
"NextToken": string,
"Labels": [
{
"Timestamp": number,
"Label":
{
"Name": string,
"Confidence": float
}
}
],
...
JobId

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Boat 99.3%
Plant 95.1%
Harbor 94.8%
Yacht 78.1%
Dock 75.7%
City 72.4%
Architecture 71.8%
Urban 63.9%
Building 62.3%
Marina 60.3%
Plaza 51.1%
Spire 50.8%
Neighborhood 50.7%
Flower 50.6%
Waterfront 94.8%
Object and Scene detection

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Blowing a candle Drinking
Object and Activity detection

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Before After
Influencer Marketing
2.6x more insights, on 30% more posts

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Context from user-generated content
Person 99.2%
Dog 95.7%
Person 99.2%
Snowboarding 98.1%

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Zach Schwitzky
Co-Founder and CEO
Predictive Analytics for Video

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Our Mission
▪ We believe that machine learning can inspire human creativity.
▪ Our mission is to create products that transform how content
creators use data.
▪ Limbik powers data-driven video for leading brands, publishers,
media companies, and creative agencies.

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Who we are & What we do
▪ NYC-based startup founded by a data scientist, an agency executive, a
technologist, and me.
▪ The first Data Studio for video.
▪ Integrate the world’s largest set of otherwise disconnected video
content and engagement data.
▪ Combine big data, technology, and Content Science® to help clients
make and support creative decisions.

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Limbik System
Video Content
Engagement Data
Viewer Data
Limbik Annotate®
Content Science®
Content
Creators
Publishers

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Rekognition helps us predict attention
▪ The reasons people pay attention to short-form video are different, and far less
obvious than the reasons they watch TV shows and movies
▪ We need to identify every visual, audible and contextual feature to understand
the triggers of attention
▪ Amazon Rekognition delivers the most accurate and robust results; cheaper
than other solutions; easiest to integrate into our video analysis stack
▪ For a 30-second beverage/alcohol commercial, 29% of the video’s performance
can be determined by its featured drinking occasion

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Reference Architecture
Video
Amazon S3
Bucket
Limbik Video
Manager
Amazon Mechanical
Turk (Topic)
Amazon Rekognition
Image
Amazon Rekognition
Video
Amazon
Transcribe
Amazon
Comprehend
TensorFlow
on AWS
Amazon Machine
Learning
Limbik Content
Analysts
Amazon Mechanical
Turk (Topic)

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Rekognition helps us predict attention

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The power of data-driven video
▪ Creative agency; Bev/Alc client
▪ UGC prior to retaining agency; we started with the
agency on June 1
▪ 4 – 7 new Facebook videos / week
▪ Reduce duration and increase Seconds Per View
(SPV)
▪ Thumbnail image + drinking occasion
▪ Play rate up 69%; SPV up 206% (duration down
34%)

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The power of data-driven video

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The power of data-driven video
▪ Existing, underperforming recipe
video
▪ Limited to editable features
▪ Changed from 6 shot changes in
the first 10 seconds to 4
▪ 46% lift in Avg. View Time

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Customers / Data partners
Limbik by the numbers:
▪ 29 T1 IAB categories; 350+
T2 categories
▪ 1.3 billion avg. monthly
views per category (T1)
▪ 22 billion minutes of video
analyzed
▪ 40,000 unique features
extracted
▪ Thousands of decisions for
hundreds of global clients

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Image and Video Moderation
Hierarchical taxonomy provides greater control for users
Top-Level Category Second-Level Category
Explicit Nudity
Nudity
Graphic Male Nudity
Graphic Female Nudity
Sexual Activity
Partial Nudity
Suggestive
Female Swimwear Or Underwear
Male Swimwear Or Underwear
Revealing Clothes

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Reviewing user-generated content

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Image quality
Facial landmarks
Demographic data Emotions
General attributes
Facial pose
Brightness 24.0%
Sharpness 99.9%
EyeLeft,EyeRight,Nose
RightPupil,LeftPupil
MouthRight,LeftEyeBrowUp
Age Range 29–45
Gender: Male 91.6%
Happy 90.38%
Smile:True 99.8%
EyesOpen:True 99.7%
Beard:True 99.8%
Mustache:True 99.6%
Pitch 2.059
Roll 4.569
Yaw 2.970
Facial Detection and Analysis

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Support for up to 100 faces
Counting faces and measuring crowd sentiment

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Reviewing uploaded photos
No faces detected Eyes open: False 99.99% 10 faces detected

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Face Search

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Finding missing people

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Celebrity recognition

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Finding similar faces
WHHA: Presidential and First Lady Look-Alike

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Face-based authentication
• Facility access control
• Know your customer
• Remote password reset

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Device Camera
1. Images stored in
Amazon S3
AWS Lambda
Amazon DynamoDB
Amazon Rekognition
Amazon Cognito
Face collectionAmazon S3
Amazon Rekognition
2. Use Lambda to process
images with Amazon
Rekognition
3. Index faces into a face
collection, get FaceId
4. Create person name
and collection stats
metadata with FaceId5. SearchFacesByImage with
the collection using AWS SDK
6. Get back search results
with names and stats
Face Search architecture

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Best practices for faces
• Face size: At least 24–32 pixels or ~5%
• Filter out poor quality faces when indexing
• Blurry
• Too small
• Extreme pose
• For interactive use cases, send image as bytes
• Reduce image resolution for faster response
• Multiple training images per identity
• Face model versioning

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Rekognition Text in Image
Extract textual content from real-world images in various layouts, fonts, and styles

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Reading signs

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
2. Submit
Image
4. DetectFaces 7. DetectText
1. Upload
3. Store image AWS Lambda AWS Step Functions
5.DetectLabels 6.DetectModerationLabels
8. Store Metadata
& analysis Amazon
DynamoDB
Amazon ES
Content review system

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Developer resources for Amazon Rekognition
Homepage: https://aws.amazon.com/rekognition/
Amazon Machine Learning Blog – Amazon Rekognition:
https://aws.amazon.com/blogs/ai/tag/amazon-rekognition/
Serverless image recognition processing backend: https://github.com/awslabs/lambda-
refarch-imagerecognition
Reviewing user generated content demo code:
https://github.com/mbtaws/rekognition-reviewing-user-content

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Submit session feedback
1. Tap the Schedule icon.
2. Select the session you attended.
3. Tap Session Evaluation to submit your feedback.

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!