At Hulu, notifying our viewers when their favorite teams are playing helps us drive growth and improve viewer engagement. However, building this feature was a complex process. Managing our live TV metadata, while generating audiences in real time in high-scalability scenarios, posed unique challenges for the engineering team. In this session, we discuss the challenges in building our real-time notification platform, how Amazon Pinpoint helped us with our goals, and how we architected the solution for global scale and deliverability.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Game On! Building Hulu’s Real-Time
Notification Platform for Live TV
with Amazon Pinpoint
M O B 3 0 4
Ilan Sehayek
Principal Product
Manager
Amazon
Yingan Wang
Principal Engineering
Manager
Hulu
Mingkui Liu
Principal Software
Engineer
Hulu

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
About Hulu
Our goals
Our solution with Amazon Pinpoint
Key learnings
Future plans

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
About Hulu
• Premium streaming service for VOD and live TV
• Hulu originals including “The Handmaid’s Tale,” the Emmy Award-
winning drama
• > 20M subscribers
• > 1M live TV subscribers

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Our goals
• For mobile clients
• Send the push notification to Hulu users
when game starts on live channels
• Tap into the playback of the live game

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Right audience
• User has opted in push
notifications
• User has shown interest in
specific content
• User is able to watch the specific
content
• Local live TV channels make it more
challenging

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deduplication
• Only one notification when it’s
airing on multiple channels
• Notification shouldn’t be sent
again for re-run
• Notification shouldn’t be sent if
the user has watched

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Handling schedule updates
• Notification should be sent at
the right time, even when
program schedule changes
• Program can start earlier/later, or can be
canceled

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Handling schedule updates
• Notification should be sent at
the right time, even when
program schedule changes
• Program can start earlier/later, or can be
canceled

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Handling schedule updates
• Notification should be sent at
the right time, even when
program schedule changes
• Program can start earlier/later, or can be
cancelled

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SLA
• Scalability
• Audience size can be as large as > 10MM

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SLA
• Scalability
• Audience size can be as large as > 10MM
• Multiple large campaigns can be scheduled at the
same time

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SLA
• Scalability
• Audience size can be as large as > 10MM
• Multiple large campaigns can be scheduled at the
same time
• Latency
• We aim to deliver the live game notifications to
end users within 1 min

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
High-level architecture
Mobile client

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
High-level architecture
Mobile client

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why Amazon Pinpoint?
• Amazon Pinpoint can provide the expected SLA
• Especially for the dynamic content campaigns
• Importing segments
• Delivery latency
• Amazon Pinpoint has a lot of flexibility to integrate with backends and
clients to support our unique use cases
• We can use their API/SDK to register users dynamically from backend
• We can use their API/SDK to create campaigns/segments dynamically and in real time
• We are not required to use Amazon Pinpoint client SDK, which is what Hulu client teams
want to avoid (they already have too many third-party SDKs integrated)
• With these, Hulu has full control of implementing Hulu notification scenarios

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Messaging pipeline
Mobile client

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content data model

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Program schedule
• Those channels may have different schedules for the same live game
• Pre-game
• Commercials
• A lot of games could get extended at the last minute
• Schedule data
• Out of band
• In band

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Schedule ingestion
• Hulu gets content schedule
info from our content
partners
• Hulu also buys some third-
party content metadata, such
as Gracenote
• Our metadata team unifies
all signals and publishes real-
time change events

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event trigger
• We listen to the changes and store schedule info in a
Redis-sorted set
• 30 minutes before the game starts, we build the
Amazon Pinpoint segments
• When the game really starts, it triggers the campaign
delivery

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Audience building
Mobile client

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Right audience
• User has opted in push
notifications
• User has shown interest in
specific content
• User is able to watch the specific
content
• Local live TV channels make it more
challenging

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Audience indexing platform
• Functionality
• Given specific content, find all users who have interest in the content and have the rights to
watch the content
• SLA
• We want the targeting results to be accurate
• We want low latency time
• We want high throughput
• We want high uptime

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Challenges—Request pattern
• Huge spike in request pattern
• Most live channels would change/start content at the same time, either every hour or every
half hour
• Some games may trigger millions of notifications, such as the Super Bowl

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Challenges—Updates
• The user’s attributes are always changing
• User could make changes to their interests
• User could change the entitlements
• User could change the location

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep dive—Audience building solutions
• Apache Lucene
• A high-performance, full-featured text search engine
library
• Treat each user as a document and build inverted index
• Search the target users from the inverted index

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep dive—How to build inverted index

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep dive—How to build inverted index

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep dive—How to search on the index
• Example
• The game episode X belongs to team_b
• The episode X can only be watched by live subscriber (entitlement=e2), and within the Las
Vegas area (distribution area= da2)
• Request: Find all users who have an interest in team_b AND who can watch episode X

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep dive—How to search on the index
• Example
• The game episode X belongs to team_b
• The episode X can only be watched by live subscriber (entitlement=e2), and within the Las
Vegas area (distribution area= da2)
• Request: Find all users who have an interest in team_b AND who can watch episode X

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep dive—How to search on the index
• Example
• The game episode X belongs to team_b
• The episode X can only be watched by live subscriber (entitlement=e2), and within the Las
Vegas area (distribution area= da2)
• Request: Find all users who have an interest in team_b AND who can watch episode X

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep dive—How to search on the index
Parse params to get
entities, entitlement, and
location
Find user lists from index
for each index type
Join the user lists

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep dive—Why is our solution efficient?

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep dive—Why is our solution efficient?
- Indexing is an
offline module—
build once and
copy-paste to
engine

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep dive—Why is our solution efficient?
User list is ordered and stored by
skip-list
- Indexing is an
offline module—
build once and
copy-paste to
engine
- Terms are
ordered

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep dive—Why is our solution efficient?
Interest matching & availability
checking are both handled in engine
User list is ordered and stored by
skip-list
- Indexing is an
offline module—
build once and
copy-paste to
engine
- Terms are
ordered

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep dive—Real-time updates
Interests change
Location changes
Entitlement changes
Index changes

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep dive—Scalability
• For 20M users
• Search engine
• Index file size: 1 GB
• Heap memory: 4 GB
• Scalability
• Big room for memory scalability
• Easy to scale to multi-processors for high throughput

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Messaging platform
Mobile client

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Real-time delivery pipeline
• We define a list of processing components
• Deduplication
• Across different categories
• Kids
• Watch history
• Quota control
• Rolling windows
• By category

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Messaging address
• We keep track of all devices
that enabled notifications
• With the context of the
devices
• Time zone
• Device platform

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Messaging delivery
Mobile client

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Pinpoint integration
• Amazon Pinpoint & Amazon
Simple Storage Service (Amazon
S3) APIs
• Import segment: createImportJob
• Keep pulling: getImportJob
• Create campaign: createCampaign

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Pinpoint integration advice
• Load testing via silent
notification
• Large segment handling
• Error handling
• Idempotence
• Token registration
• Daily CAP limitation
• Race condition

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Monitoring with AWS
• Quick setup on Lambda function
• Elasticsearch (with Kibana)
provides:
• Delivery size
• Device distribution
• Delivery latency
• Delivery status
• Integration with Hulu metrics
platform
• Advice
• Scale Amazon Kinesis to avoid data loss

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep link resolution
• Ideally, deep link to playback

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep link resolution
• Ideally, deep link to playback
• In reality, deep link destination varies
to achieve best user experience

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep link resolution
• Ideally, deep link to playback
• In reality, deep link destination varies
to achieve best user experience

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key learnings
• Program schedule accuracy
• Audience building efficiency
• Build vs. buy
• System limitation
• Non-silent vs. silent notification
• Delivery ratio

54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Future plans
• In-game excitement notification
• Notification center
• In-app notification

55. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Yingan Wang
yingan.wang@hulu.com
Mingkui Liu
mingkui.liu@hulu.com

56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.