How to use both big and thick data, quantitative and qualitative user studies, to drive product development, design, and growth? Case studies at KKTV and examples of leveraging Amplitude Analytics.

1. Leveraging Amplitude
Case Studies @ KKTV
Triangulating Data to Drive Growth
Jason @ Growth Team | 20171110
1

2. 2
Jason Hou
Growth Team Lead at KKTV
Joined KKTV 2 months before service launch
Built Growth Team from 0 to 4
Started using Mixpanel, Google Analytics by 2013
Started using Amplitude by 2014

3. 3
Actually..
This sharing is more than
demonstrating how we use Amplitude

4. 4
Actually..
We hope to share our methodology and mindset
of triangulating different types of data
Of course, Amplitude is probably the easiest way
for most people, at the time of this sharing

5. 5
Triangulating Data Triggers Actions
Triangulate
Macro Trends
Quantitative Data
Micro Streams
Qualitative Data
Human Judgement
Industry Experience

6. ● My Highlights of Amplitude Analytics
我眼中的 Amplitde，有哪些亮點？
● Customer Support - From Macro to Micro
讓客服從「巨觀到微觀」 - 從資料點到個體紀錄
● Marketing - High Definition Custom Audience
讓行銷有「高解析度的自建受眾」 - 優化 FB 廣告名單
● Messaging Experiment - From Hypo to Actions
訊息實驗 - 從「假設到行動」
● Cuz “You Never Know” - Triangulating Data
交叉比對 - 因為永遠有意外
6
Agenda

7. 7
KKTV’s Vision: Re-Invent TV Experience

8. My Highlights of Amplitude Analytics
我眼中的 Amplitde，有哪些亮點？
8
01

9. 9
Amplitude Enable Us to..
● Get realtime data streams (no sampling), and fast reporting
● Jump between macro, aggregated trends to micro, individual streams
=> Compare two types of data: quantitative and qualitative
● Group users based on their behaviors (events) to create cohorts
=> No SQL needed to build custom audience or user segments
● Compare cohorts by applying multiple metrics and reports
=> Generate actionable insights, verify hypotheses quickly

10. 10
Powering The Entier Team
50%of the team play data every week
Among the top 10 Amplitude users in KKTV:
4from Customer Support
3from Growth Team
2from Marketing, and 1from Content Licensing

11. 11
How We Leverage Amplitude….?
Let’s see what the heavy users do !
3 cases showing you how...
● Customer support
● Marketing
● Growth / Product
use Amplitude

12. Customer Support - From Macro to Micro
讓客服從「巨觀到微觀」 - 從資料點到個體紀錄
12
02

13. 13
Whenever There’s a New Issue..
We enter the user’s info here

14. 14
This Complete User Activity History Shows Up (Fake Data)

15. 15
This Complete User Activity History Shows Up (Fake Data)

16. 16
This Complete User Activity History Shows Up (Fake Data)

17. 17
This Complete User Activity History Shows Up
Events
Event
Properties
User
Properties
(Fake Data)

18. 18
When There’s a Sudden Increase of Errors...
Oops, bugs ???
(Testing
Data)

19. 19
We Are Able to Dive In Quickly
(Testing
Data)

20. 20
Locate History Around the Error, and Share to Developers

21. 21
Recap: Jump from Macro Trends to Micro Streams
(Testing
Data)

22. 22
More Scenarios of Leveraging User Activity
● UX/UR Designers: Peek into user event history before interview
● Growth Team: QA trackings & AB testings, find growth targets
● Customer Support: Find issues, report bugs
● Developers: Trace events before & after bugs, find root cause

23. 23
Find Growth Targets: For Users Who Dropped Off ...
Where did they go?
What were they doing INSTEAD?
(Demo Data)

24. Marketing - High Definition Custom Audience
讓行銷有「高解析度的自建受眾」 - 優化 FB 廣告名單
24
03

25. 25
What is Behavioral Cohort?
Group users based on their actions
(and/or attributes)
See what they do, how they perform
This sharing is also using this concept
=> Select top Amplitude users in KKTV
=> Show what they do

26. 26
FB Custom or Lookalike Audience Is Key to Boost Ad Return
Create Custom Audience
Create Lookalike Audience

27. 27
Build Cohorts Without SQLs (Demo Data)

28. 28
Build Cohorts Without SQLs
Export, upload to FB,
and then create
Custom Audience
(Demo Data)

29. 29
● Marketing: Create Custom Audience
● Marketing: Send out targeted push notifications
● Content Operation: Discovery user persona from watch history
More Scenarios of Leveraging Behavioral Cohort
● UX/UR Designers: Send out targeted surveys
● Growth Team: Compare cohorts by applying multiple metrics

30. 30
Compare Cohorts by Applying Multiple Metrics (Demo Data)

31. 31
Compare Cohorts by Applying Multiple Metrics (Demo Data)

32. Messaging Experiment - From Hypo to Actions
訊息實驗 - 從「假設到行動」
32
04

33. 33
Assumption vs Reality
● Assumption:
○ KKTV provides cross-platform experience
○ Users will know and jump between platforms
Assumption is great, but...
..always remember to double check it with reality
OK, HOW?

34. 34
Users Who Jumped Between Platforms

35. 35
Assumption vs Reality
● Assumption:
○ KKTV provides cross-platform experience
○ Users will know and jump between platforms
● Reality: (Right after KKTV was launched)
○ We acquired a lot of users on mobile
○ Very few of them used both mobile and web apps

36. 36
Questions Trigger Actions
● Assumption:
○ KKTV provides cross-platform experience
○ Users will know and jump between platforms
● Reality: (Right after KKTV was launched)
○ We acquired a lot of users on mobile
○ Very few of them used both mobile and web apps
● Follow-up questions:
○ For users who jumped between platforms, how are they different?
○ Is it important to encourage users to do so? How?

37. 37
More Observations & Quick Validations
● From user researchers:
○ Mobile users were surprised to know there’s KKTV Web App
○ They expressed satisfaction after using it
○ They described scenarios of when and why they would use it
● By using behavioral cohorts:
○ Cohort 1: Select users who jumped between mobile & web platforms
○ Cohort 2: Select users who didn’t
○ We compare two cohorts by applying retention & conversion metrics
○ => Cohort 1 performs far better then cohort 2

38. ● For users who signed up on mobile platforms...
○ What if we notify them there’s KKTV Web App?
○ How are we going to notify them?
38
Design Experiments
● Experiment examples:
○ Send out a push notification after sign-up, then compare w/ control group
○ Display a in-app welcome message, and show a picture of Web App

39. ● We inject experiment data into Amplitude
○ Separate users in different experiment groups into cohorts
○ Compare results by applying multiple metrics
39
Analyze Experiments

40. Cuz “You Never Know” - Triangulating Data
交叉比對 - 因為永遠有意外
40
05

41. Story of “You Never Know” - 5-Day-Long Session
41
2017-08-17
2017-08-13
Data Scientist Asked:
“How is it even possible?”

42. Story of “You Never Know” - 5-Day-Long Session
42
For a session to end:
{ App is backgrounded }
AND
{ Stops sending events for 5min }

43. 43
Story of “You Never Know” - 5-Day-Long Session

44. 44
Story of “You Never Know” - 5-Day-Long Session
Looks strange..

45. 45
Story of “You Never Know” - 5-Day-Long Session

46. 46
Triangulating Data Triggers Actions
Triangulate
Macro Trends
Quantitative Data
Micro Streams
Qualitative Data
Human Judgement
Industry Experience

47. THANK YOU
47