1.
PyCon Korea 2019
, .
|
1

2.
2
/
+

3.
?

4.
?
5

5.
?
6

6.
?
7

7.
?
8

8.
?

9.
?

10.
.

11.
14
( )
?
?

12.
15
10
20
20

13.
16

14.
.
17

15.
?

16.
.
- -

17.
.
- -

18.
?

19.
22
( )
?
?

20.
:
23
( )
?
?
( ) .

21.
24
10
20
20

22.
: .
25
10
20
20
20

23.
26

24.
: .
27

25.
.

26.
?

27.
.
30
/
TP.

28.
.

29.
.
!32

30.
33
…

31.
34
- - ViewerEnd

32.
• :
CTR(%)
•
• MAB(Multi Armed Bandit)
• User Clustering
-
!35

33.
MAB(Multi Armed Bandit)
• MAB = Exploration( ) and Exploitation( ) Trade-off
• 10%( ) Feedback (impression,
click)
* ε-greedy MAB .
!36

34.
• Feedback CTR(%) .
• CTR(%) = # of clicks / # of impressions
Exploration
0.4% 4.0% 2.9% 7.3% 2.7% 8.7% 6.7% 1.0% 1.9% 8.1% 6.3%
3.6% 6.7% 8.0% 3.1% 3.6% 2.0% 4.4% 3.1% 7.3% 8.2% 2.7%
4.4% 8.1% 0.6% 5.9% 9.2% 7.3% 8.3% 8.6% 4.2% 9.9% 6.9%
* ε-greedy MAB .
!37
MAB(Multi Armed Bandit)

35.
• CTR 90% ( )
• CTR
0.4% 4.0% 2.9% 7.3% 2.7% 8.7% 6.7% 1.0% 1.9% 8.1% 6.3%
3.6% 6.7% 8.0% 3.1% 3.6% 2.0% 4.4% 3.1% 7.3% 8.2% 2.7%
4.4% 8.1% 0.6% 5.9% 9.2% 7.3% 8.3% 8.6% 4.2% 9.9% 6.9%
Exploitation8.0% 8.2%
* ε-greedy MAB .
!38
MAB(Multi Armed Bandit)

36.
0.4% 4.0% 2.9% 7.3% 2.7% 8.7% 6.7% 1.0% 1.9% 8.1% 6.3%
3.6% 6.7% 8.0% 3.1% 3.6% 2.0% 4.4% 3.1% 7.3% 8.2% 2.7%
4.4% 8.1% 0.6% 5.9% 9.2% 7.3% 8.3% 8.6% 4.2% 9.9% 6.9%
Exploitation
(10%) (90%)
&
?
:
?
: .
!39
MAB(Multi Armed Bandit)

37.
0.4% 4.0% 2.9% 7.3% 2.7% 8.7% 6.7% 1.0% 1.9% 8.1% 6.3%
3.6% 6.7% 8.0% 3.1% 3.6% 2.0% 4.4% 3.1% 7.3% 8.2% 2.7%
4.4% 8.1% 0.6% 5.9% 9.2% 7.3% 8.3% 8.6% 4.2% 9.9% 6.9%
Exploitation
Exploration(10%) Exploitation(90%)
• MAB ?
• TS-MAB
ε-greedy
UCB(Upper Confidence Bound)
Lin-UCB
Thompson Sampling
NeuralBandit
LinRel (Linear Associative Reinforcement Learning)
!40
MAB(Multi Armed Bandit)

38.
ε-Greedy MAB ε=0.10
41
10M Impressions
10%(ε)
1M Impressions( )
1 2 3 4 4 5
6 7 8 9 … 100

39.
ε-Greedy MAB ε=0.10
42
10M Impressions
10%(ε)
1M Impressions( )
1.1% 2.0% 8.2% 0.01% 4.6% 1.2%
5.2% 0.1% 0.2% 1.0% … 2.2%
(100 ) 10k Impression

40.
ε-Greedy MAB ε=0.10
43
10M Impressions
10%(ε)
1M Impressions( )
1.1% 2.0% 8.2% 0.01% 4.6% 1.2%
5.2% 0.1% 0.2% 1.0% … 2.2%
(100 ) 10k Impression
CTR = 1.5%
Best
arm
( )
3
8.2%
7
5.2%
4
4.6%
50
3.0%

41.
ε-Greedy MAB ε=0.10
44
10M Impressions
10%(ε)
1M Impressions( )
1.1% 2.0% 8.2% 0.01% 4.6% 1.2%
5.2% 0.1% 0.2% 1.0% … 2.2%
Best
arm
( )
3
8.2%
7
5.2%
4
4.6%
50
3.0%
90%(1-ε)
9M Impressions
(100 ) 10k Impression
CTR = 1.5%
CTR = 5.1%
CTR 4.74%

42.
ε-Greedy MAB ε=0.10
45
10M Impressions
10%(ε)
1M Impressions( )
1.1% 2.0% 8.2% 0.01% 4.6% 1.2%
5.2% 1.5% 0.2% 1.0% … 2.2%
Best
arm
( )
3
8.2%
7
5.2%
4
4.6%
90%(1-ε)
9M Impressions
(100 ) 10k Impression
CTR = 1.5%
CTR = 5.1%
CTR 4.74%
10k Impression
CTR
Impressions CTR (3σ)

43.
ε-Greedy MAB ε=0.10
46
10M Impressions
10%(ε)
1M Impressions( )
1.1% 2.0% 8.2% 0.01% 4.6% 1.2%
5.2% 1.5% 0.2%
10 Impressions
CTR
Impressions CTR

44.
ε-Greedy MAB ε=0.10
47
10M Impressions
10%(ε)
1M Impressions( )
1.1% 2.0% 8.2% 0.01% 4.6% 1.2%
5.2% 1.5% 0.2% 1.0% … 2.2%
Best
arm
( )
3
8.2%
7
5.2%
4
4.6%
90%(1-ε)
9M Impressions
(100 ) 10k Impression
CTR = 1.5%
CTR = 5.1%
CTR 4.74%
CTR
Impressions 99.7%(3σ)

45.
ε-Greedy MAB ε=0.10
48
10M Impressions
10%(ε)
1M Impressions( )
1.1% 2.0% 8.2% 0.01% 4.6% 1.2%
5.2% 1.5% 0.2% 1.0% … 2.2%
Best
arm
( )
3
8.2%
7
5.2%
4
4.6%
50
3.0%
90%(1-ε)
9M Impressions
(100 ) 10k Impression
CTR = 1.5%
CTR = 5.1%
CTR 4.74%
CTR
Impressions 99.7%(3σ)
CTR 3.0%
3.0%
3.0% 3.0% 3.0%
3.0%3.0%

46.
ε-Greedy MAB ε=0.10
49
10M Impressions
10%(ε)
1.1% 2.0%
5.2% 1.5% 0.2% 1.0% … 2.2%
Best
arm
( )
90%(1-ε)
(100 ) 10k Impression
CTR = 1.5%
CTR = 5.1%
CTR 4.74%
CTR
Impressions 99.7%(3σ)
3.0% 3.0%3.0%
Optimal Arm
Impressions (regret )

47.
Thompson Sampling MAB ?

48.
Thompson Sampling MAB
• (arm) CTR Beta(a,b) . ( a=click, b=unclick )
51
1
10%
Impressions : 10 50 100 200 1k 10k
2
25%
Impressions : 10 50 100 200 1k 10k

49.
Thompson Sampling MAB
• (arm) CTR Beta(a,b) ( a=click, b=unclick )
52
1
10%
( ) CTR 15%
1 (10%<15%) 100 Impressions trial
.
Impression ->
Impressions : 10 50 100 200 1k 10k

50.
Thompson Sampling MAB
• (arm) CTR Beta(a,b) ( a=click, b=unclick )
53
2
25%
2 CTR 25%>15%
( )
.
Impressions : 10 50 100 200 1k 10k

51.
• 1K Impressions
54
CTR

52.
• 10K Impressions
55
CTR

53.
• 50K Impressions
56
CTR

54.
• 100K Impressions
57
CTR

55.
• 500K Impressions
58
CTR

56.
• 1M Impressions
59
CTR

57.
• 1M Impressions
60
CTR
CTR (Arm)

58.
• 1M Impressions
61
CTR
CTR
( )
.
.

59.
• 1M Impressions
62
CTR
CTR
( )
.
.
TS-MAB
& Trade-off
Regret( ) .

60.
User Clustering
• CTR .
CTR : 7.6%
A : 25% (30 )
B : 2.1% ( )
C : 7.1% ( )
CTR
!63

61.
User Clustering
• X CTR
• CTR
200 8,000

62.
User Clustering
•
• 8
8
8,000 64,000
CTR

63.
Clustering ?
CB(image,Text)
Feature User Feature
[0.628, 0.88, 0.376, 0.065, 0.849]
[0.508, 0.268, 0.193, 0.125, 0.425]
[0.431, 0.077, 0.012, 0.07, 0.037]
[0.915, 0.294, 0.713, 0.851, 0.423]
[0.508, 0.268, 0.193, 0.125, 0.425]
[0.607, 0.639, 0.554, 0.092, 0.297]
[0.587, 0.319, 0.094, 0.173, 0.177]
[0.409, 0.458, 0.48, 0.319, 0.783]
[0.479, 0.434, 0.618, 0.297, 0.752]
[0.467, 0.206, 0.905, 0.7, 0.568]
, , ,
, ,
, , ,
!66
1
2
3
4
5
6

64.
Clustering ?
14 CB(image,Text)
Feature User Feature
[0.628, 0.88, 0.376, 0.065, 0.849]
[0.508, 0.268, 0.193, 0.125, 0.425]
[0.431, 0.077, 0.012, 0.07, 0.037]
[0.915, 0.294, 0.713, 0.851, 0.423]
[0.508, 0.268, 0.193, 0.125, 0.425]
[0.607, 0.639, 0.554, 0.092, 0.297]
[0.587, 0.319, 0.094, 0.173, 0.177]
[0.409, 0.458, 0.48, 0.319, 0.783]
[0.479, 0.434, 0.618, 0.297, 0.752]
[0.467, 0.206, 0.905, 0.7, 0.568]
, , ,
, ,
, , ,
8 (#0~#7)
?

65.
-
•
• #1, #5, #6, #7
• #0, #2 #3

66.
-
•
• #1, #5, #6, #7
• #0, #2 #3
/

67.
#1
, ,

68.
#3
, ,

69.
/
#3
, ,

70.
Tag

71.
Tag

72.
/
.

73.
?
77
#2
#1
#3
User Clustering
Targeting
CTR
1 : CTR 9.1%
2 : CTR 8.8%
3 : CTR 8.0%
4 : CTR 7.8%
5 : CTR 7.1%
6 : CTR 6.8%
7 : CTR 6.7%
…
MAB
Ranking

74.
?
78
#2
#3
User Clustering
Targeting
Ranker
MAB
Ranking
Targeter

75.
+ MAB

76.
-ViewerEnd
• :
CTR(%)
•
• (Item Feature)
• MAB(Multi Armed Bandit)
!80

77.
?
81
#2
#3
User Clustering
Targeting
Ranker
MAB
Ranking
Targeter

78.
?
82
(Item)
Targeter
Feature
Targeting
Ranker
MAB
Ranking

79.
-ViewerEnd
(CF)
(Text)
(Image)
!83
1 2 3

80.
- Item Features
/ Image
(1) Image Feature
, Text
(2) Text Feature
Feedback
(3) CF-Feature
!84

81.
?
!85

82.
?
!86

83.
?
!87

84.
1 3 41 1 92 1
1 3 41 1 92 1
1 3 41 1 92 1
Image
Text( )
CF( )
!88
1 3 41 1 92 1Image Style

85.
Image
Text( )
CF( )
Image Style
Style transfer network
!89
1 3 41 1 92 1

86.
Image Style
Text( )
CF( )
Image Object Detection Task
Image Object
Pre-trained VGG19 Model
!90
1 3 41 1 92 1

87.
Image Style
Image
CF( )
,
Keyword
Word2Vec
!91
1 3 41 1 92 1

88.
Image Style
Image
Text( )
CF( )
Matrix Factorization(ALS)
with implicit feedback
(Feedback) Item-User
!92
1 3 41 1 92 1

89.
!93
1 3 41 1 92 1
1 3 41 1 92 1
1 3 41 1 92 1
1 3 41 1 92 1

90.
“ ”
(%)
!94

91.
0.4% 4.0% 2.9% 7.3% 2.7% 8.7% 6.7% 1.0% 1.9% 8.1%
0.4% 4.0% 2.9% 7.3% 2.7% 8.7% 6.7% 1.0% 1.9% 8.1%
0.4% 4.4% 2.9% 7.3% 2.3% 8.7% 0.2% 1.0% 1.9% 8.1%
0.4% 6.0% 2.9% 7.3% 2.7% 5.6% 6.7% 1.0% 1.9% 8.1%
MAB
(%)
!95

92.
0.4% 4.0% 2.9% 7.3% 2.7% 8.7% 6.7% 1.0% 1.9% 8.1%
0.4% 4.0% 2.9% 7.3% 2.7% 8.7% 6.7% 1.0% 1.9% 8.1%
0.4% 4.4% 2.9% 7.3% 2.3% 8.7% 0.2% 1.0% 1.9% 8.1%
0.4% 6.0% 2.9% 7.3% 2.7% 5.6% 6.7% 1.0% 1.9% 8.1%
!96

93.
?
97
(Item)
Targeting
CTR
1 : CTR 9.1%
2 : CTR 8.8%
3 : CTR 8.0%
4 : CTR 7.8%
5 : CTR 7.1%
6 : CTR 6.8%
7 : CTR 6.7%
…
MAB
Ranking

94.
+ MAB
!98

95.
!99

96.
100
1 2 3 4 5 6
…
89 90
(Clicks)
Impression

97.
101
1 2 3 4 5 6
…
89 90
(Use Coin)
Impression

98.
102
User Cluster +
+
MAB
MAB
RankerTargeter
=
=

99.
103
+
MAB
Conditional Bandit
Exponential Smoothing
Seen Decay
Soft User Clustering
Retention Model
Unbiased Most Popular
Feature Matching
Targeter
=
=
Ranker

100.
104
• MAB
- Bandit Algorithm = Thompson Sampling(
- Reward = Click (with Unclick )
- Play Arms = Cluster Most Popular
- None Stationary = Exponential Decaying
• 2
- = # of clicks / # of impressions
- = # of use_coins / # of impressions
1. MAB

101.
105
• MAB
- Bandit Algorithm = Thompson Sampling
- Reward = Click (with Unclick )
- Play Arms = Cluster Most Popular
- None Stationary = Exponential Decaying
• 2
- = # of clicks / # of impressions
- = # of use_coins / # of impressions
1. MAB
Use Coin( )
MAB Reward Use Coin, Click + User Coin
by @brandon.lim

102.
106
1. MAB ?
(Beta) (Alpha) -20% —>
? ?
MAB ?
-20%

103.
2. Conditional Bandit
107
1 2 3 4 5 6
…
89 90
by @troye.kwon

104.
2. Conditional Bandit
108
1 2 3 4 5 6
…
89 90
Impressions
Reward=Click( )
α=click, β=unclick
MAB
by @troye.kwon

105.
2. Conditional Bandit
109
1 2 3 4 5 6
…
89
Impressions
Reward=Click( )
α=click, β=unclick
MAB
Reward=Use Coin( )
α=use-coin, β=click
MAB
by @troye.kwon

106.
2. Conditional Bandit ?
110
(Beta) (alpha)
? ?
- MAB .

107.
3. Retention Model
• : .
,
“ ” .
•
• MAB
111
by @jinny.k
+
MAB
Targeter Ranker

108.
3. Retention Model ?
112
by @jinny.k
(CTR) (CVR)
? (CTR) ?

109.
4. Seen decay
• : Negative Feedback
• click impression Ranker
• : (alpha) (Beta)
113

110.
-> CTR
114
•
• Hard Clustering(k-Means) —> Soft Clustering (pLSI)
• Feature Matching
• Targetting Genre/Tag Matching
• MAB non-stationary Exponential Smoothing
• Targeting Unbiased Most Popular
• MAB Hyper parameter Turning

111.
(%)
!115
Soft Clustering (pLSI) Feature Matching
Conditional Bandit Retention Model
Exponential Smoothing
MABUnbiased Most Popular

112.
?
?
!116

113.
!117

114.
?
118

115.
?
119

116.
?
120

117.
?
121

118.
?
122

119.
0.1%
123
= 3.96%
= 6.70%
= 2.63%
= 0.07%

120.
-> 4.56
124
= 3.96%
= 6.70%
= 2.63%
= 0.07%
4.56

121.
-> 4.56
125
= 3.96%
= 6.70%
= 2.63%
= 0.07%
4.56
AB
(<0.001)
Feedback (>4.56days)

122.
/
,
?

123.
.
127
Base : Editor’s ( X) 1.9%
Alpha : 1 4.8%
Beta : 2 5.5%
Gamma : 3 6.5%
CTR
1 2 3 4
…

124.
.
128
Base : Editor’s ( X) 1.9%
Alpha : 1 4.8%
Beta : 2 5.5%
Gamma : 3 6.5%
CTR
+ 242% + 42%
1 2 3 4
…
CTR

125.
.
?

126.
/ ?
?

127.
/ ? ->
? ->

128.
133
YOU
?

129.
.
!134
|