정부환(ben.hur) / kakao --- 온라인 광고는 현재 인터넷 비즈니스 모델을 가능케하는 핵심 동인이다. 새로운 서비스가 등장하고 시장 규모가 커짐에 따라서 온라인 광고 생태계도 함께 진화하고 여러 기술적 도전이 있다. 지면 중심에서 오디언스 중심으로, 단순 광고 노출과 클릭에서 광고주의 필요를 반영한 다양한 전환으로, 그리고 수의 계약에서 실시간 자동 입찰로 온라인 광고 생태계가 진화하고 있다. 이런 변화의 흐름에서 여러 지면에 접속하는 사용자들의 성향을 즉시 분석해서 가장 적합한 광고를 실시간으로 선택하고 노출하는 것은 기술적으로 매우 어려운 문제다. 본 발표는 카카오의 광고 랭킹에 필요한 데이터와 알고리즘을 간략히 소개한다. ------------- <Glossary> - Audience: 광고 도메인에서 사용자 (User)를 뜻함 - Publisher: 앱이나 웹 등의 광고 지면(inventory)/매체(media)을 제공하는 사람 - SSP: Supplier-side platform - DSP: Demand-side platform - DMP: Data management platform - MAT: Mobile app tracking - Pixel: 광고의 전화을 추척하기 위해서 웹에 심어두는 스크립트 - AdX: Ad Exchange, 광고의 mediation이 이뤄지는 마켓 - RTB: Real-time bidding - Programmatic buying: AdX에서 프로그램에 의해서 (자동으로) 광고 입찰 및 낙찰이 이뤄지는 것 - Impression: 광고 노출 - ROAS: Return on ad spending - eCPM: effective cost per mille (1천회 노출당 기대 비용/수익) - CPM/CPC/CPA: Cost Per mille/click/acquisition(action, conversion) - CTR/CVR: Clickthrough rate, (post-click) conversion rate - SGD: Stochastic gradient descent - FTRL: Follow-the-regularized-leader - FM/FFM/FwFM: Factorization machines / Field-aware FM / Field-weighted FM - DCN: Deep & cross network - LDA: Latent dirichlet allocation - DNN: Deep neural network (DL) - AE: Auto-encoder - GBDT: Gradient-boosting decision tree - Targeting: 광고주가 자신의 광고가 노출될 오디언스 (사용자)를 제한하는 것 - Retargeting: 사용자의 특정 행동 (i.e., 광고주 사이트 방문)에 반응해서 광고를 노출/제한하는 것 - LookALike: 광고주가 제공한 오디언스 그룹과 유사한 특징을 갖는 오디언스군 (유사확장타게팅) - PPC: Pay per click 클릭당 과금액 - RIG: Relative information gain - NE: Normalized entropy - AUC: Area under ROC Curve - GSP/VCG: Generalized second price auction / Vickrey-Clarke-Groves auction - DNT: Do not track

1. Intelligence on Kakao Advertising
beyond state-of-the-art
ben.hur@kakaocorp.com

2. Nothing is certain but death and taxes.
- Benjamin Franklin

3. Nothing is certain but death and taxes.
- Benjamin Franklin
“AD”.
- Benjamin Franklin

4. User Advertiser
Publisher
Contract

5. Audience
(User)
Advertiser
Publisher
AdTech Eco (AdX/RTB/Programmatic Buying)
Traffic Req4Bid
BidImpression
Visit
Ad Selection
- Filtering
- Ranking
- Pricing
Mediation
(Auction)
** Visit-to-Impression is done within 1~200ms
for at least ten-thousand requests per second.
SSP DSP
DMP
Audience Tracking
(MAT/SDK/Pixel)
Transaction log (train)
Audience Info. (target)
Log
Ad
Media
Log
SSP: Supply-side platform
DSP: Demand-side platform
DMP: Data management platform
MAT: Mobile app tracking
AdX: Ad exchange
RTB: Real-time bidding

6. Different dreams among {Audience, Advertiser, Publisher, Platform}
AD
Audience Advertiser
Publisher Platform
Annoying vs Information Marketing channel (ROAS)
Reserve Price Revenue (Adv - Pub)

7. Balance between Revenue and Relevance
Audience Provide relevant and interesting information
Advertiser Gather new or loyal customers through a low cost channel
Publisher Guarantee a stable and predictable revenue source
Platform Maximize overall welfare (& utility) with user satisfaction

8. Effective Cost Per Mille (eCPM)
Expected/estimated revenue per (a thousand) impression(s)

9. eCPM = Bid Amount x Likeness
Revenue:
Amount that an advertiser is willing to pay for desired actions
Relevance:
How much does a user like to do the actions

10. In CPC,
eCPM = BAclick x pCTR
number of clicks / number of impressions

11. Leave (y = 0) Click (y = 1)
Ad | X
X: Traffic properties (ADxUSRxPLx…)

12. Pr(y = 1 | x, ad)
Aggregation of historical data
Learning from historical data
Reactive method vs Predictive method

13. More likely to click
Sum of traffic properties (WTX)
Logistic Regression
(Maximum entropy)
Pr(y = 1|x) =
1
1 + exp(−wTx)
Softmax of binary (1/0) output
Less likely to click

14. Pr(y = 1|x) =
1
1 + exp(−wTx)
Loss = ∥y − ̂y∥2
y<latexit sha1_base64="paQhm8QH9RuJYjMoRm3VlxatzsM=">AAAB6HicdVDLSsNAFJ3UV62vqks3g0VwFSY1tHVXdOOyBfuANpTJdNKOnUzCzEQIoV/gxoUibv0kd/6Nk7aCih64cDjnXu69x485UxqhD6uwtr6xuVXcLu3s7u0flA+PuipKJKEdEvFI9n2sKGeCdjTTnPZjSXHoc9rzZ9e537unUrFI3Oo0pl6IJ4IFjGBtpHY6KleQfdmoVd0aRDZCdafq5KRady9c6BglRwWs0BqV34fjiCQhFZpwrNTAQbH2Miw1I5zOS8NE0RiTGZ7QgaECh1R52eLQOTwzyhgGkTQlNFyo3ycyHCqVhr7pDLGeqt9eLv7lDRIdNLyMiTjRVJDloiDhUEcw/xqOmaRE89QQTCQzt0IyxRITbbIpmRC+PoX/k27VdpDttN1K82oVRxGcgFNwDhxQB01wA1qgAwig4AE8gWfrznq0XqzXZWvBWs0cgx+w3j4BR9uNPw==</latexit><latexit sha1_base64="paQhm8QH9RuJYjMoRm3VlxatzsM=">AAAB6HicdVDLSsNAFJ3UV62vqks3g0VwFSY1tHVXdOOyBfuANpTJdNKOnUzCzEQIoV/gxoUibv0kd/6Nk7aCih64cDjnXu69x485UxqhD6uwtr6xuVXcLu3s7u0flA+PuipKJKEdEvFI9n2sKGeCdjTTnPZjSXHoc9rzZ9e537unUrFI3Oo0pl6IJ4IFjGBtpHY6KleQfdmoVd0aRDZCdafq5KRady9c6BglRwWs0BqV34fjiCQhFZpwrNTAQbH2Miw1I5zOS8NE0RiTGZ7QgaECh1R52eLQOTwzyhgGkTQlNFyo3ycyHCqVhr7pDLGeqt9eLv7lDRIdNLyMiTjRVJDloiDhUEcw/xqOmaRE89QQTCQzt0IyxRITbbIpmRC+PoX/k27VdpDttN1K82oVRxGcgFNwDhxQB01wA1qgAwig4AE8gWfrznq0XqzXZWvBWs0cgx+w3j4BR9uNPw==</latexit><latexit sha1_base64="paQhm8QH9RuJYjMoRm3VlxatzsM=">AAAB6HicdVDLSsNAFJ3UV62vqks3g0VwFSY1tHVXdOOyBfuANpTJdNKOnUzCzEQIoV/gxoUibv0kd/6Nk7aCih64cDjnXu69x485UxqhD6uwtr6xuVXcLu3s7u0flA+PuipKJKEdEvFI9n2sKGeCdjTTnPZjSXHoc9rzZ9e537unUrFI3Oo0pl6IJ4IFjGBtpHY6KleQfdmoVd0aRDZCdafq5KRady9c6BglRwWs0BqV34fjiCQhFZpwrNTAQbH2Miw1I5zOS8NE0RiTGZ7QgaECh1R52eLQOTwzyhgGkTQlNFyo3ycyHCqVhr7pDLGeqt9eLv7lDRIdNLyMiTjRVJDloiDhUEcw/xqOmaRE89QQTCQzt0IyxRITbbIpmRC+PoX/k27VdpDttN1K82oVRxGcgFNwDhxQB01wA1qgAwig4AE8gWfrznq0XqzXZWvBWs0cgx+w3j4BR9uNPw==</latexit><latexit sha1_base64="paQhm8QH9RuJYjMoRm3VlxatzsM=">AAAB6HicdVDLSsNAFJ3UV62vqks3g0VwFSY1tHVXdOOyBfuANpTJdNKOnUzCzEQIoV/gxoUibv0kd/6Nk7aCih64cDjnXu69x485UxqhD6uwtr6xuVXcLu3s7u0flA+PuipKJKEdEvFI9n2sKGeCdjTTnPZjSXHoc9rzZ9e537unUrFI3Oo0pl6IJ4IFjGBtpHY6KleQfdmoVd0aRDZCdafq5KRady9c6BglRwWs0BqV34fjiCQhFZpwrNTAQbH2Miw1I5zOS8NE0RiTGZ7QgaECh1R52eLQOTwzyhgGkTQlNFyo3ycyHCqVhr7pDLGeqt9eLv7lDRIdNLyMiTjRVJDloiDhUEcw/xqOmaRE89QQTCQzt0IyxRITbbIpmRC+PoX/k27VdpDttN1K82oVRxGcgFNwDhxQB01wA1qgAwig4AE8gWfrznq0XqzXZWvBWs0cgx+w3j4BR9uNPw==</latexit>
ˆy<latexit sha1_base64="QmQDjeeN4gpKWLfKwkS/Fz5qGt4=">AAAB7nicdVDLSgNBEOyNrxhfUY9eBoPgKcwEMckt6MVjBPOAZAmzk9lkyOyDmVlhWfIRXjwo4tXv8ebfOJtEUNGChqKqm+4uL5ZCG4w/nMLa+sbmVnG7tLO7t39QPjzq6ihRjHdYJCPV96jmUoS8Y4SRvB8rTgNP8p43u8793j1XWkThnUlj7gZ0EgpfMGqs1BtOqcnS+ahcwVWMMSEE5YTUL7ElzWajRhqI5JZFBVZoj8rvw3HEkoCHhkmq9YDg2LgZVUYwyeelYaJ5TNmMTvjA0pAGXLvZ4tw5OrPKGPmRshUatFC/T2Q00DoNPNsZUDPVv71c/MsbJMZvuJkI48TwkC0X+YlEJkL572gsFGdGppZQpoS9FbEpVZQZm1DJhvD1KfqfdGtVgqvk9qLSulrFUYQTOIVzIFCHFtxAGzrAYAYP8ATPTuw8Oi/O67K14KxmjuEHnLdP/reQAA==</latexit><latexit sha1_base64="QmQDjeeN4gpKWLfKwkS/Fz5qGt4=">AAAB7nicdVDLSgNBEOyNrxhfUY9eBoPgKcwEMckt6MVjBPOAZAmzk9lkyOyDmVlhWfIRXjwo4tXv8ebfOJtEUNGChqKqm+4uL5ZCG4w/nMLa+sbmVnG7tLO7t39QPjzq6ihRjHdYJCPV96jmUoS8Y4SRvB8rTgNP8p43u8793j1XWkThnUlj7gZ0EgpfMGqs1BtOqcnS+ahcwVWMMSEE5YTUL7ElzWajRhqI5JZFBVZoj8rvw3HEkoCHhkmq9YDg2LgZVUYwyeelYaJ5TNmMTvjA0pAGXLvZ4tw5OrPKGPmRshUatFC/T2Q00DoNPNsZUDPVv71c/MsbJMZvuJkI48TwkC0X+YlEJkL572gsFGdGppZQpoS9FbEpVZQZm1DJhvD1KfqfdGtVgqvk9qLSulrFUYQTOIVzIFCHFtxAGzrAYAYP8ATPTuw8Oi/O67K14KxmjuEHnLdP/reQAA==</latexit><latexit sha1_base64="QmQDjeeN4gpKWLfKwkS/Fz5qGt4=">AAAB7nicdVDLSgNBEOyNrxhfUY9eBoPgKcwEMckt6MVjBPOAZAmzk9lkyOyDmVlhWfIRXjwo4tXv8ebfOJtEUNGChqKqm+4uL5ZCG4w/nMLa+sbmVnG7tLO7t39QPjzq6ihRjHdYJCPV96jmUoS8Y4SRvB8rTgNP8p43u8793j1XWkThnUlj7gZ0EgpfMGqs1BtOqcnS+ahcwVWMMSEE5YTUL7ElzWajRhqI5JZFBVZoj8rvw3HEkoCHhkmq9YDg2LgZVUYwyeelYaJ5TNmMTvjA0pAGXLvZ4tw5OrPKGPmRshUatFC/T2Q00DoNPNsZUDPVv71c/MsbJMZvuJkI48TwkC0X+YlEJkL572gsFGdGppZQpoS9FbEpVZQZm1DJhvD1KfqfdGtVgqvk9qLSulrFUYQTOIVzIFCHFtxAGzrAYAYP8ATPTuw8Oi/O67K14KxmjuEHnLdP/reQAA==</latexit><latexit sha1_base64="QmQDjeeN4gpKWLfKwkS/Fz5qGt4=">AAAB7nicdVDLSgNBEOyNrxhfUY9eBoPgKcwEMckt6MVjBPOAZAmzk9lkyOyDmVlhWfIRXjwo4tXv8ebfOJtEUNGChqKqm+4uL5ZCG4w/nMLa+sbmVnG7tLO7t39QPjzq6ihRjHdYJCPV96jmUoS8Y4SRvB8rTgNP8p43u8793j1XWkThnUlj7gZ0EgpfMGqs1BtOqcnS+ahcwVWMMSEE5YTUL7ElzWajRhqI5JZFBVZoj8rvw3HEkoCHhkmq9YDg2LgZVUYwyeelYaJ5TNmMTvjA0pAGXLvZ4tw5OrPKGPmRshUatFC/T2Q00DoNPNsZUDPVv71c/MsbJMZvuJkI48TwkC0X+YlEJkL572gsFGdGppZQpoS9FbEpVZQZm1DJhvD1KfqfdGtVgqvk9qLSulrFUYQTOIVzIFCHFtxAGzrAYAYP8ATPTuw8Oi/O67K14KxmjuEHnLdP/reQAA==</latexit>

15. Find w that minimizes the negative log likelihood (w/ L2 regularization)
Control model complexity
NLL for logistic regression
arg min
w
n
∑
i=1
log(1 + exp(−yiwT
xi)) +
λ
2
∥w∥2
2

16. Stochastic Gradient Descent (SGD)
⌘t<latexit sha1_base64="SU/TSRqhSNKT3zfwyFM+mpJHyjY=">AAAB73icbVBNS8NAEJ3Ur1q/qh69BIvgqSQi6LHoxWMF+wFtKJvttF262cTdiVBC/4QXD4p49e9489+4bXPQ1gcDj/dmmJkXJlIY8rxvp7C2vrG5Vdwu7ezu7R+UD4+aJk41xwaPZazbITMohcIGCZLYTjSyKJTYCse3M7/1hNqIWD3QJMEgYkMlBoIzslK7i8R6GU175YpX9eZwV4mfkwrkqPfKX91+zNMIFXHJjOn4XkJBxjQJLnFa6qYGE8bHbIgdSxWL0ATZ/N6pe2aVvjuItS1F7lz9PZGxyJhJFNrOiNHILHsz8T+vk9LgOsiESlJCxReLBql0KXZnz7t9oZGTnFjCuBb2VpePmGacbEQlG4K//PIqaV5Ufa/q319Wajd5HEU4gVM4Bx+uoAZ3UIcGcJDwDK/w5jw6L86787FoLTj5zDH8gfP5A11OkCs=</latexit><latexit sha1_base64="SU/TSRqhSNKT3zfwyFM+mpJHyjY=">AAAB73icbVBNS8NAEJ3Ur1q/qh69BIvgqSQi6LHoxWMF+wFtKJvttF262cTdiVBC/4QXD4p49e9489+4bXPQ1gcDj/dmmJkXJlIY8rxvp7C2vrG5Vdwu7ezu7R+UD4+aJk41xwaPZazbITMohcIGCZLYTjSyKJTYCse3M7/1hNqIWD3QJMEgYkMlBoIzslK7i8R6GU175YpX9eZwV4mfkwrkqPfKX91+zNMIFXHJjOn4XkJBxjQJLnFa6qYGE8bHbIgdSxWL0ATZ/N6pe2aVvjuItS1F7lz9PZGxyJhJFNrOiNHILHsz8T+vk9LgOsiESlJCxReLBql0KXZnz7t9oZGTnFjCuBb2VpePmGacbEQlG4K//PIqaV5Ufa/q319Wajd5HEU4gVM4Bx+uoAZ3UIcGcJDwDK/w5jw6L86787FoLTj5zDH8gfP5A11OkCs=</latexit><latexit sha1_base64="SU/TSRqhSNKT3zfwyFM+mpJHyjY=">AAAB73icbVBNS8NAEJ3Ur1q/qh69BIvgqSQi6LHoxWMF+wFtKJvttF262cTdiVBC/4QXD4p49e9489+4bXPQ1gcDj/dmmJkXJlIY8rxvp7C2vrG5Vdwu7ezu7R+UD4+aJk41xwaPZazbITMohcIGCZLYTjSyKJTYCse3M7/1hNqIWD3QJMEgYkMlBoIzslK7i8R6GU175YpX9eZwV4mfkwrkqPfKX91+zNMIFXHJjOn4XkJBxjQJLnFa6qYGE8bHbIgdSxWL0ATZ/N6pe2aVvjuItS1F7lz9PZGxyJhJFNrOiNHILHsz8T+vk9LgOsiESlJCxReLBql0KXZnz7t9oZGTnFjCuBb2VpePmGacbEQlG4K//PIqaV5Ufa/q319Wajd5HEU4gVM4Bx+uoAZ3UIcGcJDwDK/w5jw6L86787FoLTj5zDH8gfP5A11OkCs=</latexit><latexit sha1_base64="SU/TSRqhSNKT3zfwyFM+mpJHyjY=">AAAB73icbVBNS8NAEJ3Ur1q/qh69BIvgqSQi6LHoxWMF+wFtKJvttF262cTdiVBC/4QXD4p49e9489+4bXPQ1gcDj/dmmJkXJlIY8rxvp7C2vrG5Vdwu7ezu7R+UD4+aJk41xwaPZazbITMohcIGCZLYTjSyKJTYCse3M7/1hNqIWD3QJMEgYkMlBoIzslK7i8R6GU175YpX9eZwV4mfkwrkqPfKX91+zNMIFXHJjOn4XkJBxjQJLnFa6qYGE8bHbIgdSxWL0ATZ/N6pe2aVvjuItS1F7lz9PZGxyJhJFNrOiNHILHsz8T+vk9LgOsiESlJCxReLBql0KXZnz7t9oZGTnFjCuBb2VpePmGacbEQlG4K//PIqaV5Ufa/q319Wajd5HEU4gVM4Bx+uoAZ3UIcGcJDwDK/w5jw6L86787FoLTj5zDH8gfP5A11OkCs=</latexit>
gt<latexit sha1_base64="P/GvVIeqVKiemQWRJKaaMovVVQM=">AAAB9XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbduKxgH9DWMplO2qGTSZi5UUrIf7hxoYhb/8Wdf+OkzUJbDwwczrmXe+b4sRQGXffbWVldW9/YLG2Vt3d29/YrB4ctEyWa8SaLZKQ7PjVcCsWbKFDyTqw5DX3J2/7kJvfbj1wbEal7nMa8H9KREoFgFK300Aspjv0gHWWDFLNBperW3BnIMvEKUoUCjUHlqzeMWBJyhUxSY7qeG2M/pRoFkzwr9xLDY8omdMS7lioactNPZ6kzcmqVIQkibZ9CMlN/b6Q0NGYa+nYyT2kWvVz8z+smGFz1U6HiBLli80NBIglGJK+ADIXmDOXUEsq0sFkJG1NNGdqiyrYEb/HLy6R1XvPcmnd3Ua1fF3WU4BhO4Aw8uIQ63EIDmsBAwzO8wpvz5Lw4787HfHTFKXaO4A+czx9Ca5L+</latexit><latexit sha1_base64="P/GvVIeqVKiemQWRJKaaMovVVQM=">AAAB9XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbduKxgH9DWMplO2qGTSZi5UUrIf7hxoYhb/8Wdf+OkzUJbDwwczrmXe+b4sRQGXffbWVldW9/YLG2Vt3d29/YrB4ctEyWa8SaLZKQ7PjVcCsWbKFDyTqw5DX3J2/7kJvfbj1wbEal7nMa8H9KREoFgFK300Aspjv0gHWWDFLNBperW3BnIMvEKUoUCjUHlqzeMWBJyhUxSY7qeG2M/pRoFkzwr9xLDY8omdMS7lioactNPZ6kzcmqVIQkibZ9CMlN/b6Q0NGYa+nYyT2kWvVz8z+smGFz1U6HiBLli80NBIglGJK+ADIXmDOXUEsq0sFkJG1NNGdqiyrYEb/HLy6R1XvPcmnd3Ua1fF3WU4BhO4Aw8uIQ63EIDmsBAwzO8wpvz5Lw4787HfHTFKXaO4A+czx9Ca5L+</latexit><latexit sha1_base64="P/GvVIeqVKiemQWRJKaaMovVVQM=">AAAB9XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbduKxgH9DWMplO2qGTSZi5UUrIf7hxoYhb/8Wdf+OkzUJbDwwczrmXe+b4sRQGXffbWVldW9/YLG2Vt3d29/YrB4ctEyWa8SaLZKQ7PjVcCsWbKFDyTqw5DX3J2/7kJvfbj1wbEal7nMa8H9KREoFgFK300Aspjv0gHWWDFLNBperW3BnIMvEKUoUCjUHlqzeMWBJyhUxSY7qeG2M/pRoFkzwr9xLDY8omdMS7lioactNPZ6kzcmqVIQkibZ9CMlN/b6Q0NGYa+nYyT2kWvVz8z+smGFz1U6HiBLli80NBIglGJK+ADIXmDOXUEsq0sFkJG1NNGdqiyrYEb/HLy6R1XvPcmnd3Ua1fF3WU4BhO4Aw8uIQ63EIDmsBAwzO8wpvz5Lw4787HfHTFKXaO4A+czx9Ca5L+</latexit><latexit sha1_base64="P/GvVIeqVKiemQWRJKaaMovVVQM=">AAAB9XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbduKxgH9DWMplO2qGTSZi5UUrIf7hxoYhb/8Wdf+OkzUJbDwwczrmXe+b4sRQGXffbWVldW9/YLG2Vt3d29/YrB4ctEyWa8SaLZKQ7PjVcCsWbKFDyTqw5DX3J2/7kJvfbj1wbEal7nMa8H9KREoFgFK300Aspjv0gHWWDFLNBperW3BnIMvEKUoUCjUHlqzeMWBJyhUxSY7qeG2M/pRoFkzwr9xLDY8omdMS7lioactNPZ6kzcmqVIQkibZ9CMlN/b6Q0NGYa+nYyT2kWvVz8z+smGFz1U6HiBLli80NBIglGJK+ADIXmDOXUEsq0sFkJG1NNGdqiyrYEb/HLy6R1XvPcmnd3Ua1fF3WU4BhO4Aw8uIQ63EIDmsBAwzO8wpvz5Lw4787HfHTFKXaO4A+czx9Ca5L+</latexit>
wt<latexit sha1_base64="wQsvs8XlfPgJ6APhixgXICv3Sn0=">AAAB9XicbVDLSsNAFL2pr1pfVZdugkVwVRIRdFl047KCfUAby2Q6aYdOJmHmxlJC/sONC0Xc+i/u/BsnbRbaemDgcM693DPHjwXX6DjfVmltfWNzq7xd2dnd2z+oHh61dZQoylo0EpHq+kQzwSVrIUfBurFiJPQF6/iT29zvPDGleSQfcBYzLyQjyQNOCRrpsR8SHPtBOs0GKWaDas2pO3PYq8QtSA0KNAfVr/4woknIJFJBtO65ToxeShRyKlhW6SeaxYROyIj1DJUkZNpL56kz+8woQzuIlHkS7bn6eyMlodaz0DeTeUq97OXif14vweDaS7mME2SSLg4FibAxsvMK7CFXjKKYGUKo4iarTcdEEYqmqIopwV3+8ippX9Rdp+7eX9YaN0UdZTiBUzgHF66gAXfQhBZQUPAMr/BmTa0X6936WIyWrGLnGP7A+vwBWvuTDg==</latexit><latexit sha1_base64="wQsvs8XlfPgJ6APhixgXICv3Sn0=">AAAB9XicbVDLSsNAFL2pr1pfVZdugkVwVRIRdFl047KCfUAby2Q6aYdOJmHmxlJC/sONC0Xc+i/u/BsnbRbaemDgcM693DPHjwXX6DjfVmltfWNzq7xd2dnd2z+oHh61dZQoylo0EpHq+kQzwSVrIUfBurFiJPQF6/iT29zvPDGleSQfcBYzLyQjyQNOCRrpsR8SHPtBOs0GKWaDas2pO3PYq8QtSA0KNAfVr/4woknIJFJBtO65ToxeShRyKlhW6SeaxYROyIj1DJUkZNpL56kz+8woQzuIlHkS7bn6eyMlodaz0DeTeUq97OXif14vweDaS7mME2SSLg4FibAxsvMK7CFXjKKYGUKo4iarTcdEEYqmqIopwV3+8ippX9Rdp+7eX9YaN0UdZTiBUzgHF66gAXfQhBZQUPAMr/BmTa0X6936WIyWrGLnGP7A+vwBWvuTDg==</latexit><latexit sha1_base64="wQsvs8XlfPgJ6APhixgXICv3Sn0=">AAAB9XicbVDLSsNAFL2pr1pfVZdugkVwVRIRdFl047KCfUAby2Q6aYdOJmHmxlJC/sONC0Xc+i/u/BsnbRbaemDgcM693DPHjwXX6DjfVmltfWNzq7xd2dnd2z+oHh61dZQoylo0EpHq+kQzwSVrIUfBurFiJPQF6/iT29zvPDGleSQfcBYzLyQjyQNOCRrpsR8SHPtBOs0GKWaDas2pO3PYq8QtSA0KNAfVr/4woknIJFJBtO65ToxeShRyKlhW6SeaxYROyIj1DJUkZNpL56kz+8woQzuIlHkS7bn6eyMlodaz0DeTeUq97OXif14vweDaS7mME2SSLg4FibAxsvMK7CFXjKKYGUKo4iarTcdEEYqmqIopwV3+8ippX9Rdp+7eX9YaN0UdZTiBUzgHF66gAXfQhBZQUPAMr/BmTa0X6936WIyWrGLnGP7A+vwBWvuTDg==</latexit><latexit sha1_base64="wQsvs8XlfPgJ6APhixgXICv3Sn0=">AAAB9XicbVDLSsNAFL2pr1pfVZdugkVwVRIRdFl047KCfUAby2Q6aYdOJmHmxlJC/sONC0Xc+i/u/BsnbRbaemDgcM693DPHjwXX6DjfVmltfWNzq7xd2dnd2z+oHh61dZQoylo0EpHq+kQzwSVrIUfBurFiJPQF6/iT29zvPDGleSQfcBYzLyQjyQNOCRrpsR8SHPtBOs0GKWaDas2pO3PYq8QtSA0KNAfVr/4woknIJFJBtO65ToxeShRyKlhW6SeaxYROyIj1DJUkZNpL56kz+8woQzuIlHkS7bn6eyMlodaz0DeTeUq97OXif14vweDaS7mME2SSLg4FibAxsvMK7CFXjKKYGUKo4iarTcdEEYqmqIopwV3+8ippX9Rdp+7eX9YaN0UdZTiBUzgHF66gAXfQhBZQUPAMr/BmTa0X6936WIyWrGLnGP7A+vwBWvuTDg==</latexit>
wt+1<latexit sha1_base64="r/oTFMkBdTR9L3i5TOvM6rbcAK4=">AAAB+XicbVDLSsNAFL2pr1pfUZduBosgCCURQZdFNy4r2Ae0IUymk3boZBJmJpUS8iduXCji1j9x5984abPQ1gMDh3Pu5Z45QcKZ0o7zbVXW1jc2t6rbtZ3dvf0D+/Coo+JUEtomMY9lL8CKciZoWzPNaS+RFEcBp91gclf43SmVisXiUc8S6kV4JFjICNZG8m17EGE9DsLsKfczfeHmvl13Gs4caJW4JalDiZZvfw2GMUkjKjThWKm+6yTay7DUjHCa1wapogkmEzyifUMFjqjysnnyHJ0ZZYjCWJonNJqrvzcyHCk1iwIzWeRUy14h/uf1Ux3eeBkTSaqpIItDYcqRjlFRAxoySYnmM0MwkcxkRWSMJSbalFUzJbjLX14lncuG6zTch6t687asowoncArn4MI1NOEeWtAGAlN4hld4szLrxXq3PhajFavcOYY/sD5/ALPpk68=</latexit><latexit sha1_base64="r/oTFMkBdTR9L3i5TOvM6rbcAK4=">AAAB+XicbVDLSsNAFL2pr1pfUZduBosgCCURQZdFNy4r2Ae0IUymk3boZBJmJpUS8iduXCji1j9x5984abPQ1gMDh3Pu5Z45QcKZ0o7zbVXW1jc2t6rbtZ3dvf0D+/Coo+JUEtomMY9lL8CKciZoWzPNaS+RFEcBp91gclf43SmVisXiUc8S6kV4JFjICNZG8m17EGE9DsLsKfczfeHmvl13Gs4caJW4JalDiZZvfw2GMUkjKjThWKm+6yTay7DUjHCa1wapogkmEzyifUMFjqjysnnyHJ0ZZYjCWJonNJqrvzcyHCk1iwIzWeRUy14h/uf1Ux3eeBkTSaqpIItDYcqRjlFRAxoySYnmM0MwkcxkRWSMJSbalFUzJbjLX14lncuG6zTch6t687asowoncArn4MI1NOEeWtAGAlN4hld4szLrxXq3PhajFavcOYY/sD5/ALPpk68=</latexit><latexit sha1_base64="r/oTFMkBdTR9L3i5TOvM6rbcAK4=">AAAB+XicbVDLSsNAFL2pr1pfUZduBosgCCURQZdFNy4r2Ae0IUymk3boZBJmJpUS8iduXCji1j9x5984abPQ1gMDh3Pu5Z45QcKZ0o7zbVXW1jc2t6rbtZ3dvf0D+/Coo+JUEtomMY9lL8CKciZoWzPNaS+RFEcBp91gclf43SmVisXiUc8S6kV4JFjICNZG8m17EGE9DsLsKfczfeHmvl13Gs4caJW4JalDiZZvfw2GMUkjKjThWKm+6yTay7DUjHCa1wapogkmEzyifUMFjqjysnnyHJ0ZZYjCWJonNJqrvzcyHCk1iwIzWeRUy14h/uf1Ux3eeBkTSaqpIItDYcqRjlFRAxoySYnmM0MwkcxkRWSMJSbalFUzJbjLX14lncuG6zTch6t687asowoncArn4MI1NOEeWtAGAlN4hld4szLrxXq3PhajFavcOYY/sD5/ALPpk68=</latexit><latexit sha1_base64="r/oTFMkBdTR9L3i5TOvM6rbcAK4=">AAAB+XicbVDLSsNAFL2pr1pfUZduBosgCCURQZdFNy4r2Ae0IUymk3boZBJmJpUS8iduXCji1j9x5984abPQ1gMDh3Pu5Z45QcKZ0o7zbVXW1jc2t6rbtZ3dvf0D+/Coo+JUEtomMY9lL8CKciZoWzPNaS+RFEcBp91gclf43SmVisXiUc8S6kV4JFjICNZG8m17EGE9DsLsKfczfeHmvl13Gs4caJW4JalDiZZvfw2GMUkjKjThWKm+6yTay7DUjHCa1wapogkmEzyifUMFjqjysnnyHJ0ZZYjCWJonNJqrvzcyHCk1iwIzWeRUy14h/uf1Ux3eeBkTSaqpIItDYcqRjlFRAxoySYnmM0MwkcxkRWSMJSbalFUzJbjLX14lncuG6zTch6t687asowoncArn4MI1NOEeWtAGAlN4hld4szLrxXq3PhajFavcOYY/sD5/ALPpk68=</latexit>
Loss/Cost function (w)
(Global) minimum
(Local) minimum
ηt =
α
β + ∑
t
s=1
g2
s
wt+1 = wt − ηtgt

17. FTRL-Proximal (Online)
Follow-the-leaders
Proximal (convexity for stability)
Regularization (sparsity)
Reference: Ad click prediction: a view from the trenches (2013)
wt+1 = arg min
w
(g1:t ⋅ w +
1
2
t
∑
s=1
σs∥w − ws∥2
2 + λ1∥w∥1)
wt+1 = wt − ηtgt

18. Recent advances on response prediction
— FM/FFM/FwFM (**Factorization Machines)
— Deep & Cross Network (DCN)
— Model ensemble
Interaction & latent
Pr(y=1|X) = 1 / (1 + exp(-y*FM(X)))
Nonlinear embedding
New state-of-the-art

19. Pr(y = 1 | x)
= 1 / (1 + exp(-wTx))
** The pictograms are only for explanation, in that it does not imply that Kakao uses such audience information.

20. (Hopefully) Almost activities in Kakao (+ 𝛂)
with cautious treatments
— Law and guidance (i.e., Privacy)
— De-identification and k-anonymity
— Abstraction: Estimation and aggregation
— No merging between internal and external user data
— & technical and economical barriers

21. M F 20 30 40 50 ADF SUBS …
1 0 0 1 0 0 0 1 0 0 0 0 1 0 0
Estimation Aggregation
Classification (Bayesian inference) Gender/Age-band estimation
Simple but syntactic
Clustering / LDA (topic modeling)
Hashing trick
Gradient Boosting Tree
DNN / AE
Effective but slow
Promising as others do

22. GBDT+LR (Facebook)
DCN (Google)
DNN + GBDT (MS)
LDA + LR (Kakao)
2 {0, 1}d
<latexit sha1_base64="YL/bYKPJDajrGI4enkt5M5HXN5E=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBg5REBD0WvXisYD+giWWz2bRLN5uwuymUkH/ixYMiXv0n3vw3btsctPXBwOO9GWbmBSlnSjvOt1VZW9/Y3Kpu13Z29/YP7MOjjkoySWibJDyRvQArypmgbc00p71UUhwHnHaD8d3M706oVCwRj3qaUj/GQ8EiRrA20sC2PSaQlzsXyPWKpzwsBnbdaThzoFXilqQOJVoD+8sLE5LFVGjCsVJ910m1n2OpGeG0qHmZoikmYzykfUMFjqny8/nlBTozSoiiRJoSGs3V3xM5jpWaxoHpjLEeqWVvJv7n9TMd3fg5E2mmqSCLRVHGkU7QLAYUMkmJ5lNDMJHM3IrICEtMtAmrZkJwl19eJZ3Lhus03IerevO2jKMKJ3AK5+DCNTThHlrQBgITeIZXeLNy68V6tz4WrRWrnDmGP7A+fwAt7JK1</latexit><latexit sha1_base64="YL/bYKPJDajrGI4enkt5M5HXN5E=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBg5REBD0WvXisYD+giWWz2bRLN5uwuymUkH/ixYMiXv0n3vw3btsctPXBwOO9GWbmBSlnSjvOt1VZW9/Y3Kpu13Z29/YP7MOjjkoySWibJDyRvQArypmgbc00p71UUhwHnHaD8d3M706oVCwRj3qaUj/GQ8EiRrA20sC2PSaQlzsXyPWKpzwsBnbdaThzoFXilqQOJVoD+8sLE5LFVGjCsVJ910m1n2OpGeG0qHmZoikmYzykfUMFjqny8/nlBTozSoiiRJoSGs3V3xM5jpWaxoHpjLEeqWVvJv7n9TMd3fg5E2mmqSCLRVHGkU7QLAYUMkmJ5lNDMJHM3IrICEtMtAmrZkJwl19eJZ3Lhus03IerevO2jKMKJ3AK5+DCNTThHlrQBgITeIZXeLNy68V6tz4WrRWrnDmGP7A+fwAt7JK1</latexit><latexit sha1_base64="YL/bYKPJDajrGI4enkt5M5HXN5E=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBg5REBD0WvXisYD+giWWz2bRLN5uwuymUkH/ixYMiXv0n3vw3btsctPXBwOO9GWbmBSlnSjvOt1VZW9/Y3Kpu13Z29/YP7MOjjkoySWibJDyRvQArypmgbc00p71UUhwHnHaD8d3M706oVCwRj3qaUj/GQ8EiRrA20sC2PSaQlzsXyPWKpzwsBnbdaThzoFXilqQOJVoD+8sLE5LFVGjCsVJ910m1n2OpGeG0qHmZoikmYzykfUMFjqny8/nlBTozSoiiRJoSGs3V3xM5jpWaxoHpjLEeqWVvJv7n9TMd3fg5E2mmqSCLRVHGkU7QLAYUMkmJ5lNDMJHM3IrICEtMtAmrZkJwl19eJZ3Lhus03IerevO2jKMKJ3AK5+DCNTThHlrQBgITeIZXeLNy68V6tz4WrRWrnDmGP7A+fwAt7JK1</latexit><latexit sha1_base64="YL/bYKPJDajrGI4enkt5M5HXN5E=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBg5REBD0WvXisYD+giWWz2bRLN5uwuymUkH/ixYMiXv0n3vw3btsctPXBwOO9GWbmBSlnSjvOt1VZW9/Y3Kpu13Z29/YP7MOjjkoySWibJDyRvQArypmgbc00p71UUhwHnHaD8d3M706oVCwRj3qaUj/GQ8EiRrA20sC2PSaQlzsXyPWKpzwsBnbdaThzoFXilqQOJVoD+8sLE5LFVGjCsVJ910m1n2OpGeG0qHmZoikmYzykfUMFjqny8/nlBTozSoiiRJoSGs3V3xM5jpWaxoHpjLEeqWVvJv7n9TMd3fg5E2mmqSCLRVHGkU7QLAYUMkmJ5lNDMJHM3IrICEtMtAmrZkJwl19eJZ3Lhus03IerevO2jKMKJ3AK5+DCNTThHlrQBgITeIZXeLNy68V6tz4WrRWrnDmGP7A+fwAt7JK1</latexit>
⌃<latexit sha1_base64="xl6gt5zesTVLaQIKxfcD+hK+ntk=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKEI9BLx4jmgckS5idzCZj5rHMzAphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntJ6oNU/LBThIaCjyULGYEWye1evdsKHC/XPGr/hxolQQ5qUCORr/81RsokgoqLeHYmG7gJzbMsLaMcDot9VJDE0zGeEi7jkosqAmz+bVTdOaUAYqVdiUtmqu/JzIsjJmIyHUKbEdm2ZuJ/3nd1MZXYcZkkloqyWJRnHJkFZq9jgZMU2L5xBFMNHO3IjLCGhPrAiq5EILll1dJ66Ia+NXg7rJSv87jKMIJnMI5BFCDOtxCA5pA4BGe4RXePOW9eO/ex6K14OUzx/AH3ucPa1+PAw==</latexit><latexit sha1_base64="xl6gt5zesTVLaQIKxfcD+hK+ntk=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKEI9BLx4jmgckS5idzCZj5rHMzAphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntJ6oNU/LBThIaCjyULGYEWye1evdsKHC/XPGr/hxolQQ5qUCORr/81RsokgoqLeHYmG7gJzbMsLaMcDot9VJDE0zGeEi7jkosqAmz+bVTdOaUAYqVdiUtmqu/JzIsjJmIyHUKbEdm2ZuJ/3nd1MZXYcZkkloqyWJRnHJkFZq9jgZMU2L5xBFMNHO3IjLCGhPrAiq5EILll1dJ66Ia+NXg7rJSv87jKMIJnMI5BFCDOtxCA5pA4BGe4RXePOW9eO/ex6K14OUzx/AH3ucPa1+PAw==</latexit><latexit sha1_base64="xl6gt5zesTVLaQIKxfcD+hK+ntk=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKEI9BLx4jmgckS5idzCZj5rHMzAphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntJ6oNU/LBThIaCjyULGYEWye1evdsKHC/XPGr/hxolQQ5qUCORr/81RsokgoqLeHYmG7gJzbMsLaMcDot9VJDE0zGeEi7jkosqAmz+bVTdOaUAYqVdiUtmqu/JzIsjJmIyHUKbEdm2ZuJ/3nd1MZXYcZkkloqyWJRnHJkFZq9jgZMU2L5xBFMNHO3IjLCGhPrAiq5EILll1dJ66Ia+NXg7rJSv87jKMIJnMI5BFCDOtxCA5pA4BGe4RXePOW9eO/ex6K14OUzx/AH3ucPa1+PAw==</latexit><latexit sha1_base64="xl6gt5zesTVLaQIKxfcD+hK+ntk=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKEI9BLx4jmgckS5idzCZj5rHMzAphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntJ6oNU/LBThIaCjyULGYEWye1evdsKHC/XPGr/hxolQQ5qUCORr/81RsokgoqLeHYmG7gJzbMsLaMcDot9VJDE0zGeEi7jkosqAmz+bVTdOaUAYqVdiUtmqu/JzIsjJmIyHUKbEdm2ZuJ/3nd1MZXYcZkkloqyWJRnHJkFZq9jgZMU2L5xBFMNHO3IjLCGhPrAiq5EILll1dJ66Ia+NXg7rJSv87jKMIJnMI5BFCDOtxCA5pA4BGe4RXePOW9eO/ex6K14OUzx/AH3ucPa1+PAw==</latexit>
<latexit sha1_base64="WVBNru380J4XCjCR9Lt9t8h8e1I=">AAAB7XicbVDLSgNBEOyNrxhfqx69DAbBU9gVQY9BLx4jmAckS5idzCZj5rHMzAphyT948aCIV//Hm3/jJNmDJhY0FFXddHfFKWfGBsG3V1pb39jcKm9Xdnb39g/8w6OWUZkmtEkUV7oTY0M5k7RpmeW0k2qKRcxpOx7fzvz2E9WGKflgJymNBB5KljCCrZNaPcOGAvf9alAL5kCrJCxIFQo0+v5Xb6BIJqi0hGNjumGQ2ijH2jLC6bTSywxNMRnjIe06KrGgJsrn107RmVMGKFHalbRorv6eyLEwZiJi1ymwHZllbyb+53Uzm1xHOZNpZqkki0VJxpFVaPY6GjBNieUTRzDRzN2KyAhrTKwLqOJCCJdfXiWti1oY1ML7y2r9poijDCdwCucQwhXU4Q4a0AQCj/AMr/DmKe/Fe/c+Fq0lr5g5hj/wPn8AnF+PIw==</latexit><latexit sha1_base64="WVBNru380J4XCjCR9Lt9t8h8e1I=">AAAB7XicbVDLSgNBEOyNrxhfqx69DAbBU9gVQY9BLx4jmAckS5idzCZj5rHMzAphyT948aCIV//Hm3/jJNmDJhY0FFXddHfFKWfGBsG3V1pb39jcKm9Xdnb39g/8w6OWUZkmtEkUV7oTY0M5k7RpmeW0k2qKRcxpOx7fzvz2E9WGKflgJymNBB5KljCCrZNaPcOGAvf9alAL5kCrJCxIFQo0+v5Xb6BIJqi0hGNjumGQ2ijH2jLC6bTSywxNMRnjIe06KrGgJsrn107RmVMGKFHalbRorv6eyLEwZiJi1ymwHZllbyb+53Uzm1xHOZNpZqkki0VJxpFVaPY6GjBNieUTRzDRzN2KyAhrTKwLqOJCCJdfXiWti1oY1ML7y2r9poijDCdwCucQwhXU4Q4a0AQCj/AMr/DmKe/Fe/c+Fq0lr5g5hj/wPn8AnF+PIw==</latexit><latexit sha1_base64="WVBNru380J4XCjCR9Lt9t8h8e1I=">AAAB7XicbVDLSgNBEOyNrxhfqx69DAbBU9gVQY9BLx4jmAckS5idzCZj5rHMzAphyT948aCIV//Hm3/jJNmDJhY0FFXddHfFKWfGBsG3V1pb39jcKm9Xdnb39g/8w6OWUZkmtEkUV7oTY0M5k7RpmeW0k2qKRcxpOx7fzvz2E9WGKflgJymNBB5KljCCrZNaPcOGAvf9alAL5kCrJCxIFQo0+v5Xb6BIJqi0hGNjumGQ2ijH2jLC6bTSywxNMRnjIe06KrGgJsrn107RmVMGKFHalbRorv6eyLEwZiJi1ymwHZllbyb+53Uzm1xHOZNpZqkki0VJxpFVaPY6GjBNieUTRzDRzN2KyAhrTKwLqOJCCJdfXiWti1oY1ML7y2r9poijDCdwCucQwhXU4Q4a0AQCj/AMr/DmKe/Fe/c+Fq0lr5g5hj/wPn8AnF+PIw==</latexit><latexit sha1_base64="WVBNru380J4XCjCR9Lt9t8h8e1I=">AAAB7XicbVDLSgNBEOyNrxhfqx69DAbBU9gVQY9BLx4jmAckS5idzCZj5rHMzAphyT948aCIV//Hm3/jJNmDJhY0FFXddHfFKWfGBsG3V1pb39jcKm9Xdnb39g/8w6OWUZkmtEkUV7oTY0M5k7RpmeW0k2qKRcxpOx7fzvz2E9WGKflgJymNBB5KljCCrZNaPcOGAvf9alAL5kCrJCxIFQo0+v5Xb6BIJqi0hGNjumGQ2ijH2jLC6bTSywxNMRnjIe06KrGgJsrn107RmVMGKFHalbRorv6eyLEwZiJi1ymwHZllbyb+53Uzm1xHOZNpZqkki0VJxpFVaPY6GjBNieUTRzDRzN2KyAhrTKwLqOJCCJdfXiWti1oY1ML7y2r9poijDCdwCucQwhXU4Q4a0AQCj/AMr/DmKe/Fe/c+Fq0lr5g5hj/wPn8AnF+PIw==</latexit>
𝝈

23. Conversion really matters.
eCPM = BAconv. * pCTR * pCVR
Very similar to CTR but totally different from CTR

24. Impression Click Conversion
CTR >> CVR Rare event
𝜟t << 𝜟t Conversion dalay
Click
Installation, registration,
purchase, subscription, …
Variety
Segment Personal Granularity
One action Sequence of actions Context (hurdles)
SSP/DSP MAT/SDK/Pixel Data Integrity

25. Features over Algorithms
— Conversion proxy: Retargeting
— Conversion-driven LookALike

26. More topics for better response prediction
— Multi-task learning (for each ad, objective)
— Transfer learning
— Landscape forecasting
— Multi-touch attribution
— Cold-start
— Exploitation vs Exploration
— Thompson sampling

27. eCPM = BA x pCTR
How much to bid?
How to dynamically adjust bids?
Is every audience equally valuable to me?

28. Budget smoothing and auto-bidding

29. Creative BAclk pCTR eCPM Rank
*PPC
(GSP)
A 1,000 17% 170 1 941
B 1,500 6% 90 4 **500
C 1,200 9% 108 3 1,000
D 800 20% 160 2 540
* PPC = BA * (next eCPM / own eCPM) = next eCPM / own pCTR
** Reserve price = 500

30. Research Offline Test Online Test Production
• Model validity
• Effect simulation
• Validity & revenue
• A/A Test
• A/B/C/… Test
• Random bucket
Problem & ideation Complexity & Stability

31. In theory (model validation)
— Loss (logLoss)
— RIG = 1 - NE (Entropy)
— Calibration = predicted / actual
— AUC
In reality
— Revenue per request

33. 10
More topics beyond this presentation
— Yield optimization (SSP)
— Auction design (GSP/VCG, reserve price/bid floor)
— Fraud/abusing detection
— Targeting/retargeting/LookALike
— Frequency/recency capping
— AdBlock & DNT: Usefulness vs Annoying
— Dynamic/personalized creative generation
— System consideration (e.g., distributed system)
— Knowledge representation beyond audience

34. Inventory-buying Audience-buying
Impression Click Conversion
Guaranteed
AdX & RTB
Non-Guaranteed
One Platform

35. Bid amount (BA) Response Rate eCPM
≫ Manual setting (by Adv.)
≫ Auto-bidding (BA*)
≫ Impressions (≒ requests)
≫ Viewable impression
≫ Clickthrough rate (CTR)
≫ Conversion rate (CR,
Landscape forecasting (ARIMA, Prophet)
Probability (Logistic Regression)
— Input feature (X)
— Model weight (W)

36. Ranking eCPM = BAimp = BAclk * pCTR = BAconv * pCTR * pCVR
Data/Feature Privacy-free audience data
Embedding Classification (Bayesian) + Topic Modeling (LDA)
Prediction Logistic Regression
Training (Online) FTRL-Proximal
BidAmount Auto-bidding
Targeting Conversion-driven LookALike

37. Ideality vs Reality
Hard works from theory to production
Privacy is our top priority.

38. Some references
- Ad click prediction: a view from the trenches
- Practical lessons from predicting clicks on ads at facebook
- Simple and scalable response prediction for display advertising
- Modeling delayed feedback in display advertising
- Latent dirichlet allocation
- Factorization machines
- Field-aware factorization machines for CTR prediction
- Field-weighted factorization machines for click-through rate prediction in display advertising
- Deep and cross network for ad click predictions
- Model ensemble for click prediction in bing search ads
- Optimal real-time bidding for display advertising
- Feature hashing for large scale multitask learning
- Score lookalike audiences