정부환(ben.hur) / kakao
---
온라인 광고는 현재 인터넷 비즈니스 모델을 가능케하는 핵심 동인이다. 새로운 서비스가 등장하고 시장 규모가 커짐에 따라서 온라인 광고 생태계도 함께 진화하고 여러 기술적 도전이 있다. 지면 중심에서 오디언스 중심으로, 단순 광고 노출과 클릭에서 광고주의 필요를 반영한 다양한 전환으로, 그리고 수의 계약에서 실시간 자동 입찰로 온라인 광고 생태계가 진화하고 있다. 이런 변화의 흐름에서 여러 지면에 접속하는 사용자들의 성향을 즉시 분석해서 가장 적합한 광고를 실시간으로 선택하고 노출하는 것은 기술적으로 매우 어려운 문제다. 본 발표는 카카오의 광고 랭킹에 필요한 데이터와 알고리즘을 간략히 소개한다.
•
------------
<Glossary>
- Audience: 광고 도메인에서 사용자 (User)를 뜻함
- Publisher: 앱이나 웹 등의 광고 지면(inventory)/매체(media)을 제공하는 사람
- SSP: Supplier-side platform
- DSP: Demand-side platform
- DMP: Data management platform
- MAT: Mobile app tracking
- Pixel: 광고의 전화을 추척하기 위해서 웹에 심어두는 스크립트
- AdX: Ad Exchange, 광고의 mediation이 이뤄지는 마켓
- RTB: Real-time bidding
- Programmatic buying: AdX에서 프로그램에 의해서 (자동으로) 광고 입찰 및 낙찰이 이뤄지는 것
- Impression: 광고 노출
- ROAS: Return on ad spending
- eCPM: effective cost per mille (1천회 노출당 기대 비용/수익)
- CPM/CPC/CPA: Cost Per mille/click/acquisition(action, conversion)
- CTR/CVR: Clickthrough rate, (post-click) conversion rate
- SGD: Stochastic gradient descent
- FTRL: Follow-the-regularized-leader
- FM/FFM/FwFM: Factorization machines / Field-aware FM / Field-weighted FM
- DCN: Deep & cross network
- LDA: Latent dirichlet allocation
- DNN: Deep neural network (DL)
- AE: Auto-encoder
- GBDT: Gradient-boosting decision tree
- Targeting: 광고주가 자신의 광고가 노출될 오디언스 (사용자)를 제한하는 것
- Retargeting: 사용자의 특정 행동 (i.e., 광고주 사이트 방문)에 반응해서 광고를 노출/제한하는 것
- LookALike: 광고주가 제공한 오디언스 그룹과 유사한 특징을 갖는 오디언스군 (유사확장타게팅)
- PPC: Pay per click 클릭당 과금액
- RIG: Relative information gain
- NE: Normalized entropy
- AUC: Area under ROC Curve
- GSP/VCG: Generalized second price auction / Vickrey-Clarke-Groves auction
- DNT: Do not track
More Related Content
카카오의 광고지능 (Intelligence on Kakao Advertising)
1.
1. Intelligence on Kakao Advertising beyond state-of-the-art ben.hur@kakaocorp.com
2.
2. Nothing is certain but death and taxes. - Benjamin Franklin
3.
3. Nothing is certain but death and taxes. - Benjamin Franklin “AD”. - Benjamin Franklin
4.
4. User Advertiser Publisher Contract
5.
5. Audience (User) Advertiser Publisher AdTech Eco (AdX/RTB/Programmatic Buying) Traffic Req4Bid BidImpression Visit Ad Selection - Filtering - Ranking - Pricing Mediation (Auction) ** Visit-to-Impression is done within 1~200ms for at least ten-thousand requests per second. SSP DSP DMP Audience Tracking (MAT/SDK/Pixel) Transaction log (train) Audience Info. (target) Log Ad Media Log SSP: Supply-side platform DSP: Demand-side platform DMP: Data management platform MAT: Mobile app tracking AdX: Ad exchange RTB: Real-time bidding
6.
6. Different dreams among {Audience, Advertiser, Publisher, Platform} AD Audience Advertiser Publisher Platform Annoying vs Information Marketing channel (ROAS) Reserve Price Revenue (Adv - Pub)
7.
7. Balance between Revenue and Relevance Audience Provide relevant and interesting information Advertiser Gather new or loyal customers through a low cost channel Publisher Guarantee a stable and predictable revenue source Platform Maximize overall welfare (& utility) with user satisfaction
8.
8. Effective Cost Per Mille (eCPM) Expected/estimated revenue per (a thousand) impression(s)
9.
9. eCPM = Bid Amount x Likeness Revenue: Amount that an advertiser is willing to pay for desired actions Relevance: How much does a user like to do the actions
10.
10. In CPC, eCPM = BAclick x pCTR number of clicks / number of impressions
11.
11. Leave (y = 0) Click (y = 1) Ad | X X: Traffic properties (ADxUSRxPLx…)
12.
12. Pr(y = 1 | x, ad) Aggregation of historical data Learning from historical data Reactive method vs Predictive method
13.
13. More likely to click Sum of traffic properties (WTX) Logistic Regression (Maximum entropy) Pr(y = 1|x) = 1 1 + exp(−wTx) Softmax of binary (1/0) output Less likely to click
14.
14. Pr(y = 1|x) = 1 1 + exp(−wTx) Loss = ∥y − ̂y∥2 y<latexit sha1_base64="paQhm8QH9RuJYjMoRm3VlxatzsM=">AAAB6HicdVDLSsNAFJ3UV62vqks3g0VwFSY1tHVXdOOyBfuANpTJdNKOnUzCzEQIoV/gxoUibv0kd/6Nk7aCih64cDjnXu69x485UxqhD6uwtr6xuVXcLu3s7u0flA+PuipKJKEdEvFI9n2sKGeCdjTTnPZjSXHoc9rzZ9e537unUrFI3Oo0pl6IJ4IFjGBtpHY6KleQfdmoVd0aRDZCdafq5KRady9c6BglRwWs0BqV34fjiCQhFZpwrNTAQbH2Miw1I5zOS8NE0RiTGZ7QgaECh1R52eLQOTwzyhgGkTQlNFyo3ycyHCqVhr7pDLGeqt9eLv7lDRIdNLyMiTjRVJDloiDhUEcw/xqOmaRE89QQTCQzt0IyxRITbbIpmRC+PoX/k27VdpDttN1K82oVRxGcgFNwDhxQB01wA1qgAwig4AE8gWfrznq0XqzXZWvBWs0cgx+w3j4BR9uNPw==</latexit><latexit sha1_base64="paQhm8QH9RuJYjMoRm3VlxatzsM=">AAAB6HicdVDLSsNAFJ3UV62vqks3g0VwFSY1tHVXdOOyBfuANpTJdNKOnUzCzEQIoV/gxoUibv0kd/6Nk7aCih64cDjnXu69x485UxqhD6uwtr6xuVXcLu3s7u0flA+PuipKJKEdEvFI9n2sKGeCdjTTnPZjSXHoc9rzZ9e537unUrFI3Oo0pl6IJ4IFjGBtpHY6KleQfdmoVd0aRDZCdafq5KRady9c6BglRwWs0BqV34fjiCQhFZpwrNTAQbH2Miw1I5zOS8NE0RiTGZ7QgaECh1R52eLQOTwzyhgGkTQlNFyo3ycyHCqVhr7pDLGeqt9eLv7lDRIdNLyMiTjRVJDloiDhUEcw/xqOmaRE89QQTCQzt0IyxRITbbIpmRC+PoX/k27VdpDttN1K82oVRxGcgFNwDhxQB01wA1qgAwig4AE8gWfrznq0XqzXZWvBWs0cgx+w3j4BR9uNPw==</latexit><latexit sha1_base64="paQhm8QH9RuJYjMoRm3VlxatzsM=">AAAB6HicdVDLSsNAFJ3UV62vqks3g0VwFSY1tHVXdOOyBfuANpTJdNKOnUzCzEQIoV/gxoUibv0kd/6Nk7aCih64cDjnXu69x485UxqhD6uwtr6xuVXcLu3s7u0flA+PuipKJKEdEvFI9n2sKGeCdjTTnPZjSXHoc9rzZ9e537unUrFI3Oo0pl6IJ4IFjGBtpHY6KleQfdmoVd0aRDZCdafq5KRady9c6BglRwWs0BqV34fjiCQhFZpwrNTAQbH2Miw1I5zOS8NE0RiTGZ7QgaECh1R52eLQOTwzyhgGkTQlNFyo3ycyHCqVhr7pDLGeqt9eLv7lDRIdNLyMiTjRVJDloiDhUEcw/xqOmaRE89QQTCQzt0IyxRITbbIpmRC+PoX/k27VdpDttN1K82oVRxGcgFNwDhxQB01wA1qgAwig4AE8gWfrznq0XqzXZWvBWs0cgx+w3j4BR9uNPw==</latexit><latexit sha1_base64="paQhm8QH9RuJYjMoRm3VlxatzsM=">AAAB6HicdVDLSsNAFJ3UV62vqks3g0VwFSY1tHVXdOOyBfuANpTJdNKOnUzCzEQIoV/gxoUibv0kd/6Nk7aCih64cDjnXu69x485UxqhD6uwtr6xuVXcLu3s7u0flA+PuipKJKEdEvFI9n2sKGeCdjTTnPZjSXHoc9rzZ9e537unUrFI3Oo0pl6IJ4IFjGBtpHY6KleQfdmoVd0aRDZCdafq5KRady9c6BglRwWs0BqV34fjiCQhFZpwrNTAQbH2Miw1I5zOS8NE0RiTGZ7QgaECh1R52eLQOTwzyhgGkTQlNFyo3ycyHCqVhr7pDLGeqt9eLv7lDRIdNLyMiTjRVJDloiDhUEcw/xqOmaRE89QQTCQzt0IyxRITbbIpmRC+PoX/k27VdpDttN1K82oVRxGcgFNwDhxQB01wA1qgAwig4AE8gWfrznq0XqzXZWvBWs0cgx+w3j4BR9uNPw==</latexit> ˆy<latexit sha1_base64="QmQDjeeN4gpKWLfKwkS/Fz5qGt4=">AAAB7nicdVDLSgNBEOyNrxhfUY9eBoPgKcwEMckt6MVjBPOAZAmzk9lkyOyDmVlhWfIRXjwo4tXv8ebfOJtEUNGChqKqm+4uL5ZCG4w/nMLa+sbmVnG7tLO7t39QPjzq6ihRjHdYJCPV96jmUoS8Y4SRvB8rTgNP8p43u8793j1XWkThnUlj7gZ0EgpfMGqs1BtOqcnS+ahcwVWMMSEE5YTUL7ElzWajRhqI5JZFBVZoj8rvw3HEkoCHhkmq9YDg2LgZVUYwyeelYaJ5TNmMTvjA0pAGXLvZ4tw5OrPKGPmRshUatFC/T2Q00DoNPNsZUDPVv71c/MsbJMZvuJkI48TwkC0X+YlEJkL572gsFGdGppZQpoS9FbEpVZQZm1DJhvD1KfqfdGtVgqvk9qLSulrFUYQTOIVzIFCHFtxAGzrAYAYP8ATPTuw8Oi/O67K14KxmjuEHnLdP/reQAA==</latexit><latexit sha1_base64="QmQDjeeN4gpKWLfKwkS/Fz5qGt4=">AAAB7nicdVDLSgNBEOyNrxhfUY9eBoPgKcwEMckt6MVjBPOAZAmzk9lkyOyDmVlhWfIRXjwo4tXv8ebfOJtEUNGChqKqm+4uL5ZCG4w/nMLa+sbmVnG7tLO7t39QPjzq6ihRjHdYJCPV96jmUoS8Y4SRvB8rTgNP8p43u8793j1XWkThnUlj7gZ0EgpfMGqs1BtOqcnS+ahcwVWMMSEE5YTUL7ElzWajRhqI5JZFBVZoj8rvw3HEkoCHhkmq9YDg2LgZVUYwyeelYaJ5TNmMTvjA0pAGXLvZ4tw5OrPKGPmRshUatFC/T2Q00DoNPNsZUDPVv71c/MsbJMZvuJkI48TwkC0X+YlEJkL572gsFGdGppZQpoS9FbEpVZQZm1DJhvD1KfqfdGtVgqvk9qLSulrFUYQTOIVzIFCHFtxAGzrAYAYP8ATPTuw8Oi/O67K14KxmjuEHnLdP/reQAA==</latexit><latexit sha1_base64="QmQDjeeN4gpKWLfKwkS/Fz5qGt4=">AAAB7nicdVDLSgNBEOyNrxhfUY9eBoPgKcwEMckt6MVjBPOAZAmzk9lkyOyDmVlhWfIRXjwo4tXv8ebfOJtEUNGChqKqm+4uL5ZCG4w/nMLa+sbmVnG7tLO7t39QPjzq6ihRjHdYJCPV96jmUoS8Y4SRvB8rTgNP8p43u8793j1XWkThnUlj7gZ0EgpfMGqs1BtOqcnS+ahcwVWMMSEE5YTUL7ElzWajRhqI5JZFBVZoj8rvw3HEkoCHhkmq9YDg2LgZVUYwyeelYaJ5TNmMTvjA0pAGXLvZ4tw5OrPKGPmRshUatFC/T2Q00DoNPNsZUDPVv71c/MsbJMZvuJkI48TwkC0X+YlEJkL572gsFGdGppZQpoS9FbEpVZQZm1DJhvD1KfqfdGtVgqvk9qLSulrFUYQTOIVzIFCHFtxAGzrAYAYP8ATPTuw8Oi/O67K14KxmjuEHnLdP/reQAA==</latexit><latexit sha1_base64="QmQDjeeN4gpKWLfKwkS/Fz5qGt4=">AAAB7nicdVDLSgNBEOyNrxhfUY9eBoPgKcwEMckt6MVjBPOAZAmzk9lkyOyDmVlhWfIRXjwo4tXv8ebfOJtEUNGChqKqm+4uL5ZCG4w/nMLa+sbmVnG7tLO7t39QPjzq6ihRjHdYJCPV96jmUoS8Y4SRvB8rTgNP8p43u8793j1XWkThnUlj7gZ0EgpfMGqs1BtOqcnS+ahcwVWMMSEE5YTUL7ElzWajRhqI5JZFBVZoj8rvw3HEkoCHhkmq9YDg2LgZVUYwyeelYaJ5TNmMTvjA0pAGXLvZ4tw5OrPKGPmRshUatFC/T2Q00DoNPNsZUDPVv71c/MsbJMZvuJkI48TwkC0X+YlEJkL572gsFGdGppZQpoS9FbEpVZQZm1DJhvD1KfqfdGtVgqvk9qLSulrFUYQTOIVzIFCHFtxAGzrAYAYP8ATPTuw8Oi/O67K14KxmjuEHnLdP/reQAA==</latexit>
15.
15. Find w that minimizes the negative log likelihood (w/ L2 regularization) Control model complexity NLL for logistic regression arg min w n ∑ i=1 log(1 + exp(−yiwT xi)) + λ 2 ∥w∥2 2
16.
16. Stochastic Gradient Descent (SGD) ⌘t<latexit sha1_base64="SU/TSRqhSNKT3zfwyFM+mpJHyjY=">AAAB73icbVBNS8NAEJ3Ur1q/qh69BIvgqSQi6LHoxWMF+wFtKJvttF262cTdiVBC/4QXD4p49e9489+4bXPQ1gcDj/dmmJkXJlIY8rxvp7C2vrG5Vdwu7ezu7R+UD4+aJk41xwaPZazbITMohcIGCZLYTjSyKJTYCse3M7/1hNqIWD3QJMEgYkMlBoIzslK7i8R6GU175YpX9eZwV4mfkwrkqPfKX91+zNMIFXHJjOn4XkJBxjQJLnFa6qYGE8bHbIgdSxWL0ATZ/N6pe2aVvjuItS1F7lz9PZGxyJhJFNrOiNHILHsz8T+vk9LgOsiESlJCxReLBql0KXZnz7t9oZGTnFjCuBb2VpePmGacbEQlG4K//PIqaV5Ufa/q319Wajd5HEU4gVM4Bx+uoAZ3UIcGcJDwDK/w5jw6L86787FoLTj5zDH8gfP5A11OkCs=</latexit><latexit sha1_base64="SU/TSRqhSNKT3zfwyFM+mpJHyjY=">AAAB73icbVBNS8NAEJ3Ur1q/qh69BIvgqSQi6LHoxWMF+wFtKJvttF262cTdiVBC/4QXD4p49e9489+4bXPQ1gcDj/dmmJkXJlIY8rxvp7C2vrG5Vdwu7ezu7R+UD4+aJk41xwaPZazbITMohcIGCZLYTjSyKJTYCse3M7/1hNqIWD3QJMEgYkMlBoIzslK7i8R6GU175YpX9eZwV4mfkwrkqPfKX91+zNMIFXHJjOn4XkJBxjQJLnFa6qYGE8bHbIgdSxWL0ATZ/N6pe2aVvjuItS1F7lz9PZGxyJhJFNrOiNHILHsz8T+vk9LgOsiESlJCxReLBql0KXZnz7t9oZGTnFjCuBb2VpePmGacbEQlG4K//PIqaV5Ufa/q319Wajd5HEU4gVM4Bx+uoAZ3UIcGcJDwDK/w5jw6L86787FoLTj5zDH8gfP5A11OkCs=</latexit><latexit sha1_base64="SU/TSRqhSNKT3zfwyFM+mpJHyjY=">AAAB73icbVBNS8NAEJ3Ur1q/qh69BIvgqSQi6LHoxWMF+wFtKJvttF262cTdiVBC/4QXD4p49e9489+4bXPQ1gcDj/dmmJkXJlIY8rxvp7C2vrG5Vdwu7ezu7R+UD4+aJk41xwaPZazbITMohcIGCZLYTjSyKJTYCse3M7/1hNqIWD3QJMEgYkMlBoIzslK7i8R6GU175YpX9eZwV4mfkwrkqPfKX91+zNMIFXHJjOn4XkJBxjQJLnFa6qYGE8bHbIgdSxWL0ATZ/N6pe2aVvjuItS1F7lz9PZGxyJhJFNrOiNHILHsz8T+vk9LgOsiESlJCxReLBql0KXZnz7t9oZGTnFjCuBb2VpePmGacbEQlG4K//PIqaV5Ufa/q319Wajd5HEU4gVM4Bx+uoAZ3UIcGcJDwDK/w5jw6L86787FoLTj5zDH8gfP5A11OkCs=</latexit><latexit sha1_base64="SU/TSRqhSNKT3zfwyFM+mpJHyjY=">AAAB73icbVBNS8NAEJ3Ur1q/qh69BIvgqSQi6LHoxWMF+wFtKJvttF262cTdiVBC/4QXD4p49e9489+4bXPQ1gcDj/dmmJkXJlIY8rxvp7C2vrG5Vdwu7ezu7R+UD4+aJk41xwaPZazbITMohcIGCZLYTjSyKJTYCse3M7/1hNqIWD3QJMEgYkMlBoIzslK7i8R6GU175YpX9eZwV4mfkwrkqPfKX91+zNMIFXHJjOn4XkJBxjQJLnFa6qYGE8bHbIgdSxWL0ATZ/N6pe2aVvjuItS1F7lz9PZGxyJhJFNrOiNHILHsz8T+vk9LgOsiESlJCxReLBql0KXZnz7t9oZGTnFjCuBb2VpePmGacbEQlG4K//PIqaV5Ufa/q319Wajd5HEU4gVM4Bx+uoAZ3UIcGcJDwDK/w5jw6L86787FoLTj5zDH8gfP5A11OkCs=</latexit> gt<latexit sha1_base64="P/GvVIeqVKiemQWRJKaaMovVVQM=">AAAB9XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbduKxgH9DWMplO2qGTSZi5UUrIf7hxoYhb/8Wdf+OkzUJbDwwczrmXe+b4sRQGXffbWVldW9/YLG2Vt3d29/YrB4ctEyWa8SaLZKQ7PjVcCsWbKFDyTqw5DX3J2/7kJvfbj1wbEal7nMa8H9KREoFgFK300Aspjv0gHWWDFLNBperW3BnIMvEKUoUCjUHlqzeMWBJyhUxSY7qeG2M/pRoFkzwr9xLDY8omdMS7lioactNPZ6kzcmqVIQkibZ9CMlN/b6Q0NGYa+nYyT2kWvVz8z+smGFz1U6HiBLli80NBIglGJK+ADIXmDOXUEsq0sFkJG1NNGdqiyrYEb/HLy6R1XvPcmnd3Ua1fF3WU4BhO4Aw8uIQ63EIDmsBAwzO8wpvz5Lw4787HfHTFKXaO4A+czx9Ca5L+</latexit><latexit sha1_base64="P/GvVIeqVKiemQWRJKaaMovVVQM=">AAAB9XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbduKxgH9DWMplO2qGTSZi5UUrIf7hxoYhb/8Wdf+OkzUJbDwwczrmXe+b4sRQGXffbWVldW9/YLG2Vt3d29/YrB4ctEyWa8SaLZKQ7PjVcCsWbKFDyTqw5DX3J2/7kJvfbj1wbEal7nMa8H9KREoFgFK300Aspjv0gHWWDFLNBperW3BnIMvEKUoUCjUHlqzeMWBJyhUxSY7qeG2M/pRoFkzwr9xLDY8omdMS7lioactNPZ6kzcmqVIQkibZ9CMlN/b6Q0NGYa+nYyT2kWvVz8z+smGFz1U6HiBLli80NBIglGJK+ADIXmDOXUEsq0sFkJG1NNGdqiyrYEb/HLy6R1XvPcmnd3Ua1fF3WU4BhO4Aw8uIQ63EIDmsBAwzO8wpvz5Lw4787HfHTFKXaO4A+czx9Ca5L+</latexit><latexit sha1_base64="P/GvVIeqVKiemQWRJKaaMovVVQM=">AAAB9XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbduKxgH9DWMplO2qGTSZi5UUrIf7hxoYhb/8Wdf+OkzUJbDwwczrmXe+b4sRQGXffbWVldW9/YLG2Vt3d29/YrB4ctEyWa8SaLZKQ7PjVcCsWbKFDyTqw5DX3J2/7kJvfbj1wbEal7nMa8H9KREoFgFK300Aspjv0gHWWDFLNBperW3BnIMvEKUoUCjUHlqzeMWBJyhUxSY7qeG2M/pRoFkzwr9xLDY8omdMS7lioactNPZ6kzcmqVIQkibZ9CMlN/b6Q0NGYa+nYyT2kWvVz8z+smGFz1U6HiBLli80NBIglGJK+ADIXmDOXUEsq0sFkJG1NNGdqiyrYEb/HLy6R1XvPcmnd3Ua1fF3WU4BhO4Aw8uIQ63EIDmsBAwzO8wpvz5Lw4787HfHTFKXaO4A+czx9Ca5L+</latexit><latexit sha1_base64="P/GvVIeqVKiemQWRJKaaMovVVQM=">AAAB9XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbduKxgH9DWMplO2qGTSZi5UUrIf7hxoYhb/8Wdf+OkzUJbDwwczrmXe+b4sRQGXffbWVldW9/YLG2Vt3d29/YrB4ctEyWa8SaLZKQ7PjVcCsWbKFDyTqw5DX3J2/7kJvfbj1wbEal7nMa8H9KREoFgFK300Aspjv0gHWWDFLNBperW3BnIMvEKUoUCjUHlqzeMWBJyhUxSY7qeG2M/pRoFkzwr9xLDY8omdMS7lioactNPZ6kzcmqVIQkibZ9CMlN/b6Q0NGYa+nYyT2kWvVz8z+smGFz1U6HiBLli80NBIglGJK+ADIXmDOXUEsq0sFkJG1NNGdqiyrYEb/HLy6R1XvPcmnd3Ua1fF3WU4BhO4Aw8uIQ63EIDmsBAwzO8wpvz5Lw4787HfHTFKXaO4A+czx9Ca5L+</latexit> wt<latexit sha1_base64="wQsvs8XlfPgJ6APhixgXICv3Sn0=">AAAB9XicbVDLSsNAFL2pr1pfVZdugkVwVRIRdFl047KCfUAby2Q6aYdOJmHmxlJC/sONC0Xc+i/u/BsnbRbaemDgcM693DPHjwXX6DjfVmltfWNzq7xd2dnd2z+oHh61dZQoylo0EpHq+kQzwSVrIUfBurFiJPQF6/iT29zvPDGleSQfcBYzLyQjyQNOCRrpsR8SHPtBOs0GKWaDas2pO3PYq8QtSA0KNAfVr/4woknIJFJBtO65ToxeShRyKlhW6SeaxYROyIj1DJUkZNpL56kz+8woQzuIlHkS7bn6eyMlodaz0DeTeUq97OXif14vweDaS7mME2SSLg4FibAxsvMK7CFXjKKYGUKo4iarTcdEEYqmqIopwV3+8ippX9Rdp+7eX9YaN0UdZTiBUzgHF66gAXfQhBZQUPAMr/BmTa0X6936WIyWrGLnGP7A+vwBWvuTDg==</latexit><latexit sha1_base64="wQsvs8XlfPgJ6APhixgXICv3Sn0=">AAAB9XicbVDLSsNAFL2pr1pfVZdugkVwVRIRdFl047KCfUAby2Q6aYdOJmHmxlJC/sONC0Xc+i/u/BsnbRbaemDgcM693DPHjwXX6DjfVmltfWNzq7xd2dnd2z+oHh61dZQoylo0EpHq+kQzwSVrIUfBurFiJPQF6/iT29zvPDGleSQfcBYzLyQjyQNOCRrpsR8SHPtBOs0GKWaDas2pO3PYq8QtSA0KNAfVr/4woknIJFJBtO65ToxeShRyKlhW6SeaxYROyIj1DJUkZNpL56kz+8woQzuIlHkS7bn6eyMlodaz0DeTeUq97OXif14vweDaS7mME2SSLg4FibAxsvMK7CFXjKKYGUKo4iarTcdEEYqmqIopwV3+8ippX9Rdp+7eX9YaN0UdZTiBUzgHF66gAXfQhBZQUPAMr/BmTa0X6936WIyWrGLnGP7A+vwBWvuTDg==</latexit><latexit sha1_base64="wQsvs8XlfPgJ6APhixgXICv3Sn0=">AAAB9XicbVDLSsNAFL2pr1pfVZdugkVwVRIRdFl047KCfUAby2Q6aYdOJmHmxlJC/sONC0Xc+i/u/BsnbRbaemDgcM693DPHjwXX6DjfVmltfWNzq7xd2dnd2z+oHh61dZQoylo0EpHq+kQzwSVrIUfBurFiJPQF6/iT29zvPDGleSQfcBYzLyQjyQNOCRrpsR8SHPtBOs0GKWaDas2pO3PYq8QtSA0KNAfVr/4woknIJFJBtO65ToxeShRyKlhW6SeaxYROyIj1DJUkZNpL56kz+8woQzuIlHkS7bn6eyMlodaz0DeTeUq97OXif14vweDaS7mME2SSLg4FibAxsvMK7CFXjKKYGUKo4iarTcdEEYqmqIopwV3+8ippX9Rdp+7eX9YaN0UdZTiBUzgHF66gAXfQhBZQUPAMr/BmTa0X6936WIyWrGLnGP7A+vwBWvuTDg==</latexit><latexit sha1_base64="wQsvs8XlfPgJ6APhixgXICv3Sn0=">AAAB9XicbVDLSsNAFL2pr1pfVZdugkVwVRIRdFl047KCfUAby2Q6aYdOJmHmxlJC/sONC0Xc+i/u/BsnbRbaemDgcM693DPHjwXX6DjfVmltfWNzq7xd2dnd2z+oHh61dZQoylo0EpHq+kQzwSVrIUfBurFiJPQF6/iT29zvPDGleSQfcBYzLyQjyQNOCRrpsR8SHPtBOs0GKWaDas2pO3PYq8QtSA0KNAfVr/4woknIJFJBtO65ToxeShRyKlhW6SeaxYROyIj1DJUkZNpL56kz+8woQzuIlHkS7bn6eyMlodaz0DeTeUq97OXif14vweDaS7mME2SSLg4FibAxsvMK7CFXjKKYGUKo4iarTcdEEYqmqIopwV3+8ippX9Rdp+7eX9YaN0UdZTiBUzgHF66gAXfQhBZQUPAMr/BmTa0X6936WIyWrGLnGP7A+vwBWvuTDg==</latexit> wt+1<latexit sha1_base64="r/oTFMkBdTR9L3i5TOvM6rbcAK4=">AAAB+XicbVDLSsNAFL2pr1pfUZduBosgCCURQZdFNy4r2Ae0IUymk3boZBJmJpUS8iduXCji1j9x5984abPQ1gMDh3Pu5Z45QcKZ0o7zbVXW1jc2t6rbtZ3dvf0D+/Coo+JUEtomMY9lL8CKciZoWzPNaS+RFEcBp91gclf43SmVisXiUc8S6kV4JFjICNZG8m17EGE9DsLsKfczfeHmvl13Gs4caJW4JalDiZZvfw2GMUkjKjThWKm+6yTay7DUjHCa1wapogkmEzyifUMFjqjysnnyHJ0ZZYjCWJonNJqrvzcyHCk1iwIzWeRUy14h/uf1Ux3eeBkTSaqpIItDYcqRjlFRAxoySYnmM0MwkcxkRWSMJSbalFUzJbjLX14lncuG6zTch6t687asowoncArn4MI1NOEeWtAGAlN4hld4szLrxXq3PhajFavcOYY/sD5/ALPpk68=</latexit><latexit sha1_base64="r/oTFMkBdTR9L3i5TOvM6rbcAK4=">AAAB+XicbVDLSsNAFL2pr1pfUZduBosgCCURQZdFNy4r2Ae0IUymk3boZBJmJpUS8iduXCji1j9x5984abPQ1gMDh3Pu5Z45QcKZ0o7zbVXW1jc2t6rbtZ3dvf0D+/Coo+JUEtomMY9lL8CKciZoWzPNaS+RFEcBp91gclf43SmVisXiUc8S6kV4JFjICNZG8m17EGE9DsLsKfczfeHmvl13Gs4caJW4JalDiZZvfw2GMUkjKjThWKm+6yTay7DUjHCa1wapogkmEzyifUMFjqjysnnyHJ0ZZYjCWJonNJqrvzcyHCk1iwIzWeRUy14h/uf1Ux3eeBkTSaqpIItDYcqRjlFRAxoySYnmM0MwkcxkRWSMJSbalFUzJbjLX14lncuG6zTch6t687asowoncArn4MI1NOEeWtAGAlN4hld4szLrxXq3PhajFavcOYY/sD5/ALPpk68=</latexit><latexit sha1_base64="r/oTFMkBdTR9L3i5TOvM6rbcAK4=">AAAB+XicbVDLSsNAFL2pr1pfUZduBosgCCURQZdFNy4r2Ae0IUymk3boZBJmJpUS8iduXCji1j9x5984abPQ1gMDh3Pu5Z45QcKZ0o7zbVXW1jc2t6rbtZ3dvf0D+/Coo+JUEtomMY9lL8CKciZoWzPNaS+RFEcBp91gclf43SmVisXiUc8S6kV4JFjICNZG8m17EGE9DsLsKfczfeHmvl13Gs4caJW4JalDiZZvfw2GMUkjKjThWKm+6yTay7DUjHCa1wapogkmEzyifUMFjqjysnnyHJ0ZZYjCWJonNJqrvzcyHCk1iwIzWeRUy14h/uf1Ux3eeBkTSaqpIItDYcqRjlFRAxoySYnmM0MwkcxkRWSMJSbalFUzJbjLX14lncuG6zTch6t687asowoncArn4MI1NOEeWtAGAlN4hld4szLrxXq3PhajFavcOYY/sD5/ALPpk68=</latexit><latexit sha1_base64="r/oTFMkBdTR9L3i5TOvM6rbcAK4=">AAAB+XicbVDLSsNAFL2pr1pfUZduBosgCCURQZdFNy4r2Ae0IUymk3boZBJmJpUS8iduXCji1j9x5984abPQ1gMDh3Pu5Z45QcKZ0o7zbVXW1jc2t6rbtZ3dvf0D+/Coo+JUEtomMY9lL8CKciZoWzPNaS+RFEcBp91gclf43SmVisXiUc8S6kV4JFjICNZG8m17EGE9DsLsKfczfeHmvl13Gs4caJW4JalDiZZvfw2GMUkjKjThWKm+6yTay7DUjHCa1wapogkmEzyifUMFjqjysnnyHJ0ZZYjCWJonNJqrvzcyHCk1iwIzWeRUy14h/uf1Ux3eeBkTSaqpIItDYcqRjlFRAxoySYnmM0MwkcxkRWSMJSbalFUzJbjLX14lncuG6zTch6t687asowoncArn4MI1NOEeWtAGAlN4hld4szLrxXq3PhajFavcOYY/sD5/ALPpk68=</latexit> Loss/Cost function (w) (Global) minimum (Local) minimum ηt = α β + ∑ t s=1 g2 s wt+1 = wt − ηtgt
17.
17. FTRL-Proximal (Online) Follow-the-leaders Proximal (convexity for stability) Regularization (sparsity) Reference: Ad click prediction: a view from the trenches (2013) wt+1 = arg min w (g1:t ⋅ w + 1 2 t ∑ s=1 σs∥w − ws∥2 2 + λ1∥w∥1) wt+1 = wt − ηtgt
18.
18. Recent advances on response prediction — FM/FFM/FwFM (**Factorization Machines) — Deep & Cross Network (DCN) — Model ensemble Interaction & latent Pr(y=1|X) = 1 / (1 + exp(-y*FM(X))) Nonlinear embedding New state-of-the-art
19.
19. Pr(y = 1 | x) = 1 / (1 + exp(-wTx)) ** The pictograms are only for explanation, in that it does not imply that Kakao uses such audience information.
20.
20. (Hopefully) Almost activities in Kakao (+ 𝛂) with cautious treatments — Law and guidance (i.e., Privacy) — De-identification and k-anonymity — Abstraction: Estimation and aggregation — No merging between internal and external user data — & technical and economical barriers
21.
21. M F 20 30 40 50 ADF SUBS … 1 0 0 1 0 0 0 1 0 0 0 0 1 0 0 Estimation Aggregation Classification (Bayesian inference) Gender/Age-band estimation Simple but syntactic Clustering / LDA (topic modeling) Hashing trick Gradient Boosting Tree DNN / AE Effective but slow Promising as others do
22.
22. GBDT+LR (Facebook) DCN (Google) DNN + GBDT (MS) LDA + LR (Kakao) 2 {0, 1}d <latexit sha1_base64="YL/bYKPJDajrGI4enkt5M5HXN5E=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBg5REBD0WvXisYD+giWWz2bRLN5uwuymUkH/ixYMiXv0n3vw3btsctPXBwOO9GWbmBSlnSjvOt1VZW9/Y3Kpu13Z29/YP7MOjjkoySWibJDyRvQArypmgbc00p71UUhwHnHaD8d3M706oVCwRj3qaUj/GQ8EiRrA20sC2PSaQlzsXyPWKpzwsBnbdaThzoFXilqQOJVoD+8sLE5LFVGjCsVJ910m1n2OpGeG0qHmZoikmYzykfUMFjqny8/nlBTozSoiiRJoSGs3V3xM5jpWaxoHpjLEeqWVvJv7n9TMd3fg5E2mmqSCLRVHGkU7QLAYUMkmJ5lNDMJHM3IrICEtMtAmrZkJwl19eJZ3Lhus03IerevO2jKMKJ3AK5+DCNTThHlrQBgITeIZXeLNy68V6tz4WrRWrnDmGP7A+fwAt7JK1</latexit><latexit sha1_base64="YL/bYKPJDajrGI4enkt5M5HXN5E=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBg5REBD0WvXisYD+giWWz2bRLN5uwuymUkH/ixYMiXv0n3vw3btsctPXBwOO9GWbmBSlnSjvOt1VZW9/Y3Kpu13Z29/YP7MOjjkoySWibJDyRvQArypmgbc00p71UUhwHnHaD8d3M706oVCwRj3qaUj/GQ8EiRrA20sC2PSaQlzsXyPWKpzwsBnbdaThzoFXilqQOJVoD+8sLE5LFVGjCsVJ910m1n2OpGeG0qHmZoikmYzykfUMFjqny8/nlBTozSoiiRJoSGs3V3xM5jpWaxoHpjLEeqWVvJv7n9TMd3fg5E2mmqSCLRVHGkU7QLAYUMkmJ5lNDMJHM3IrICEtMtAmrZkJwl19eJZ3Lhus03IerevO2jKMKJ3AK5+DCNTThHlrQBgITeIZXeLNy68V6tz4WrRWrnDmGP7A+fwAt7JK1</latexit><latexit sha1_base64="YL/bYKPJDajrGI4enkt5M5HXN5E=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBg5REBD0WvXisYD+giWWz2bRLN5uwuymUkH/ixYMiXv0n3vw3btsctPXBwOO9GWbmBSlnSjvOt1VZW9/Y3Kpu13Z29/YP7MOjjkoySWibJDyRvQArypmgbc00p71UUhwHnHaD8d3M706oVCwRj3qaUj/GQ8EiRrA20sC2PSaQlzsXyPWKpzwsBnbdaThzoFXilqQOJVoD+8sLE5LFVGjCsVJ910m1n2OpGeG0qHmZoikmYzykfUMFjqny8/nlBTozSoiiRJoSGs3V3xM5jpWaxoHpjLEeqWVvJv7n9TMd3fg5E2mmqSCLRVHGkU7QLAYUMkmJ5lNDMJHM3IrICEtMtAmrZkJwl19eJZ3Lhus03IerevO2jKMKJ3AK5+DCNTThHlrQBgITeIZXeLNy68V6tz4WrRWrnDmGP7A+fwAt7JK1</latexit><latexit sha1_base64="YL/bYKPJDajrGI4enkt5M5HXN5E=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBg5REBD0WvXisYD+giWWz2bRLN5uwuymUkH/ixYMiXv0n3vw3btsctPXBwOO9GWbmBSlnSjvOt1VZW9/Y3Kpu13Z29/YP7MOjjkoySWibJDyRvQArypmgbc00p71UUhwHnHaD8d3M706oVCwRj3qaUj/GQ8EiRrA20sC2PSaQlzsXyPWKpzwsBnbdaThzoFXilqQOJVoD+8sLE5LFVGjCsVJ910m1n2OpGeG0qHmZoikmYzykfUMFjqny8/nlBTozSoiiRJoSGs3V3xM5jpWaxoHpjLEeqWVvJv7n9TMd3fg5E2mmqSCLRVHGkU7QLAYUMkmJ5lNDMJHM3IrICEtMtAmrZkJwl19eJZ3Lhus03IerevO2jKMKJ3AK5+DCNTThHlrQBgITeIZXeLNy68V6tz4WrRWrnDmGP7A+fwAt7JK1</latexit> ⌃<latexit sha1_base64="xl6gt5zesTVLaQIKxfcD+hK+ntk=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKEI9BLx4jmgckS5idzCZj5rHMzAphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntJ6oNU/LBThIaCjyULGYEWye1evdsKHC/XPGr/hxolQQ5qUCORr/81RsokgoqLeHYmG7gJzbMsLaMcDot9VJDE0zGeEi7jkosqAmz+bVTdOaUAYqVdiUtmqu/JzIsjJmIyHUKbEdm2ZuJ/3nd1MZXYcZkkloqyWJRnHJkFZq9jgZMU2L5xBFMNHO3IjLCGhPrAiq5EILll1dJ66Ia+NXg7rJSv87jKMIJnMI5BFCDOtxCA5pA4BGe4RXePOW9eO/ex6K14OUzx/AH3ucPa1+PAw==</latexit><latexit sha1_base64="xl6gt5zesTVLaQIKxfcD+hK+ntk=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKEI9BLx4jmgckS5idzCZj5rHMzAphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntJ6oNU/LBThIaCjyULGYEWye1evdsKHC/XPGr/hxolQQ5qUCORr/81RsokgoqLeHYmG7gJzbMsLaMcDot9VJDE0zGeEi7jkosqAmz+bVTdOaUAYqVdiUtmqu/JzIsjJmIyHUKbEdm2ZuJ/3nd1MZXYcZkkloqyWJRnHJkFZq9jgZMU2L5xBFMNHO3IjLCGhPrAiq5EILll1dJ66Ia+NXg7rJSv87jKMIJnMI5BFCDOtxCA5pA4BGe4RXePOW9eO/ex6K14OUzx/AH3ucPa1+PAw==</latexit><latexit sha1_base64="xl6gt5zesTVLaQIKxfcD+hK+ntk=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKEI9BLx4jmgckS5idzCZj5rHMzAphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntJ6oNU/LBThIaCjyULGYEWye1evdsKHC/XPGr/hxolQQ5qUCORr/81RsokgoqLeHYmG7gJzbMsLaMcDot9VJDE0zGeEi7jkosqAmz+bVTdOaUAYqVdiUtmqu/JzIsjJmIyHUKbEdm2ZuJ/3nd1MZXYcZkkloqyWJRnHJkFZq9jgZMU2L5xBFMNHO3IjLCGhPrAiq5EILll1dJ66Ia+NXg7rJSv87jKMIJnMI5BFCDOtxCA5pA4BGe4RXePOW9eO/ex6K14OUzx/AH3ucPa1+PAw==</latexit><latexit sha1_base64="xl6gt5zesTVLaQIKxfcD+hK+ntk=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKEI9BLx4jmgckS5idzCZj5rHMzAphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntJ6oNU/LBThIaCjyULGYEWye1evdsKHC/XPGr/hxolQQ5qUCORr/81RsokgoqLeHYmG7gJzbMsLaMcDot9VJDE0zGeEi7jkosqAmz+bVTdOaUAYqVdiUtmqu/JzIsjJmIyHUKbEdm2ZuJ/3nd1MZXYcZkkloqyWJRnHJkFZq9jgZMU2L5xBFMNHO3IjLCGhPrAiq5EILll1dJ66Ia+NXg7rJSv87jKMIJnMI5BFCDOtxCA5pA4BGe4RXePOW9eO/ex6K14OUzx/AH3ucPa1+PAw==</latexit> <latexit sha1_base64="WVBNru380J4XCjCR9Lt9t8h8e1I=">AAAB7XicbVDLSgNBEOyNrxhfqx69DAbBU9gVQY9BLx4jmAckS5idzCZj5rHMzAphyT948aCIV//Hm3/jJNmDJhY0FFXddHfFKWfGBsG3V1pb39jcKm9Xdnb39g/8w6OWUZkmtEkUV7oTY0M5k7RpmeW0k2qKRcxpOx7fzvz2E9WGKflgJymNBB5KljCCrZNaPcOGAvf9alAL5kCrJCxIFQo0+v5Xb6BIJqi0hGNjumGQ2ijH2jLC6bTSywxNMRnjIe06KrGgJsrn107RmVMGKFHalbRorv6eyLEwZiJi1ymwHZllbyb+53Uzm1xHOZNpZqkki0VJxpFVaPY6GjBNieUTRzDRzN2KyAhrTKwLqOJCCJdfXiWti1oY1ML7y2r9poijDCdwCucQwhXU4Q4a0AQCj/AMr/DmKe/Fe/c+Fq0lr5g5hj/wPn8AnF+PIw==</latexit><latexit sha1_base64="WVBNru380J4XCjCR9Lt9t8h8e1I=">AAAB7XicbVDLSgNBEOyNrxhfqx69DAbBU9gVQY9BLx4jmAckS5idzCZj5rHMzAphyT948aCIV//Hm3/jJNmDJhY0FFXddHfFKWfGBsG3V1pb39jcKm9Xdnb39g/8w6OWUZkmtEkUV7oTY0M5k7RpmeW0k2qKRcxpOx7fzvz2E9WGKflgJymNBB5KljCCrZNaPcOGAvf9alAL5kCrJCxIFQo0+v5Xb6BIJqi0hGNjumGQ2ijH2jLC6bTSywxNMRnjIe06KrGgJsrn107RmVMGKFHalbRorv6eyLEwZiJi1ymwHZllbyb+53Uzm1xHOZNpZqkki0VJxpFVaPY6GjBNieUTRzDRzN2KyAhrTKwLqOJCCJdfXiWti1oY1ML7y2r9poijDCdwCucQwhXU4Q4a0AQCj/AMr/DmKe/Fe/c+Fq0lr5g5hj/wPn8AnF+PIw==</latexit><latexit sha1_base64="WVBNru380J4XCjCR9Lt9t8h8e1I=">AAAB7XicbVDLSgNBEOyNrxhfqx69DAbBU9gVQY9BLx4jmAckS5idzCZj5rHMzAphyT948aCIV//Hm3/jJNmDJhY0FFXddHfFKWfGBsG3V1pb39jcKm9Xdnb39g/8w6OWUZkmtEkUV7oTY0M5k7RpmeW0k2qKRcxpOx7fzvz2E9WGKflgJymNBB5KljCCrZNaPcOGAvf9alAL5kCrJCxIFQo0+v5Xb6BIJqi0hGNjumGQ2ijH2jLC6bTSywxNMRnjIe06KrGgJsrn107RmVMGKFHalbRorv6eyLEwZiJi1ymwHZllbyb+53Uzm1xHOZNpZqkki0VJxpFVaPY6GjBNieUTRzDRzN2KyAhrTKwLqOJCCJdfXiWti1oY1ML7y2r9poijDCdwCucQwhXU4Q4a0AQCj/AMr/DmKe/Fe/c+Fq0lr5g5hj/wPn8AnF+PIw==</latexit><latexit sha1_base64="WVBNru380J4XCjCR9Lt9t8h8e1I=">AAAB7XicbVDLSgNBEOyNrxhfqx69DAbBU9gVQY9BLx4jmAckS5idzCZj5rHMzAphyT948aCIV//Hm3/jJNmDJhY0FFXddHfFKWfGBsG3V1pb39jcKm9Xdnb39g/8w6OWUZkmtEkUV7oTY0M5k7RpmeW0k2qKRcxpOx7fzvz2E9WGKflgJymNBB5KljCCrZNaPcOGAvf9alAL5kCrJCxIFQo0+v5Xb6BIJqi0hGNjumGQ2ijH2jLC6bTSywxNMRnjIe06KrGgJsrn107RmVMGKFHalbRorv6eyLEwZiJi1ymwHZllbyb+53Uzm1xHOZNpZqkki0VJxpFVaPY6GjBNieUTRzDRzN2KyAhrTKwLqOJCCJdfXiWti1oY1ML7y2r9poijDCdwCucQwhXU4Q4a0AQCj/AMr/DmKe/Fe/c+Fq0lr5g5hj/wPn8AnF+PIw==</latexit> 𝝈
23.
23. Conversion really matters. eCPM = BAconv. * pCTR * pCVR Very similar to CTR but totally different from CTR
24.
24. Impression Click Conversion CTR >> CVR Rare event 𝜟t << 𝜟t Conversion dalay Click Installation, registration, purchase, subscription, … Variety Segment Personal Granularity One action Sequence of actions Context (hurdles) SSP/DSP MAT/SDK/Pixel Data Integrity
25.
25. Features over Algorithms — Conversion proxy: Retargeting — Conversion-driven LookALike
26.
26. More topics for better response prediction — Multi-task learning (for each ad, objective) — Transfer learning — Landscape forecasting — Multi-touch attribution — Cold-start — Exploitation vs Exploration — Thompson sampling
27.
27. eCPM = BA x pCTR How much to bid? How to dynamically adjust bids? Is every audience equally valuable to me?
28.
28. Budget smoothing and auto-bidding
29.
29. Creative BAclk pCTR eCPM Rank *PPC (GSP) A 1,000 17% 170 1 941 B 1,500 6% 90 4 **500 C 1,200 9% 108 3 1,000 D 800 20% 160 2 540 * PPC = BA * (next eCPM / own eCPM) = next eCPM / own pCTR ** Reserve price = 500
30.
30. Research Offline Test Online Test Production • Model validity • Effect simulation • Validity & revenue • A/A Test • A/B/C/… Test • Random bucket Problem & ideation Complexity & Stability
31.
31. In theory (model validation) — Loss (logLoss) — RIG = 1 - NE (Entropy) — Calibration = predicted / actual — AUC In reality — Revenue per request
32.
32. 10 More topics beyond this presentation — Yield optimization (SSP) — Auction design (GSP/VCG, reserve price/bid floor) — Fraud/abusing detection — Targeting/retargeting/LookALike — Frequency/recency capping — AdBlock & DNT: Usefulness vs Annoying — Dynamic/personalized creative generation — System consideration (e.g., distributed system) — Knowledge representation beyond audience
33.
33. Inventory-buying Audience-buying Impression Click Conversion Guaranteed AdX & RTB Non-Guaranteed One Platform
34.
34. Bid amount (BA) Response Rate eCPM ≫ Manual setting (by Adv.) ≫ Auto-bidding (BA*) ≫ Impressions (≒ requests) ≫ Viewable impression ≫ Clickthrough rate (CTR) ≫ Conversion rate (CR, Landscape forecasting (ARIMA, Prophet) Probability (Logistic Regression) — Input feature (X) — Model weight (W)
35.
35. Ranking eCPM = BAimp = BAclk * pCTR = BAconv * pCTR * pCVR Data/Feature Privacy-free audience data Embedding Classification (Bayesian) + Topic Modeling (LDA) Prediction Logistic Regression Training (Online) FTRL-Proximal BidAmount Auto-bidding Targeting Conversion-driven LookALike
36.
36. Ideality vs Reality Hard works from theory to production Privacy is our top priority.
37.
37. Some references - Ad click prediction: a view from the trenches - Practical lessons from predicting clicks on ads at facebook - Simple and scalable response prediction for display advertising - Modeling delayed feedback in display advertising - Latent dirichlet allocation - Factorization machines - Field-aware factorization machines for CTR prediction - Field-weighted factorization machines for click-through rate prediction in display advertising - Deep and cross network for ad click predictions - Model ensemble for click prediction in bing search ads - Optimal real-time bidding for display advertising - Feature hashing for large scale multitask learning - Score lookalike audiences