From DataEngConf NYC 2018 https://www.datacouncil.ai/speaker/scaling-personalization-via-machine-learned-assortment-optimization -- Machine learning has revolutionized the capability of businesses to create personalized experiences via real-time, individual predictions and recommendations. But what happens when one must make thousands of decisions for thousands of individuals at the same time? At Dia&Co, a plus-size women’s styling service, we recently faced such an obstacle when building out a brand new product line for the business. This talk will explore how we combined modern machine learning with classical operations research techniques to scale personalization in the face of constraints inherent to a retail business. The basics of operations research will be introduced before demonstrating how to solve a simple version of our real-world problem using all open source libraries. I will then reveal the gory details of productionizing this work, from testing to gracefully handling failures of convergence. Finally, I will cover the journey from the coldest of starts, with zero data, to synthesizing machine learning with the operations research problem.

1. Scaling Personalization via
Machine-Learned
Assortment Optimization
Ethan Rosenthal
Dia&Co
DataEngConf NYC 2018
11/8/2018
eprosenthal
EthanRosenthal
www.ethanrosenthal.com

3. Nearly 70% of women in the U.S.
are plus-size, but they represent
only 16% of apparel spend

6. Algorithmically style all
boxes for a day at once

8. Greedy Humans-in-
the-loop

10. Queue

11. Queue

12. Queue

13. Queue

14. Queue

15. Queue

16. Queue

17. Queue

18. Queue

19. Pre-distribution of
Wealth

20. Batch

21. Batch

22. ML + OR

23. Analytics
Machine
Learning
Operations
Research
Machine
Learning
+
Operations
Research

24. Analytics
Machine
Learning
Operations
Research
Machine
Learning
+
Operations
Research
Send emails at 9 AM
on Tuesdays because
they have the highest
open rates

25. Analytics
Machine
Learning
Operations
Research
Machine
Learning
+
Operations
Research
Each customer
receives email at time
that they are most
likely to open

26. Analytics
Machine
Learning
Operations
Research
Machine
Learning
+
Operations
Research
Given deliverability
and throughput,
distribute email sends
across the week

27. Analytics
Machine
Learning
Operations
Research
Machine
Learning
+
Operations
Research
Given deliverability,
throughput, and
customer’s behavior
maximize opens

28. Integer
Programming 101

31. <math><code>

32. Decision Variables
from pulp import LpVariable
users = range(N)
items = range(M)
user_indicators = {}
for u in users:
user_indicators[u] = LpVariable(f'user_{u}', cat='Binary')
user_item_indicators = {}
for u in users:
for i in items:
user_item_indicators[u, i] = LpVariable(
f'user_{u}_item_{i}', cat='Binary'
)

34. Objective Function
from pulp import lpSum, lpProblem, lpMaximize
problem = lpProblem('BoxMaker', lpMaximize)
objective_function = []
for u in users:
for i in items:
value = scores[u, i] * user_item_indicators[u, i]
objective_function.append(value)
problem += lpSum(objective_function)
Machine
Learning

36. Constraints
for u in users:
users_item_count = lpSum(
user_item_indicators[u, i] for i in items
)
problem += users_item_count == 5 * user_indicators[u]
for i in items:
quantity_used = lpSum(
user_item_indicators[u, i] for u in users
)
problem += quantity_used <= quantity_available[i]

38. Solve
problem.solve()¯_(ツ)_/¯

39. </math></code>

40. Productionizing

41. Productionalizing

42. Making this thing
automatically and
reliably run in the
cloud

43. ● Software Architecture
● Testing
● Interfaces
● Failure Planning
● Iteration

44. ● Software Architecture
● Testing
● Interfaces
● Failure Planning
● Iteration

48. Objective Function

49. Deterministic Scorers
“If item is in-budget, +50 points”
Problem
Solution
Pre-calculate and aggregate

50. Stochastic Scorers
“If `outfit` in box, +100 points”
Problem
Solution
Create mini-optimization problem
with new decision variables and
constraints

52. Deterministic Constraints
“User can only receive items that fit”
Problem
Solution
Don’t make these decision variables

53. Stochastic Constraints
“Customers must receive between X and Y
shirts”
Problem
Mental gymnastics to define constraints
Solution

54. ● Software Architecture
● Testing
● Interfaces
● Failure Planning
● Iteration

55. Unit Tests are Hard

56. Test for Behavior

57. Invest in Test Setup
Functions

58. ● Software Architecture
● Testing
● Interfaces
● Failure Planning
● Iteration

59. Microservices

61. ● Software Architecture
● Testing
● Interfaces
● Failure Planning
● Iteration

62. Perfect is the
enemy of good

63. Minimum Viable Solution

64. ● Software Architecture
● Testing
● Interfaces
● Failure Planning
● Iteration

65. Frequent
Deployments

66. Model
Improvements

67. Tests

69. Points -> Coefficients
Each scorer becomes an indicator “feature”
Each observation/sample is a box
Learn to predict some box-level metric

70. f(X) = X * w ~ y
box 1
box 2
.
.
.
box n
0 4
51
.
.
.
.
.
.
31
. .
. .
.
.
.
.
.
.
. .
w_0
w_1
.
.
.
w_n
?
?
.
.
.
?
~*

71. Thank You!
We’re hiring! www.dia.com/careers
eprosenthal
EthanRosenthal
www.ethanrosenthal.com