Building a good experimental pipeline and flow for machine learning experimentation is tricky. By focusing on good underlying data platform architecture, DevOps pipelining, experimental scope, and product discoverability, teams can put experiments before customers and learn quickly with a hyper-light process.
1. FROM NEURONS TO A
BRAIN: INTEGRATING
AI INTO YOUR
STARTUP’S
WORKFLOW
2. I am Emily Dresner
I am the CTO of Upside
Travel.
You can find me at
@multiplexer.
3. ▪ AI-powered Business Travel Startup in
Washington, DC.
▪ Travel is a huge ML opportunity and a big data
business.
▫ Shopping and Booking
▫ Aggregated Itineraries
▫ Fare Construction
▪ Millions to billions of itineraries daily.
4. Our goal is to
learn from real
people doing real
work on our
platform.
5. ▪ Air and Hotel Sort
Algorithms
▪ Recommendation Engines
▪ Personalization
▪ Proactively Detect, Alert
and Adjust
▪ Optimum Pricing both for
Customer and for Upside
(steering)
▪ Margin Calculations
▪ Fare Construction
▪ Upsell/Cross-sell
We are in constant tension between supply, profitability and our customer.
We want to make sure everyone is somewhat happy. It’s realistic ML.
6. ▪ Machine Learning Engineering - Shipping Models, Auto-Training
models, A/B testing
▪ Data Pipelining and the ETL - Make Data Useable and Useful
▪ Product Management - Understanding the Models and Turning
them into Product that Delights and Finds “Model-Market Fit."
7. A system to derive hypotheses, run Machine
Learning experiments, and learn quickly with live
customers without negatively impacting the product.
8. DevOps/Data
Fully automated CI/CD
infrastructure.
ML model training pipeline
and deployment fully
automated.
ETL’d Tracks and travel
itinerary data flows
automatically into data
warehouse for continuous
training.
(New!) Experiment data
mixes with track data for end
to end product analysis.
Data Science
Hypothesis construction.
Data selection for research
and development.
Algorithm development,
model training and data
analysis.
Extracts meaning from
experimental results.
Communication with
visualization (Looker)
Product Engineering
Builds tools and services for
deploying models to labs
production.
Builds data pipelines and
schemas for the collection
and extraction of data.
Implements experiments in
the product flow.
Brings successful
experiments to the product
and customers.
9.
10. Hypothesis
Idea!
Standard
Intake Form
Wiki Page
• Testable Hypothesis
• Requirements
• Launch Checklist
• Priori Work
Booking
Pre-Trip
On-Trip
Post-Trip
Total Trip Profit Product
Process Goal: Capture every idea
we can and categorize it at the
intersection of the UX Journey x
Product Theme to help better
understand stakeholders and
visibility.
15. ▪ Culture of experimentation across Engineering and
Product
▪ Faster learning – “pruning the tree.”
▪ Continuous improving our ETL processes for cleaner,
better data
▫ QA-ing the data in the pipeline
▫ Better alerting/monitoring
16. ▪ We are at upside.com!
▪ We host an Engineering and Data Science blog
at engineering.upside.com
▪ Check out our experiments at labs.upside.com
▪ You can reach me at emily@upside.com.
Hinweis der Redaktion
Hello! This is me!
Everything in the travel industry is data. Inventory is data. Pricing is data. Packaging is data.
We’re heavily awash in data, and it’s not always coherent data. It comes from dozens of sources, and every source has its own file and transport format with its own data and its own priorities.
We have invested heavily in ETL (Matillion and Snowflake) to help smooth out that problem.
This is me and what I believe.
Our goal is to build engines of innovation around our data and ML to learn, grow, and give customers what they want.
Our process is an ML Innovation Engine.
Customer empathy is paramount.
A nice selection of our current ML research work. We’re heavily focused on pricing, margins and personalization.
We have limited resources, so we want to apply our people where we’re going to get.
And we have a constant tensions between supply, profitability and our customer. Everyone ends up somewhat happy. It’s not optimized laboratory ML. It’s realistic, real-world ML.
This is a list of the challenges directly out of the posting for the talk.
We do like the phrase “model-market” fit! It means finding a model that works so well for the customer that the customer wants more of your product.
* It is a way to validate product hypothesis against real customers.
* It forces us to curate the hypothesis so that they’re measurable to the customer.
* And these environments are usually only available to large environments ie IBM. We built it small and lean and effective on few people.
One thing to emphasize – you cannot have a post doc in a corner as your AI launching system. This took a small army.
This does not happen in silos.
We put them all in a barn.
Don’t spend much time here. If people want to learn more, they can get the deck after the fact.
Capturing context and learning from the context.
We have difficulty understanding the context of the customer at the time of purchase.
We need to be as deliberate as possible.
We have to marginalize against the context the data was generated in.
HMMM is ML DEBT.
What are the tools we can provide anyone that they can go and run experiments on their own or with little handholding?
It’s not autoML but it is contextual ML and templatized ML in that it is contextual for our problem.
Travel is very domain specific ML problem.
This is our Innovation Filtering Process.
Also gives us WHAT IT IS NOT.
Gives us clear guidelines on experimentation.
Can use the hypothesis as defense against scope creep.
You can get velocity through scope management.
Controlling scope is key
#1 is the customer. These fields make it very clear what the MVP is.
We need a document that lives beyond the team.
Polish is very important. That’s where all the scope creep goes.
Anything not necessary to measure the hypothesis is scope creep/
Launch Checklist is a big innovation. It forces a pattern to notify, measure, and does our measurement work?
We found standardization helped us control scope.
Innovation Engine in Action
Take your hypothesis and break it up.
An experiment really should not be more than two hours.
Think about your smallest possible unit and make it smaller.
Then pass through this cycle until you learn something useful.
You can pop out at any of these points.
It’s ok to bail if this not going to work!
This is the evaluation!
Does it get into the product?!?
Exit points!
There are feedback loops to this process. Even if it fails, there is a feedback loop.
We have lots of opportunity for improvement and optimization
You have to have someone looking over all of it
We have taught non-data scientists this process for innovation
Exactly what it says on the slide.
We are accelerating research by controlling scope, leaning into customer empathy, focusing on learning, feedback cycles, and focusing our time on results.
Despite dirty data and travel industry issues, we have pushed 50+ ML experiments to customers live.
This works for any industry anywhere.