Triple Your Experiment Velocity by Integrating Optimizely with Your Data Warehouse
Melden
OptimizelyFolgen
1. Oct 2019•0 gefällt mir•505 views
Triple Your Experiment Velocity by Integrating Optimizely with Your Data Warehouse
1. Oct 2019•0 gefällt mir•505 views
Downloaden Sie, um offline zu lesen
Melden
Business
Harnessing the power of data and experimentation is central to Updater’s goal of revolutionizing the moving industry. However, while scaling their experimentation program, the team at Updater had to overcome the challenge that many of their conversions happen offline or on third-party sites and therefore couldn't be used in testing. During this session, you’ll learn how Updater fixed this blind spot in their funnels, tripled experiment velocity, and discovered deep user insights by integrating their experimentation platform and data warehouse.
OptimizelyFolgen
Recomendados
Program Management 101: Best Practices from Optimizely-on-OptimizelyOptimizely
Quantify the ROI of Your Experimentation ProgramOptimizely
Testing Across the Enterprise: How Cox Automotive Scales Experimentation to M...Optimizely
Metrics that Matter: Measure Experiment Impact with Optimizely's Data PlatformOptimizely
What Your Customers Really Do Online: 5 Ways to Remove the GuessworkOptimizely
Under the Hood Webinar Series: B2B Experimentation & Personalization at Optim...Optimizely
Más contenido relacionado
Was ist angesagt?
Driving Customer Acquisition and Growth with ExperimentationOptimizely
Speed Matters - Strategies to Improve Your Site PerformanceOptimizely
Optimism Webinar 1: Improving your digital experiences - what's next in 2019?Optimizely
Building a Culture of Experimentation at HPOptimizely
Building a Technology Architecture for Experience OptimizationOptimizely
An Experimentation Framework: How to Position for Triple Digit GrowthOptimizely
Similar a Triple Your Experiment Velocity by Integrating Optimizely with Your Data Warehouse
What Your Customers Really Do Online: 5 Ways to Remove the GuessworkOptimizely
Conducting the Experimentation OrchestraOptimizely
Beyond Simple A/B testingRatio
Clover Rings Up Digital Growth to Drive ExperimentationOptimizely
Intuit - How to Scale Your Experimentation ProgramOptimizely
Starter Kit for Collaboration from Karuana @ Microsoft ITKaruana Gatimu
![[Webinar] Visa's Journey to a Culture of Experimentation](https://cdn.slidesharecdn.com/ss_thumbnails/optimizelywebinar-visasjourneytoacultureofexperimentation-180227210737-thumbnail.jpg?width=1600&height=908&fit=bounds)
Más de Optimizely
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Optimizely
Atlassian's Mystique CLI, Minimizing the Experiment Development CycleOptimizely
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...Optimizely
Zillow + Optimizely: Building the Bridge to $20 Billion RevenueOptimizely
The Future of Optimizely for Technical TeamsOptimizely
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...Optimizely
Último
Transforming Smartsheet into a Strategic Portfolio Management Solution with O...OnePlan Solutions
Grand Challenges & Platform EcosystemsPaavo Ritala
MSA Website Slideshow (1).pdfHamzaKhan46086
140 Book Marketing Ideas to Help Authors Increase SalesAdrienne Jack
Gravyty Presentation for Bloomerang Year End Stewardship 9-26-23.pdfBloomerang
Profile Mstar Corp 2023 ENG Mstar Corp
Triple Your Experiment Velocity by Integrating Optimizely with Your Data Warehouse
- 1. Congratulations to our Outperform Award Winners! Most Dramatic Business Impact Most Customer- Obsessed Company Culture Most Transformative Innovation Most Inspiring Social Impact
- 2. 2 Sessions today will be recorded and will be available after Opticon Join the conversation on Twitter at #Opticon19 Like what you’ve seen today? Give feedback and rate sessions on the mobile app
- 3. Chris Byington Director of Analytics, Updater Dan Storms VP of Product, Updater Triple Your Experiment Velocity by Integrating Optimizely with Your Data Warehouse
- 4. 4 Agenda 1. Updater & experimentation 2. A tale of two experimentation programs 3. Updater’s challenge 4. Our solution 5. Example outcomes 6. Conclusion & key takeaways 7. Q & A
- 5. 5 Build technology that is so good, people won’t move without it. • Revolutionizing the way 40 million people move every year in the U.S. • Offering new and innovative solutions to make relocation more efficient for everyone
- 6. 6 Updater’s business model Create value by bridging the communication gap Mover BusinessUpdater Save time, money, headache, & backache Reach consumers at a crucial behavior change moment
- 7. 7 Experimentation at Updater Learn Drive Release how to solve problems for our users revenue features with confidence The goal is to…
- 8. 8 Experimentation at Updater Crank the Flywheel • Faster experimentation process • Quicker feature development • More feature releases
- 9. 9 — Jeff Bezos Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day.
- 10. 10 The blind spot Our true outcome metrics often occur off-platform, creating a blind spot in our funnels Off-platform metrics
- 11. 11
- 12. 12 • Weak off-platform transaction tracking • Slow experiments (if done at all) • Shallow user insights • Quick experimentation • Higher confidence experiments • Deeper user insights
- 13. 13 The dream… • Experiment more quickly • Experiment with higher confidence • Gain deeper user insights The reality • Bottlenecks in the funnel • Lack of visibility into what users are doing off-platform
- 14. 14 Experiment bottlenecks and lack of visibility Example: Setting Up TV & Internet Sign Up Enter TV/Internet Flow Answer Preference Questions Browse Plans Make a Reservation Call the Call Center Schedule Installation Installatio n Occurs On Platform Off-Platform Track and experiment No tracking
- 15. 15 Major hurdles Long evaluation cycle Low confidence in results Loss in team productivity
- 16. 16 Can it be better?The Challenge Analyst hours to evaluate 4 hours Elapsed time to evaluate 3 days Ticket required to evaluate Yes Lines of SQL needed ~35 Medium of evaluation MS Excel Ability to self-serve results No Maximum confidence level 50% Maximum experiment riskiness Medium Experimentation Process Effectiveness
- 17. 17 Our solution Identify the problem Combine forces Work with 3rd parties Enable feed into data platform Integrate data platform with Optimizely 1 2 3 4 5
- 18. 18 Our solution — Step 1 Identify Combine Work with Enable feed Integrate 1 2 3 4 5 the problem forces 3rd parties into data platform data platform with Optimizely
- 19. 19 Our solution — Step 2 Identify Combine Work with Enable feed Integrate 1 2 3 4 5 the problem forces 3rd parties into data platform data platform with Optimizely
- 20. 20 Our solution — Step 3 Identify Combine Work with Enable feed Integrate 1 2 3 4 5 the problem forces 3rd parties into data platform data platform with Optimizely
- 21. 21 Our solution — Step 4 Identify Combine Work with Enable feed Integrate 1 2 3 4 5 the problem forces 3rd parties into data platform data platform with Optimizely
- 22. 22 Our solution — Step 5 Identify Combine Work with Enable feed Integrate 1 2 3 4 5 the problem forces 3rd parties into data platform data platform with Optimizely
- 23. 23 Data platform Source Systems Extract & Load Data Lake & Warehouse Data Consumption & Informed Team Members UserExperienceSaaS Event Data Off-Platform Data Application DBs EventBus DataStoreOrchestration Snapshots Denormalized Tables Structured Data Business Intelligence Business Partners
- 25. 25 Outcomes We’ve unlocked the ability to… Run Have Automate Gain many more experiments higher confidence a manual process deeper user insights
- 26. 26 Experimentation Process Effectiveness The Challenge The Solution Analyst hours to evaluate 4 hours 0 hours Elapsed time to evaluate 3 days 0 days Ticket required to evaluate Yes No Lines of SQL needed ~35 0 Medium of evaluation MS Excel Optimizely Ability to self-serve results No Yes Maximum confidence level 50% >99% Maximum experiment riskiness Medium Very high
- 27. 27 Experiment velocity gains Number of Experiments Launched per Month Number of Experiments Aug 2018 Sep 2018 Oct 2018 Nov 2018 Dec 2018 Jan 2019 Feb 2019 Mar 2019 Apr 2019 May 2019 Jun 2019 Optimizely Data Platform Integration
- 28. 28 Results A successful experiment Offline booking of TV and internet services Control: Plan Summary Only 15% Variant: Show Additional Plan Details off-platform Flat on platform
- 29. 29 Learning what doesn’t work A shorter user workflow Control: Include a Reservation Step Variant: Skip Reservation Page Results 23% conversion impact
- 30. 30 Experiment Runtime Syncing experiment results from Optimizely Analyzing a detailed funnel of user behavior… while the test is still running Sync user-variant mapping data Further optimization based on learnings Control: Book a New Service Variant: Transfer an Existing Service Extract experiment data Results 67% UX result improvement By 2/3 decrease in poor experience
- 31. 31 Experimentation velocity is crucial to success.So act like it.
- 32. 32 There’s power in combining forces Data + Product =
- 33. Thank you!
- 34. Up Next… Head to the Expo Hall for a refreshment break Fireside Chat with Ashton Kutcher at 4pm! Happy Hour in the Expo Hall at 4:45pm
Hinweis der Redaktion
- We’ll show something here to highlight our business model to provide context for the presentation.
- Graphic representation of a flywheel… Making this process fast = making feature development fast = releasing more features ==> testing quickly is crucial to Updater’s success
- As a result, there’s real value in experiment velocity Be creative and thoughtful to improve it Aspire to drive a step-change increase in experimentation speed rather than optimizing around the edges
- a choice between the least worst of: Test using intermediate metrics that eroded confidence in test results & feature release Download user-variant lists for each experiment, request one-off reporting from 3rd party conversion sites, wait for them to send the reporting, and then tabulate experiment results by hand before evaluating experiments (took days to evaluate experiment once reached sample size)
- We wanted to experiment more quickly & with higher confidence and gain deeper insights about our users, but time and time again hit a bottleneck of not knowing what was happening off-platform -- blind spot in our funnels. This had a few consequences (consequences in the following slide):
- If our systems were integrated, we’d be able to treat offline conversions as first-class success metrics in Optimizely and learn and make decisions – with confidence – more quickly, allowing us to run riskier, higher-potential experiments, fail, and be more creative.
- Step 1. Identified the problem - Data team realized Experimentation speed is crucial to speed of product development It’s a bottleneck to not have offline conversion events in Optimizely in near-real time. Making this change will dramatically increase testing speed and confidence, and allow us to more deeply analyze user behavior in Data Platform to boot
- Step 2. Product & Data teams partner to vet & prioritize the project
- Step 3. Worked with third party partners to gain access to conversion event data feed
- Step 4. Data Engineering wrote ELT process to ingest this data feed into Data Platform
- Step 5. Take 3rd party conversion data in Data Platform & send it into Optimizely in a way that can be used for AB testing; get AB testing data back from Optimizely into our Data Platform so it can be used for deeper analysis
- Data Platform Core data store (Snowflake) Surrounding services Fivetran for ELT of stateful data Production databases 3rd party data like Salesforce, Iterable, etc. Consolidated event bus for all event data Apache Airflow for orchestration Optimizely
- Show the NEW user journey next, as it’s a result of the outcomes.
- Example: Displaying additional TV / Internet package info in-app, drove an increase in call center performance. Our internet booking product uses a call center to complete the transaction, and the funnel has two offline steps: calls to the call center and conversions. We made an in-product change to surface additional information about individual internet plans to the user, which we expected to increase conversion to the endpoint of our in-product experience. However, we saw a flat impact on in-product conversion. Thanks to having offline events in Optimizely in near real time, we were surprised to see a 15% increase in call-to-conversion rate, meaning a 15% lift to revenue for this product overall! But for this data integration, we would have written this experiment off as a loser and would not have known this product change helped our users and revenue at the same time.
- Example: removing the last page in our in-app user workflow for TV / Internet doesn’t help our users connect with our call center at a higher rate, even though there are fewer in-app steps. We decided to test removing the page in our user workflow where the user can view which internet package they reserved, the hypothesis being fewer pages would make it easier for users to get to the endpoint and convert. We saw the opposite, i.e. a negative impact on end-conversions. Thanks to being able to see end-conversion events in Optimizely, we were able to see that our treatment variant was performing worse than control well before our test reached its target sample size and turn the test off, thereby significantly limiting the negative impact the test would have had on our overall conversion rate.
- Example: analyzing a detailed funnel of user behavior in an experiment while the test is still running Again for our internet booking product, we tested adding an entirely new user workflow in parallel to our current workflow - one to book new service, one to transfer existing service Each workflow ends with a call to our call center, so in Optimizely we were quickly able to see that giving users the option to transfer resulted in more calls - this was great because it gave users something they needed while also generating revenue Never satisfied with our own success, while the test was running, using user-variant mapping data synced into Snowflake from Optimizely we analyzed each step of both funnels to more deeply understand how users were performing within each funnel We quickly identified an area for further optimization: transfer users calling our call center were converting at about ½ the rate that other users were. Armed with this insight, we collaborated with our call center to understand the driver of this lower conversion rate, and released optimizations to the product which significantly increased conversion rate from calls, further increasing revenue on top of the lift we saw in our initial experiment. Specifically, we made one particularly impactful discovery: Many users calling our call center were trying to transfer with a provider which wasn’t available at their new home, meaning that transfer was not possible and our alternative new service workflow was better suited to these users. We used this information to more conspicuously surface available providers to users within our app, decreasing this false transfer attempt rate by ⅔
- Moving as quickly as possible from observation >> hypothesis >> test >> evaluation >> action will map to releasing better product experiences to users faster
- Importance of Data/Product teams working together/joining forces Marry business understanding with technical understanding Data is a crucial source of learning, so Product teams that value learning quickly can benefit from partnering closely with Data teams Creativity and enthusiasm for the world of data can have a massive impact. Don’t just think, “how do we make more revenue with this one experiment?” - think about the second- and third-order business-outcome-driving questions such as “how do we increase experiment velocity to impact overall revenue and UX?”