Scaling experiments from 1 to 100+ with Optimizely

•Download as PPTX, PDF•

0 likes•264 views

Are you struggling to go from 1 to 10 or from 100 to 1000 experiments? Optimizely Full Stack is a powerful tool, but knowing best practices for implementation is key to 10x-ing your experimentation program. In this workshop, we'll describe best practices like installing and wrapping the SDK, strategies for code footprint, and fitting a product experimentation culture into your team's process. Come learn how to best leverage Full Stack to scale your product experimentation program.

Technology

2
Scaling Fullstack
Going from 1 to 100+ experiments
Asa Schachar
Engineering Manager
Ali Rizvi
Staff Software Engineer
Thilo Richter
Solutions Architect Manager

Simple Experimentation Setup
Application
SDK
Datafile
A
B
Remote
Configuration
Analytics

Application Backend
app.optimizely.com
Analytics
Backend
Frontend Application
app.optimizely.com
Your App
SDK
Datafile
Optimizely User
Your Users
Events
A
B

Application Backend
app.optimizely.com
Analytics
Backend
Frontend Application
app.optimizely.com

Application Backend
app.optimizely.com
Analytics
Backend
Frontend Application
app.optimizely.com
SDKSDK
SDK

Installing SDKs
Application Backend
app.optimizely.com
Analytics
Backend
Frontend Application
app.optimizely.com
SDKSDK
SDK

Datafile Polling
Application
SDK
Datafile
A
B
5 min
CDN

Datafile Polling
Application
SDK
Datafile
A
B
Remote
Configuration
Analytics
CDN
5 min

Example #1
SDK Stateless
Bucketing Logic
app thread #1
Analytics Backend
Request #1SDK
SDK app thread #2 Request #2
SDK app thread #3 Request #3

Example #1
Analytics Backend
app thread #1 Request #1
app thread #2 Request #2
Wrappe
r
Stateful, Thread-safe,
Singleton Wrapper
Wrapper
SDK
SDK Stateless Bucketing
Logic

100%
100%
Local
(local computer)
Staging
(staging website)
Production
(production website)
Rollback
23
0% 0%
Environments

Challenges of 10-100
Experiments
1. Centralized Metrics Tracking
2.

Scaling experiments from 1 to 100+ with Optimizely

What's hot

Code-to-Cloud Visibility: An Essential Framework for DevOps SuccessJadeCampbell13

Engineering OperationsCybera Inc.

LilaPapiernikResume.DeveloperIILila Papiernik

DevOps Powered by Splunk Hands-OnSplunk

Discovering and Fixing Dependency Vulnerabilities for Kubernetes apps with Sn...Codefresh

ATAGTR2017 Security Testing / IoT Testing in Real WorldAgile Testing Alliance

Why monitoring is an analytics problemPhillip Liu

ATAGTR2017 Machine Learning telepathy for Shift Right approach of testingAgile Testing Alliance

Enterprise Ready Test Execution Platform for Mobile AppsVijayan Srinivasan

Spool manager pipe weld shop Yakup Bozkurt

Software Supply Chain Observability with Grafeas and KritisAysylu Greenberg

ATAGTR2017 Protractor Cucumber BDD ApproachAgile Testing Alliance

ATAGTR2017 CDC Tests - Integration Tests cant be made simpler than this!Agile Testing Alliance

ATAGTR2017 Analytics TestingAgile Testing Alliance

IBM Index Conference - 10 steps to build token based API SecuritySenthilkumar Gopal

ATAGTR2017 HikeRunner: Load Test FrameworkAgile Testing Alliance

Shhh!: Secret Management Practices for Infrastructure as Code Akond Rahman

Turning Cloud Metrics into ResultsInfluxData

OSMC 2013 | The future of Nagios by Andreas EricssonNETWAYS

Characterizing Defective Configuration Scripts Used for Continuous DeploymentAkond Rahman

What's hot (20)

Code-to-Cloud Visibility: An Essential Framework for DevOps Success

Engineering Operations

LilaPapiernikResume.DeveloperII

DevOps Powered by Splunk Hands-On

Discovering and Fixing Dependency Vulnerabilities for Kubernetes apps with Sn...

ATAGTR2017 Security Testing / IoT Testing in Real World

Why monitoring is an analytics problem

ATAGTR2017 Machine Learning telepathy for Shift Right approach of testing

Enterprise Ready Test Execution Platform for Mobile Apps

Spool manager pipe weld shop

Software Supply Chain Observability with Grafeas and Kritis

ATAGTR2017 Protractor Cucumber BDD Approach

ATAGTR2017 CDC Tests - Integration Tests cant be made simpler than this!

ATAGTR2017 Analytics Testing

IBM Index Conference - 10 steps to build token based API Security

ATAGTR2017 HikeRunner: Load Test Framework

Shhh!: Secret Management Practices for Infrastructure as Code

Turning Cloud Metrics into Results

OSMC 2013 | The future of Nagios by Andreas Ericsson

Characterizing Defective Configuration Scripts Used for Continuous Deployment

Similar to Scaling experiments from 1 to 100+ with Optimizely

DevOps Powered by SplunkSplunk

Platform 4.0 Meetup Launch EventFokke Zandbergen

Dynatrace: Going beyond APM and soaring to the futureDynatrace

Predix Builder RoadshowPredix

Data-Driven Software Engineering for Agile TeamsTechWell

Technical Architect on Embedded System.Prasad Roy Raju

Big Data And HTML5 (DevCon TLV 2012)Ido Green

Breaking the 2 Pizza Paradox with your Platform as an ApplicationMark Rendell

Optimizely Agent: Scaling Resilient Feature DeliveryOptimizely

Oracle APEX, Low Code for Data Driving AppsFranco Ucci

Paypal teradata gimel_thrift_serverAnisha Nainani

Release Readiness Validation with Keptn for Austrian Online Banking SoftwareAndreas Grabner

Sharepoint 2013 applied architecture from the field (v2)Tihomir Ignatov

Cirrculum_Vitaevemulakonda

What's new in android 2018 (dev fest)Shady Selim

GDG Cloud Southlake 31: Santosh Chennuri and Festus Yeboah: Empowering Develo...James Anderson

SoftwareQualityControlProfessional_12Yrs_SreekrishnaHPanditSreekrishna Pandit

AWS Startup Insights SingaporeAmazon Web Services

Resume ram-krishnaRam Krishna | BitHostIn

Microsoft Cognitive Service, Tap into the Power of Machine Learning with Easy...IDEAS - Int'l Data Engineering and Science Association

Similar to Scaling experiments from 1 to 100+ with Optimizely (20)

DevOps Powered by Splunk

Platform 4.0 Meetup Launch Event

Dynatrace: Going beyond APM and soaring to the future

Predix Builder Roadshow

Data-Driven Software Engineering for Agile Teams

Technical Architect on Embedded System.

Big Data And HTML5 (DevCon TLV 2012)

Breaking the 2 Pizza Paradox with your Platform as an Application

Optimizely Agent: Scaling Resilient Feature Delivery

Oracle APEX, Low Code for Data Driving Apps

Paypal teradata gimel_thrift_server

Release Readiness Validation with Keptn for Austrian Online Banking Software

Sharepoint 2013 applied architecture from the field (v2)

Cirrculum_Vitae

What's new in android 2018 (dev fest)

GDG Cloud Southlake 31: Santosh Chennuri and Festus Yeboah: Empowering Develo...

SoftwareQualityControlProfessional_12Yrs_SreekrishnaHPandit

AWS Startup Insights Singapore

Resume ram-krishna

Microsoft Cognitive Service, Tap into the Power of Machine Learning with Easy...

Recently uploaded

WordPress Websites for Engineers: Elevate Your Brandgvaughan

Anypoint Exchange: It’s Not Just a Repo!Manik S Magar

TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc

DevEX - reference for building teams, processes, and platformsSergiu Bodiu

Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University

SALESFORCE EDUCATION CLOUD | FEXLE SERVICESmohitsingh558521

"ML in Production",Oleksandr BaganFwdays

A Journey Into the Emotions of Software DevelopersNicole Novielli

Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada

Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3

A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3

Artificial intelligence in cctv survelliance.pptxhariprasad279825

What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett

The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3

What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina

DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy

Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos

Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3

Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3

New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada

Recently uploaded (20)

WordPress Websites for Engineers: Elevate Your Brand

Anypoint Exchange: It’s Not Just a Repo!

TrustArc Webinar - How to Build Consumer Trust Through Data Privacy

DevEX - reference for building teams, processes, and platforms

Nell’iperspazio con Rocket: il Framework Web di Rust!

SALESFORCE EDUCATION CLOUD | FEXLE SERVICES

"ML in Production",Oleksandr Bagan

A Journey Into the Emotions of Software Developers

Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024

Moving Beyond Passwords: FIDO Paris Seminar.pdf

A Deep Dive on Passkeys: FIDO Paris Seminar.pptx

Artificial intelligence in cctv survelliance.pptx

What's New in Teams Calling, Meetings and Devices March 2024

The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx

What is DBT - The Ultimate Data Build Tool.pdf

DevoxxFR 2024 Reproducible Builds with Apache Maven

Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)

Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx

Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx

New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024

Scaling experiments from 1 to 100+ with Optimizely

2. 2 Scaling Fullstack Going from 1 to 100+ experiments Asa Schachar Engineering Manager Ali Rizvi Staff Software Engineer Thilo Richter Solutions Architect Manager

3. 1 to 10 Experiments

5. Simple Experimentation Setup Application SDK Datafile A B Remote Configuration Analytics

6. Application Backend app.optimizely.com Analytics Backend Frontend Application app.optimizely.com Your App SDK Datafile Optimizely User Your Users Events A B

7. Application Backend app.optimizely.com Analytics Backend Frontend Application app.optimizely.com

8. Application Backend app.optimizely.com Analytics Backend Frontend Application app.optimizely.com SDKSDK SDK

9. Installing SDKs Application Backend app.optimizely.com Analytics Backend Frontend Application app.optimizely.com SDKSDK SDK

10. Datafile Polling Application SDK Datafile A B 5 min CDN

11. Datafile Polling Application SDK Datafile A B Remote Configuration Analytics CDN 5 min

12. 10 to 100 Experiments

13.

14. Example #1 SDK Stateless Bucketing Logic app thread #1 Analytics Backend Request #1SDK SDK app thread #2 Request #2 SDK app thread #3 Request #3

15. Example #1 Analytics Backend app thread #1 Request #1 app thread #2 Request #2 Wrappe r Stateful, Thread-safe, Singleton Wrapper Wrapper SDK SDK Stateless Bucketing Logic

16. Example #2

17. Example #2

18. Example #2

19. Example #2

20. 100+ Experiments

21.

22.

23. 100% 100% Local (local computer) Staging (staging website) Production (production website) Rollback 23 0% 0% Environments

24.

25. What are your challenges?

26. Challenges of 1-10 Experiments 1. 2.

27. Challenges of 10-100 Experiments 1. Centralized Metrics Tracking 2.