This document provides an overview of Optimizely for mobile apps. It discusses how mobile usage is growing rapidly and time spent on mobile devices now exceeds desktop. It then shares customer stories from French Girls and Secret Escapes who significantly improved key metrics like activation and signups through mobile A/B testing. The document demonstrates Optimizely's visual editor and code capabilities. It proposes a framework to test across acquisition, engagement, retention and monetization. Specific mobile testing ideas are outlined for each area like layouts, feature placement, checkout flows, permissions and reviews. Best practices are shared around integrating Optimizely, planning a testing roadmap, and verifying experiments.

1. Getting Started with
Optimizely for Mobile
Apps
Suneet Shah
Senior Product Manager,
Optimizely
@suneets
suneet@optimizely.com
#opticon2015

2. #opticon2015
Suneet Shah

3. In The Next 45 Minutes
1. Customer Stories
2. Demos
3. Test Ideas
4. Best Practices for getting started
5. Q & A

4. • iOS Developer Preview
Launched
• A/B testing a new concept for
mobile apps
Last Year

5. Today
• iOS and Android
• New Visual Editor
• Tons of customer feedback
• Hundreds of apps on iOS and
Android

6. Time spent per adult user per day (USA)
Source: Mary Meeker internet trends report 2015
0
1.5
3
4.5
6
2012 2013 2014 2015 (YTD)
0.40.3
0.3
0.3
2.82.6
2.3
1.6
2.42.42.32.5
Desktop/Laptop Mobile Other Connected Devices
} 51%
} 42%
HoursPerDay

7. Another 1b people will come online through mobile
0
1
2
3
4
1995 2000 2014 2020
People Online Smartphones
Source: a16z Benedict Evans “Mobile is eating the world”
BillionPeopleOnline

8. Cost Per Install (CPI) rising
April 2014 April 2015
Cost Per Install (iOS) $1.38 $2.13
Cost Per Install
(Android)
$1.31 $2.08
Source: FiKSU mobile marketing index https://www.fiksu.com/resources/fiksu-indexes#analysis
+54%

9. On average, apps lose 80% of their new users
Source: Andrew Chen’s blog, http://andrewchen.co/

10. The top apps follow the same pattern
Source: Andrew Chen’s blog, http://andrewchen.co/

11. If you’re not testing your
app yet, you should be!

12. Customer Stories

14. French Girls Optimizes Activation
• Activation is the difference
between 1-2 days of use and 7+
days of use
• Activation rate of 25%

15. Baseline activation rate of 25%

16. Baseline activation rate of 25%

17. Winning variation doubled activation to 50%
Activation
50%
Copy changes
+16%

18. “Optimization is no longer an
afterthought, but a necessary
part of every feature”
- Jeff Farkas, French Girls

19. French Girls
• Used code blocks to increase activation
from 25% to 50%
• Subsequent visual changes found an
additional 16.5% improvement
• Total improvement: from 25% to 66.5%

21. Secret Escapes Optimizes Acquisition
• To force sign up, or not to force
sign up
• Looking to optimize LTV, sign up
rates, and justify mobile ad spend

22. Experiment Setup
Version 1 Version 2
Sign up gate with
mandatory sign in on first
user experience
Sign up not required,
added a “skip” button to
close the sign up form

23. Results
• Defying all expectations, a mandatory
signup gate was by far the optimal
experience
• Sign up rates doubled, LTV average
increased, and LTV:CAC ratio was
positive

24. “If the skip button was part of
the onboarding flow, we
would not be able to do paid
marketing on mobile.”
– Sebastian Fallert, Secret
Escapes

25. Optimizely for iOS and Android
Demo

26. Demo
• SDK
• Visual Editor
• Live Variables
• Code Blocks

27. SDK Installation
• Optimizely SDK needs
to be integrated and
deployed
• 1 line integration

29. Visual Changes
• Copy changes, calls to
action, images and layout
changes
• Create tests without
releasing a new app
• No code required

30. Live Variables & Code Blocks
• Have to be built into the
app before release
• Easy setup for developers
• Allow you to test anything

31. Body text goes here.
Live Variables
Regular Variable
private static final String ratingsPromptText = "Do you love our app?”;
alertDialog.setText(ratingsPromptText);
Optimizely Live Variable
private static LiveVariable<String> ratingsPromptText =
Optimizely.stringVariable("RatingsPrompt1","Do you love our app?”);
alertDialog.setText(ratingsPromptText.get())

32. Body text goes here.
Live Variables
Regular Variable
private static final String ratingsPromptText = "Do you love our app?”;
alertDialog.setText(ratingsPromptText);
Optimizely Live Variable
private static LiveVariable<String> ratingsPromptText =
Optimizely.stringVariable("RatingsPrompt1","Do you love our app?”);
alertDialog.setText(ratingsPromptText.get())

33. OptimizelyCodeBlocksKeyWithBlockNames(CheckoutFunnel,
@"Default Checkout”,
@"One Stage Checkout”,
@"Paypal Checkout”);
Code Blocks
[Optimizely codeBlocksWithKey:CheckoutFunnel
blockOne:^{
[self showOneStageCheckout];
}
blockTwo:^{
[self showPaypalCheckout];
}
defaultBlock:^{
[self showDefaultCheckout];
}];

34. Mobile Testing Ideas

35. Framework for Mobile Apps
Acquisition Engagement
Retention Monetization

36. Framework for Mobile Apps
Acquisition Engagement
Retention Monetization

37. Framework for Mobile Apps
Acquisition Engagement
Retention Monetization

38. Engagement
• Layouts and navigation
patterns
• Discoverability and
placement of key features
• Test every new feature

39. Facebook layout test
Source: LukeW’s Design blog, http://www.lukew.com

40. LukeW’s Polar App
Source: LukeW’s Design blog, http://www.lukew.com

41. Test every new feature
Phased Rollouts

42. Test key feature placements
1. Date selector
2. Repeat map CTA as floating button
3. Narrower focus + information
hierarchy
2
3
1

43. Framework for Mobile Apps
Acquisition Engagement
Retention Monetization

44. Monetization
• Checkout flows and purchase
funnels
• Ad networks
• Virality flows and sharing
features

45. Calls to action in your purchase flow

46. Payment Options

47. Payment Options

48. Ad Placements

49. Social/sharing

50. Framework for Mobile Apps
Acquisition Engagement
Retention Monetization

51. Retention
• Permissions - Push
Notifications, Location,
Camera
• Reviews

52. Permissions

53. Only one chance

54. Asking for push notification permissions

55. When and how you ask

56. Reviews
Source: Circa’s blog article, https://medium.com/circa/the-right-way-to-ask-users-to-review-your-app-9a32fd604fca

57. Optimizing Reviews
3 Things They Optimized:
Threshold (Timing)
of Request Location of Request Flow
of Request
10 Sessions
3x in 1
Week

58. Retention for top apps

60. The best way to improve
retention is to improve
onboarding and
activation

61. Framework for Mobile Apps
Acquisition Engagement
Retention Monetization

62. Best Practices

63. 5 Key Steps To Get Started
Integrate the SDK
Start using in development
Plan Testing Roadmap
What metrics are you improving?
Implement First 3 Tests
Before going to the app store
QA Experiments
Verify metrics
Submit to the App Store
And start your experiments
1
2
3
4
5

64. Plan Ahead
• Make testing part of your release cycle
• Plan tests upfront
• Keep an experiment backlog

65. Start With 3 Tests
• Get familiar with Optimizely
• Visual Change
• Code Blocks
• Live Variables

66. • QA your experiments the way your users will see them
• Verify conversions and visitors
• Use SDKs’ developer features in QA process
• Use Preview feature
Experiment QA

67. #opticon2015
Suneet Shah
Senior Product Manager,
Optimizely
@suneets
suneet@optimizely.com
Thank you!

68. #opticon2015
10 Secrets to Building an Amazing Mobile Testing Roadmap
Thurs 2:30 - 3:15 | Coit Tower Room
Optimizely Product Feedback Roundtable Discussions
Thurs 11:25 - 12:10 | Expo Hall
Roundtable - Optimizing for Mobile Retention
Thurs 2:35 – 3:12 | Expo Hall