Mobile a/b testing is here to stay. Check out the top recipes mobile product managers, marketers and developers use to optimize the user experience.

1. 10 Delicious Recipes
for Lean a/bs
leanplum
For Consumer Brands and mCommerce

3. From re-activation to increasing mobile engagement and
monetization, we hope these delicious A/B testing recipes
will help your team optimize the mobile user experience.
The Leanplum Team
Leanplum unleashes the value of your user data by making it easy to A/B test and
personalize mobile content and messaging. Get immediate feedback on experiments
with metrics you can act on today.
leanplum

4. Table of Recipes
Re-Activate
First Time User Experience (FTUE)
HOLIDAY PROMOTION
ONBOARDING FLOW
SOCIAL SIGN IN
NEW FEATURE ANNOUNCEMENT
Push Notification
CART ABANDONMENT
3
7
9
1
5
Few opportunities exist that allow you
to re-engage your users with a push
notification without irritating them. These
recipes provide the right opportunities.
The cost to re-activate and keep an
existing user is much lower than the
cost to acquire a new user, so use this
channel strategically.
User flows are the lifeblood of your app.
If there are any arterial blockages, users
can churn after asking themselves “Now
What?” The most important user flow to
optimize is the first 10 seconds when a
user enters your app (the first time user
experience), because a majority of users
will leave within this time period and
never come back.

5. Engage
Monetize
NEW FEATURE ANNOUNCEMENT
In-App Message
OPTIMIZE FOR
SOCIAL BEHAVIOR
NEW FEATURE DEPLOYMENT
APP RATING SCREEN
PRICE SENSITIVITY
11
13
15
17
19
No mobile app is truly static. Being
lean means iterating, learning and
gaining insights through data and
generating new ideas to test. User
engagement encompasses a broad
swath of recipes, from individual
UI elements within screens (call
to action, copy, colors, images) to
funnel analysis and optimizing for
specific goals or achievements (social
sharing, posting content, uploading
photos, leveling up, etc).
App monetization is critical for any
company that relies on ROI to keep the
lights on. Make sure you know who your
most avid and loyal users are. They
are the ones most likely to monetize, or
keep monetizing.

6. NEW FEATURE ANNOUNCEMENT
Push Notification
WHAT TO MEASURE:
As a Product Manager, I want to notify users of recently launched new
content and features via push notification to maximize engagement
and retention.
• Push notification:
- text
- deep link location
- in-app message (aka landing page)
Re-Activate
As you develop a cadence of announcements, measure and get
feedback on which features and content different user segments find
the most relevant and engaging. Be sure to measure and optimize
your campaigns by the most responsive user segments and A/B test
your messaging.
1
Difficulty level:
Novice
Impact:
3/5
WHAT TO A/B TEST:
GOAL:
leanplum
Pro-Tip:

7. WHAT TO MEASURE:
WHO TO TARGET:
• Push notification:
- total deliveries
- total open
- open rate
• New feature or content clicks
per user
• DAU/MAU
• Day 1, 3, 7, 14, 30 retention
• ARPU, ARPPU
All users

8. HOLIDAY PROMOTION
As a Marketer, I want to re-engage inactive users with a seasonal
promotion (aka tentpole event) push notification.
• Push notification:
- text
- discount coupon
- in-app message (aka landing page)
- deep link location
Re-Activate
Sending special offers via push notification during seasonal events
like holidays is a good tactic to re-activate users who may be more
receptive during these times.
3
Difficulty level:
Basic
Impact:
2/5
WHAT TO A/B TEST:
leanplum
Pro-Tip:
GOAL:

9. WHAT TO MEASURE:
WHO TO TARGET:
• Push notification:
- total deliveries
- total opens
- open rate
• Promoted item clicks per user
• Time spent in app per daily
user
• DAU/MAU
• Day 1, 3, 7, 14, 30 retention
• ARPU / ARPPU
All users during the seasonal
event

10. CART ABANDONMENT
As a Marketer, I want to re-engage users who added items to the cart
but did not go through with their purchase.
• Push notification
- text
- discount coupon
Re-Activate
This recipe is a must-do for marketers. Use targeted local push
notifications as a best practice to re-engage with a user’s purchase
intent. Offering a discount or special offer may persuade them
to come back and complete their purchase. Pair this with a
corresponding in-app message to maintain consistency in offer
messaging and incentives.
5
Difficulty level:
Intermediate
Impact:
3/5
WHAT TO A/B TEST:
leanplum
Pro-Tip:
GOAL:

11. WHAT TO MEASURE:
WHO TO TARGET:
• Push notification:
- total deliveries
- total opens
- open rate
• Checkout occurrences per
user
• Time spent in app per daily
user
• DAU/MAU
• Day 1, 3, 7, 14, 30 retention
• ARPU / ARPPU
All users who add items to a
cart but don’t complete the
purchase within a certain
period of time

12. ONBOARDING FLOW
As a Product Manager, I want to experiment with a completely new
re-designed onboarding flow to optimize the completion rate.
• Onboarding flow
- Number of onboarding steps
- Onboarding completion incentive on / off
- New onboarding flow on / off
FirstTimeUserExperience(FTUE)
A large majority of users will abandon your app in the first 30
seconds. Continuously optimizing the on-boarding experience
ensures there is no “now what” moment. Lifts in on-boarding
completion rates can have a much larger impact downstream.
7
Difficulty level:
Intermediate
Impact:
5/5
WHAT TO A/B TEST:
leanplum
Pro-Tip:
GOAL:

13. WHAT TO MEASURE:
WHO TO TARGET:
• Funnel Analysis:
- onboarding completion rate
- step-to-step completion rate
- time spent in each step
- time spent from first to last
step
• Time spent in app per daily
user
• DAU/MAU
• Day 1, 3, 7, 14, 30 retention
• ARPU / ARPPU
New users

14. SOCIAL SIGN-IN
As a Product Manager, I want to maximize the number of new users
who sign in / create a profile with a social plug-in (FB / Twitter /
Google+).
• Number of registration form fields
• Position of social sign in button
• Sign in screen text, look & feel
FirstTimeUserExperience(FTUE)
One-tap social sign-ins reduce friction when compared to email
registration, and enable app developers to capture a richer set of
user data to better tailor the user experience. As a best practice, you
should always tell your users why you want them to login with a social
sign-in so they have context for the request.
9
Difficulty level:
Intermediate
Impact:
4/5
WHAT TO A/B TEST:
leanplum
Pro-Tip:
GOAL:

15. WHAT TO MEASURE:
WHO TO TARGET:
• Funnel Analysis:
- social sign in rate
- manual sign in rate
- step-to-step completion rate
- time spent in each step
- time spent from first to final
step
• Avg. 1st session length
• Time spent in app per daily
user
• DAU/MAU
• Day 1, 3, 7, 14, 30 retention
• ARPU / ARPPU
New users

16. NEW FEATURE DEPLOYMENT
As a Product Manager, I want to test new content or features on a
subset of my users first to avoid hurting engagement or monetization.
• On-off switch for the new content or feature
Engage
Sophisticated mobile product teams rollout new features and content
gradually first to a small percentage of their users, and then step-by-
step to 100% of their user base. Using this method mitigates the risk
of any new release negatively affecting your user experience.
11
Difficulty level:
Intermediate
Impact:
5/5
WHAT TO A/B TEST:
leanplum
Pro-Tip:
GOAL:

17. WHAT TO MEASURE:
WHO TO TARGET:
• New feature or content:
- total views
- total clicks
- click-through rate
• Time spent in app per daily
user
• DAU/MAU
• Day 1, 3, 7, 14, 30 retention
• ARPU / ARPPU
Smaller percent of users,
potentially limited to Tier 2
regions or markets first, then
100% of users.

18. NEW FEATURE ANNOUNCEMENT
In-App Message
WHAT TO MEASURE:
As a Product Manager, I want to notify returning users of recently
launched new content and features via in-app messaging to maximize
engagement and retention.
• In-app message
- headline text
- body text
- background image
- call-to-action color, copy, placement
• In-app message on / off
Engage
Your app is never static. You want your users to know there is always
something new and exciting in store for them. Not everyone will
be interested in every new feature, so make sure you target your
announcement to the most relevant user segment.
13
Difficulty level:
Novice
Impact:
2/5
WHAT TO A/B TEST:
leanplum
Pro-Tip:
GOAL:

19. WHAT TO MEASURE:
WHO TO TARGET:
• In-app message:
- total views
- total clicks
- click-through rate
• New content or feature clicks
per user
• Time spent in app per daily
user
• DAU / MAU
• Day 1, 3, 7, 14, 30 retention
• ARPU, ARPPU
Returning users when the app
starts, but only once per user

20. OPTIMIZE FOR SOCIAL BEHAVIOR
As a Product Manager, I want to maximize social engagement
opportunities for my most viral users but limit them for my less social
users.
• Social sharing features on / off
Engage
You want to optimize virality as much as possible for highly social
users. However, non-social users will not respond to requests to
share/post achievements and should have limited exposure to social
requests in order to prevent them from churning.
15
Difficulty level:
Intermediate
Impact:
3/5
WHAT TO A/B TEST:
leanplum
Pro-Tip:
GOAL:

21. WHAT TO MEASURE:
WHO TO TARGET:
• Social sharing features:
- total views
- total clicks
- click-through rate
• Social vs. non-social user
funnel analysis
• Time spent in app per daily
user
• DAU/MAU
• Day 1, 3, 7, 14, 30 retention
Start with all users, then target
social / non-social personas

22. APP RATING SCREEN
As a Product Manager, I want to optimize for a 5 star rating by
requesting happy users to rate my app at the appropriate time.
• In-app message:
- headline text
- body text
- background image
- call-to-action color, copy,
placement
• In-app message on / off
Engage
High average app ratings are a driving force behind gaining more
organic and viral installs of your app. Low ratings can hurt the
adoption of your app. If you are running paid marketing campaigns,
low ratings can significantly increase your overall cost-per-
acquisition. To optimize for the highest rating, try showing the rating
screen at moments of high user engagement or flow.
17
Difficulty level:
Basic
Impact:
3/5
WHAT TO A/B TEST:
leanplum
Pro-Tip:
GOAL:

23. WHAT TO MEASURE:
WHO TO TARGET:
• Total 4 or 5 star ratings
• Time spent in app per daily
user
• DAU/MAU
• Day 1, 3, 7, 14, 30 retention
• ARPU, ARPPU
All users who pass an
emotional milestone event and
spent some time in the app.
Limit to once per user.

24. PRICE SENSITIVITY
As a Product Manager, I want to maximize total revenue by testing
price points and discounts for items in my inventory.
• Price point • Storefront
- look and feel
- text
- call-to-action button placement,
text, color, size
- background image
Monetize
Be sure to slice your data and zero in on specific user segments
when doing analysis. For example, you may find a meaningful lift in
purchase rates and revenue amongst males ages 18-24 whereas
there may be no noticeable effect on females aged 46-55 for the
same experiment.
19
Difficulty level:
Intermediate
Impact:
3/5
WHAT TO A/B TEST:
leanplum
Pro-Tip:
GOAL:

25. WHAT TO MEASURE:
WHO TO TARGET:
• Time spent in app per daily
user
• DAU/MAU
• Day 1, 3, 7, 14, 30 retention
• ARPU / ARPPU
All users. Consider targeting by
country.

26. contact@leanplum.com
+1 415-230-0397