NRF NXT 2019 July 23, 2019 Noah Freeman, CTO and Co-Founder, Social Fulcrum Andrew Krebs-Smith, CEO and Founder, Social Fulcrum

1. Noah Freeman, CTO & Co-Founder
Social Fulcrum
July 23, 2019
SESSION:
Incremental Lift Testing: Determine the
True Impact of Your Ad Campaigns
Deep Dive

2. 2
Lift Testing: An Example

3. What % is real? True ROI
30% 3x
100% 4x
50% 3.5x
Audience In-Platform ROI
Cart abandoners 10x
New customers 4x
Total 7x
True ROI is above target,
so increase total budgets
Raw data tells us to move
money to cart abandoners....
...but real data says
the opposite!
Imagine knowing the real impact of each ad...
3
$
$

4. Who Can Lift Test
4

5. 5
Lift testing: available for online & in-store purchases
Purchase location Online In-store
Time period Raw ROI Lift-adjusted ROI Raw ROI Lift-adjusted ROI
Pre-2015 ✓
2015-2016 ✓ ✓ ✓
2017-now ✓ ✓ ✓ ✓

6. Why We Lift Test
6

7. Attribution tells you if an ad preceded a purchase,
NOT if it caused the purchase
7
Saw ad
Made purchase
Did the pixel
give credit?
Did the ad cause
the purchase?
Monday 9am
Tuesday 8am
Yes
Yes!
Monday 9am
No purchase
No
No!
Monday 9am
8 days later
No
Yes - but 8 days
later
Monday 9am
Tuesday 8am
Yes
No - would have
bought anyway

8. Lift Based Attribution
Time Based
Attribution
Lift testing tells us what %
of the reported purchases
are caused by the ad
8
Lift testing lets us give correct, partial credit to purchases
across a huge time window

9. How We Lift Test
9

10. 20%
Control
StartingAudience
See real
ad
10 purchases per
1K audience
80%
Exposure
Incremental Lift =
7 Purchases
See
charity
ad
3 purchases per
1K audience
■ Before: 100% of credit for
7 days of purchases
■ After: 70% of credit for 90
days of purchases
■ Outcome: More accurate,
higher ROI, more scale
10
To run a lift test, you show real ads to your
exposure group, and fake ads to a control group

11. 11
To do a great test, you need to get a few things right
03
02
05
01
04
Design the test to give you
audience-level lift results
Make sure you can measure
ALL purchases, anywhere
Use good software, or big
control groups
Make sure your control group
is dark for 1-2 months pre-test
Find the right partner
• Most systems give only “whole-test” level incrementality
• Incrementality by audience is better
• Customers wander through your funnels in 90 days
• If you can’t measure all your checkouts, data suffers
• Control groups cost money and customers
• Accurate, small control groups are only possible with great third-party
software
• Customers who switch to control ads have “purchase momentum” from
previous real ads
• This momentum overstates control group purchases, and shows incorrect
low incrementality results
• Lift tests: easy to try, hard to do well
• You need to choose the right partner

12. 12
You can choose from many partner types
Ad platforms
Pro: cheapest choice,
simple setup
Con: perception of bias,
HUGE control groups, no
audience-level data
Example: Facebook’s
“Conversion Lift Test”
Big measure
vendors
Pro: You may already
have a contract in place
Con: Expensive, no
audience-level data, long
delays on reporting
Example: ODC
Measure
boutiques
Pro: Granular data, small
control groups
Con: None
Example: Social Fulcrum

13. What Results Do We Typically See?
13

14. 14
Lift tests show incrementality between 25% and 95%,
leading to real ROIs from 3-25x
Ad type
% of reported
revenues that are
real
Prospecting 90-95%
Retargeting
(Site Traffic)
25-35%
Retention/reactivation
(Past Customers)
35-45%
Advertiser Real ROI
5-10x
low scale 5-15x
high scale 3-5x
15-25x

15. ● Lift testing tells you the real impact of your digital ads
● Recent tests for retailers have shown tremendous ROI
● You need to choose the right partner to meet your needs and deliver
the details that are important to you
Now what?
15

16. Q & A
16

17. ● Do you ever doubt the results your platforms are reporting? Why?
● How have you dealt with this before?
● Have you run a lift test? How did it improve your decision-making?
● For those who have not run a lift test, how could lift testing help your
business?
Discussion questions
17