Holiday Catalog Testing Strategies

1. Holiday Catalog Testing Catalog University Pub 7/20/2016

2. Holiday Catalog Testing • Why testing this year matters next year • Best practices for test set up and results assessment that empower you with dependable data for predicting future performance. • Test variables that provide actionable learnings most likely to most significantly improve performance

6. $1.0M $1.2M $1.5M $1.8M $1.8M 2.0M 2.0M 2.5M 3.0M 3.0M $4.0M $6.0M $6.3M $7.5M $7.2M Spend Circulation Revenue Year 1 Actual Year 2 Actual Year 3 Actual Year 4 Fcst Year 4 Actual Test Winner More Costly Device Offer Lists Lists Winner Rollout Status Quo More Costly Device Better Offer Expanded Circ Spend $1M $1.2M $1.5M $1.8M $1.8M CPP $0.50 $0.60 $0.60 $0.60 $0.60 Circ 2M 2M 2.5M 3M 3M Incr $/Circ $2.00 $3.00 $2.50 $2.50 $2.40 Incr $ $4.0M $6.0M $6.3M $7.5M $7.2M Incr ROAS 4.0 5.0 4.2 4.2 4.0 $ Chg YOY 50% 4% 20% 15% 20% 0% 50% 25% 25% 4% 20% 20% 20% 20% 20% 15%Change YOY  ChangevsYear1 80% 80% 50% Thisiswherewewanttobe… Assumes a margin and ROI target that is acceptable for this fictitious business. Conservative long-term vision… what any manager would like to see

9. $1.0M $1.2M $1.5M $1.8M $1.8M 2.0M 2.0M 2.5M 3.0M 3.0M $4.0M $6.0M $6.3M $7.5M $7.2M Spend Circulation Revenue Year 1 Actual Year 2 Actual Year 3 Actual Year 4 Fcst Year 4 Actual Test Winner More Costly Device Offer Lists Lists Lists Winner Rollout Status Quo More Costly Device Better Offer Expanded Circ Expanded Circ Spend $1M $1.2M $1.5M $1.8M $1.8M CPP $0.50 $0.60 $0.60 $0.60 $0.60 Circ 2M 2M 2.5M 3M 3M Incr $/Circ $2.00 $3.00 $2.50 $2.50 $2.40 Incr $ $4.0M $6.0M $6.3M $7.5M $7.2M Incr ROAS 4.0 5.0 4.2 4.2 4.0 $ Chg YOY 50% 4% 20% 15% 20% 0% 50% 25% 25% 4% 20% 20% 20% 20% 20% 15%Change YOY  ChangevsYear1 80% 80% 50% Thisiswherewewanttobe… Assumes a margin and ROI target that is acceptable for this fictitious business. Conservative long-term vision… what any manager would like to see

10. Conservative long-term vision… what any manager would like to see $1.0M $1.2M $1.5M $1.8M $1.8M 2.0M 2.0M 2.5M 3.0M 3.0M $4.0M $6.0M $6.3M $7.5M $7.2M Spend Circulation Revenue Year 1 Actual Year 2 Actual Year 3 Actual Year 4 Fcst Year 4 Actual Test Winner More Costly Device Offer Lists Lists Winner Rollout Status Quo More Costly Device Better Offer Expanded Circ Spend $1M $1.2M $1.5M $1.8M $1.8M CPP $0.50 $0.60 $0.60 $0.60 $0.60 Circ 2M 2M 2.5M 3M 3M Incr $/Circ $2.00 $3.00 $2.50 $2.50 $2.40 Incr $ $4.0M $6.0M $6.3M $7.5M $7.2M Incr ROAS 4.0 5.0 4.2 4.2 4.0 $ Chg YOY 50% 4% 20% 15% 20% 0% 50% 25% 25% 4% 20% 20% 20% 20% 20% 15%Change YOY  ChangevsYear1 80% 80% 50% Thisiswherewewanttobe… Assumes a margin and ROI target that is acceptable for this fictitious business.

11. Execute roll-outs and tests each year with a view across at least 3 holiday seasons… • THIS year’s ROLL-OUTS are informed by LAST year’s test WINNERS. – That’s just hard work – execution. Has to be done perfectly. • THIS year’s TESTS are selected informed by LAST year’s test results (unless you’re testing for the first time) – plus totally new ideas – all directed towards maximizing performance NEXT year. – Testing is the vital – and fun – part if we are to achieve long-term growth, and it can easily be overlooked since it distracts from the execution and performance of THIS year’s campaign. – Don’t let testing get lost! Make testing a priority! – Plus – it’s FUN! It’s THE fun! This Year Next YearLast Year

17. Simple Campaign Structure Mailed Circulation Audience Device Offer IHD Expiration Date Mail Control 1 25,000 Catalog Save $10 14-Nov 31-Dec Mail Control 2 25,000 Catalog Save $10 14-Nov 31-Dec Test A - Costlier Device 25,000 Catalog + Bells/Whistles Save $10 14-Nov 31-Dec Test B – Better Offer 25,000 Catalog Save $30 14-Nov 31-Dec Test C – Later IHD 25,000 Catalog Save $10 1-Dec 31-Dec Test D - Cheaper Device 25,000 Plastic Postcard Save $10 14-Nov 31-Dec Test E – Earlier Exp. Date 25,000 Catalog Save $10 14-Nov 15-Dec Rollout Control 1,825,000 Catalog Save $10 14-Nov 31-Dec Total Circulation Mailed 2,000,000 - - - - - - - - - - - - No Mail Control Audience - - - - - - Start End No Mail Control 1 25,000 - - - - - - 14-Nov 31-Dec No Mail Control 2 25,000 - - - - - - 14-Nov 31-Dec Total No Mail Control 50,000 - - - - - - - - - - - - Audience Audience - - - - - - - - - - - - Total Required 2,050,000 - - - - - - - - - - - -

26. Building Testpools • For our example: – Take the 2.05M Households – Split into 10 cohorts, n-thing off for equal balance, so that if all cohorts were treated equally in the mail they should all behave the same • We need 9 cohorts of 25k and 1 rollout cohort of 1.825M – For house cohorts, check that they’re balanced by checking each cohort’s transaction history – they should each have the same 0-6 month revenue per household – For prospect cohorts, test that they’re balanced by checking if zips are equally represented

30. Example of n-thing We have an audience of 100k We want 5 testpools of 10k each • No Mail Control • Mail Control • Test 1 of 3 • Test 2 of 3 • Test 3 of 3 We want the remainder, 50k, for the rollout 50% 10.0% 10.0% 10.0% 10.0% 10.0% Household Rollout No Mail Control Mail Control Test 1 of 3 Test 2 of 3 Test 3 of 3 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 1 10 1 11 1 12 1 13 1 14 1 15 1 16 1 17 1 18 1 19 1 20 1 Etc….

31. Test Pool Size: What’s Optimal? • Best Practice – Work with an analyst who is fully informed on your campaign structure and your testing objectives to determine optimal test pool size • Second-Best Practice – 100 orders minimum “rule of thumb” • Have your test pool size large enough that you are confident that you will definitely get 100 orders or more • Assume prospect response will be just 20-30% of house response • If possible, shoot for test pools sizes that will deliver much more than 100 orders – particularly if the results of your tests will inform decisions on a grand scale – Example: we don’t want make a change to a 20M rollout campaign that would significantly impact costs or revenue based on a test conducted with a testpool size of just 5,000

34. No Mail Control vs. Mail Control Results Audience Fully Loaded Matchback Fully Loaded Matchback/Household Incremental $/Household Incremental Revenue No Mail Control 1 25,000 298,750$ 11.95$ No Mail Control 2 25,000 296,250$ 11.85$ Total No Mail Control 50,000 595,000$ 11.90$ Mail Control 1 25,000 373,750$ 14.95$ 3.00$ 75,000$ Mail Control 2 25,000 376,250$ 15.05$ 3.20$ 80,000$ Total Mail Control 50,000 750,000$ 15.00$ 3.10$ 155,000$ Rollout 1,825,000 27,375,000$ 15.00$ 3.10$ 5,657,500$ Total Mail Controls/Rollout 1,875,000 28,125,000$ 15.00$ 3.10$ 5,812,500$ $15.00 - $11.90 = $3.10 – Matchback Incremental Lift Assessment Model (above) • Mail Control Matchback $ or Orders minus No Mail Control Matchback $ or Orders (you can also add in Pass-Along Promo Code Redemption $ or Orders) = Incremental Result

38. Tests vs. Mail Control (Rollout) Results • Test A delivered $2.50 more in incremental revenue vs. Mail Control (Rollout). • Next year, when Test A is rolled out to the 2M circulation, it will deliver $5M more in incremental revenue vs. this year (but cut that back 15%). • Test B Offer improved results by $0.80 • Test C Later In-Home Date was a LOSER ($2.50) • Test D Cheaper Device was a LOSER ($1.10) • Test E Earlier Expiration Date had NO IMPACT Audience Fully Loaded Matchback Fully Loaded Matchback/ Household Incremental $/Household Incremental vs. Incr Mail Control Total No Mail Control 50,000 595,000$ 11.90$ Total Mail Control 50,000 750,000$ 15.00$ 3.10$ Test A - Costlier Device 25,000 437,500$ 17.50$ 5.60$ 2.50$ Test B - Offer 25,000 395,000$ 15.80$ 3.90$ 0.80$ Test C - In-Home Date 25,000 312,500$ 12.50$ 0.60$ (2.50)$ Test D - Cheaper Device 25,000 347,500$ 13.90$ 2.00$ (1.10)$ Test E - Expiration Date 25,000 375,000$ 15.00$ 3.10$ -$ $5.60 - $3.10 = $2.50

43. Tests vs. Mail Control (Rollout) Results • Test A delivered $2.50 more in incremental revenue vs. Mail Control (Rollout). • Next year, when Test A is rolled out to the 2M circulation, it will deliver $5M more in incremental revenue vs. this year (but cut that back 5-15%). • Test B Offer improved results by $0.80 • Test C Later In-Home Date was a LOSER ($2.50) • Test D Cheaper Device was a LOSER ($1.10) • Test E Earlier Expiration Date had NO IMPACT Audience Fully Loaded Matchback Fully Loaded Matchback/ Household Incremental $/Household Incremental vs. Incr Mail Control Total No Mail Control 50,000 595,000$ 11.90$ Total Mail Control 50,000 750,000$ 15.00$ 3.10$ Test A - Costlier Device 25,000 437,500$ 17.50$ 5.60$ 2.50$ Test B - Offer 25,000 395,000$ 15.80$ 3.90$ 0.80$ Test C - In-Home Date 25,000 312,500$ 12.50$ 0.60$ (2.50)$ Test D - Cheaper Device 25,000 347,500$ 13.90$ 2.00$ (1.10)$ Test E - Expiration Date 25,000 375,000$ 15.00$ 3.10$ -$ $5.60 - $3.10 = $2.50

48. Results Assessment Models • 2 primary ways to measure results – Matchback Incremental Lift Model • Match back your Mail and No Mail Controls • Mail minus No Mail Control + Pass-Alongs = Incremental Revenue – Promotion Code Redemptions • Match Back Redemptions • Measure Pass-Alongs – promo code usage by customers not mailed – somehow they received the coupon • Total Redemptions • Redemption Entries (entered into account but not used) vs. Usage

49. Results Assessment Models • 2 primary ways to measure results – Matchback Incremental Lift Model • Match back your Mail and No Mail Controls • Mail minus No Mail Control + Pass-Alongs = Incremental Revenue – Promotion Code Redemptions • Match Back Redemptions • Measure Pass-Alongs – promo code usage by customers not mailed – somehow they received the coupon • Total Redemptions • Redemption Entries (entered into account but not used) vs. Usage

50. Results Assessment Models • 2 primary ways to measure results – Matchback Incremental Lift Model (Best) • Match back your Mail and No Mail Controls • Mail minus No Mail Control + Pass-Alongs = Incremental Revenue – Promotion Code Redemptions • Match Back Redemptions • Measure Pass-Alongs – promo code usage by customers not mailed – somehow they received the coupon • Total Redemptions • Redemption Entries (entered into account but not used) vs. Usage

51. What results can show and not show • Matchbacks Incremental Lift Model only tells you incremental number of responders and revenue – we don’t know WHO was the incremental responder – And your catalog drives not just incremental responders, but incremental revenue from those responders who were going to order anyway whether they received the catalog or not – they spent more than they might have if they hadn’t received it – You can look at incremental revenue by product category • Promo Code Redemptions tell you who engaged – But NOT who would have engaged anyway if you hadn’t mailed at all

55. How many tests in holiday season? • Conduct as many tests as practical within bounds of circ size, budget, keeping it simple, and prioritizing tests likely to improve performance the most • Some tests will be winners, most will be losers vs your roll-out control – You may lose some short-term revenue due to testing but that’s the cost of increasing revenue next year. • One chance per year – don’t miss it – so certainly don’t under-test. • More tests increase chance you find something that will move the meter and have a meaningful impact on next year’s results

59. What to test? Here are come categories: – Lists – Devices – Promotional Offers – Merchandise – Timing – Expiration Dates – Covers – Messaging – Design – Color – Frequency – Digital Integration (Catalog + Email, Catalog + Display)

62. What matters most? Factors most likely to increase performance • In my experience, what will move the meter the most is the interaction between: (Assuming timing & merch mix are adequate.) – Lists • Relevance • Breadth • Depth – Device “in-the-mail” • You may call it a “Form Factor”? • Catalog / Pages / Merch / Bells & Whistles / Wraps / Wallet Cards / Blow-Ins / Inserts / Envelopes / Things to Peel / Greeting Card / Postcard – Promotional Offer (Incentive) • Save! Free! Survey! Exclusive Access! • Be uncomfortably generous

66. Test: House Lists • Segmentation RFMP – Recency – Frequency – Monetary – Product – Seasonal – Inquiries / Non-Buyers • Reactivation Scoring (lapsed “look-alike” modeling) • Match your e-mail only customers to postal addresses and mail them • Match your anonymous site visitors to postal addresses and mail them

70. Test: Prospect Lists • Test from many list categories: – Vertical Lists (ex: Pottery Barn Buyers) – Compiled Lists (ex: Kindergarten Teachers) – Look-Alike Modeled Lists (co-ops) – Subscriber Lists (magazines) – Association Lists (conferences) – Business Partner Lists • Do you sell to same target, but not competitive products? • Use each other’s lists – Competitor List Swaps • Trust, good relations, and a common belief that both parties benefit (that both sell more) by mailing each other’s lists are required. This is a rare arrangement, but it works for those who partner to make it possible.

73. Use Promo Codes AND Matchbacks for list results analysis • Promo Codes are very helpful for LIST analysis – NMCs (No Mail Controls) may be too small at list segment level to meaningfully measure the incremental revenue – You may not want to pay for the prospect NMC names for every list – particularly at large cohort sizes that provide statistically significant data – You will at least get a read on the promo code performance of list segments relative to each other – If your match back reports end up unreliable for some reason, promo code redemption usage is a back-up path to results analysis (though not preferrable vs. match back and incremental lift model). – In my experience, the incremental performance of prospect lists lands at 90-110% of the promo code redemptions – not too far off (house lists not so) • Therefore, assign a separate promo code group for each list test, each segment depth

74. Test: Device • Test devices that make you feel uncomfortably radical – don’t just be a “catalog” – They stand out in the mailbox, violate, and engage, triggering a pause for further consideration, an interception on the way to the trash can – What sparkles? Catches the eye? Feeling of touch? • Thick plastic postcards – What conveys value? • Wallet Cards (they look like credit or gift cards) – What conveys a trigger to take action? • Sticky Notes (anything that peels) • It’s only a test! • Remind everyone at your company that only a small portion of the mailing will receive this radial device – it won’t “destroy the brand” if mailed once to a small cohort OBJECTIVE: Intercept the trash can and get recipient from the direct mailer / catalog to your web site. Everything about your device should be serving this objective!

75. Costlier Devices can PAY out BIG time! • Costlier devices can have significant impact in the mailbox. – You’re not just competing with your compeitors in your sector, you’re competing with whatever else arrives in the recipient’s mailbox that day – are you standing out? • If $0.10 extra spend per piece in the mail for some violating bells & whistles engagement brings in $2.50 more revenue, you have a ROAS of 25 on that incremental $0.10 spend – an insanely high ROAS. • A blow-in card, with an extra ink-jetted promo code promoting a featured product, costs about $0.01. If that lifts your result $0.05 you have a ROAS of 5.

78. Test: Promotional Offers • Here’s what I am seeing from my experience: – Save 10-30% isn’t meaningfully increasing response – Save $ offers seem to perform better than Save % offers – BOGO where margins make it possible tends to be powerful – Free Shipping isn’t meaningfully increasing response – Free Gifts aren’t meaningfully increasing response – Free Products – if you can give products away, do! – Multiple offers (promo codes sprinkled throughout book, product targeted, on top of an overall order offer) tends to increase response – Surveys – do you have a credible reason to ask your customers or prospects to give you their opinion? Do you have an incentive for them to do it without any obligation to purchase anything at all for that incentive? These can be very powerful, depending on the sector and target market. – Invitations – premium offers for special events or access to special information. Sneak peeks. These are effective with folks who care about access, information and membership than about price and promotions.

87. Test: Merchandise • More or less merch? • Breadth? Depth? – Pack more merch into your existing catalog size – Cut pages – can you spend less? – Add in a centerfold stitched-in insert of extra pages, a “gift guide” booklet – Try a postcard instead • Plastic postcards perform 2-3x better than card-stock postcards – Try a greeting card instead – very little merch • Invitation • Wallet Card • Works great with existing customers

88. Test: Promotional Offer Expiration Dates • I’ve seen both short-term expiration dates and extended expiration dates perform well • Test to see what works best for your business • Don’t test small differences – test big differences: – Nov 30 • Careful! Don’t run the risk that your offer expires before it lands in mailboxes! – Dec15 – Dec20 – Dec31 – Jan15

89. Test: Timing – when you’re landing in mailboxes • Holiday catalogs are hitting homes in October these days…what works best for your catalog? • Don’t guess! Test! • Test a variety of dates – Pre-Halloween – Post-Halloween – Pre-Thanksgiving – Land in mailbox Thanksgiving weekend – Post-Thanksgiving

90. Test: Covers • Covers are difficult to test because a control won’t help you; you can’t repeat the same cover twice (or the same cover next year) • Cover wraps, on the other hand, can be repeated and serve as a roll-out control to be beat each year • If you’re going to test cover imagery, try: – “product” vs. “aspirational” • Bottles of Wine VS. • Beautiful people laughing and drinking wine at a Vineyard – If “aspirational” works best, then always use “aspirational” covers of people partying at a Vineyard until you find some other approach that works better – Variations on the same theme: every cover is of friends on a dramatic outdoor excursion

95. Test: Messaging • In my experience, messaging relevant to a targeted audience can increase results a bit – 5%. – Never ordered? – Ordered just once? – Previous products purchased indicates they qualify for relevant upsell?

96. Test: Colors • Color in a vacuum has no impact except to the degree that it: – Helps or impairs the conveyance to the recipient of your intended message – Helps recipients recognize your company, such as Shutterfly, Netflix

100. Test: Digital Integration • Email Test – Cohort A: Send Catalog Only – Cohort B: Send Email Only – Cohort C: Send Catalog + Email • Display – Cohort A: Send Catalog Only – Cohort B: Serve Display Only – Cohort C: Send Catalog + Serve Display • In my experience this integration isn’t a major lever, is a lot of work and cost with little payout – Difficult to read results, so much noise in those channels already – Most catalog recipient households are probably getting hit with emails and display ads anyway. – Not clear this integration has any impact, but it does cost!

101. Test: Digital Integration • Email Test – Cohort A: Send Catalog Only – Cohort B: Send Email Only – Cohort C: Send Catalog + Email • Display – Cohort A: Send Catalog Only – Cohort B: Serve Display Only – Cohort C: Send Catalog + Serve Display • In my experience this integration isn’t a major lever, is a lot of work and cost with little payout – Difficult to read results, so much noise in those channels already – Most catalog recipient households are probably getting hit with emails and display ads anyway. – Not clear this integration has any impact, but it does cost!

102. Test: Digital Integration • Email Test – Cohort A: Send Catalog Only – Cohort B: Send Email Only – Cohort C: Send Catalog + Email • Display – Cohort A: Send Catalog Only – Cohort B: Serve Display Only – Cohort C: Send Catalog + Serve Display • In my experience this integration isn’t a major lever, is a lot of work and cost with little payout – Difficult to read results, so much noise in those channels already – Most catalog recipient households are probably getting hit with emails and display ads anyway. – Not clear this integration has any impact, but it does cost! • Don’t conclude from my experience – try it yourself…

103. Test: Digital Integration • Email Test – Cohort A: Send Catalog Only – Cohort B: Send Email Only – Cohort C: Send Catalog + Email • Display – Cohort A: Send Catalog Only – Cohort B: Serve Display Only – Cohort C: Send Catalog + Serve Display • In my experience this integration isn’t a major lever, is a lot of work and cost with little payout – Difficult to read results, so much noise in those channels already – Most catalog recipient households are probably getting hit with emails and display ads anyway. – Not clear this integration has any impact, but it does cost! • Don’t conclude from my experience – try it yourself…

104. Test: Frequency – Test A • Single-Touch Mail Control Touch 1 of 1 (NOV15) vs. – Test B – The total combined revenue of… • 2-Touch Mail Touch 1 of 2 (NOV15) • 2-Touch Mail Touch 2 of 2 (DEC1) • Trigger Direct Mail – 24-hour follow-up to account activity (log-in, placed order, new customer)

105. Test: Frequency – Test A • Single-Touch Mail Control Touch 1 of 1 (NOV15) vs. – Test B – The total combined revenue of… • 2-Touch Mail Touch 1 of 2 (NOV15) • 2-Touch Mail Touch 2 of 2 (DEC1) • Trigger Direct Mail – 24-hour follow-up to account activity (log-in, placed order, new customer)

106. Take-aways • If you’re not testing, take some steps now to test something meaningful this year – it can be simple • If you can test: – be sure to set up tests using best practices so that the results are actionable – Prioritize • Lists • Promotional Offers • Devices • Don’t believe what anyone else says about what is working – test it yourself before taking action. – Believe in the best practice methods for testing. – Believe in what you have learned yourself by testing using best practices!

109. Thanks! Contact Info: Christopher Werler christopher@amherstdirect.com 781-962-7902

Holiday Catalog Testing Strategies

Recommended

Recommended

More Related Content

Similar to Holiday Catalog Testing Strategies

Similar to Holiday Catalog Testing Strategies (20)

More from Sarah Fletcher

More from Sarah Fletcher (11)

Recently uploaded

Recently uploaded (20)

Holiday Catalog Testing Strategies