How Shutterstock Tests Everything and Stays Nimble
•
13 likes•6,475 views
This document discusses lean testing approaches at Shutterstock. It emphasizes that data is a competitive advantage and that testing everything allows the company to stay nimble. Key points include running hundreds to thousands of small tests, choosing one key metric to track, and maintaining a culture of experimentation where teams are empowered to make decentralized decisions based on data. Testing covers areas like pricing, search, and contributor experiences. The goal is a high test volume with around a 30% win rate to drive continuous growth.
Recommended
Recommended
More Related Content
What's hot
What's hot (7)
Viewers also liked
Viewers also liked (20)
Similar to How Shutterstock Tests Everything and Stays Nimble
Similar to How Shutterstock Tests Everything and Stays Nimble (20)
More from Lean Startup Co.
More from Lean Startup Co. (20)
Recently uploaded
Recently uploaded (20)
How Shutterstock Tests Everything and Stays Nimble
- 1. GROWTH Is simply a series of experiments Wyatt Jenkins – VP Product @wyatt_earp_ @shutterstock
- 2. Lean Testing How Shutterstock Tests Everything and Stays Nimble
- 3. Wyatt Jenkins VP, Product Development Shutterstock Computer science, Interaction Design, Product Strategy M.I.T. & University of Colorado Partner / Board Member of Beatport @wyatt_earp_
- 5. Data is your product The more Data you have, the better your product can be. The more usage you have, the faster you can achieve confidence on tests
- 6. Data is a competitive advantage 3 Downloads / second 1 TB per day
- 7. Tips & Types
- 8. A/B Testing
- 9. Smoke Tests
- 10. Multi-Variate
- 13. Teams • • • • • • Keep „em small – 3 ppl. for testing No hand-offs Let go of “Strategy / Process” Metrics driven – choose one Data highly accessible Decision making de-centralized
- 14. Qualitative –vs- Quantitative Idea Generation Validation Qualitative Quantitative
- 15. Testing Lessons
- 16. Results – Run significance on win / flat / lose Cell Keystone Redesign Content Types: Visual Content Types: Text-Only Browse % XX% XX% XX% Flat 0.02% 48.63% 96.10% Win 0.01% 38.88% 2.02% Lose 99.98% 12.49% 1.87%
- 17. Volume of Tests • Testing win $ amount unreliable metric • 100s to 1000s in future • 30% win rate
- 18. Testing Lessons • Leave design principles at the door • Test small changes • Don‟t forget performance • Hard to test strategy • Re-Test ideas.
- 19. Metrics Choose one, view many
- 20. Tools
- 21. Devent
- 22. Lil‟ Brother
- 23. Absinthe
- 24. DataViz
- 25. Test Examples
- 27. Browse Modal
- 28. Iteration & Big Swing
- 29. What else do we test? • • • • • Linguistics (Language detection) Relevance / Popular Localized experiences SEM landing pages Contributor sign up conversion
- 30. Search Grid
- 39. Challenges: Can‟t Optimize Big Swings * Josh Porter: http://bokardo.com/talks/metrics-driven-design/
- 40. Testing = Growth & Data = Competitive Advantage Wyatt Jenkins @wyatt_earp_ @shutterstock
Editor's Notes
- Growth = ExperimentationI’m going to make you better at testing – whether you are at a big company or a small, I’ve got some information about testing that can help.Testing & Experimentation is important – we take it very seriouslyHere’s a few things that can help
- Your product is made up of a bunch of interactions with real people who value the service you provide. Your product is data. The more of this data you have, the faster you achieve confidence on tests. ** Different companies in the audience will be of different sizes, but for Shutterstock in a highly commoditized industry, data is everything.
- 1TB per day3 downloads a second50
- Edimology of smoke tests – you plug in rado
- Why does this matter? Sometimes it’s fun to put things on a scale to show differences in opinion. Of course none of these brands would appreciate me doing this, but let’s check it out to prove a point. Problem to solve – Unclear lines between marketing and productMarketing design not yet good at conversionMarketing assets look different than websiteHow do we create a testing culture at Shutterstock
- Leadership team isn’t involved in tactics of testing.
- This comes into effect if a test is strategic. Because you might be testing strategy – than a flat to win is good. If you are running a revenue test, than only win is what we go on. Because there is a cost to switching to a new design.
- 1-2mSmall changes matter. Spot the difference, it’s worth 2million dollars.
- Got our ass kicked on this – lost potentially 3m Browse account modal- existing features don’t always work
- It’s important to test both iterative changes and big swings in some cases, or you may only make small changes.
- Everthing
- There’s an “ideas” list sitting on on our internal wiki with 200+ test ideas waiting to go.
- Define Highest paid person opinion
- Don’t’ overbuild or conduct full redesigns when all you need are optimizations. It’s too much work to take a swing like that.
- Here’s a slide from Joshua Porter – really bright guy Why do I drive this point home – because I’ve seen testing cultures where the teams literally can’t launch a new feature. They are so entrenched in the mindset of optimization that a new product is out of the question.