An obsession with data, efficiency, and delivering incredible customer experiences are just a few things that the CNN Consumer Science and Software Engineering teams have in common. Simple A/B testing practices evolved into a culture of experimentation, sparking new development practices across the organization. Learn how they drive results across their entire platform from websites to mobile apps.
6. X6
EARLY DAYS
â˘2014 Redesign Beta
- Simple redirect test
â˘Headline testing POC
- Optimizely
- Visual Revenue
â˘Optimizing template monetization
â˘Homepage zone arrangements
â˘Ad Hoc experiments
WE DIDNâT KNOW MUCH
7. X7
AWARENESS TURNING POINT
1.30%
-29.26%
0.80%
-47.30%
2.99%
-0.97% 0.26%
1.02%
Var.1: Replace âTOP STORIESâ - Desktop Var.2: Replace âNews and Buzzâ- Desktop Var.3: Replace âMore from CNNâ - Desktop Mobile Web â Replace More from CNN
-20.38%
-1%
+34%
Overall Footer:
-61.43%
- 86%
+24%
+46%
8. X8
CHALLENGES WE FACED
â˘News cycle
- Audience mix changes with big news events
â˘Content matters and changes quickly
â˘Optimal layout depends on content
â˘Advertising business model centers on attention
- Proxy metrics to habit are hard to identify
â˘Every single interaction represents a micro-transaction
â˘No paid subscription means no big payback conversions
EASIER SAID THAN DONE
14. X14
ORGANIZATIONAL EVOLUTION
â˘Creation of Audience Development
⢠Focus on growth and habit
⢠Lifecycle management
â˘Consumer Science approach
- Road towards COE
â˘Productâs adoption of cross functional
team approach
â˘Build up of UX Research team
â˘Scaling the organization
CHANGING HOW TEAMS INTERACT
15. X15
CULTURAL EVOLUTION
⢠Discovery driven Product Development
⢠Right people, right problems
⢠Hiring the right team
⢠Audience segmentation awareness
⢠Cold, Casual, Core
⢠Audience metrics
⢠Building habit
⢠Measuring satisfaction
⢠Retention Proxy metrics
GROWTH OF PRODUCT AND ENGINEERING
16. X16
PROCESS
⢠Starting with what we know
⢠Data / observations / feedback
⢠Earlier involvement of
Consumer Science team
⢠Relentless focus on refining the
process
EMPHASIS ON TEST & LEARN
DATA
QUAL
TESTING
RELEAS
E
Validate
and repeat
QUANT
TESTING
âTest
against
Dataâ
âTest
against few
volunteersâ
âTest at
scaleâ
Fidelity & effort
17. X17
TECHNOLOGY
⢠Optimizely Web
⢠Optimizely Full Stack
⢠Improved event instrumentation
⢠List Vars on Adobe
⢠Custom metrics on Optimizely
⢠Program documentation and
measurement
⢠Snowflake, Databricks, Tableau & Looker
22. X22
5
APP TESTING
â˘Important to monitor adoption
rates before launching
experiment
â˘Launch to larger percentage of
app audience to reach
sufficient sample size faster
â˘Simultaneously test on mobile
web to help inform app testing
WHAT WEâVE LEARNED
23. X23
8
NEXT STEPS
â˘Continue optimizing iOS mobile app test
process
â˘Launch Android mobile app tests
â˘Evaluate OTT testing requirements
â˘Enable server-side testing on both web
and apps
24. X24
8
LONG TERM
â˘Dynamic experience testing (Server-side)
â˘Automated and faster analysis
â˘Complex editorial testing
â˘Cross platform experiments
â˘Continued evolution of user experience
25. X25
â˘Trustworthiness is fundamental
â˘Move fast and methodically
â˘Get involved from inception
â˘Leverage qualitative to refine
solutions before coding
â˘Define success before testing
â˘Communicate broadly
â˘Focus on process
KEY
LEARNINGS