Show & Prove
Balancing Data-Driven & ‘Genius’
March 8th, 2010
Five minutes of fury.
Creative freedom is not deﬁned
by getting to pick where the
“sign up” button goes.
No genius required.
‘Genius design’ should focus
upon articulating the holistic
It should build the novel, the
elegant, and the compelling.
For all the rest, default to data.
What do you do with your data
How closely are you paying
attention to the metrics of
conversion, discovery and
“Metrics are people, too.”
Each of these data points tells a
story about the success or
failure of the experience you’ve
built for your audience.
Each of these data points can
be positively impacted by data-
driven design decisions.
What role can data play in
helping teams create better
experiences for users?
Data can empower user
experience teams to design
more creatively “risky” things
than they believed to be
They can implement the most
audacious design decisions,
while at the same time lowering
the risk of those decisions.
“Try insane sh*t.”
By only showing 5% of your
Split testing is your new BFF.
Split testing and user surveys
on live sites can move metrics
quickly and effectively, with
less expense than traditional
Behavioral data will become the
primary driver of tactical design
decisions within your
“Objectives & Key
Track the lifespan of the design
decisions you make.
Don’t go overboard.
This isn’t about 41 shades of
Data liberates us from the
illusion that we completely
understand the needs and
desires of our audience.
“But what should we test?”
Why not start with page
structure and messaging?
“You can’t test within the conﬁnes of a style
When we lack the resolve to
experiment, we doom ourselves
to a series of ongoing
Blow it up if you have to.
Iterate without sentimentality*
Iterate without sentimentality*
* especially when you’re successful.
Thank you for your attention.
Thank you for your attention.
You should follow me on Twitter:
Hello, I&#x2019;m Ryan Freitas from Second Verse.
Enough with the Oxbridge pleasantries. This is a 5 minute minute madness which means...
I&#x2019;ve got to get to my point rather quickly. When I worked with Peter at AP his usual guidance when writing a presentation to was to be sure there was a recognizable point to what you were trying to say. He called it the &#x201C;so what&#x201D;, and it was one of the most helpful lessons he ever taught me. So what&#x2019;s the &#x201C;so what&#x201D;?
Arguments over tactical design decisions take up too much time for all experience design teams. Minute decisions around messaging, structure and even color derail our efforts from ensuring what should be our priority, creating the best possible experience for our audience.
This may be the most successful homepage on the web. Two actionable areas, sign in or register. Minimal messaging... tactical decisions focused on GETTING PEOPLE IN THE DOOR. The real heavy lifting, the &#x2018;genius design&#x2019; is the experience once you&#x2019;re through the gates...
Which is really what its all about. Our time and energy should not be spent fighting out tactical battles over the inventory and positioning of elements on the home page.
We should be inventing and refining, focusing on decisions that create deeper, more impressive levels of interaction with our audience.
I&#x2019;m hoping that you&#x2019;re at least getting occasional reports on visits, views and your bounce-rate. There might even be somebody who&#x2019;s compelling you to increase one while reducing another.
But there&#x2019;s a whole wealth of information you can gain access to if you know where to look.
This is Eric Reis, author, speaker, and advocate of the lean startup movement. And he&#x2019;s got a very good point.
It&#x2019;s all in there. Yes of course you can do research and ask individual users and groups about their experience. But the numbers tell their own story, and while it may not be as deep or nuanced, it is ACTIONABLE, and potentially more valuable than you understand.
You can increase conversion by experimenting with your calls to action. In a genius design methodology, we come up with many possible messages and roll out ONE. In data driven design, for tactical choices WE ROLL OUT ALL OF THEM.
If you&#x2019;re not already convinced of the value of data, you might be asking this question.
The best answer is from Avinash Kaushik of Google. He&#x2019;s their Analytics Evangelist, and one of the smartest people talking about the roll of data in design right now. (you should follow him on twitter)
What if putting things out in to the world to get REAL reactions from users was trivial? What if it didn&#x2019;t require writing a protocol, recruitment, or any other element of research?
This is Hiten Shah, founder of KISSMetrics and another lean startup advocate. He&#x2019;s also brilliant. (you should follow him on twitter)
The technical requirements to do multivariate or split testing on your site are minimal - there&#x2019;s server and client side solutions, and both can ensure that only a sliver of your audience sees any given tactical change. Just enough to get feedback on the failure or success of that change.
Split testing opens the way to experiment on our products in the real world and track the impact.
Testing yields bigger, more actionable data in less time for prep and deployment than any comparable method, including remote usability.
Why am I telling you this? Because if you don&#x2019;t believe it already, you need to soon. You are the Highest Paid Person&#x2019;s Opinion in many of the tactical design decisions that are getting made. And we all know that HiPPOs usually get their way. I&#x2019;d like to encourage you to be advocates for data as the determiner of how to make tactical design decisions.
... because I think that if you don&#x2019;t, the next year or so is going to get very, very difficult for you and your team. This isn&#x2019;t a prediction. This is how business is getting done.
So focus. Find the areas of the site where you can move the dial with minor changes. See what impact you can have, and be sure to measure it appropriately.
Many of you already have these, but they are as outdated as the pageview model for advertising that&#x2019;s being thrown out as we speak. Focus on conversion and engagement, and align them into your OKRs.
I can&#x2019;t stress this enough. If you decide to go with only one field for passwords, and don&#x2019;t make users enter them twice, you might drive down the time to complete a registration form. You&#x2019;ll never know if it was the right decision until you pay attention to the fact that your &#x201C;I forgot my password&#x201D; clicks just doubled since the change. Experiences bleed over.
This is simple. Don&#x2019;t try to test everything at once. Don&#x2019;t push your designers out of the product loop by testing without their input. And most important, don&#x2019;t overtest the same issue into the realm of ridicule.
For most of you, 4 shades will probably do.
Complete information is rare in our line of work. Even with the best research on our audience, we cannot assume we know everything they need. Allow their ACTUAL BEHAVIOR to be your guide, and refine the experience for them based on their interaction with your product.
If you&#x2019;re looking for a place to get started, pick an underperforming landing page and see what can be accomplished with just a few changes to how the page looks, and what it says. Come up with 3 alternatives, and give them to 5% of your audience each.
Remember that sometimes the solution will lie outside of the realm of the constraints you set up for yourself. Try not to go too nuts. It&#x2019;s a balancing act.
Past design decisions are a sunk cost. Move past them as quickly as you can.
You should be more afraid of what you don&#x2019;t know than what might happen if you make the wrong decision. Which leads me to my final piece of advice...
Unless you&#x2019;ve got a perfect conversion rate, you can always improve. The best design teams constantly iterate, because someone out there is building something without all the compromises that created the current solution.