October 2013 - Public legal education (PLE) is increasingly delivered online. This webinar will look at how to leverage a number of free or low-cost online tools (including Google Analytics and iPerceptions surveys) to acquire data to measure your impact and align with your key performance indicators or KPIs. Other tools that will be discussed include online user testing tools and what metrics matter when it comes social media evaluation.

1. Using online tools to
help us assess our PLE work
Presented by Fiona MacCool,
Your Legal Rights Project Manager
CLEO (Community Legal Education
Ontario / Éducation juridique
communautaire Ontario)
October, 2013

2. What we’ll cover:
•
•
•
•
•
•
Introduction to Key Performance Indicators
Google Analytics (The Basics!)
Online User Testing
User Experience Surveys
Social Media Evaluation
Questions and Discussion

3. Let’s think about evaluation not as a single
measurement in time nor as a vast collection of
statistics you collect for funders. Evaluation,
especially when it comes to online tools or
websites, should be an ongoing process that
ensures that continuous improvement is
happening in ways that are measureable,
evidence-based, and that contribute to the
strategic goals of the organization.
(EASIER SAID THAN DONE!)

4. What are Key Performance
Indicators?
Performance measures (or KPIs the terms are
more often used interchangeably these days)
are objective evidence of the degree to which
a performance result is happening over time.
They are feedback about your progress in
improving aspects of your organization.
(Source: http://staceybarr.com)

5. Why use KPIs?
• They allow us to set targets and improve our
site/projects over time
• They allow us to take action to improve
• They let us measure the viability of time consuming
activities – should we keep doing what we are doing?
• They help us answer the question - What is the
point? Why are we doing this? Are we delivering on
our promises?

6. Key Learning(s)…
• This is not something you should be left to do on
your own as the “web person” or project manager.
You should brainstorm every element of this process
with others, including board and management
• There is no one magic tool that can do this. You
need to come at with combination of approaches and
modify them regularly.
• We need to share our strategies and techniques
with one another as we are breaking new ground
here.

7. The Process
1. Why - Look at your Vision Statement (for the org) – how can
your website help you deliver on this objective (along with
other channels)
2. How - Strategic Plan, or Services Review Plans (one way our
sector has advantages over the private sector) – Integrate key
initiatives and where the org is heading to meet core goals and
objectives
3. Outcomes/Results - “Defining Success” – Looking at why and
how – what outcomes or results will create the collective “fist
bump” – what does victory/success look like for the org and for
the project?

8. What makes a good KPI?
“We know we will be successful when this RESULT is achieved”

9. Types of Success Evidence
1.
2.
3.
4.
5.
Financial – online purchases, tickets sold, fundraising $ - Success
Evidence – “Met fundraising goals”
Behavioural – Are people “doing” what you hoped they would
(differently than without your site/project) - Success Evidence –
“Increase in registrations for and repeat participation/viewings of
webinars over time”
Experiential – Is the process better, are they able to complete their
task more easily? – Success Evidence: Time required to submit an
online request is reduced, client intake interview less time consuming
Attitudinal – Getting a new perspective on how things are done or
what steps to take – Evidence of Success “Organizations perceive you
to be collaborative and responsive to feedback”
Technical – slow site – needs to be faster - Evidence of Success –
“Average page load time reduced” – Fewer “page not found” errors

10. Examples of KPI Metrics
• % of content greater than 1 year old – (insert
appropriate time frame based on sector/topic)
• % of content voted up or down or rated
• % Average # of edits prior to publish
• % Average time to respond to email
questions/enquiries
• % of users able to complete their task
• % of users reporting satisfaction and/or that they
would recommend the site to others over time

11. Key learning(s)…
• It is worth the time making a few customizations or
configurations to your website or online project early
on to make sure are actually able to collect what you
plan to measure . Google Analytics is great, but it’s
probably not enough and nor is a simple user survey.
Ask your developers about this early on if you can.
• Raw numbers of stats are less helpful and not easily
actionable. It is better to establish a way to measure
percentages and shifts in percentage patterns.
• Try to measure monthly averages or annual
benchmarks – comparisons over time are key.

12. How to get started…
•
•
•
•
•
Think about a project in your organization…
Think of what the goal of the project is?
Think of ONE result of that you would like to see
for the project.
Think of one piece of evidence you might use to
prove whether you have achieved this goal.
Draft that into a KPI (Key Performance Indicator)

13. Website Sample
Project
Goal
Result
Measurement
CLEONet.ca –a PLEI
site for service
providers was rebranded and
launched as Your
Legal Rights, a site
aimed at the general
public in Ontario
Expand site
audience from
service providers to
also reach “General
Public”
Within 1 year of the
launch of YLR, we
should see a 30 – 50
% increase in visits
over the last year of
CLEONet.ca
- Google Analytics
Visits and Page
Views
Within 1 year of
launch we should
see an increase in
the % of users who
report using the site
to find information
for a friend or family
member rather than
in their work or to
help a client.
- User survey asking
question of
audience/purpose
on CLEONet and
then on YLR for key
periods - % shifts in
breakdown of
purpose over time.

14. Project Sample
Project
Goal
Result
Measurement
PLEI Webinars for
service providers in
Ontario
Increase the reach
of PLEI from local inperson workshops
to wider online
availability through
YLR
Within 1 year of
project launch we
should deliver 10
-12 webinars
reaching over 100
people live and over
1000 watching the
archived versions.
- # of webinars
conducted and
archived to the site
- # of live
participants
- # of views of
archived webinars
- Each of the metrics
looked at as % from
year to year
Within 1 year of
project launch,
identify and partner
with 4-6
organizations to
develop and deliver
webinars
- # of organizations
presenting webinars
with CLEO
Surveys on whether
webinars met user’s
learning needs

15. Project Sample
Project
Goal
Result
Measurement
At Your Legal Rights
we want to make
sure we are
responding to
outreach emails in a
timely and helpful
manner
YLR should be an
authoritative PLEI
site for Ontario and
must therefore
provide information
that is relevant.
% User survey
respondents that
agree/strongly
agree that YLR’s
response was
relevant to their
inquiry
Follow up survey sent
to people who
received a response
from YLR’s outreach
team. (Monthly)
Count of survey
respondents that
“Agree” or “Strongly
Agree” that YLR’s
response to their
inquiry was relevant
Count of responses
to the question vs.
count of outreach
emails sent
Trends examined
over time

16. Benchmarking and Baselines
Benchmarking vs. Baseline
•
•
Benchmarking – comparing one’s business process against competitors
(industry standards – difficult in this sector to know)
Baseline – initial known value which is used for comparison with later data
(don’t forget to establish/retain stats before re-launching or redesigning
your site)
Measurement – planning development processes testing
•
•
•
Baseline from launch – For example: last 3 months of CLEONet to first
three months of Yourlegalrights.ca, then year to year etc.
Numbers of likes on your FB page or followers on Twitter over time
Target your baseline to your audience (service providers in Ontario,
general public) – Note that Google Analytics does Country and City only

17. Featured Tools
What are users doing on your site and how?
• Google Analytics – Tracks use and can follow “goals” or
“events” set by you based on KPIs
• Heat maps – Crazyegg.com and Clicktale as well as some
online usability testing tools show you who is clicking what,
how quickly and how often
• Clicktale – records user sessions on web site – free account
for 45 recorded sessions a month – shows you what/where
users clicked – should help you see recurring problem areas
• Google Adwords – Great tool to track keywords and compare
your page rankings and click-rates

18. Featured Tools
Online User Testing
• Usability testing – You can pay third party sites/services to set
up task-based tests your site with unknown people based on
demographics you select, or you can do it yourself with tools
like VerifyApp based on invitations to volunteers (cheaper
than running them yourselves, especially if you serve a large
Geographic area)
• A/B testing – Google’s A/B testing tool lets you split traffic in
half so one half of users see one site/page and the others see
a second version – reports will tell you that the new site
reaches more goals – Try this with your email bulletins with
MailChimp

19. Featured Tools
Voice of the User
• 4Q/IPerceptions – Survey that is used widely and
asks 4 basic questions to random visitors specific to
the purpose of their visit and their task completion
and site satisfaction – This tool has a variety of
monthly plans with add-on features
• Online Polls – asking a single question at key
intervals let’s you get quick answers to questions
about your users – Examples – Poll Daddy and Poll
Everywhere

20. Google Analytics – Basics
Bounce rate is the percentage of visits that go only one page before
exiting a site: If the point of our work is to get people to find information
quickly and leave our site(s) to get what they need, how can we know if
a bounce is a success or whether they have lost interest or found the
site useless/frustrating ?

21. Custom Dashboards
Set widgets for the stats you care about
most – In this case, visits, bounce rates,
sources and goals

22. Use Google Analytics to Calculate
Percentage Shifts Over Time

23. Set “Goals” in Google Analytics
• If there are things you really want your users to “do” on your
site, set them up as “Goals” in Google Analytics. Each time a user
completes a Goal, a conversion is logged in your Google Analytics
account.
• Give each goal a monetary value merely to rank them in
relationship to one another. If a visit to a particular area of your
site is more important/strategic than another give it a higher $
value. This allows you to answer the question “Which
pages/sections are worth our time?”
• Improve a section of your site and see the Goal value rise and
fall “at a glance”

24. Examples of Goals Types and Reports

25. Heat Maps and Screen Recordings

26. Google Adwords - Keywords
Adwords is a great tool to suggest keywords and help you know which
words are commonly used in Ontario. It also tells you your site’s
average position in Google search results.

27. User testing Best Practices
• Should be task-based rather than “do you like
our site?”
• Should not be “guided” or assisted in any way
• Results should be observable and you only
need enough “testers” to identify trends or
repeated problems
• Should be done early enough that you can
make changes

28. In-Person User Testing
• Should be facilitated or led by a neutral 3rd
party that can provide tasks and instructions
and summarize findings
• Ideally, testers should verbalize their process,
their confusion, their opinions as they go
• And again, ideally you should be able to watch
and observe these results to make your final
conclusions

29. Virtual or Online User Testing
• Because we are trying to reach a vast
geographic area with limited funds, we
elected to conduct our user testing for Your
Legal Rights online in two phases
• Phase 1: Testing with wireframes – Designfree blue prints
• Phase 2: Re-testing with design mock-ups

30. Online User Testing: A Case Study
• In April and then in July 2011 we conducted sets
of online usability tests for the Your Legal Rights
website using a tool called VerifyApp.
• In April, we presented 15 volunteers with a set of
test based on 3 – 6 wireframes. We then
analyzed results and made revisions to our design
plans. In July and then conducted a set of linked
tests with 3 - 6 design mock ups including an
option for users to annotate the page with
comments or suggestions. For these later tests,
we put out a wider call and had closer to 25
volunteers.

31. Wireframe Test: Where would you click
to find information on eviction?

32. Mock-up Test: Where would you click to find
information on eviction?

33. Sample Test Result Showing Heat Map
Where would you click to find information on Eviction?

34. Examples of
Comments Reports

35. You can use these tools to test
accessibility, navigation,
emotional responses, user’s
memory, and A/B testing
Samples here are from
http://verifyapp.com/

36. A/B Testing
• Create two “competing” versions of a page and
then have Google create a test where 50% of
users see one and 50% see the other and then
examine results. Which page had a higher bounce
rate? Which page led to the next level you were
hoping users would find?
• Create two “competing” designs of your email
newsletter or e-bulletin and see which one has a
higher open or click-through rate. (MailChimp)

37. User Experience Surveys
• Create a post-visit survey and set an invitation
rate (20 to 50% of visitors) for key time
periods throughout the year.
• Ask users what their primary task was,
whether they completed it, what their “next
step” is, and if they would recommend your
site. They can also leave comments

38. IPerceptions (formerly 4Q)

39. Rate of return based on Invitation Rate
We set the Invitation
rate to 20% for desktop,
10% for mobile and 20%
for tablet and leave it in
place for 4-6 months
each year to compare
and benchmark success

40. User Experience Survey

41. Comments by Purpose of Visit
Challenge:
People keep
“anonymously”
telling me what
they wish they
had found and I
can’t help them
find it!
Anecdotal
Observation:
Now that our site
is not for service
providers, users
can be more
negative

42. Social Media – Followers/Users
Twitter
As of July 31, 2012 we have sent out 3900 tweets and have 1,560
followers.
As of March 31, 2012 we have sent out 5,035 Tweets to 1,998 Followers
This represents a 28.07% change in the # of followers to @legalrightsON
on Twitter.
Facebook
As of the end of July 2012, the Your Legal Rights Facebook page had 600
fans or “likes”.
As of the end of March 2013, the Your Legal Rights Facebook page had
2,031 fans or “likes”.
This represents a 238.49% increase.

43. Social Media– Measuring Engagement
Twitter - “Re-tweets” and “Mentions” – Hard to measure this over time, better to
log it on a schedule for benchmarking or baseline purposes. Lots of tools give you a
“glance” of the last 90 days but for our sector, we need longer timeframes.
Facebook - Facebook Insights Reports for Pages shows “Impressions” which refers
to the number of people who saw your postings as well as user engagement in
terms of the number of people who clicked on any of your postings or created
stories based on content posted to your Facebook page.

44. Social Media – Facebook Insights

45. Social Media – Klout Score

46. Tweetreach
https://tweetreach.com/reports/7661919?v=2

47. Discussion Questions?
How do we figure out who the audience is and why they
are visiting our sites or using our PLEI? It is more than
“purpose” and can’t really be captured in demographic
data…
How do we know what success really looks like without
any “industry benchmarks” or with such goals so difficult
to measure because of client privacy?
When our sites cover vast ranges of topics and given the
gaps in what legal info is needed, what insights can we
bring to the reasons certain topics or keywords are most
visited/searched?

48. Let’s do a sample together…
•
•
•
•
•
Think about a project in your organization…
Think of what the goal of the project is?
Think of ONE result of that you would like to see
for the project.
Think of one piece of evidence you might use to
prove whether you have achieved this goal.
Draft that into a KPI (Key Performance Indicator)

49. Keep in Touch!
Fiona MacCool
Your Legal Rights Project Manager
Tel: 416-408-4420 ext. 829
Email: fionamaccool@yourlegalrights.on.ca
Web site: http://www.yourlegalrights.on.ca
Twitter Feed: http://twitter.com/legalrightsON
Facebook: http://www.facebook.com/yourlegalrights