Technical Class: Monday, March 4 2:15 PM - 3:30 PM

1. SharePoint Moneyball – The Art of Winning the
SharePoint Metrics Game
March 4, 2013
©2013 SUSAN HANLEY LLC

2. About Me
• President, Susan Hanley LLC
• Led national Portals,
Management Collaboration,
and Content practice for Dell
• Director of Knowledge
Management at American
Management Systems
• Governance
• User Adoption
• Metrics
• Information Architecture
• Knowledge Management
• Portals
• Collaboration Solutions sue@susanhanley.com
susanhanley

3. Why Measure? – The Four “F” Words
Feedback
Funding
Follow-on
Focus
3

4. Measurement throughout the life-cycle
Before During After
Make the business Provide a target Develop
case benchmarks
Make tradeoffs
Develop lessons
Tune the learned
implementation
process
4

5. Make your case for the solution SharePoint enables
5

6. Measurement Process
4. What are the
1. What are the Modify the METRICS and how
BUSINESS OBJECTIVES? measures should we PRESENT
them?
2. How should the
SOLUTION be Modify the 5. How can we COLLECT
DESIGNED to meet the process or tool the metrics?
objectives?
6. What do the metrics
3. Who are the metrics Aid decision
TELL US about how we
STAKEHOLDERS? making
need to CHANGE?
6

7. 1. What are the BUSINESS OBJECTIVES?
 Without a critical business initiative …
 …“Career limiting move”
7

8. Be the main event

9. It’s easy to go for the “motherhood” objectives …
 More innovative products and services
 More effective marketing
 Better access to knowledge
 Lower cost of doing business – reduction in travel and other operational
costs
 Higher revenues
 Improved employee, customer, and partner satisfaction
9

10. It’s better to Get SMART!
Specific (concrete and well-defined)
Measurable (quantifiable, comparable)
Achievable (feasible, actionable)
Realistic (consider resources)
Time-bound (deadline driven)
10

11. SMART objective for a proposal library
Reduce the average amount of time it takes to produce complex
Reduce the average amount oftime it takes to to produce
time it takes produce complex
Reduce the average amount ofoftime it takes to produce
average amount
proposals by 10% in 10% in the
complex proposals the next yearnext year
proposals by 10% by next year
complexproposalsin by 10% in the next year
Specific
Measurable
Achievable
Realistic
Time-bound
11

12. Considering SharePoint 2013?
Search Connecting Mobile Cloud Support
People
12

13. 2. How should the solution be DESIGNED to meet these objectives?
Site Technical
Features
Architecture Infrastructure
Customization Security
Governance Roles and Responsibilities
Training and Communications
13

14. Your business case is personal

15. 3. Who are the metrics STAKEHOLDERS?
 They’re at all levels - especially in the
middle
 They care about different things
15

16. For each key stakeholder, ask …
 What counts?
 What keeps you up at night?
 What do you already use?
 What do I need to tell you?
16

17. 4. What are the METRICS and how should we PRESENT them?
Identify the type
• Quantitative Consider the life-cycle
• Qualitative
Establish a baseline
Gain commitment Decide the best way to
about targets communicate
17

18. Good metrics come in multiple types … plan on both
 Quantitative
 Performance between points
 Spot trends
 Qualitative
 Provide context
 Used when numbers aren’t easy (storytelling)
 Used at early project stages (future scenarios)
 Richer (“serious anecdotes”)
18

19. Return on Investment
 Benefit > Cost
 Be careful: whoever controls the spreadsheet and the
assumptions can make an ROI that can justify anything.
 Resources:
 Total Economic Impact™ of Microsoft SharePoint Server 2010 (Forrester)
http://bit.ly/cWfeyN
 Best for platform investment, less helpful for individual solutions
 Cloud vs. On Premise Calculator (Andrew McAfee and Google Analytics)
http://bit.ly/R6jlsZ (for small to medium businesses)
19

20. ROI is only PART of the story
Good metrics are:
Related to outcome
Relevant to stakeholders
Collected at low cost
Balanced
20

21. Consider two other types of quantitative metrics
BUSINESS METRICS
SYSTEM METRICS
21

22. Sample Business Metrics
 Hours per week to execute a process
 Number of Proposals/Contracts per year
 Number of “[My Organization]-All” emails
 Number of email attachments
 Average application training costs
 Cost savings to retire a legacy application
22

23. Business Metrics Example: Best Practice Library
• Save time, improve consistency by not re-
Business Goal inventing the wheel
• SMARTer Objective: Reduce the number of
employee reported hours per week
searching for examples of prior work.
Approach System Metrics
• OK: Number of views per document
• Better: Ratings
Business Metrics
• Best: Combine both with survey to
seek out specific re-use cases
23

24. Metrics Example: Prioritizing Resources for Projects
Allocate limited SharePoint Resources for
Business Goal •
Process Improvement Projects
• SMART Objective Example – Reduce the
amount of time for a task by x% in 90 days
Approach
x x x
T = time on task (in minutes)
E = number of employees performing that task
N = number of times per year a typical
employee performs that task
S = average employee’s loaded salary per minute
24

25. Qualitative Metrics – the stories that drive it home
Keep it real
In the storyteller’s
words
Serious Anecdotes
25

26. Serious Anecdote | Consulting
 I joined the organization on March 16 without previous experience. After
one week of training, I joined a project team.
 After one day of training on the project, I was assigned a task to learn a
particular technology that was new to everyone on the team. I was
given a bunch of books and three days, my tasklearn how to
Instead of told that I had three days to was
create a project using this technology.
completed in 4 hours.
 In my first week of training, I learned about the company’s intranet where
people described their expertise. I sent an email to four people I found
with a search for that technology asking for their help. One of them sent
me a link to a document containing exactly what I needed.
26

27. Serious Anecdote | Pharma – The Need
 A scientist with
Thrombotic & Joint
Diseases in Germany
began a project to isolate
and culture macrophages
and needed some help.
 Meanwhile, two scientists in
the US had deep experience in
protocols for this area.
27

28. Serious Anecdote | Pharma – The Result
 The German scientist consulted
the expertise directory to find
that expertise existed within the
company and contacted the two
US scientists he found in his
search.
Both scientists quickly
responded with assistance. One Benefit: The German scientist was able to
helped him with culturing leverage existing internal expertise and,
protocols and the other helped in the process, reduce his research effort
him with information on
magnetic cell sorting. by four weeks.
28

29. Different Measures for Different Stages
Enterprise Growth Phase
Use mixture of metrics to show
value across the organization.
Pilot Project Phase
Solution Maturity
Use definitive metrics to show real
value for business objectives.
Startup Phase
Use scenarios and simulations to
explore projected results and effects.
Pre-Planning Phase
Use scenarios and simulations to
explore projected results and effects.
Time
29

30. How do you spell success? Have a Baseline and Target
30

31. Presenting Metrics
Balanced Scorecard
Dashboard
“Report Card”
31

32. Balanced Scorecard Dimensions
Business Value
Health
Capabilities
32

33. Balanced Scorecard Example | Expertise Location
Metric Target Pilot Outcome
# of Anecdotes 10 serious 10 serious + 12 transactional
Business Minimum $ value/anecdote $X $2X
Value:
Estimated time saved X months X + 3 months
# searches/user/week .25 .58
Usefulness rating 3.5 out of 5 3.6 out of 5
Health: % of users who say “Don’t 66% 83%
take it away”
Usability/friendliness rating 3.5 out of 5 4.1 out of 5
# Anecdotes (repeat metric) 10 serious 22
Capabilities
& Culture: % of participants attending 75% 85%
training
33

34. 5. How can we collect the metrics?
 Try not to over-achieve – balance counting with “doing”
 Automate where possible
 Get creative when it comes to qualitative metrics
 Ask
 Survey
 Usability Testing
 Active Listening
 Seek
 Send out a “journalist”
 Track
 Classify by type
 Keep storyteller value metrics – what was the benefit to you?
34

35. Example Survey Questions
 If given the choice, would you KEEP it?
 How does this COMPARE?
 How EASY was it to …?
35

36. Tip: Collect Qualitative Metrics with Ratings
Was it helpful?
Were you able to get value?
Trigger a Can we call Follow up
survey at 4-5 you? conversation
36

37. 6. What do the metrics tell us about how we need to change?
 Are we doing the right thing?
 What areas are most successful?
 What areas should we be promoting?
 In which areas should we be investing?
 Which initiatives should we discontinue?
37

38. Keep in mind
Metrics alone won’t make your program
successful
A person who is
A person whose job it accountable for making
is to monitor them changes based on
analysis
It’s as important to have a plan for acting on
metrics as it is to have a plan for collecting them!
38

39. Call to Action
Make sure metrics
Develop a plan to capture quantitative
are part of
and qualitative metrics.
someone’s job.
Identify baseline measures – and gain commitment on
targets – before you start!!
Develop a library or list to
Develop an approach to
capture and categorize
produce and promote metrics.
qualitative metrics.
39

40. Susan Hanley
Susan Hanley LLC
sue@susanhanley.com
301 469 0770 (o)
301 442 0127 (m)
@susanhanley
www.susanhanley.com
http://www.networkworld.com/
community/sharepoint
40

41. Extras
41

42. White Paper
 For a white paper that explains the concepts in this presentation in more detail – with
lots more examples, please go to http://www.susanhanley.com.
42

43. Examples of Intranet System Measures
Key Measure Objective Metric
Which features of the  Knowing which pages are most used can help to prioritize which pages should  Page Hits
intranet are most be improved or developed.  “Dwell” Time (Time on
important?  You can also see which business units are the biggest intranet users and which Page/Site)
business unit’s content is used the most.
Which features are not  If certain pages have low usage numbers, it is an indication that either the page  Page Hits
being used? is not very popular—and therefore should be a lower priority to develop—or  Document Downloads
that people are just not aware of its existence (which might be a
communications or "promotion" problem).
Is the site navigation  A high number of hits on a page that is not easily accessible from the main page  Page Hits on pages deep
effective? indicate that the popular page should be moved up in the hierarchy. in the hierarchy
 Search results with no hits present opportunities to both promote content and
add best bets.
Which team sites  Sites that have not been accessed in the past 12 months might be candidates for  Page Hits
should be archived or archival or deletion if the content is no longer useful.
deleted?
What are the  Monitoring usage trends helps identify patterns or problems and potentially  Usage by time
peak/low usage alerts the Exchange Business Owner and Portal Administrator of potential user or
times? performance issues – ideally, before they become a problem.
How is usage  Trending reports are available for a limited period of time within SharePoint  Number of users and
trending? 2010. Third-party tools are required to do multi-year detailed trend analysis. number of unique users
over time 43

44. Sample System Metrics (“out of the box” SharePoint 2010)
Metric Objective
Number of Unique •Provides a proxy for adoption, which is a loose proxy for value.
Users (month to
month)
Most Viewed •Provides a proxy for the most valuable content.
Pages/Sites •Sites not being used help identify content that might either need to be
promoted or deleted.
Top Queries (search) •Identifies “trending “ content.
•Top queries can also provide insights about what content should be
promoted to the home page.
Failed Queries / No •Identifies candidates for best bets or synonyms and identifies emerging
Results Queries business terms or concepts.
Best Bet Suggestion •Helps the business owner improve user outcomes by identifying URLs as
Report most likely results for search queries.
Best Bet Usage •Helps identify which best bets are adding value and as an input to
determine new best bets or best bets that need updating.
44

45. Additional Useful System Metrics (third-party for 2010)
Metric Objective
Most Viewed Documents  Provides a proxy for the most valuable content.
Document Contribution/Editing Analysis  Provides a way to measure sustained adoption from the
perspective of employee engagement.
Team Site Summary Information  Provides a way to understand which sites are actively being
 Total Number of Team Sites used to monitor the health of the collaborative team sites.
 Viewed in Past 30 days  Can be used to identify which sites are no longer being
 Modified in Past 30 days used and might be able to be deleted or archived.
 Sites with no access in past 12  Provides a proxy for whether or not team sites are adding
months value.
 Trend of the number of team sites
created
My Site Summary and Trends  Identifies adoption of people-to-people collaboration
 Total number of My Sites features.
 Viewed in past 30 days  Proxy for employee engagement.
 Modified in past 30 days
 Average size 45

46. System Metrics in SharePoint 2013 Online
46

47. One sample metric for each stage (more in White Paper)
Life-cycle Example Quantitative Metric Example Qualitative Metric Sources
Stage
Planning • Time to perform current • "Day in the life" future • Work measurement studies
process stories • Interviews of key
stakeholders
Start up • N/A • Immediate term “day in the • Employee surveys
life” stories
Pilot • Same metrics you used for • Usage anecdotes –specific • Follow up work
Conclusion baselines examples from pilot measurement studies
• Surveys and follow up
interviews
Ongoing • Additional metrics relevant • Usage anecdotes with a • New solution system
to the business problem “serious” punch line that metrics
available with new process you collect and catalogue • Employee surveys and
on an ongoing basis follow up interviews
47

48. Balanced Scorecard Framework
Objectives Critical Success Factors Source Sample Metrics
Gain frequent and • High volume of needs that can’t be • System metrics • # of searches per week
sustained met through existing channels • User Surveys • # of average users per week
Solution adoption of • Positive impact on existing workload • # unique users per week
Health solution or work processes • # of “hits” on key pages/sites
• “Usefulness rating” from user surveys
• % of users who say “don’t take it away”
at the end of the pilot
Provide reliable, • Solution user-friendliness and • System metrics • “Usefulness rating” from user surveys
easy-to-use intuitiveness • User Surveys • # of searches per week
technology that • Solution reliability • Direct measurement • # of average users per week
can be • Integration of the solution with work • # unique users per week
incorporated into processes and existing tools • # of “hits” on key pages/sites
work processes • # Help Desk calls/week
Capabilities
Ensure users • User training • System metrics • % of users trained
understand • Effective help resources • User Surveys • % of pilot milestones achieved
objectives and • Persistent, clear communications • Direct measurement • # of communications events/activities
how to leverage • Active, sustained management
the solution support
• Incorporation of collaboration into
performance objectives and
evaluations
Demonstrate clear • Tangible, quantifiable examples of • “Serious” Anecdotes • # of anecdotes
Business value with respect reductions in process cycle time collected via surveys • $ value of anecdotes
Value to the business • Estimates and/or • Cycle time improvement (in hours)
strategy direct measurement
of cycle time 48

49. Survey Example: Post-Pilot Usability
Usability Question Metric
If presented the choice, do you want to keep the  “Don’t Take it
solution? Away”
 Don't take it away  Take it away
Usability/friendliness - how does the usability of this  “User Friendliness
solution compare to other solutions you use on a Rating”
regular basis?
 Much easier to use  Harder to use
 Easier to use  Much harder to use
 About the same
How easy and intuitive was the solution to use for each  “Intuitiveness
of the following [specific task]? Rating”
 Very easy  Difficult
 Easy  Very Difficult
 Moderate
49

50. Other Resources
 How to Measure Anything: Finding the Value of Intangibles in Business by Douglas
Hubbard
 Jakob Nielen’s Alert Box - Current Issues in Web Usability:
http://www.useit.com/alertbox/
 Determining the Value of Social Business ROI: Myths, Facts, and Potentially High
Returns: http://www.idc.com/getdoc.jsp?containerId=225497
 Social Software for Business Performance: The missing link in social software:
Measureable business process performance: http://www.deloitte.com/assets/Dcom-
UnitedStates/Local%20Assets/Documents/TMT_us_tmt/us_tmt_socialsoftwareexecsu
mmary_021411.pdf
 SharePoint Lifecycle Management Solution with Project Server 2010:
http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=17058
50