SlideShare ist ein Scribd-Unternehmen logo
1 von 10
Downloaden Sie, um offline zu lesen
1
Recommendations for Research Networking Systems (RNS)
Usage Tracking – Version 1, July 2013
Foreword
A subgroup of the CTSA Research Networking Affinity Group (RNAG) has developed a set of recommendations for SGC3, focused on measuring
the usage of research networking systems (RNS).
Introduction
Biomedical institutions have widely deployed institutional expertise discovery and researcher networking systems (RNS), including VIVO, Profiles
RNS, and SciVal Experts. Institutions have invested in networking systems drawn by the promise of discovering complementary expertise,
enabling collaboration, facilitating funding opportunity matching, and other key needs for accelerating translational research.
Evidence is currently limited on the extent of usage of RNS tools and the extent to which they promote research-oriented networking. However,
it is possible to measure usage of these systems using web traffic analytics tools, via analysis of web logs. The most well-known of these tools is
Google Analytics, but a number of attractive alternatives are emerging.1
In addition, qualitative methods can be employed to collect feedback
with respect to value and utility, especially from the user perspective. The recommendations here reflect current measures being used by 8
CTSA consortium member institutions and one commercial entity:
• Harvard University
• Medical University of South Carolina
• Northwestern University
• University of Iowa
• University of Minnesota
• University of California, San Francisco*
• Vanderbilt University
• Washington University*
• Elsevier*
This set of recommendations is a start. This document is meant to be a living and changing set of recommendations that will inform institutions
and SGC3 of the value and power of research networking systems for accelerating translational research. For any specific institution, adopting a
subset of these measures can help to shape strategy, decision making, and practical tactics.
1
Some notable examples include Google Analytics (http://www.google.com/analytics/), Site AI (http://www.siteai.com/), Woopra (http://www.woopra.com/),
with NewRelic (http://newrelic.com ) being used for performance monitoring. See Appendix 1 for more on tools.
*Authored Version 1 of this document
2
The structure of this document is as follows: The set of recommended RNS Usage Measures below a description of each measure and an
accompanying rationale and explanation as to why the measure should be tracked. Institutional examples and case studies then describe
successful strategies and tactics that have evolved from the analysis of the data, which range from informing the overall marketing strategy for
adoption, to targeting specific faculty meetings for presentations and training, to gaining stakeholder and financial support. For ease of
discussion and presentation, Google Analytics will be used as an example of an analytics tool to track tool usage.
Recommended RNS Usage Measures
For all measures:
Track trends
R1
Establish a baseline and
track over time
Track the same measures over time to illustrate patterns and responsiveness to actions (e.g., a new marketing
campaign).
Set expectations
R2
Set expectation for trend
over time
For trends, set expectations for how the measures
will trend.
At UCSF, while they hope to see the number of 2+
minute on-campus visits per day increase over time,
they expect this measure to plateau over time if the
majority of the core user community comes to make
appropriate use of the tool whenever needed.
Separate internal from external usage
R3
Distinguish between traffic
internal and external to the
institution
It easy to distinguish between institutional and
extra-institutional traffic with Google Analytics by
looking at the visitor’s service provider. By
creating advanced segments that show only traffic
from your institution’ service provider(s) vs. those
from all other service providers, you can filter
UCSF set up two advanced segments: users with service
provider “university of california san francisco,” and
those with any other service provider. They then look at
every statistic via both segments. This shows, for
example, that they get over ten times as many off-
campus visits as they do on-campus visits, and that on-
3
every statistic through an internal vs. external
lens. (Be sure to that your team’s own accesses to
the site are being ignored by filtering out your IP
addresses as an account-level filter.)
campus visitors are more likely to arrive via campus
websites than external visitors.
The basics:
Implement Google Analytics
R4
Track basic web traffic
statistics via Google
Analytics
Google Analytics affords easy access to basic usage analytics s so you can determine:
1. Visits to your site
2. Page views
3. Unique visitors
4. % new vs. % returning visitors
5. Bounce rate on home and profile pages
6. # pages looked at per visit
7. Average visit duration
Understand search traffic
R5
Use search traffic statistics
to inform marketing and
outreach opportunities
Google Analytics shows not only the amount of
traffic from public search engines like Google and
Bing, but also the keywords that brought users to
your site. This can be used to understand search
behavior and intent. For example, if search traffic
is low, it might be an opportunity do basic search
engine optimization; or if users are coming to the
site after searching for people’s name on Google,
it may provide an opportunity to highlight their
networks on the landing page.
UCSF received only 2,500 visits a month from external
search engines during launch month. They subsequently
did search engine optimization work, and now receive
over 40,000 visits a month from external search engines
— 72% of all visits.
4
Delve into referral traffic
R6
Use referral traffic statistics
to inform marketing and
outreach opportunities
Google Analytics also provides data on which
website refer the most traffic to your system. By
understanding these patterns, you can look at
institutional traffic and sites and it can help to
inform your outreach and marketing efforts. For
example, if a web resource popular in the local
research community does not refer much traffic it
could be an opportunity for outreach to get links
back to your system added at the appropriate
spots.
Comparisons across institutions can also help here. For
example, the 2nd highest referring site at UCSF is the
campus directory. UCSF partnered with the campus
group responsible for each person’s page on the main
online people directory to link back to the corresponding
UCSF Profiles page; the result was a substantial increase
in traffic to UCSF Profiles -- about 1,700 visits a month.
Understand what’s popular
R7
Understand traffic patterns
to different parts of the site
to understand preferential
usage, and prioritize
outreach and development
Google Analytics allows users to see what parts of
the site are getting the most traffic, and how users
behave on each page, e.g. by looking at bounce.
Understanding these measures is critical to
understanding usage of the site.
A month after launch, UCSF staff were surprised to
discover that only 9% of visits begin on the home page,
with most users arriving directly on profile pages. This
provided the impetus to spend substantially more time
improving data on profile pages, and deemphasize tweaks
to the home page.
Look at your geographic breakdown
R8
Examine regional and
global traffic patterns to
inform potential economic
development impact
Google Analytics’s audience location demographics will show your traffic from within your region, state,
nation, and the world. If you’re like many institutions, you might be surprised at how much of the traffic to
your RNS comes from outside of your institution, let alone your state.
In addition, look both at city, (for the institution’s city and the surrounding suburbs), as well as network
provider which is often be linked to the university, sometimes even down to school or college within the
university.
5
Benchmarking:
Define meaningful engagement
R9A & R9B
Track the number of “X+”
minute visits to the RNS
from within the institution
(where “X+” represents the
length of time you deem
meaningful)
Track the number of visits
where users from within
the institution view at least
“Y” pages (where “Y”
represents the # pages you
deem meaningful)
For context, we are using the model of an RNS that
does not require log-in to search the network. As
such, discuss and define a surrogate for what
constitutes meaningful engagement with your
RNS. Once defined, these measures can be found
via Google Analytics; complex queries can be used
through advanced segments and/or custom
dashboards.
At UCSF, they pay particular attention to the average
number of visits from within the UCSF campus network
lasting 2 minutes or longer (Google Analytics defines a
visit as a period of continuous usage where the gap
between subsequent page views or interactions is under
half an hour). Because users do not need to log in to
search UCSF Profiles, they use these 2+ minute on-
campus visits as a proxy for deep active use of the
product by researchers, administrators, and other staff on
campus.
Benchmark the traffic
R10
Benchmark relative to
other campus web
properties, both on and off
campus
One way to infer relative usefulness of your RNS is
to track and benchmark the RNS traffic relative to
other campus properties such as the directory or
the main institutional web site.
At UCSF, the RNS gets as much as 60% of the visits of the
campus directory from all visitors, and 12% of the visits of
the campus directory from UCSF campus visitors. This
signals the relative importance of this system as a
resource.
6
Segment populations
R11
Segment and compare
populations of profiled
users, and of visitors (when
possible)
Using the data in the RNS, segment populations of
engaged users to determine differing or similar
patterns of use. The differences can inform
outreach and training opportunities, and value can
be inferred. Examples of segments: faculty rank,
department, school, etc. Segments with lesser
adoption could be candidates for outreach and
training, whereas segments with greater adoption
are candidates for interviews on why and what is
going well. If the system is behind a login wall,
there may also be opportunities to segment end
users.
By looking at UCSF Profile adoption rates (where adoption
is defined as a profile that has some curation or
customization completed such as an uploaded photo),
they found that faculty at all levels – junior to senior –
have adopted their profiles in close to similar proportions.
This seems to indicate equal importance of the tool
across the segments, and helps as they plan outreach and
marketing for the system.
Analyze where people click
R12
See what your users are
clicking on and use the data
to improve your product
A heatmap is an easy way to understand what
users do on your site. It’s a visual representation
showing where users click and what they do. This
can help you determine what aspects of your RNS
are most popular & what is overlooked. Google
Analytics provides this data via in-page analytics
feature, or a separate heatmap tool may be used.
If certain classes of links on your site are being
systematically ignored, this data can help inform
user interface changes and/or marketing
messages.
UCSF has installed an inexpensive heat mapping tool and
used that along with Google Analytics in-page analytics to
prioritize improvements to their RNS home page.
7
Qualitative measures:
R13
Measure traffic impact
resulting from various
communications (e.g., PR
and marketing campaigns)
and then judiciously use
the successful strategies in
the future
Analyze traffic to your RNS before and after
various marketing campaigns such as emails or
trainings.
Try different messages using A/B testing and
measure which ones are most successful.
Partner with other groups on campus so all of the
messages don’t come from one source. Measure
traffic impact from these various sources.
UCSF noticed a big spike in traffic after the Executive Vice
Chancellor & Provost (EVCP) sent out an email
announcing UCSF Profiles. This traffic spike was larger
than any other email campaign and speaks to the reach
and value of communications from UCSF’s EVCP.
Also, at UCSF, groups across campus from the Library to
University Relations have helped to spread the word
about UCSF Profiles. These groups are on board to help
promote UCSF Profiles because data mined from UCSF
Profiles have facilitated those groups’ projects.
Survey users on value & usage
R14
Survey users for direct
feedback; use the feedback
to inform strategy and gain
support for your system
To really find out how people use the system and
what value they garner from your RNS, you will
need to ask them. The feedback will identify gaps
and improvements, inform planning and hopefully
help gain support to sustain and enhance your
system.
A number of SciVal Experts’ sites track numbers of
inquiries and/or profile corrections as additional evidence
of how many people are really looking at the site and its
content.
REACH-NC (NC state SciVal Experts portal) created a
contact form that users fill out when trying to contact a
researcher and they do not display direct contact
information. The contact form goes to the campus liaison
who coordinates the contact. In addition, a copy of the
request goes to the REACH-NC Executive Director and
Program Coordinator so they can track request and
monitor connections. In addition, inquiries and feedback
from profiled researcher come to a group mail box that
the ELS Account Development Manager personally
manages, instead of directly going to the general ELS
helpdesk.
8
Arizona State University had a ‘Submit Feedback’ button
on the Experts site that linked to an end-user survey.
(They felt that the information they were receiving was
biased toward the negative and also unhelpful, so they
discontinued use of the survey.) Note: this offers up a
lesson on being specific about what kind of feedback you
solicit, and not necessarily a warning against soliciting
feedback in this way
UCSF has implemented a pop up survey on UCSF Profiles
that while annoying to many, has resulted in the
collection of substantive feedback such as:
“Great resource for finding potential research
collaborators and for PhD dissertation
committees”
“I am hoping that it just helped me find a
mentor...”
“It has helped find new nursing research
problems”
Interventional networking:
Given the significant lag time between identifying a potential collaborator and the first outcome of a successful collaboration, such as a
published paper or even a funded grant award, it is difficult to directly link RNS usage to interventional networking. Additional studies need to
be done to link short-term RNS usage patterns to longer-term outcome measures such as:
• Number of new introductions/meetings/active collaborations
• Interdisciplinary nature of new introductions/meetings/collaborators
• Resultant joint grant applications and publications
Enable & track collaboration right from your RNS
9
R15
Enable collaboration from
within your RNS
Identify potential collaborations and enable
communication between researchers directly from
your RNS.
The Michigan Corporate Relations Network (MCRN) portal
functions like a multi-university Experts site that allows
for searching across participating universities in Michigan
(U of MI, University of Michigan’s Dearborn Campus,
Michigan State University, Wayne State University at
launch, and Western Michigan University shortly
thereafter, and likely also Michigan Tech sometime after
that). Thinking of it like an ecommerce metaphor, the
site allows a businessperson interested in working with
state of Michigan researchers to add researchers of
interest to a list (‘shopping cart’) and the call to action
(‘checkout’) process routes the researchers of interest
along with a user-entered problem statement to the
appropriate university business relations offices. These
will be tracked centrally and by each office, hopefully with
some evaluation of quantity of contacts and ‘quality’ of
contacts (was it a legitimate research collaboration
interest or a solicitor trying to sell something? did it turn
into a collaboration?)
UCSF has an operational pilot that has integrated UCSF
Profiles with Salesforce Chatter (Chatter is UCSF’s
enterprise social networking tool). This allows one to
“follow” a person directly from UCSF Profiles and also
allows a person to select a few people of interest and
create a “Chatter Group” directly from UCSF Profiles.
10
Appendix 1
Notes / community comments on analytic tools
From various sources, including: https://wiki.duraspace.org/display/VIVO/VIVO+Web+Analytics
Analytic Tool Comments
Google Analytics Used as default Profiles solution
Also used as default VIVO solution – see googleAnalytics.ftl FreeMarker template
Woopra "all-in-one web application performance tool"
Paul Albert at Weill notes: "I've personally used Woopra before. It seems to be better than Google Analytics about tracking outbound
links and for watching people click through your site in real time. (Paul Albert, WCMC)
New Relic: For performance monitoring, we use New Relic, which you can get a sense of here:
https://dl.dropboxusercontent.com/u/2014679/NewRelic.png and https://dl.dropboxusercontent.com/u/2014679/EmailAlert.png.
The best part about New Relic is that it will send a notice to other team members when you have tried to execute several malformed
SPARQL queries in a row." (Paul Albert, WCMC)
Site AI: "The problem with dashboards is they don’t directly provide insights or deliver knowledge about the data. Even worse, most
visualizations require the user go through the mental exercise of interpreting the results. Site Ai does the analysis for you and presents
the information in plain English."
example: http://www.siteai.com/share/2672/2013/5/22/fcsjiggewdpechaafwpfjedlkvjhvipu
(Mark Fallu, research-hub.griffith.edu.au at Griffith University)
Mixpanel: "Page view counts are popular because they are easy to report, but ultimately cannot tell you how engaged your visitors are. Mixpanel
lets you measure what customers do in your app by reporting actions, not page views." (Alex Viggio, University of Colorado Boulder)
Chartbeat See demo
CrazyEgg CrazyEgg provides data on what links users click on once they arrive on key pages or selected profiles. Click statistics are provided in
several formats: heatmap, scrollmap, overlay, and ‘confetti.’ This allows us to improve design of page elements strategically to
encourage interaction. (Leslie Yuan, UCSF)
Qualaroo User feedback is collected via a popup survey tool; detailed reports are available online. We ask “How has UCSF Profiles helped you?”
to solicit success stories and better understand use cases, but the survey functions as a customer service access point as well. Users
may enter an email address if they’d like a response. (Leslie Yuan, UCSF)

Weitere ähnliche Inhalte

Andere mochten auch

One example of a feature built as an OpenSocial gadget
One example of a feature built as an OpenSocial gadgetOne example of a feature built as an OpenSocial gadget
One example of a feature built as an OpenSocial gadgetlesliey
 
Plumage: Open Source Web App Leveraging eagle-i to Make Core Resources Discov...
Plumage: Open Source Web App Leveraging eagle-i to Make Core Resources Discov...Plumage: Open Source Web App Leveraging eagle-i to Make Core Resources Discov...
Plumage: Open Source Web App Leveraging eagle-i to Make Core Resources Discov...anirvanchatterjee
 
VIVO2015 - Delivering Trending Publications
VIVO2015 - Delivering Trending PublicationsVIVO2015 - Delivering Trending Publications
VIVO2015 - Delivering Trending PublicationsBrian Turner
 
Improving research networking usability at UCSF
Improving research networking usability at UCSFImproving research networking usability at UCSF
Improving research networking usability at UCSFanirvanchatterjee
 
AMIA Panel: Usability Enhancements
AMIA Panel: Usability EnhancementsAMIA Panel: Usability Enhancements
AMIA Panel: Usability EnhancementsBrian Turner
 
Intro to UCSF Profiles for RadOnc
Intro to UCSF Profiles for RadOncIntro to UCSF Profiles for RadOnc
Intro to UCSF Profiles for RadOncBrian Turner
 
Reduce the pain of updating faculty web pages
Reduce the pain of updating faculty web pages Reduce the pain of updating faculty web pages
Reduce the pain of updating faculty web pages lesliey
 
Growth Hacking 101 for Research Networking (for VIVO Implementation & Dev call)
Growth Hacking 101 for Research Networking (for VIVO Implementation & Dev call)Growth Hacking 101 for Research Networking (for VIVO Implementation & Dev call)
Growth Hacking 101 for Research Networking (for VIVO Implementation & Dev call)anirvanchatterjee
 
Deconstructing a Dashboard:
 Inside the UCSF Profiles Team’s Monthly Key Metrics
Deconstructing a Dashboard:
 Inside the UCSF Profiles Team’s Monthly Key MetricsDeconstructing a Dashboard:
 Inside the UCSF Profiles Team’s Monthly Key Metrics
Deconstructing a Dashboard:
 Inside the UCSF Profiles Team’s Monthly Key Metricsanirvanchatterjee
 
VIVO2015 - Leveraging Personalized Google Analytics for Greater RNS Engagement
VIVO2015 - Leveraging Personalized Google Analytics for Greater RNS EngagementVIVO2015 - Leveraging Personalized Google Analytics for Greater RNS Engagement
VIVO2015 - Leveraging Personalized Google Analytics for Greater RNS EngagementBrian Turner
 
Fostering Collaboration at UCSF - presented at Univ of Melbourne Research Pro...
Fostering Collaboration at UCSF - presented at Univ of Melbourne Research Pro...Fostering Collaboration at UCSF - presented at Univ of Melbourne Research Pro...
Fostering Collaboration at UCSF - presented at Univ of Melbourne Research Pro...lesliey
 
5 Secrets of a Successful UCSF Profiles page 2015
5 Secrets of a Successful UCSF Profiles page 20155 Secrets of a Successful UCSF Profiles page 2015
5 Secrets of a Successful UCSF Profiles page 2015anirvanchatterjee
 
UCSF Profiles, more Moneyball, less Field of Dreams, presented at U of Chicag...
UCSF Profiles, more Moneyball, less Field of Dreams, presented at U of Chicag...UCSF Profiles, more Moneyball, less Field of Dreams, presented at U of Chicag...
UCSF Profiles, more Moneyball, less Field of Dreams, presented at U of Chicag...lesliey
 
10 simple ways UCSF Profiles has been used to win funding, find collaborators...
10 simple ways UCSF Profiles has been used to win funding, find collaborators...10 simple ways UCSF Profiles has been used to win funding, find collaborators...
10 simple ways UCSF Profiles has been used to win funding, find collaborators...lesliey
 
UCCSC 2013 Presentation on UCSF Profiles
UCCSC 2013 Presentation on UCSF Profiles UCCSC 2013 Presentation on UCSF Profiles
UCCSC 2013 Presentation on UCSF Profiles lesliey
 

Andere mochten auch (15)

One example of a feature built as an OpenSocial gadget
One example of a feature built as an OpenSocial gadgetOne example of a feature built as an OpenSocial gadget
One example of a feature built as an OpenSocial gadget
 
Plumage: Open Source Web App Leveraging eagle-i to Make Core Resources Discov...
Plumage: Open Source Web App Leveraging eagle-i to Make Core Resources Discov...Plumage: Open Source Web App Leveraging eagle-i to Make Core Resources Discov...
Plumage: Open Source Web App Leveraging eagle-i to Make Core Resources Discov...
 
VIVO2015 - Delivering Trending Publications
VIVO2015 - Delivering Trending PublicationsVIVO2015 - Delivering Trending Publications
VIVO2015 - Delivering Trending Publications
 
Improving research networking usability at UCSF
Improving research networking usability at UCSFImproving research networking usability at UCSF
Improving research networking usability at UCSF
 
AMIA Panel: Usability Enhancements
AMIA Panel: Usability EnhancementsAMIA Panel: Usability Enhancements
AMIA Panel: Usability Enhancements
 
Intro to UCSF Profiles for RadOnc
Intro to UCSF Profiles for RadOncIntro to UCSF Profiles for RadOnc
Intro to UCSF Profiles for RadOnc
 
Reduce the pain of updating faculty web pages
Reduce the pain of updating faculty web pages Reduce the pain of updating faculty web pages
Reduce the pain of updating faculty web pages
 
Growth Hacking 101 for Research Networking (for VIVO Implementation & Dev call)
Growth Hacking 101 for Research Networking (for VIVO Implementation & Dev call)Growth Hacking 101 for Research Networking (for VIVO Implementation & Dev call)
Growth Hacking 101 for Research Networking (for VIVO Implementation & Dev call)
 
Deconstructing a Dashboard:
 Inside the UCSF Profiles Team’s Monthly Key Metrics
Deconstructing a Dashboard:
 Inside the UCSF Profiles Team’s Monthly Key MetricsDeconstructing a Dashboard:
 Inside the UCSF Profiles Team’s Monthly Key Metrics
Deconstructing a Dashboard:
 Inside the UCSF Profiles Team’s Monthly Key Metrics
 
VIVO2015 - Leveraging Personalized Google Analytics for Greater RNS Engagement
VIVO2015 - Leveraging Personalized Google Analytics for Greater RNS EngagementVIVO2015 - Leveraging Personalized Google Analytics for Greater RNS Engagement
VIVO2015 - Leveraging Personalized Google Analytics for Greater RNS Engagement
 
Fostering Collaboration at UCSF - presented at Univ of Melbourne Research Pro...
Fostering Collaboration at UCSF - presented at Univ of Melbourne Research Pro...Fostering Collaboration at UCSF - presented at Univ of Melbourne Research Pro...
Fostering Collaboration at UCSF - presented at Univ of Melbourne Research Pro...
 
5 Secrets of a Successful UCSF Profiles page 2015
5 Secrets of a Successful UCSF Profiles page 20155 Secrets of a Successful UCSF Profiles page 2015
5 Secrets of a Successful UCSF Profiles page 2015
 
UCSF Profiles, more Moneyball, less Field of Dreams, presented at U of Chicag...
UCSF Profiles, more Moneyball, less Field of Dreams, presented at U of Chicag...UCSF Profiles, more Moneyball, less Field of Dreams, presented at U of Chicag...
UCSF Profiles, more Moneyball, less Field of Dreams, presented at U of Chicag...
 
10 simple ways UCSF Profiles has been used to win funding, find collaborators...
10 simple ways UCSF Profiles has been used to win funding, find collaborators...10 simple ways UCSF Profiles has been used to win funding, find collaborators...
10 simple ways UCSF Profiles has been used to win funding, find collaborators...
 
UCCSC 2013 Presentation on UCSF Profiles
UCCSC 2013 Presentation on UCSF Profiles UCCSC 2013 Presentation on UCSF Profiles
UCCSC 2013 Presentation on UCSF Profiles
 

Ähnlich wie Recommendations for usage tracking for research networking systems, v.1. July 2013

IRJET- Strength and Workability of High Volume Fly Ash Self-Compacting Concre...
IRJET- Strength and Workability of High Volume Fly Ash Self-Compacting Concre...IRJET- Strength and Workability of High Volume Fly Ash Self-Compacting Concre...
IRJET- Strength and Workability of High Volume Fly Ash Self-Compacting Concre...IRJET Journal
 
IRJET- Implementing Social CRM System for an Online Grocery Shopping Platform...
IRJET- Implementing Social CRM System for an Online Grocery Shopping Platform...IRJET- Implementing Social CRM System for an Online Grocery Shopping Platform...
IRJET- Implementing Social CRM System for an Online Grocery Shopping Platform...IRJET Journal
 
Intranet focus search strategy a z - from Findability Day 2014
Intranet focus search strategy a z - from Findability Day 2014Intranet focus search strategy a z - from Findability Day 2014
Intranet focus search strategy a z - from Findability Day 2014Findwise
 
IRJET- Rating based Recommedation System for Web Service
IRJET- Rating based Recommedation System for Web ServiceIRJET- Rating based Recommedation System for Web Service
IRJET- Rating based Recommedation System for Web ServiceIRJET Journal
 
Enhanced Web Usage Mining Using Fuzzy Clustering and Collaborative Filtering ...
Enhanced Web Usage Mining Using Fuzzy Clustering and Collaborative Filtering ...Enhanced Web Usage Mining Using Fuzzy Clustering and Collaborative Filtering ...
Enhanced Web Usage Mining Using Fuzzy Clustering and Collaborative Filtering ...inventionjournals
 
A Systematic Literature Survey On Recommendation System
A Systematic Literature Survey On Recommendation SystemA Systematic Literature Survey On Recommendation System
A Systematic Literature Survey On Recommendation SystemGina Rizzo
 
Query Recommendation by using Collaborative Filtering Approach
Query Recommendation by using Collaborative Filtering ApproachQuery Recommendation by using Collaborative Filtering Approach
Query Recommendation by using Collaborative Filtering ApproachIRJET Journal
 
NOW! Get the internet to work for you!
NOW! Get the internet to work for you!NOW! Get the internet to work for you!
NOW! Get the internet to work for you!Philip Hannah
 
IRJET- Virtual Business Analyst using a Progressive Web Application
IRJET- Virtual Business Analyst using a Progressive Web ApplicationIRJET- Virtual Business Analyst using a Progressive Web Application
IRJET- Virtual Business Analyst using a Progressive Web ApplicationIRJET Journal
 
IJRET : International Journal of Research in Engineering and TechnologyImprov...
IJRET : International Journal of Research in Engineering and TechnologyImprov...IJRET : International Journal of Research in Engineering and TechnologyImprov...
IJRET : International Journal of Research in Engineering and TechnologyImprov...eSAT Publishing House
 
APPLICATION OF SENTIMENT ANALYSIS IN WEB DATA ANALYTICS
APPLICATION OF SENTIMENT ANALYSIS IN WEB DATA ANALYTICSAPPLICATION OF SENTIMENT ANALYSIS IN WEB DATA ANALYTICS
APPLICATION OF SENTIMENT ANALYSIS IN WEB DATA ANALYTICSJim Jimenez
 
VIVO 2014: Google Analytics, Email Marketing & Vanity to Increase User Engage...
VIVO 2014: Google Analytics, Email Marketing & Vanity to Increase User Engage...VIVO 2014: Google Analytics, Email Marketing & Vanity to Increase User Engage...
VIVO 2014: Google Analytics, Email Marketing & Vanity to Increase User Engage...CTSI at UCSF
 
VIVO 2014: Google Analytics, Email Marketing & Vanity to Increase User Engage...
VIVO 2014: Google Analytics, Email Marketing & Vanity to Increase User Engage...VIVO 2014: Google Analytics, Email Marketing & Vanity to Increase User Engage...
VIVO 2014: Google Analytics, Email Marketing & Vanity to Increase User Engage...Nooshin Latour
 
LyonALMProposal20041018.doc
LyonALMProposal20041018.docLyonALMProposal20041018.doc
LyonALMProposal20041018.docbutest
 
LyonALMProposal20041018.doc
LyonALMProposal20041018.docLyonALMProposal20041018.doc
LyonALMProposal20041018.docbutest
 
Google Analytics ( Digital Marketing ) Dashboard , Sessions, Technology
Google Analytics ( Digital Marketing ) Dashboard , Sessions, Technology Google Analytics ( Digital Marketing ) Dashboard , Sessions, Technology
Google Analytics ( Digital Marketing ) Dashboard , Sessions, Technology IMM Graduate School
 
Using Ontology to Capture Supply Chain Code Halos
Using Ontology to Capture Supply Chain Code HalosUsing Ontology to Capture Supply Chain Code Halos
Using Ontology to Capture Supply Chain Code HalosCognizant
 

Ähnlich wie Recommendations for usage tracking for research networking systems, v.1. July 2013 (20)

AN EFFECTIVE FRAMEWORK FOR GENERATING RECOMMENDATIONS
AN EFFECTIVE FRAMEWORK FOR GENERATING RECOMMENDATIONSAN EFFECTIVE FRAMEWORK FOR GENERATING RECOMMENDATIONS
AN EFFECTIVE FRAMEWORK FOR GENERATING RECOMMENDATIONS
 
dabblr_report_final
dabblr_report_finaldabblr_report_final
dabblr_report_final
 
IRJET- Strength and Workability of High Volume Fly Ash Self-Compacting Concre...
IRJET- Strength and Workability of High Volume Fly Ash Self-Compacting Concre...IRJET- Strength and Workability of High Volume Fly Ash Self-Compacting Concre...
IRJET- Strength and Workability of High Volume Fly Ash Self-Compacting Concre...
 
IRJET- Implementing Social CRM System for an Online Grocery Shopping Platform...
IRJET- Implementing Social CRM System for an Online Grocery Shopping Platform...IRJET- Implementing Social CRM System for an Online Grocery Shopping Platform...
IRJET- Implementing Social CRM System for an Online Grocery Shopping Platform...
 
Intranet focus search strategy a z - from Findability Day 2014
Intranet focus search strategy a z - from Findability Day 2014Intranet focus search strategy a z - from Findability Day 2014
Intranet focus search strategy a z - from Findability Day 2014
 
IRJET- Rating based Recommedation System for Web Service
IRJET- Rating based Recommedation System for Web ServiceIRJET- Rating based Recommedation System for Web Service
IRJET- Rating based Recommedation System for Web Service
 
Enhanced Web Usage Mining Using Fuzzy Clustering and Collaborative Filtering ...
Enhanced Web Usage Mining Using Fuzzy Clustering and Collaborative Filtering ...Enhanced Web Usage Mining Using Fuzzy Clustering and Collaborative Filtering ...
Enhanced Web Usage Mining Using Fuzzy Clustering and Collaborative Filtering ...
 
A Systematic Literature Survey On Recommendation System
A Systematic Literature Survey On Recommendation SystemA Systematic Literature Survey On Recommendation System
A Systematic Literature Survey On Recommendation System
 
Query Recommendation by using Collaborative Filtering Approach
Query Recommendation by using Collaborative Filtering ApproachQuery Recommendation by using Collaborative Filtering Approach
Query Recommendation by using Collaborative Filtering Approach
 
NOW! Get the internet to work for you!
NOW! Get the internet to work for you!NOW! Get the internet to work for you!
NOW! Get the internet to work for you!
 
IRJET- Virtual Business Analyst using a Progressive Web Application
IRJET- Virtual Business Analyst using a Progressive Web ApplicationIRJET- Virtual Business Analyst using a Progressive Web Application
IRJET- Virtual Business Analyst using a Progressive Web Application
 
IJRET : International Journal of Research in Engineering and TechnologyImprov...
IJRET : International Journal of Research in Engineering and TechnologyImprov...IJRET : International Journal of Research in Engineering and TechnologyImprov...
IJRET : International Journal of Research in Engineering and TechnologyImprov...
 
APPLICATION OF SENTIMENT ANALYSIS IN WEB DATA ANALYTICS
APPLICATION OF SENTIMENT ANALYSIS IN WEB DATA ANALYTICSAPPLICATION OF SENTIMENT ANALYSIS IN WEB DATA ANALYTICS
APPLICATION OF SENTIMENT ANALYSIS IN WEB DATA ANALYTICS
 
VIVO 2014: Google Analytics, Email Marketing & Vanity to Increase User Engage...
VIVO 2014: Google Analytics, Email Marketing & Vanity to Increase User Engage...VIVO 2014: Google Analytics, Email Marketing & Vanity to Increase User Engage...
VIVO 2014: Google Analytics, Email Marketing & Vanity to Increase User Engage...
 
VIVO 2014: Google Analytics, Email Marketing & Vanity to Increase User Engage...
VIVO 2014: Google Analytics, Email Marketing & Vanity to Increase User Engage...VIVO 2014: Google Analytics, Email Marketing & Vanity to Increase User Engage...
VIVO 2014: Google Analytics, Email Marketing & Vanity to Increase User Engage...
 
Carpenter, "RA21 Update"
Carpenter, "RA21 Update"Carpenter, "RA21 Update"
Carpenter, "RA21 Update"
 
LyonALMProposal20041018.doc
LyonALMProposal20041018.docLyonALMProposal20041018.doc
LyonALMProposal20041018.doc
 
LyonALMProposal20041018.doc
LyonALMProposal20041018.docLyonALMProposal20041018.doc
LyonALMProposal20041018.doc
 
Google Analytics ( Digital Marketing ) Dashboard , Sessions, Technology
Google Analytics ( Digital Marketing ) Dashboard , Sessions, Technology Google Analytics ( Digital Marketing ) Dashboard , Sessions, Technology
Google Analytics ( Digital Marketing ) Dashboard , Sessions, Technology
 
Using Ontology to Capture Supply Chain Code Halos
Using Ontology to Capture Supply Chain Code HalosUsing Ontology to Capture Supply Chain Code Halos
Using Ontology to Capture Supply Chain Code Halos
 

Mehr von lesliey

Designing Digital Products for Diverse Populations
Designing Digital Products for Diverse PopulationsDesigning Digital Products for Diverse Populations
Designing Digital Products for Diverse Populationslesliey
 
Delivering AI in real time to clinicians
Delivering AI in real time to cliniciansDelivering AI in real time to clinicians
Delivering AI in real time to clinicianslesliey
 
Research Networking SEO state of the union 2015
Research Networking SEO state of the union 2015Research Networking SEO state of the union 2015
Research Networking SEO state of the union 2015lesliey
 
Incentivize use of UCSF Profiles
Incentivize use of UCSF ProfilesIncentivize use of UCSF Profiles
Incentivize use of UCSF Profileslesliey
 
Extending UCSF Profiles with OpenSocial
Extending UCSF Profiles with OpenSocial Extending UCSF Profiles with OpenSocial
Extending UCSF Profiles with OpenSocial lesliey
 
Intro to UCSF Profiles
Intro to UCSF Profiles Intro to UCSF Profiles
Intro to UCSF Profiles lesliey
 

Mehr von lesliey (6)

Designing Digital Products for Diverse Populations
Designing Digital Products for Diverse PopulationsDesigning Digital Products for Diverse Populations
Designing Digital Products for Diverse Populations
 
Delivering AI in real time to clinicians
Delivering AI in real time to cliniciansDelivering AI in real time to clinicians
Delivering AI in real time to clinicians
 
Research Networking SEO state of the union 2015
Research Networking SEO state of the union 2015Research Networking SEO state of the union 2015
Research Networking SEO state of the union 2015
 
Incentivize use of UCSF Profiles
Incentivize use of UCSF ProfilesIncentivize use of UCSF Profiles
Incentivize use of UCSF Profiles
 
Extending UCSF Profiles with OpenSocial
Extending UCSF Profiles with OpenSocial Extending UCSF Profiles with OpenSocial
Extending UCSF Profiles with OpenSocial
 
Intro to UCSF Profiles
Intro to UCSF Profiles Intro to UCSF Profiles
Intro to UCSF Profiles
 

Kürzlich hochgeladen

How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxYounusS2
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIUdaiappa Ramachandran
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfAnna Loughnan Colquhoun
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 

Kürzlich hochgeladen (20)

How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptx
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 

Recommendations for usage tracking for research networking systems, v.1. July 2013

  • 1. 1 Recommendations for Research Networking Systems (RNS) Usage Tracking – Version 1, July 2013 Foreword A subgroup of the CTSA Research Networking Affinity Group (RNAG) has developed a set of recommendations for SGC3, focused on measuring the usage of research networking systems (RNS). Introduction Biomedical institutions have widely deployed institutional expertise discovery and researcher networking systems (RNS), including VIVO, Profiles RNS, and SciVal Experts. Institutions have invested in networking systems drawn by the promise of discovering complementary expertise, enabling collaboration, facilitating funding opportunity matching, and other key needs for accelerating translational research. Evidence is currently limited on the extent of usage of RNS tools and the extent to which they promote research-oriented networking. However, it is possible to measure usage of these systems using web traffic analytics tools, via analysis of web logs. The most well-known of these tools is Google Analytics, but a number of attractive alternatives are emerging.1 In addition, qualitative methods can be employed to collect feedback with respect to value and utility, especially from the user perspective. The recommendations here reflect current measures being used by 8 CTSA consortium member institutions and one commercial entity: • Harvard University • Medical University of South Carolina • Northwestern University • University of Iowa • University of Minnesota • University of California, San Francisco* • Vanderbilt University • Washington University* • Elsevier* This set of recommendations is a start. This document is meant to be a living and changing set of recommendations that will inform institutions and SGC3 of the value and power of research networking systems for accelerating translational research. For any specific institution, adopting a subset of these measures can help to shape strategy, decision making, and practical tactics. 1 Some notable examples include Google Analytics (http://www.google.com/analytics/), Site AI (http://www.siteai.com/), Woopra (http://www.woopra.com/), with NewRelic (http://newrelic.com ) being used for performance monitoring. See Appendix 1 for more on tools. *Authored Version 1 of this document
  • 2. 2 The structure of this document is as follows: The set of recommended RNS Usage Measures below a description of each measure and an accompanying rationale and explanation as to why the measure should be tracked. Institutional examples and case studies then describe successful strategies and tactics that have evolved from the analysis of the data, which range from informing the overall marketing strategy for adoption, to targeting specific faculty meetings for presentations and training, to gaining stakeholder and financial support. For ease of discussion and presentation, Google Analytics will be used as an example of an analytics tool to track tool usage. Recommended RNS Usage Measures For all measures: Track trends R1 Establish a baseline and track over time Track the same measures over time to illustrate patterns and responsiveness to actions (e.g., a new marketing campaign). Set expectations R2 Set expectation for trend over time For trends, set expectations for how the measures will trend. At UCSF, while they hope to see the number of 2+ minute on-campus visits per day increase over time, they expect this measure to plateau over time if the majority of the core user community comes to make appropriate use of the tool whenever needed. Separate internal from external usage R3 Distinguish between traffic internal and external to the institution It easy to distinguish between institutional and extra-institutional traffic with Google Analytics by looking at the visitor’s service provider. By creating advanced segments that show only traffic from your institution’ service provider(s) vs. those from all other service providers, you can filter UCSF set up two advanced segments: users with service provider “university of california san francisco,” and those with any other service provider. They then look at every statistic via both segments. This shows, for example, that they get over ten times as many off- campus visits as they do on-campus visits, and that on-
  • 3. 3 every statistic through an internal vs. external lens. (Be sure to that your team’s own accesses to the site are being ignored by filtering out your IP addresses as an account-level filter.) campus visitors are more likely to arrive via campus websites than external visitors. The basics: Implement Google Analytics R4 Track basic web traffic statistics via Google Analytics Google Analytics affords easy access to basic usage analytics s so you can determine: 1. Visits to your site 2. Page views 3. Unique visitors 4. % new vs. % returning visitors 5. Bounce rate on home and profile pages 6. # pages looked at per visit 7. Average visit duration Understand search traffic R5 Use search traffic statistics to inform marketing and outreach opportunities Google Analytics shows not only the amount of traffic from public search engines like Google and Bing, but also the keywords that brought users to your site. This can be used to understand search behavior and intent. For example, if search traffic is low, it might be an opportunity do basic search engine optimization; or if users are coming to the site after searching for people’s name on Google, it may provide an opportunity to highlight their networks on the landing page. UCSF received only 2,500 visits a month from external search engines during launch month. They subsequently did search engine optimization work, and now receive over 40,000 visits a month from external search engines — 72% of all visits.
  • 4. 4 Delve into referral traffic R6 Use referral traffic statistics to inform marketing and outreach opportunities Google Analytics also provides data on which website refer the most traffic to your system. By understanding these patterns, you can look at institutional traffic and sites and it can help to inform your outreach and marketing efforts. For example, if a web resource popular in the local research community does not refer much traffic it could be an opportunity for outreach to get links back to your system added at the appropriate spots. Comparisons across institutions can also help here. For example, the 2nd highest referring site at UCSF is the campus directory. UCSF partnered with the campus group responsible for each person’s page on the main online people directory to link back to the corresponding UCSF Profiles page; the result was a substantial increase in traffic to UCSF Profiles -- about 1,700 visits a month. Understand what’s popular R7 Understand traffic patterns to different parts of the site to understand preferential usage, and prioritize outreach and development Google Analytics allows users to see what parts of the site are getting the most traffic, and how users behave on each page, e.g. by looking at bounce. Understanding these measures is critical to understanding usage of the site. A month after launch, UCSF staff were surprised to discover that only 9% of visits begin on the home page, with most users arriving directly on profile pages. This provided the impetus to spend substantially more time improving data on profile pages, and deemphasize tweaks to the home page. Look at your geographic breakdown R8 Examine regional and global traffic patterns to inform potential economic development impact Google Analytics’s audience location demographics will show your traffic from within your region, state, nation, and the world. If you’re like many institutions, you might be surprised at how much of the traffic to your RNS comes from outside of your institution, let alone your state. In addition, look both at city, (for the institution’s city and the surrounding suburbs), as well as network provider which is often be linked to the university, sometimes even down to school or college within the university.
  • 5. 5 Benchmarking: Define meaningful engagement R9A & R9B Track the number of “X+” minute visits to the RNS from within the institution (where “X+” represents the length of time you deem meaningful) Track the number of visits where users from within the institution view at least “Y” pages (where “Y” represents the # pages you deem meaningful) For context, we are using the model of an RNS that does not require log-in to search the network. As such, discuss and define a surrogate for what constitutes meaningful engagement with your RNS. Once defined, these measures can be found via Google Analytics; complex queries can be used through advanced segments and/or custom dashboards. At UCSF, they pay particular attention to the average number of visits from within the UCSF campus network lasting 2 minutes or longer (Google Analytics defines a visit as a period of continuous usage where the gap between subsequent page views or interactions is under half an hour). Because users do not need to log in to search UCSF Profiles, they use these 2+ minute on- campus visits as a proxy for deep active use of the product by researchers, administrators, and other staff on campus. Benchmark the traffic R10 Benchmark relative to other campus web properties, both on and off campus One way to infer relative usefulness of your RNS is to track and benchmark the RNS traffic relative to other campus properties such as the directory or the main institutional web site. At UCSF, the RNS gets as much as 60% of the visits of the campus directory from all visitors, and 12% of the visits of the campus directory from UCSF campus visitors. This signals the relative importance of this system as a resource.
  • 6. 6 Segment populations R11 Segment and compare populations of profiled users, and of visitors (when possible) Using the data in the RNS, segment populations of engaged users to determine differing or similar patterns of use. The differences can inform outreach and training opportunities, and value can be inferred. Examples of segments: faculty rank, department, school, etc. Segments with lesser adoption could be candidates for outreach and training, whereas segments with greater adoption are candidates for interviews on why and what is going well. If the system is behind a login wall, there may also be opportunities to segment end users. By looking at UCSF Profile adoption rates (where adoption is defined as a profile that has some curation or customization completed such as an uploaded photo), they found that faculty at all levels – junior to senior – have adopted their profiles in close to similar proportions. This seems to indicate equal importance of the tool across the segments, and helps as they plan outreach and marketing for the system. Analyze where people click R12 See what your users are clicking on and use the data to improve your product A heatmap is an easy way to understand what users do on your site. It’s a visual representation showing where users click and what they do. This can help you determine what aspects of your RNS are most popular & what is overlooked. Google Analytics provides this data via in-page analytics feature, or a separate heatmap tool may be used. If certain classes of links on your site are being systematically ignored, this data can help inform user interface changes and/or marketing messages. UCSF has installed an inexpensive heat mapping tool and used that along with Google Analytics in-page analytics to prioritize improvements to their RNS home page.
  • 7. 7 Qualitative measures: R13 Measure traffic impact resulting from various communications (e.g., PR and marketing campaigns) and then judiciously use the successful strategies in the future Analyze traffic to your RNS before and after various marketing campaigns such as emails or trainings. Try different messages using A/B testing and measure which ones are most successful. Partner with other groups on campus so all of the messages don’t come from one source. Measure traffic impact from these various sources. UCSF noticed a big spike in traffic after the Executive Vice Chancellor & Provost (EVCP) sent out an email announcing UCSF Profiles. This traffic spike was larger than any other email campaign and speaks to the reach and value of communications from UCSF’s EVCP. Also, at UCSF, groups across campus from the Library to University Relations have helped to spread the word about UCSF Profiles. These groups are on board to help promote UCSF Profiles because data mined from UCSF Profiles have facilitated those groups’ projects. Survey users on value & usage R14 Survey users for direct feedback; use the feedback to inform strategy and gain support for your system To really find out how people use the system and what value they garner from your RNS, you will need to ask them. The feedback will identify gaps and improvements, inform planning and hopefully help gain support to sustain and enhance your system. A number of SciVal Experts’ sites track numbers of inquiries and/or profile corrections as additional evidence of how many people are really looking at the site and its content. REACH-NC (NC state SciVal Experts portal) created a contact form that users fill out when trying to contact a researcher and they do not display direct contact information. The contact form goes to the campus liaison who coordinates the contact. In addition, a copy of the request goes to the REACH-NC Executive Director and Program Coordinator so they can track request and monitor connections. In addition, inquiries and feedback from profiled researcher come to a group mail box that the ELS Account Development Manager personally manages, instead of directly going to the general ELS helpdesk.
  • 8. 8 Arizona State University had a ‘Submit Feedback’ button on the Experts site that linked to an end-user survey. (They felt that the information they were receiving was biased toward the negative and also unhelpful, so they discontinued use of the survey.) Note: this offers up a lesson on being specific about what kind of feedback you solicit, and not necessarily a warning against soliciting feedback in this way UCSF has implemented a pop up survey on UCSF Profiles that while annoying to many, has resulted in the collection of substantive feedback such as: “Great resource for finding potential research collaborators and for PhD dissertation committees” “I am hoping that it just helped me find a mentor...” “It has helped find new nursing research problems” Interventional networking: Given the significant lag time between identifying a potential collaborator and the first outcome of a successful collaboration, such as a published paper or even a funded grant award, it is difficult to directly link RNS usage to interventional networking. Additional studies need to be done to link short-term RNS usage patterns to longer-term outcome measures such as: • Number of new introductions/meetings/active collaborations • Interdisciplinary nature of new introductions/meetings/collaborators • Resultant joint grant applications and publications Enable & track collaboration right from your RNS
  • 9. 9 R15 Enable collaboration from within your RNS Identify potential collaborations and enable communication between researchers directly from your RNS. The Michigan Corporate Relations Network (MCRN) portal functions like a multi-university Experts site that allows for searching across participating universities in Michigan (U of MI, University of Michigan’s Dearborn Campus, Michigan State University, Wayne State University at launch, and Western Michigan University shortly thereafter, and likely also Michigan Tech sometime after that). Thinking of it like an ecommerce metaphor, the site allows a businessperson interested in working with state of Michigan researchers to add researchers of interest to a list (‘shopping cart’) and the call to action (‘checkout’) process routes the researchers of interest along with a user-entered problem statement to the appropriate university business relations offices. These will be tracked centrally and by each office, hopefully with some evaluation of quantity of contacts and ‘quality’ of contacts (was it a legitimate research collaboration interest or a solicitor trying to sell something? did it turn into a collaboration?) UCSF has an operational pilot that has integrated UCSF Profiles with Salesforce Chatter (Chatter is UCSF’s enterprise social networking tool). This allows one to “follow” a person directly from UCSF Profiles and also allows a person to select a few people of interest and create a “Chatter Group” directly from UCSF Profiles.
  • 10. 10 Appendix 1 Notes / community comments on analytic tools From various sources, including: https://wiki.duraspace.org/display/VIVO/VIVO+Web+Analytics Analytic Tool Comments Google Analytics Used as default Profiles solution Also used as default VIVO solution – see googleAnalytics.ftl FreeMarker template Woopra "all-in-one web application performance tool" Paul Albert at Weill notes: "I've personally used Woopra before. It seems to be better than Google Analytics about tracking outbound links and for watching people click through your site in real time. (Paul Albert, WCMC) New Relic: For performance monitoring, we use New Relic, which you can get a sense of here: https://dl.dropboxusercontent.com/u/2014679/NewRelic.png and https://dl.dropboxusercontent.com/u/2014679/EmailAlert.png. The best part about New Relic is that it will send a notice to other team members when you have tried to execute several malformed SPARQL queries in a row." (Paul Albert, WCMC) Site AI: "The problem with dashboards is they don’t directly provide insights or deliver knowledge about the data. Even worse, most visualizations require the user go through the mental exercise of interpreting the results. Site Ai does the analysis for you and presents the information in plain English." example: http://www.siteai.com/share/2672/2013/5/22/fcsjiggewdpechaafwpfjedlkvjhvipu (Mark Fallu, research-hub.griffith.edu.au at Griffith University) Mixpanel: "Page view counts are popular because they are easy to report, but ultimately cannot tell you how engaged your visitors are. Mixpanel lets you measure what customers do in your app by reporting actions, not page views." (Alex Viggio, University of Colorado Boulder) Chartbeat See demo CrazyEgg CrazyEgg provides data on what links users click on once they arrive on key pages or selected profiles. Click statistics are provided in several formats: heatmap, scrollmap, overlay, and ‘confetti.’ This allows us to improve design of page elements strategically to encourage interaction. (Leslie Yuan, UCSF) Qualaroo User feedback is collected via a popup survey tool; detailed reports are available online. We ask “How has UCSF Profiles helped you?” to solicit success stories and better understand use cases, but the survey functions as a customer service access point as well. Users may enter an email address if they’d like a response. (Leslie Yuan, UCSF)