We believe that the quality and efficiency of all phases of the clinical and translational research (CTR) process can potentially be increased by using digital practices and tools in open and networked contexts. However, most CT researchers lack the training to take advantage of the benefits that the Internet and the social Web provide. Standardized training in digital practices and tools (Digital Scholarship) to conduct CTR has not been formalized through structured curriculum, learning approaches, and evaluation. Our overall goal is to develop a robust curriculum to train CTR researchers in digital scholarship. Here we present preliminary data from a qualitative study that describes the range of key stakeholders’ perspectives on the need to: (A) formalize educational efforts in digital scholarship among CTR trainees; and (B) develop an educational framework that defines core competencies, methods, and evaluation methods. Presented at Translational Science 2018 conference in Washington, DC on April 20, 2018.

1. INTRODUCTION
• The quality and efficiency of all phases of the clinical and translational research (CTR) process can potentially
be increased by using digital practices and tools in open and networked contexts.
• More traditional phases include hypothesis generation, recruitment efforts, analysis, and more innovative
phases include discoverability of research outputs online through search engine optimization and
crowdsourcing.
• Standardized training in digital practices and tools (Digital Scholarship [1,2,3]) to conduct CTR has not
been formalized through structured curriculum, learning approaches, and evaluation.
Overall goal: to develop a robust curriculum to train CTR researchers in digital scholarship.
• Here we present preliminary data from a qualitative study that describes the range of key stakeholders’
perspectives on the need to: (A) formalize educational efforts in digital scholarship among CTR trainees; and
(B) develop an educational framework that defines core competencies, methods, and evaluation methods.
RESULTS
METHODS
Participants: We recruited via email from June-November, 2017 using purposive and snowball sampling
methods across Digital Scholarship Experts, KL2/TL1 Educators and Trainees, and SC CTSI administrators.
Methods: Online focus groups were conducted using a semi-structured, open-ended interview guide via Google
Hangouts and a conference call interface. Transcripts were audio recorded, transcribed verbatim, and analyzed
using Atlas.ti (Version 8.1.3).
Analysis/Coding: Initial “a priori” codes and definitions were based on the interview guide and “Master’s CTR
Core Competency Areas” [4]. Two coders identified emergent codes and concepts using a framework approach
[5].
Reliability: The coders independently evaluated the first four transcripts to resolve discrepancies and reach
consensus. Inter-rater reliability was used to evaluate consistency between the two coders (kappa=0.68). Coded
transcripts were imported into a single hermeneutic unit in Atlas.ti and coding reports were created.
CONCLUSIONS
• There was consistent support for a structured program to train CTR scientists to develop competency in
digital research practices and approaches.
• Participants considered all existing CTR and additional suggested competency areas relevant.
• New training areas suggested by participants: network analysis, citizen science, crowdsourcing, machine
learning, digital health training, computing, data management and licensing, among others.
• An education program focused on digital scholarship should include a step-wise approach to meet different
research and training goals. allowing attendees to increase their awareness through flexible online learning
modules and lectures as well as specialized in-person and hands-on experience, also including consultations,
coaching, and networking.
• The development of a clear definition for “digital scholarship” in the context of CTR, program goals and
evaluation metrics represents challenges that require further exploration.
• Next steps: To establish consensus among the focus groups attendees regarding necessary components of
a CTR-focused curriculum through a follow-up questionnaire.
REFERENCES
1. Ayers, E. (2004). Doing Scholarship on the Web: Ten Years of Triumphs — and a Disappointment. Journal of Scholarly Publishing, 35(3), 143–147.
http://doi.org/10.3138/jsp.35.3.143. Accessed 16 May 2017
2. Andersen, D., & Trinkle, D. (2004). Valuing digital scholarship in the tenure, promotion, and review process - A survey of academic historians. In D. Andersen (Ed.), Digital
Scholarship in the Tenure, Promotion and Review Process (pp. 61–77). New York: M.E. Sharpe.
3. Rumsey, A. S. (2011). New-Model Scholarly Communication: Road Map for Change. Web link: http://uvasci.org/institutes-2003-2011/SCI-9-Road-Map-for-Change.pdf.
4. CTSA consortium. Core Competencies for Clinical and Translational Research. 2015. Web link: https://clic-ctsa.org/resources/education/core-competencies.
5. Ritchie, J & Spencer, L (1994) Qualitative data analysis for applied policy research Analysing Qualitative Data 173 – 194 Bryman A Burgess RG London Routledge.
ACKNOWLEDGEMENTS
This work was supported by the grant UL1TR001855 and KL2 TR00185402 from the National Center for Advancing Translational Science (NCATS) of the U.S. National Institutes
of Health. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
The image part with relationship ID rId4 was not found in the
file.
Perspectives on Increasing Competency in Using Digital Practices and Approaches to
Enhance Clinical Translational Research: Preliminary Results of a Qualitative Study
Reuter K1,2, Simpson K1, Le N2, Bluthenthal RN1, Patino CM1,2
1 Keck School of Medicine of USC, Department of Preventive Medicine, University of Southern California (USC)
2Southern California Clinical and Translational Science Institute (SC CTSI), University of Southern California (USC)
Study Population: 164 individuals were approached and 66 participants accepted to participate and included: 1.
International experts from academic and private sector institutions with experience in using digital practices and
approaches in research (n=36), 2. KL2/TL1 Educators (n=8), 3. KL2/TL1 Trainees (n=13), and 4. SC CTSI
Administrators (n=9). The average age was 41.2 years SD ± 9.3, 59% females, 97%, non-Hispanic, 72% White,
and 67% with doctoral degree. Of the 66 participants, 85% reported experience in teaching digital practices and
approaches in research, although 70% were currently not teaching in this field.
RESULTS
Table 1. “Digital scholarship” definition. (Broad definition was provided: “Digital scholars can see the options made
available to them, their students, and their research by new, digital technologies. They have a rationale for taking the
time to learn these technologies and for applying them to their scholarship.”)
Table 3. Competency areas. (Existing CTR competency areas were provided: Literature review, Literature critique, Study
design, Research implementation, Sources of error, Statistical approaches, Biomedical informatics, Regulatory support,
Responsible conduct of research, Scientific communication, Cultural diversity, Translational teamwork, Leadership,
Cross-disciplinary training, Community engagement. New suggested areas were provided: Obtain research
funding/crowdfunding, Research participant recruitment, Internet research ethics, Lab management, Project
management, Performance evaluation, Use of novel scholarly metrics/Altmetrics, Online reputation management.)
Table 4. Educational didactics.
Table 2. Metrics.
Themes Subthemes Representative Quotes
General
definition
• Term clarification
• New discipline vs. additional
competency area for CT
researchers
• Broad term including many
activities and users
• More focused definition
• Separate scholar vs. user
• “I think that’s a very succinct definition because technology is forever evolving. I
think the idea of standing still and not looking for fresher ways of working, even if
you use a (…) technology to solve a tiny part of your process, then it’s a useful
kind of idea. I think with technology you can’t stand still if you want to engage with
it. You have to be prepared to get things wrong. And you have to be prepared to
experiment. So I think that that definition you’ve got kind of captures [this] (…).”
• “I would say it’s basically [that] digital scholarship is the use of digital objects in the
process of scholarship, which involves finding them, using them, producing them,
sharing them. It would be any object that can be represented as a series of bits.”
• “I don’t think everybody who uses this is a scholar.”
Definition
in CTR
context
• CTR data
• CTR populations
• “The big thing for me with CTR has to do with the nature of privacy and security
when it comes to data. (…) It is a particular issue when you’re dealing with patient
confidentiality.”
Themes Subthemes Representative Quotes
CTR
competency
areas
• All CTR competency areas
amenable to align to a digital
practices/approach
• “I think there’s something digital that can run through all of these.”
• “The idea of taking every aspect of the competency and finding out if there
are digital tools that relate to each aspect of those competencies I think
would be highly beneficial.”
• “I think that all of these topics, the core competency areas, including the
ones that you’ve listed as additional, are actually very important to address
from day one. I think that there are certain, basically you should address all
of the questions, and find a way to allow people who wanna dig into certain
topics to do more maybe on their own time, because that is a lot of ground
to cover, if you look at the whole list of competencies.”
Competency
level
• Basic vs. advanced • "On a very basic level, it seems to me that you should know something
about how digital methods apply in the context of all of these competencies
on a very basic level. Maybe knowing how to become advanced in those
areas is the advanced section that can be self-study if designed correctly.”
Themes Subthemes Representative Quotes
General
didactic
approach
• Concepts vs. tools • “[It’s] not how we do the training but what we do the training on. There’s I think a
pretty profound difference between teaching people ideas and using specific tools to
reinforce the learning about those ideas, versus teaching people just how to use
specific tools. It’s likely that people who come out of that course are not going to be
using exactly those same tools or when they go to work somewhere.”
Online
training
• Diverse scholar-centric
formats and platforms
• “You can provide information in a didactic form (…) some kind of video that anybody
can access. But that’s just gonna be an introduction and that’s just gonna lead people
towards the right tool, or maybe help them troubleshoot specific questions, but then
they’ll still need some kind of coaching.”
In-person
training
• Human interaction to
consolidate learning
• Hands-on experience
• “We try to create (…) spaces for them to do that work so that they’re not making
horrific mistakes in public, but that really enable them to do the hands-on exploring.
I’m providing the guidance in how to use [digital tools] appropriately and effectively.”
• “Connect people who are using the same tools so that they can help each other.”
• “Where you meet with someone, you talk about your research and your needs, and
they then identify the types of resources that would be useful for you and here’s ways
to get that information. ‘Cause I think it is very individual in terms of what people need
to learn or what they need for their research projects.”
Lecture • Flipped classroom model • “It’s more efficient for [students] to watch any lectures at home and not for me to
lecture in class, because that way they can go fast on the things they know, slow it
down for the things they don’t understand, watch it on their own time whenever they
have time. I like the flipped classroom model, where anything that’s kind of a didactic
lecture, you put a module up and they watch it on their own, but then anything that’s
hands on you do in class.”
Themes Subthemes Representative Quotes
General
approach
and issues
related to
evaluation
• Focus on program goals
and desired outcomes
• Complexity of
competencies
• Lack of understanding of
metrics
• Less focus on metrics
• “I think the harder part is how do you measure whether a participant now using this
digital scholarship has actually enhanced their work? How do you quantify that?”
• “Ultimately I think measuring outcomes to the extent that you can rather than just
usage would be most helpful.”
• “I don’t necessarily think that someone should be judged on how well it ended up
working, especially when you’re just trying to master a skill.”
Suggested
metrics
• Focus on different
metrics and outcomes
• Traditional vs. non-
conventional metrics
• “Maybe I would go back to literacy. I think that’s such a great way to phrase it. Maybe
look at what’s the metric that’s accepted when we consider somebody to become
literate.”
• “Maybe build into the curriculum they have to demonstrate somehow that they’re
actually doing what they say they’re, not just a questionnaire, but to have them tell
you what digital tools they used to accomplish various things.”
• “Ultimately it’d be great to know whether or not there’s an actual effect two or three
years down the road to be able to take data on the people that went through the
program and published and whether or not they’re getting more hits on their articles
or more site views on their projects than people who didn’t go through the program.