While a number of professional organizations have produced valuable guidelines for evaluation of digital work, many colleges and universities have yet to establish clear protocols and practices for applying them. Alison Byerly, College Professor and former Provost and Executive Vice President at Middlebury College, who has co-led workshops on evaluating digital scholarship at the MLA convention, will review major issues to be considered in the evaluation of digital work, such as: presentation of medium-specific materials, documentation of multiple roles in collaborative work, changing forms of peer review, and identification of appropriate reviewers. She will then talk briefly about how these issues can best be approached from the perspective of the candidate who wishes to present his or her work effectively to review committees, as well as from the perspective of colleagues who wish to provide a well-informed evaluation of such work.

1. Evaluating Digital Scholarship
Alison Byerly, College Professor, Middlebury
College
NITLE Seminar, October 10, 2012

2. Evaluating Digital Scholarship
NITLE Seminar, Oct. 10, 2012
Alison Byerly , Middlebury College
http://blogs.middlebury.edu/alisonbyerly/

3. Challenges of evaluating digital scholarship
May differ from traditional scholarship in:
(1) analytic approach or content
and/or
(2) format, medium, method of publication or dissemination
Digital humanities: not really a field or discipline - an approach, an array of
practices. These practices are often more typical of sciences and social
sciences than of traditional humanities
• data-focused or object-focused
• visualization and spatial analysis
• collaborative; -open-access and non-proprietary
• process-oriented rather than product-oriented

4. Evaluation
• Good evaluation depends on shared context: clear expectations, defined
roles, recognized measures of success
• Both parties – candidate and evaluator(s) –have a role to play in
establishing the appropriate context
• Ideally, this work begins the moment a new colleague is hired – if not
sooner

5. Essential tasks for digital scholars
Educate your colleagues
• Define the audience of peers addressed by your work, and actively engage
that audience
• Document your specific role and effort, particularly if it is collaborative
• Explain the significance of specific achievements, recognition, or prestige
markers
• Look for opportunities to create narratives describing your work

6. Essential tasks for evaluators
Educate yourselves
• Review and assess any project in the medium in which it was
created.
• Recognize the intrinsically collaborative nature of digital
projects.
• Consult specialists in relevant disciplines regarding the various
components of a candidate’s work
• Assess candidate’s work in relation to overall institutional
expectations of quality, recognizing that specific metrics of
success will differ in new areas

7. Guidelines from professional organizations
• MLA (Modern Language Association)– Guidelines for Evaluation of Work
in Digital Humanities and Digital Media
http://www.mla.org/guidelines_evaluation_digital
• AHA (American Historical Association) – Suggested Guidelines for
Evaluating Digital Media Activities in Tenure, Review, and Promotion
http://www.historians.org/perspectives/issues/2001/0110/0110pro1.cfm
• 2011 volume of MLA's Profession contains some excellent essays (
http://www.mlajournals.org/toc/prof/2011/1),
• -NINES ‘ Evaluating Digital Scholarship site, "Guidelines for Promotion and
Tenure Committees in Judging Digital Work" (
http://institutes.nines.org/docs/2011-documents/guidelines-for-promotion-and
).

8. New publication and presentation formats
websites, online manuscripts, databases, apps…..
(1) Website
NINES: Nineteenth-Century Scholarship Online http://www.nines.org/
992,065 peer-reviewed digital objects from 119 federated sites:
The Old Bailey Online is a fully searchable edition of the largest body of texts
detailing the lives of non-elite people ever published, containing 197,745
criminal trials held at London's central criminal court.
The Yellow Nineties Online publishes digitized facsimile editions of a select
collection of periodicals…a rich historical archive of paratextual materials
related to the production and reception of these periodicals…also
publish peer-reviewed scholarship: biographies of the writers, authors,
publishers, and others associated with the period; scholarly
introductions and commentary; and essays on our process of building
the site and encoding its digital objects. All documents are marked-up
and fully searchable.

9. (2) In-process online manuscript
Complex TV: The Poetics of Contemporary Television Storytelling by Jason
Mittell
. http://mediacommons.futureofthebook.org/mcpress/complextelevision/
‘This site hosts the peer-to-peer review of the in-progress manuscript
Complex TV: The Poetics of Contemporary Television Storytelling by
Jason Mittell (copyright 2012). The book proposal was posted in March
2011, with individual chapters released in serialized installments starting
in Spring 2012. The project is currently under review contract with NYU
Press, who has allowed me to post the pieces here for pre-publication and
open-review. The draft manuscript with comments will continue to live
online here, even after the book has been published.’
See also JM’s blog post, “Thoughts on Blogging for Tenure”
http://justtv.wordpress.com/2012/01/04/thoughts-on-blogging-for-tenure/

10. (3) Database or program
Docuscope: a text analysis environment with a suite of interactive
visualization tools for corpus-based rhetorical analysis.
http://www.cmu.edu/hss/english/research/docuscope.html

11. Peer review – identifying the relevant scholarly
audience or community
• HASTAC: Humanities, Arts, Sciences, and Technology Advanced
Collaboratory http://hastac.org/ “HASTAC ("haystack") -- a network of
individuals and institutions inspired by the possibilities that new
technologies offer for shaping how we learn, teach, communicate,
create… the digital era provides rich opportunities for informal and formal
learning and for collaborative, networked research that extends across
traditional disciplines, across the boundaries of academe and community,
across the "two cultures" of humanism and technology, across the divide
of thinking versus making….”)

12. Digital Humanities – major outlets
• JDH: Journal of Digital Humanities http://journalofdigitalhumanities.org/
“A comprehensive, peer-reviewed, open access journal that features the
best scholarship, tools, and conversations produced by the digital
humanities community in the previous quarter. JDH offers expanded
coverage of the digital humanities ..by publishing scholarly work beyond
the traditional research article…by selecting content from open and
public discussions in the field…[and] by encouraging continued discussion
through peer-to-peer review.”
• DHQ: Digital Humanities Quarterly
http://www.digitalhumanities.org/dhq/vol/001/1/000007/000007.html
• Anvil Academic – A new all-digital, open-access publisher for the digital
humanities. http://chronicle.com/blogs/profhacker/digital-killed-the-
analog-star-an-interview-with-fred-moody-of-anvil-academic/42936

13. --there are some printed books, too—
•A Companion to Digital Literary Studies, ed. Susan Schreibman and
Ray Siemans. Blackwell, 2008.
•Debates in the Digital Humanities.
Ed. Matthew Gold. U of Minnesota, 2012.

14. Other resources
• MLA 2012 Workshop on Digital Scholarship
http://wiki.mla.org/index.php/WORKSHOP_2012
“The Challenges of Digital Scholarship,” Chronicle of Higher Ed, January 25,
2012 http://chronicle.com/blogs/profhacker/the-challenges-of-digital-
scholarship/38103
• See 10/5/12 #anvil twitterchat re: evaluation of digital scholarship:
http://storify.com/adelinekoh/anvil-academic-publishing-peer-reviewed-
digital-wo#publicize