1. DEFINE
•Construction
of
Bibliography
(>60 sources)
MEASURE
•Surveys
•In-person
tests
•Follow-up
questions
OBSERVE
•Generalizations
•Common
Trends
•Video capture
notes and
observations
ANALYSE
•Correlation
of data with
JSTOR’s
construction
•Design
problems
•Impact from
previous
search
algorithm
changes
IMPROVE
•Constitutes
future work
•Creation of
better
humanities
search
engine
The Evaluation of Digital Research Tools for the Humanities
Lisa Gitelman1, Chris Leslie2, Torsten Suel3, Bill Blake4
Undergraduate Students: Luciana Jaalouk, Andreína Vivas-Thomas
Abstract
An initial online survey was given to a number of subjects in order to provide feedback on their
use of search engines. Anonymously, they then searched for a specific arbitrary topic on JSTOR
where they were asked to give their impressions on their results and to suggest possible
improvements, based on how interdisciplinary their research was. The information gathered was
then analyzed on a broader scale to retrieve common trends that would help identify flaws,
qualities, or possible developments on JSTOR.
About 75% of the remarks made during the engine testing fall into the category of design;
JSTOR’s engine design does not facilitate easy, relevant, and interdisciplinary findings in regards to
the initial topic. Another more subtle realization comes with the construction of the search
algorithm used by JSTOR; this algorithm does not provide the users with enough valuable sources
within the initial searches. By carefully designing the test environment, the findings accurately
provide reliable data due to its preset constraints.
Hence, the solution indeed lies in the algorithm; improvements are needed in several data
related fields. Once all those changes are spotted and made, a new and better search engine for the
humanities could be created which would satisfy most humanities researchers’ needs.
Background
Methodology
Observations
Results
Conclusion
This study takes place to quantify the use of JSTOR (Journal STORage project), a
humanities search engine, as a research tool. This comes down to the examination of
how digital resources might be changing the method and substance of humanistic
inquiry based on how well online databases support the ideals of the humanities.
A shared digital library created in 1995, JSTOR currently includes more than 2,000 academic
journals, with a content that spans 500 years, 50 disciplines, and multiple content types.
Research has been done for the past few years on JSTOR’s interdisciplinary topics and journals;
in fact, it is a “not-for-profit organization with a mission to help the scholarly community take
advantage of advances in technology, and is supported by libraries, scholarly societies, publishers,
and foundations.” {1} This, however, establishes limits on JSTOR’s accessibility and use: it follows an
“Institutional Participation Agreement,” which restricts public access to the journals available and is
oriented to users with strictly educational purposes.
Frequency Percentage
1 14 25.00%
2 5 8.93%
3 15 26.79%
4 10 17.86%
5 11 19.64%
Other (blank) 1 1.79%
Inaccurate searches due to:
 Change in Algorithm (2005)
oPros:
Improved speed
More flexible options
oCons:
 Relevance of citations
 Filtering sources
 Interdisciplinary arrangement
 Unknown JSTOR resources
oAdvanced search
oDfR: Data for Research:
Facilitates bibliometric studies of key
terms and phrases that appear in the
full text of JSTOR articles
Supplement JSTOR's standard search
engine
 Page Layout and Design
 Measure based on majority of undergraduate students
 Surveys:
o33 questions
oProvide feedback on use of search engines:
 Previous experiences
 Search methodologies
 Pre-acquired knowledge
 Anonymous in-person test/search:
oArbitrary topic: American Spiritualism in the 19th Century
o30 minute long search
oRelevancy and content of searches rated
oImpressions and possible improvements given out loud
oData and impressions recorded
 Follow-up questions:
o7 questions after in-person search
oComments:
 Personal experience
 Expectations
 Possible improvement
 64% use JSTOR as digital resource for searches:
oMajority: Google (96%)
 Most users have an average experience with using JSTOR:
oA 3 from 1-5 scale
 JSTOR’s Advanced Search feature most known (66%)
 9% use JSTOR’s internal citation manager for
bibliographies
 27% are familiar with JSTOR’s “Similar works” feature
 Complaints:
oReviews
 Not specific to actual article
oFindings not in English
 Abstracts not in English either
oToo many in depth findings
 No chance for introductory search
 Should have preliminary knowledge
oNo sorting by publishing date
oTop results are oftentimes not relevant
 Design:
oImage view too small
oConfusing page design and layout
 Lack of diversified texts (mostly Journals):
oIronic (JSTOR is a Journal database)
 Not useful for unknown search about unfamiliar
subject
Survey Answer to Question #28:
On a scale of 1 to 5, how much experience do you have with JSTOR?
0%
5%
10%
15%
20%
25%
30%
Arts Business Humanities Mathematics
& Statistics
Regional
Studies
Sciences Social
Sciences
JSTOR DfR, by Discipline Group {3}
On another note, despite being a non-for-profit organization, JSTOR’s search engine setup
process and collaborations with numerous journals is quite costly. In developing a plan for cost
recovery, JSTOR’s first step was to define “the product.” For nonprofit organizations like JSTOR, a
key guidepost for choosing an appropriate display format is the organization’s mission; having a
clear understanding allows for flexibility in a rapidly evolving environment. Once the basic
parameters of what would be offered were framed, the question of economic viability needed to
be addressed. This development of a cost-driven, value-based pricing model should not exceed
“$2.5 million in projected annual expenses plus whatever one-time production related expenses
are incurred in converting the journals.” {2} All told, the direct cost of creating the three-part
representation (page image, electronic table of contents entry and text file) of a journal page in the
JSTOR database is approximately $0.75 to $1.00 per page.
While JSTOR’s library has received a positive reaction form many humanities scholars, the now
voluminous database still presents several complex challenges as to keeping up with today’s
changing technologies. Indeed, defining future better phases of JSTOR reflects the keystone to our
Digital Humanities research project.
Through our research, observations made on the efficiency of JSTOR as a search
engine support the ideals of the humanities; nonetheless our findings do suggest many
limitations as to the setup and interface of this humanities database.
Future suggestions may consist of a short tutorial video in order to educate test
subjects about the available JSTOR tools – such as the advanced search and modified
search – to facilitate the research of complex topics. This is indeed suggested since results
have proven that these resources are unknown and unfamiliar to JSTOR users.
In addition, the DfR (Data for Research), the “free, self-service tool that allows [users]
to select and interact with content on JSTOR” {4} is another tool within the JSTOR database
that could further expand and facilitate all searches done on JSTOR. However, DfR is
unpopular to most users, which leads us to believe that the issue with these hidden tools
is partly due to the particular design and interface of JSTOR.
Future works therefore lie in the construction of JSTOR’s search algorithm which
should encourage users to rely on JSTOR as a humanities database, and not just to ensure
the verity and random availability of an online humanities library.
References
{1} Conditions and Terms of Use." JSTOR, n.d. Web.
{2} ARL Scholarly Communication and Technology Home Page
{3} Sully, Sarah E. "JSTOR: An IP Practitioner's Perspective." D-Lib Magazine, Jan. 1997. Web.
{4} JSTOR. "JSTOR Library Digest." N.p., 2005. Web.
1. NYU College of Arts and Science, English; NYU Steinhardt, Media Culture and Communication. 2. NYU Polytechnic School of Engineering, Science and Technology Studies. 3. NYU Polytechnic School of Engineering, Computer Science and Engineering. 4. NYU College of Arts and Science, English and Drama.