Dcc endeavour-2006

a centre of expertise in data curation and preservation

Experience is a hard teacher…

Curation and the Digital Record
Chris Rusbridge
Endeavor EndUser 2006
Funded by:
This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 2.5
UK: Scotland License. To view a copy of this license, visit http://creativecommons.org/licenses/by-
nc-sa/2.5/scotland/ ; or, (b) send a letter to Creative Commons, 543 Howard Street, 5th Floor, San
Francisco, California, 94105, USA.


"Experience is a hard teacher
because she gives the test first,
the lesson afterwards”
• Vernon Sanders Law, ex baseball player

• (Or perhaps, in the case of digital preservation,
the test occurs long after you are dead?)



Contents
• Curation
• Sustainability
• Data resources
• Preservation & curation issues
• OAIS Review



Curation
• Data increasingly important as evidence
• Experimental verifiability (the basis of science)
• Unrepeatable observations & experiments
(particularly environmental in broadest sense)
• Legal, compliance & transactions
• Cultural resources

• For evidential value, data must be curated



Curation
• “Maintaining and adding value to a trusted
body of digital information for current and
future use”



Lynch remarks
• Closing the 2005 Curation Conference
• 3 views of digital curation
• Collection as a living thing
• Whole life process, evolving object(s)
• Finite process, handover to preservation



•This is what you do!



Sustainability and exit strategy
• Most critical resource for curation: present
and future money supply!
• Plan for the long term, but have a succession
plan
• Sustained approach not project mentality



Sustainability and exit strategy
• Most critical resource for curation: present
and future money supply!
• Plan for the long term, but have a succession
plan
• Sustained approach not project mentality

•This is what you do!


Some illustrations: UK census
• 1881 census (UKDA)
• Hand-written individual return forms: data conversion issue
(reference form available): digitisation and access issues
• 1961 census (TNA/NDAD)
• First using computers to analyse (first major UK-wide
computer project?); individual returns closed until 2062: data
preservation issue!!!
• 2001 census (ONS/CDU)
• Data corrections and adjustments: curation issue



Curation of emails
 Lots of metadata and context (RFC 822)
 Often highly distributed
 Split conversations
 Unknown numbers of copies
 Personal choice of clients

• Legal requirements!
• Controlled filing and controlled deletion
needed…



Online Public Access Catalogues
• Long term, curated databases
• Often high quality (not always)
• Well known interchange standards (MARC),
classification standards (several), name
authorities…
• Still significant problems combining sources



Pre 2000

© Chris Rusbridge
… The database provides indexing of 527
key international English-language business
periodicals including Business Week,
Forbes, The Wall Street Journal, The New
York Times and more. Also included are
product reviews, interviews, biographical
Post 2000 sketches, corporate profiles, reports of
sketches, corporate profiles, reports of
associations, societies and conferences.
Broad areas of coverage include
accounting, acquisitions and mergers,
advertising, banking, chemicals, …
Endeavor EndUser 2006 •Peter Buneman


• Storage
– Redundant, Distributed
– Persistent
– Readable
• Clear standards for citation
• Historical record (old data is useful)
• Well understood ownership/IP

© Chris Rusbridge … The database provides indexing of 527
•Storage periodicals including Business Week,
–Single-source Forbes, The Wall Street Journal, The New
–Volatile York Times and more. Also included are
–Centralised sketches, corporate profiles, reports of
sketches, corporate profiles, reports of
–Internal DBMS format associations, societies and conferences.
•No standards for citation Broad areas of coverage include
•No historical record accounting, acquisitions and mergers,
•Mind-boggling legal issues advertising, banking, chemicals, …
Endeavor EndUser 2006 •Peter Buneman


TWOMASS (Infrared)
SDSS (Visual)
Endeavor EndUser 2006 Slide from Rajendra Bose


Endeavor EndUser 2006 Slide from Rajendra Bose


Example…
• National Virtual Observatory
• Johns Hopkins press release: “Scientists working to create the
NVO, an online portal for astronomical research unifying dozens of
large astronomical databases, confirmed discovery of [a] new
brown dwarf recently. The star emerged from a computerized
search of information on millions of astronomical objects in two
separate astronomical databases. Thanks to an NVO prototype,
that search, formerly an endeavor requiring weeks or months of
human attention, took approximately two minutes.”



Context
• Data meaningless without context
• Linkage
• Metadata of many kinds
• Workflow!
• Provenance
• Computational lineage
• Authenticity



Access and re-use
• Ethics and rights control access
• Weak in expressing this long-term
• Collaboration tools
• Annotation, discussion, review
• Re-use leading to change and development
• “Publication”
• Not just in “print”
• Underlying data should be “published”, too
• Citation…


Citation
• Needs a stable resource to cite…

OWL Web Ontology Language
Reference
W3C Proposed Recommendation 15 December 2003
This version:
http://www.w3.org/TR/2003/PR-owl-ref-20031215/
Latest version:
http://www.w3.org/TR/owl-ref/
Previous version:
http://www.w3.org/TR/2003/CR-owl-ref-2003081



Citation…
• The date alone (as in common web citation
approaches) is not enough!
•[6] The CIA World Factbook.
•www.cia.gov/cia/publications/factbook/.
•Retrieved on 8 Jan 2006.
• Cited object likely to have changed…
• Citation should link to the cited object as it was!



Citation needs…
• An efficient way to reference and access “archived”
past states of a changing dataset (work in progress,
Buneman et al)
• Less important for original observations
• Don’t mess with those data
• Less important for incremental datasets
• Later stuff should not invalidate earlier
• Very important for revisable datasets
• Eg Genomics… datasets that result from the combined work
of curators, or contain opinions or facts likely to change



XML Archive at time t - 1
XMLArch: System Architecture
time t
Relational
XML Archiver
XML Snapshot at
Database

Pre-processor

Version
Merger
Data Extractor

XML Archive at time t
Endeavor EndUser 2006 •Carwyn Edwards


Preservation & curation
• Use preserves
• Money preserves
• Redundancy good, monoculture bad?
• LOCKSS-type & other approaches…
• Bits are fragile and robust
• Don’t rely on portable media
• Look after them well
• Technology changes…
• How fast? What impact?
• Metadata matters! (Know what you’ve got)


Formats, migration, significant
properties…
• “We MUST preserve the look and feel!”

• Well…
• Think about a book like “Kenilworth” by Walter
Scott
• Think about the BBC Domesday emulation

• You may be better with a preserved
“desiccated” version… than nothing at all!



The Project Gutenberg EBook of Kenilworth, by Sir Walter Scott

This eBook is for the use of anyone anywhere at no cost and with
almost no restrictions whatsoever. You may copy it, give it away or
re-use it under the terms of the Project Gutenberg License included
with this eBook or online at www.gutenberg.org

Title: Kenilworth
Author: Sir Walter Scott
Release Date: February 21, 2006 [EBook #1606]
Language: English
Character set encoding: ASCII

*** START OF THIS PROJECT GUTENBERG EBOOK KENILWORTH ***
Produced by An Anonymous Volunteer and David Widger

KENILWORTH.

by Sir Walter Scott, Bart.

INTRODUCTION

A certain degree of success, real or supposed, in the delineation of
Queen Mary, naturally induced the author to attempt something similar
respecting "her sister and her foe," the celebrated Elizabeth. He
will not, however, pretend to have approached the task with the same
feelings; for the candid Robertson himself confesses having felt the
prejudices with which a Scottishman is tempted to regard the subject;
…



•But there
•ARE limits!

QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.



Preservation is not cheap
• But it’s not expensive…



• Compared with the alternative!



• Compared with the alternative!

•Postcard
•Sent to me
•anonymously


Curation: whose job is it?
• Yours!
• With your archivists
• And your Records Managers
• And your scientists and scholars…



Preservation & curation
• We can’t do it alone
• Collective responsibility
• We can’t rely on anyone else
• Institutional responsibility



It’s about time…
• From the very short
• Good management (don’t under-estimate but don’t
over-estimate)
• Through the medium term
• Curation: use it or lose it
• Gather ye metadata while ye may!
• Preservation relay
• To the very long term
• High commitment, high cost, high risk
• Harder to do en masse



Supplier role?
• Work together with libraries…
• Multi-supplier, Multi-platform
• Open source mix
• The library is not simple any more
• Library 2.0?
• Power of crowds, economy of attention,
generation X…
• Wikicat?



QuickTime™ and a
TIFF (LZW) decompressor



Supplier role?
• Work together with libraries…
• Multi-supplier, Multi-platform
• Open source mix
• The library is not simple any more
• Library 2.0?
• Power of crowds, economy of attention, generation X…
• Wikicat?
• Web 2.0?
• Mix, mashup
• What you see is… not there?


BEWARE WEB 2.0!!!

QuickTime™ and a
TIFF (Uncompressed) decompressor



OAIS
• “Announcement of a Comment Period for the Five
Year Review of the Reference Model for an Open
Archival Information System (OAIS) Standard”
• “… must be reviewed every five years and a determination
made to reaffirm, modify, or withdraw the existing standard.”
• “…any revision must remain backward compatible with
regard to major terminology and concepts.”
• “… we do not plan to expand the general level of detail”
• “… reduce ambiguities and fill in any missing or weak
concepts”
• Make suggestions and express interest until 30/10/06
• OAIS-support@delight.gsfc.nasa.gov



To close…
• Your library is currently taking the curation
test…
• Your children will learn the answer!
• But


Dcc endeavour-2006

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie Dcc endeavour-2006

Ähnlich wie Dcc endeavour-2006 (20)

Mehr von Chris Rusbridge

Mehr von Chris Rusbridge (15)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Dcc endeavour-2006

Hinweis der Redaktion