Digitisation -- but particularly planning for and managing a digitisation project

1. 30 October 2014 SIPAD Workshop
Preservation of
Audiovisual Content
on Shelves
Richard Wright
SIPAD14, Mexico City, October 2014

2. 2
The need for digitisation
Preservation planning
mapping your collections;
setting priorities;
making a collection strategy,
a preservation strategy
and a preservation plan
The Preservation Factory approach
Conservation, Digitisation
and Preservation

3. Terminology

Preservation – “Everything needed to ensure
permanent access” = maintenance

Conservation: keeping what you have (for as long
as you can) = safe storage and handling

Preservation actions: interventions. Changing what
you have = repair and replace
− Making a new negative or interneg
− Copying from an old carrier to a new carrier
− Digitising

Which separates content from carrier = liberation
3

4. Conservation

packaging, handling and shelving

environmental conditions: temperature and
humidity; protection from pollution, dirt etc

protecting the masters; and

condition monitoring = checking the stock
Full description on Preservation Guide wiki:
http://wiki.prestospace.org/pmwiki.php?
n=Main.PreservationStrategy#Conservation
4

5. 5
The need for digitisation
Only three ways to preserve:
1.Keep what you have
2.Copy using same technology
3.Copy using new technology
Keep what you have: only works for film
•
Audio, Video: analogue technology obsolete
Copy on same technology: only works for film
•
Audio, Video: analogue technology obsolete

6. 6
What about vinyl?

Gramophone
records: lacquer,
shellac and vinyl

Lacquer = a master
recording (acetate,
instantaneous disc);
very fragile !

7. 7
More on gramophone records

Shellac was used for 78
rpm commercial
recordings

Also fragile – the main
risk is handling; the
shellac itself is stable
(compared to laquer)

8. 8
But, what about vinyl?

Used for 45 and 33 1/3 rpm recordings

They are fragile (though less fragile than
shellac)

Warp from heat (can be fixed with care)

Easily scratched, and can get very dirty

The groove can be damaged in playback (if the
needle and tone arm adjustment is not right)

Vinyl is soft !!!

Vinyl and shellac can suffer chemical damage

9. Vietnam Film Institute Workshops 9
Gramophone
damage

Dropping a
needle onto a
vinyl disc

Oil coming to
the surface
on a lacquer
disc

10. 10
Where was I?
Only three ways to preserve:
Keep what you have: only works for film
•
Audio, Video: analogue technology obsolete
Copy on same technology:only works for film
•
Audio, Video: analogue technology obsolete
So we are left with only one option (for audio and
video):
Copy using new technology

11. 11
Preservation of Film

Many excellent example of conservation

Film technology still exists (but in decline)

BUT – the situation is rapidly changing

Kodak in severe economic trouble

Commercial cinemas changing to digital projection
− Norway changed completely in 2011-2012

Commercial cinema will change or go out of
business

Commercial cinema will not keep old projectors

12. 12
The end of film projection?
“ We're About to Lose 1,000 Small
Theaters That Can't Convert to Digital.
Does It Matter?”
Indiewire, USA
2012

13. 13
Digitisation of Film

Now: needed for access

Internet

Digital cinema

Soon: needed for preservation

When manufacture of film stock is stopped

Result: ALL media needs to be digitised, audio
and video and film

That's a lot of content that has to be digitised

There isn't enough time

There isn't enough money

14. 14
PRESTO Digitisation
“Better Faster Cheaper” Daniel Teruggi, Institut
National de l'Audiovisuel (INA), Paris
Basic concepts:
1) Dealing with the whole collection
2) Developing a strategy

For the institution and its collection

For the preservation work
3) Developing a preservation plan

15. 15
Digitisation Planning

All the Presto information on a wiki:
www.preservationguide.co.uk/RDWiki/pmwiki.php

StrategyStrategy – what does your institution do?

What does the collection do?

What can digitisation do?

PlanningPlanning – making a preservation plan

How to estimate a budget

Building a business case

16. 16
Collection Strategy

Long-term purpose of the digitisation

Preserving the collection

Maintaining services (old business)

Creating new services (new business)

Physical Outcomes:

Digital files

Mass storage

Cheaper, better maintenance

Cheaper, better digital access copies

Web access

17. 17
Access Outcomes

Collection access from computer stations

Mediatheque approach

Collection access via the Internet

With restricted access
− British Film Institute: higher education, public libraries …
and on YouTube
− British Library Sound Archive: higher education
institutions only

Or even unrestricted access
− INA has 30 000 hours of broadcast content online !

18. 18
Restrictions on Access

Attitudes: we don't do that

Fear: somebody might complain

Rights: we don't hold the rights
A solution:
1) public institutions create public value by
opening their collections as widely as possible
2) non-fiction content has the most information
and the least rights problems
We have a public service obligation to create
access to our collections !

19. 19
Access:
Technical Requirements

A usable catalogue

Online support services: questions, linking to
online sales, dealing with faults

Logins, access control, authentication, data
protection …

Computing power to support what could be
large numbers of people requiring access

UK National Archive crashed; Europeana crashed

Managing lots of information technology

20. 20
Digital Archive: workflow

Everything changes when shelves disappear:

Acquisition: digital ingest

Cataloguing: complex metadata issues: embedded
metadata, preservation metadata, mapping

Curation: huge opportunity to create online
collections

Research: self-research, fast scanning of 'hits' –
changes the requirements of cataloguing

Access: requirements need to be built into the
digitisation workflow (access copies, public
metadata, rights clearance, censorship?)

21. 21
Getting Started

Map – The collection needs to be divided by
format, condition, purpose ...

Priorities – arrange the “areas on the map”
according to what needs to be done first
wiki: Make a Map of your Collection
Divide the collection by physical formats, and
collect the following information on each format:
age range
storage history
genre or value
physical condition
Getting Started

22. 22
BBC film examples from wiki
Map:
http://www.preservationguide.co.uk/RDWiki/pmwiki.php
?n=Main.GettingStarted
Strategy:
http://www.preservationguide.co.uk/RDWiki/pmwiki.php
?n=Main.PreservationStrategy
Plan:
http://www.preservationguide.co.uk/RDWiki/pmwiki.php
?n=Main.PresPlan
Also: tutorials on PrestoCentre website
collection strategy and preservation plans_V1.01_0.pdf
mapping your collection_V1.01_0.pdf

23. Collection Strategy & Plan
• PDFs of wiki ‘chapters’
– Collection Strategy
– Preservation Strategy
– Preservation Plan and Budget
– Collection Strategy Preservation Strategy
– Preservation Plan and Budget
• Tutorials on PrestoCentre
– collection strategy and preservation plans_V1.01_0.pdf
– mapping your collection_V1.01_0.pdf
30 October 2014 SIPAD Workshop

24. 24
The Factory Approach
The division of labour
in pin manufacturing:
(and the great increase
in the quantity of work
that results)

25. 
The division of labour in pin manufacturing:
(and the great increase in the quantity of work
that results)

Approach: NOT about cutting corners or
reducing quality

Instead: about cutting wasted time and wasted
effort

Batches: doing one thing at a time, and then
doing it again (and again, and again)

Problem: how to keep the work interesting!
25
The Factory Approach

26. 26
A BBC Preservation Factory

27. 27
The Factory Approach

“Division of labour” requires having specialists,
which means having a team

Which means having a middle-sized or larger
project

But: even a single person can use a batch
approach, and get some efficiencies

28. BBC Preservation:
£5M per year (for 20 years)
Audio: 7000 hours per year of ¼ inch tape
Also digitised 63 000 “sepmag” elements:
separate magnetic sound tracks for film;
− Transferred to CD (good!) and to polyester (bad!)
Also 35 000 shellac and vinyl recordings
Also DAT and MD digital recordings
2” videotape: 46 000 tapes, transferred to D3
or Digibeta, done over 7 years
1” videotape: 80 000 in 5 yrs, to D3/Digibeta
 U-matic: 40 000 in 3 yrs, to MPEG-2 files

29. 29

News film: 43 000 items in 3 years

Ektachrome; digitised to SD video (Digibeta)

10% of total cost is cataloguing

20 to 30% is quality control (checking)

Since 2008: transfer 40k D3 videotape to files:

Uncompressed; MXF wrapper = INGEX

Now: transfer of BetaSP and Digibeta to files

Major confusion over High Definition formats !

30. 30
Discussion

Digitisation projects of delegates

Audio, video, film

Which physical formats (eg ¼-inch audio tape,
1-inch video tape, 16mm B&W film … ) ?

Size: how much content, how many people,
how many years ?

31. 31
Time for a break
After the break – a choice !
1)Low-level: standards and best practice for
audio, video and film
2)Higher level: case studies for preservation
and access
3) Your questions on digitisation

your long-term plan?

your immediate problems?

32. 30 October 2014 SIPAD Workshop

33. Digitisation Issues:
• Audio and Video: no problems at all with the
digitisation technology – the problems are with:
– Playback of originals
– Standards: compression, formats

34. Audio holdings (TAPE data)
Carrier %
wax cylinders 0.03
coarse groove replicated disks
(‘78s’,‘shellacs’)
1.9
instantaneous disks of any kind 0.2
microgroove disks (LPs) 17.5
open reel magnetic tape 38.6
compact cassettes 20.6
R-DAT 2.5
replicated CDs, DVDs 16.5
recordable and rewritable CDs, DVDs 1.4
Minidisks 0.4
Other 0.3
Total 100

35. Audio: major problems
• Wax cylinders: condition, equipment
• Lacquer discs: condition
• Sticky shed on 6mm tape playback
• DAT recordings: technical standards, condition,
equipment
• CD ‘ripping’ – common issue for digital-digital
transfer: error concealment (and what to do about it)
• IASA TC-04: Guidelines on the Production and
Preservation of Digital Audio Objects, March 2009.
– BWAV, 96kHz, “24bit”; BBC do 44.1kHz, 16bit

36. How to digitise audio
• IASA TC-04
• 48 kHz, “24 bit”
– 96 kHz recommended
• Save as a wav file
• Broadcast Wave Format recommended

37. Film: major problems
• No problem with playback equipment !
• Some film can last centuries
• Some film already destroyed
vinegar syndrome; colour fade; mold
• Telecine out, datacine in

38. Datacine for film digitisation
• Batch process
• HD or better for 16mm
• Many archives doing 2K scanning
• No grading shot-by-shot
– so a higher dynamic range is needed: 14 bits or better
• Needs sprocketless transport to cope with fragile film,
shrinkage, splices
• Infrared technology for scratch detection
• Wet-gate for scratch concealment (on one side)
• Several new sprocketless systems, €80 to 240k

39. How to Digitise Film
B&G document on PrestoCentre
http://www.prestocentre.eu/sites/dev.prestocentre.
antenna.nl/files/filmscanning_v1.03.pdf
White Paper: Film scanning considerations
By Tom de Smet and Harm Jan Triemstra

40. Sound on Film
• Successful digital processing of optical sound
tracks now very well established by Chase
Audio, Hollywood
– But expensive
• Analogue capture most cost-effective if sound
track in reasonable condition
• Much TV archive material has separate sound
tracks: sepopt, sepmag (beware vinegar!)

41. Video: major problems
• As for audio: equipment, operators, fragile and
decaying media; sticky-shed
• Plus: time base correction and dropout ‘concealment’
not built into old equipment
• Colour: composite signals and their decoding;
– BBC: a composite original is PAL-decoded and
saved as component, even though “the original” is
composite
• Head wear, head life, head replacement – alternative
technology for reading videotape has been propose

42. Video digitisation issues
• What codec, what file format?
– Uncompressed, lossless, lossy
– Lossless: JPEG2000, FFV1 (many others)
– MOV, AVI, MXF ... (cf LOC FADGI case studies)
• Endgame: uncompressed (no codec !)
• BBC: saving uncompressed from D3 transfers
to files: INGEX open-source software (as used
by Tate Galleries, London)

43. How to digitise video
1. Save the original
2. Digitise @ SDI = 4:2:2 = 200 megabits/sec
3. Save exactly as digitised = uncompressed
4. Use an open source file format (MXF or ?)
3b – or lossless = lossless JPEG2000,
3c – or ‘mezzanine’ = high-quality lossy
(but only for low quality originals)
(digibeta is a ‘mezzanine’ format)

44. Common issue: quality control
• Computer processing exists
– in professional audio equipment: Quadrega, NOA …
• ‘fault detection’ technology from restoration
work
• Joanneum video problem detector = VidiCert
• Audio Inspector
https://www.audioinspector.com/
• New QC software from Dave Rice, BAVC
http://www.bavc.org/qctools
• Key issues: workflow integration, statistics

45. How to Digitise Film
• B&G document on PrestoCentre
• http://www.prestocentre.eu/sites/dev.prestocentr
e.antenna.nl/files/filmscanning_v1.03.pdf
White Paper: Film scanning considerations
By Tom de Smet and Harm Jan Triemstra