Eric, Mike and Steve details the UW Forward system architecture from ingest to user interface.

1. Forward in Reverse A Gentle Overview Of Forward System Architecture Eric, Mike & Steve – WiLSWorld 2010

2. Outline Intro to Forward with Demo Batch Processing (Backend) Web Application (Frontend) Challenges Q&As throughout

3. Intro & Demo

4. http://forward.library.wisconsin.edu

5. Batch Processing

6. We have gobs & gobs of data.

7. 1) Extract it

8. 1a) ILS Data

13. Sort, Deduplicate, Merge

14. Antique Style Key By Stars*Go*Blue http://www.flickr.com/photos/artbydebora/1406682449/

15. Common Identifier = OCLC Number

17. 13M MARC bibliographic records extracted

18. Approximately 14 hours

21. Formats

25. 13K “First Class” XML Objects extracted

26. Approximately 4 hours

28. Structural XML

29. MODS Bibliographic XML

31. We take raw library data and process it with MARC/XML parsing tools and local parsing rulesin order to build a Solr search index.

32. Raw data (MARC & METS XML) Parsing libraries (Java code: marc4j, SAXParser) Local code that defines parsing rules Solr index

33. 1. Raw data

34. LEADER 02000cam a22003734a 45 0 001 6939454 005 20051208125417.0 008 051104s2004 enka $b 001 0 eng 010 $a 2003045349 035 $a (OCoLC)ocm52165958 040 $aDLC $c DLC $d XMA $d BAKER $d UKM 015 $a GBA430162 $2bnb 016 7$a 012906573 $2Uk 020 $a 0754605175 (alk. paper) 024 $a 99811375970 042 $apcc 049 $a GZMA 050 00$a B3376.W564 $b W55355 2004 082 00 $a 111/.85/092 $2 21 245 00 $a Wittgenstein, aesthetics, and philosophy / $c edited by Peter B. Lewis. 260 $aAldershot, Hants, England ; $a Burlington, VT : $bAshgate, $c c2004. 300 $a xii, 255 p. : $b ill. ; $c 24 cm. 440 0 $aAshgateWittgensteinian studies 505 0$a Wittgenstein and the aesthetic domain / Kjell S. Johannessen -- 2. Wittgenstein, anti-essentialism and the definition of art / Terry Diffey -- 3. Rules, creativity and pictures : Wittgenstein's Lectures on aesthetics / David Novitz -- 4. Criticism without theory / Mark W. Rove -- 5. On aesthetic reactions and changing one's mind / Lars Hertzberg -- 6. Wittgenstein and the arts : understanding and performing / Graham McFee -- 7. Wittgenstein's music / R.A. Sharpe -- 8. Wittgenstein on music and language / Oswald Hanfling -- 9. Ethics and aesthetics are one / Carolyn Wilde -- 10. Fiction and reality in the arts / IlhamDilman -- 11. Literature, human understanding and morality / Ben Tilghman -- 12. 'The self, thinking' : Wittgenstein, Augustine and the autobiographical situation / Garry L. Hagberg 504 $a Includes bibliographical references (p. 235-247) and index.

35. 02000cam a22003734a 45 001000800000005001700008008004100025010001700066035002300083040003000106015001900136016001800155020002800173024001600201042000800217049000900225050002800234082002000262245007400282260006800356300003400424440003600458505081100494504006401305600005001369700002501419938007101444945001901515946003001534946001301564947002101577948001601598994001201614693945420051208125417.0051104s2004 enkab 001 0 eng a 2003045349 a(OCoLC)ocm52165958 aDLCcDLCdXMAdBAKERdUKM aGBA4301622bnb7 a0129065732Uk a0754605175 (alk. paper) a99811375970 apcc aGZMA00aB3376.W564bW55355 200400a111/.85/09222100aWittgenstein, aesthetics, and philosophy /cedited by Peter B. Lewis. aAldershot, Hants, England ;aBurlington, VT :bAshgate,cc2004. axii, 255 p. :bill. ;c24 cm. 0aAshgate Wittgensteinian studies0 aWittgenstein and the aesthetic domain / Kjell S. Johannessen -- 2. Wittgenstein, anti-essentialism and the definition of art / Terry Diffey -- 3. Rules, creativity and pictures : Wittgenstein's Lectures on aesthetics / David Novitz -- 4. Criticism without theory / Mark W. Rove -- 5. On aesthetic reactions and changing one's mind / Lars Hertzberg -- 6. Wittgenstein and the arts : understanding and performing / Graham McFee -- 7. Wittgenstein's music / R.A. Sharpe -- 8. Wittgenstein on music and language / Oswald Hanfling -- 9. Ethics and aesthetics are one / Carolyn Wilde -- 10. Fiction and reality in the arts / IlhamDilman -- 11. Literature, human understanding and morality / Ben Tilghman -- 12. 'The self, thinking' : Wittgenstein, Augustine and the autobiographical situation / Garry L. HagbergaIncludes bibliographical references (p. 235-247) and index.10aWittgenstein, Ludwig,d1889-1951xAesthetics.1 aLewis, Peter,d1947- aBaker & TaylorbBKTYc99.95d99.95i0754605175n0004227086sactive c1d89087961587 a714694b2005-11-23c81.86 c99.95d1 aHEUR 4801bm,stk aSCNd348032 a92bGZM

37. 2. MARC/XML parsing libraries

40. 02000cam a22003734a 45 001000800000005001700008008004100025010001700066035002300083040003000106015001900136016001800155020002800173024001600201042000800217049000900225050002800234082002000262245007400282260006800356300003400424440003600458505081100494504006401305600005001369700002501419938007101444945001901515946003001534946001301564947002101577948001601598994001201614693945420051208125417.0051104s2004 enkab 001 0 eng a 2003045349 a(OCoLC)ocm52165958 aDLCcDLCdXMAdBAKERdUKM aGBA4301622bnb7 a0129065732Uk a0754605175 (alk. paper) a99811375970 apcc aGZMA00aB3376.W564bW55355 200400a111/.85/09222100aWittgenstein, aesthetics, and philosophy /cedited by Peter B. Lewis. aAldershot, Hants, England ;aBurlington, VT :bAshgate,cc2004. axii, 255 p. :bill. ;c24 cm. 0aAshgate Wittgensteinian studies0 aWittgenstein and the aesthetic domain / Kjell S. Johannessen -- 2. Wittgenstein, anti-essentialism and the definition of art / Terry Diffey -- 3. Rules, creativity and pictures : Wittgenstein's Lectures on aesthetics / David Novitz -- 4. Criticism without theory / Mark W. Rove -- 5. On aesthetic reactions and changing one's mind / Lars Hertzberg -- 6. Wittgenstein and the arts : understanding and performing / Graham McFee -- 7. Wittgenstein's music / R.A. Sharpe -- 8. Wittgenstein on music and language / Oswald Hanfling -- 9. Ethics and aesthetics are one / Carolyn Wilde -- 10. Fiction and reality in the arts / IlhamDilman -- 11. Literature, human understanding and morality / Ben Tilghman -- 12. 'The self, thinking' : Wittgenstein, Augustine and the autobiographical situation / Garry L. HagbergaIncludes bibliographical references (p. 235-247) and index.10aWittgenstein, Ludwig,d1889-1951xAesthetics.1 aLewis, Peter,d1947- aBaker & TaylorbBKTYc99.95d99.95i0754605175n0004227086sactive c1d89087961587 a714694b2005-11-23c81.86 c99.95d1 aHEUR 4801bm,stk aSCNd348032 a92bGZM

44. 3. Local code

47. 4. http://lucene.apache.org/solr/

48. What is Solr? An XML API over a Lucene search index.

51. Access to Raw Formats Raw MARC stored for Merged record Live calls made to Fedora web services

52. Data Refresh Bibliographic: weekly Circulation status: nightly

54. For more information, see http://sdg.library.wisc.edu/blog/2010/03/03/solr-marc-indexing-based-on-diffs/

55. Web Application

56. Frontend? (X)HTML JavaScript Cascading Style Sheets Design Information Architecture User experience Chrome (images, icons, pretty)

57. Forward Colophon ActiveRecordBaseWithoutTable (Rails plugin) Apache Blacklight (Rails plugin) Blueprint CSS Bookreader (jQuery) Capistrano Crontab Engines (Rails plugin) Fedora Freebase API GeoIP (Ruby gem) Google Books API Haml (Rails plugin) Happymapper (Ruby gem) HathiTrust API jQuery Ken (Ruby gem) LowPro (Prototype JS) MARC4J Passenger (modrails) Prototype JS PostgreSQL Raphael Ruby on Rails Shibboleth Subversion Solr / Lucene Summon (Ruby gem) UW-Madison Libraries Staff Directory API UWDC (Rails plugin) Voyager API Tender love and attention

58. Campus Affiliation Users localize to a school, allows us scope many features to their campus. GeoIPRubyGem Match IP addresses with physical locations. Raphaël—JavaScript Library “Small JavaScript library that should simplify your work with vector graphics on the web”.

59. Raphaël SVG elements, like the circles and squares in the Forward splash page, can be treated as XHTML elements allowing us to manipulate them with JavaScript and CSS. http://raphaeljs.com/

61. Ruby on Rails

62. PostgreSQL

64. Ruby on Rails “Ruby on Rails is an open-source web framework that’s optimized for programmer happiness and sustainable productivity.” http://rubyonrails.org/

65. PostgreSQL “PostgreSQL is a powerful, open source object-relational database system. It has more than 15 years of active development and a proven architecture that has earned it a strong reputation for reliability, data integrity, and correctness.” http://www.postgresql.org/

66. Apache Solr “Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling.” http://lucene.apache.org/solr/

67. Results

68. Results – Three columns

69. Results – Data sources

70. Results – Facets – Solr

71. Results – Solr + PostgreSQL + APIs

72. Results – Context – APIs

73. Results – Three main columns

74. Results – CSS grid

75. Blueprint “Blueprint is a CSS framework, which aims to cut down on your development time. It gives you a solid foundation to build your project on top of, with an easy-to-use grid, sensible typography, useful plugins, and even a stylesheet for printing.” http://blueprintcss.org/

77. Show – Book

78. Show – Image

79. Show – Full Text Book

80. Show – View Full Text Book

81. BookReader “The Internet Archive BookReader is used to view books from the Internet Archive online and can also be used to view other books. ” http://github.com/openlibrary/bookreader

82. Challenges

83. Challenges Merging MARC, METS extracts Batch processing time (Time/CPU constraints) Page level indexing (Bookviewer - memory/disk constraints) Voyager API Organization challenges big project, small shop dealing with vendor silos multiple cataloging standards quality of services challenges

84. Thanks!