How we implemented exports for MeLCat Co-presented with Mary Ross at EUGM 2006

1. Voyage meets MeLCat: MC’ing the Introductions

2. MeLCat extract sequences Voyager bibout.pl bib extract patout.pl today’s extract yesterday’s extract patdiff.pl Pmarcexport patron extract to MeLCat

3. Our MeLCat implementation is unfortunately still a work in progress. The information presented here may be subject to change. The information is as current as possible, given deadline constraints. Disclaimer

4. The bib extract and patron extract programs, and the patron differences program, are all written in Perl and are designed to run on your Voyager box. A Unix box is assumed. Minimal modifications should be required in order to run on a Windows box. Notes

5. User requirements: Familiarity with a text editor, file names, and paths on your machine. You’ll need to be able to query your database (SQL provided). You will be editing the setup section at the beginning of the programs to customize settings for your institution. Notes

6. MeLCat extract sequences Voyager bibout.pl bib extract patout.pl today’s extract yesterday’s extract patdiff.pl Pmarcexport patron extract to MeLCat

7. /m1/voyager/YOURdb/sbin/Pmarcexport -oOUTPUTFILE -mBtoday-N -ttoday-N -i Bib Extract, Pmarcexport The export command is in script form in a file. Thus the parts of the command have been broken out. The trailing “ indicates the command continues on the next line.

8. /m1/voyager/ YOUR db/sbin/Pmarcexport -o OUTPUTFILE -mBtoday-N -ttoday-N -i Bib Extract, Pmarcexport Substitute your own values for words in upper case.

9. /m1/voyager/YOURdb/sbin/Pmarcexport -oOUTPUTFILE - mB today-N - t today-N - i Bib Extract, Pmarcexport The mode, m , is B , which means to select records by both create and update dates. The export target, t , gives the date range. The last parameter, i , says to ignore suppressed bib records.

10. /m1/voyager/YOURdb/sbin/Pmarcexport -oOUTPUTFILE -mB today-N -t today-N -i Bib Extract, Pmarcexport today-N specifies to look back N days from today At WMU, we used a value of 10,000 for our initial load. For ongoing loads, we use a value of 1, since we run our MeLCat bib extract daily.

11. Bib Extract, Pmarcexport problems We had problems with Pmarcexport. Our initial large bib export resulted in two files, not one! This turned out to be expected, but previously undocumented, behavior.

12. Bib Extract, Pmarcexport problems We had problems with Pmarcexport. Our initial large bib export resulted in two files, not one! This turned out to be expected, but previously undocumented, behavior. Had problems with record counts when a split occurred.

13. Bib Extract, Pmarcexport problems We had problems with Pmarcexport. Our initial large bib export resulted in two files, not one! This turned out to be expected, but previously undocumented, behavior. Had problems with record counts when a split occurred. Also had mysterious SQL error which didn’t(?) seem to affect the output. When we finalized this presentation, these issues had not yet all been resolved.

14. MeLCat extract sequences Voyager bibout.pl bib extract patout.pl today’s extract yesterday’s extract patdiff.pl Pmarcexport patron extract to MeLCat

15. bibout.pl voy_bib_export_file melcat_bib_file voy_bib_export_file is created by Voyager’s Pmarcexport program melcat_bib_file is the resulting bib file ready to be sent to MeLCat Both files are in MARC format. Bib Extract, Usage

16. Typically, Pmarcexport will create one output file. For large amounts of data, you may have two or more files. In that case, you can run bibout on each file, or possibly concatenate the Pmarcexport files into one for bibout to process. In this presentation, and in bibout.pl, Pmarcexport’s output is referred to as one file. Bib Extract, Usage

17. Two stages: test extracts and initial loads ongoing production extracts and loads The exact usage is determined via a Pmarcexport script parameter and the corresponding setting for bibout. Bib Extract, Usage

18. Bib Extract – What does it do? bibout reads the file created by Pmarcexport. For each bib, find and scan its mfhd(s) for fields 866, 867, and 868. These fields, if present, are added as is to the bib record. Get mfhd and item information associated with this bib record, or just mfhd data as a limited fallback when there is no item record.

19. Bib Extract – What does it do? Append a new field to the bib record that contains the following data: |a item id (mfhd id if no item) |b item barcode |c item status |d item type |e call number |f location |g enum info |h number of recalls This is field 999 by default.

20. Bib Extract – What does it do? |a item id When there is no item id, |a contains a mfhd id. This is indicated with a leading “M” in this field. Still to come: handling deleted records

21. Bib Extract, database access bibout needs to know how to connect to your Voyager database. At WMU, we’ve standardized on storing this information in a file, allowing for easier updates and better security. Any programs that access Voyager do not have the specific database access information in the program.

22. Bib Extract, database access The information you need for this is your: machine name database name (may be username) database login (password) Oracle SID [VGER | LIBR] Voyager 5’s SID is VGER ; the SID for previous versions is LIBR . Use the right one for your version.

23. Bib Extract, database access Put this information in the file, all on one line: your.machine.name.edu/namedb/db_login/sid (namedb may be username in your situation) The slashes, “/”, belong in there, and the bib extract program will read this data so it can access Voyager. Make sure that this file contains your information, including the SID. Then we need to tell bibout where this file is, by specifying its path and the file name.

24. Bib Extract - which database? bibout needs to know from which database we are extracting the data. Since the database name may not equate to the user, we specify the database name separately. For example, you may be connecting to your database as a read-only user, so you would have to specify the database.

25. Bib Extract bibout also needs to know how many days to look back. Give it the same value, N days, that you used in the export script. This makes sure that everyone is on the same page.

26. Bib Extract – Which locations? We wanted to restrict, by location, the items showing up in MeLCat. Here’s a sample query to see your location codes: Select the location_codes you want, and put them in a text file, one per line.

27. Bib Extract – Which locations? After creating the location codes file, you need to tell bibout where this file is (the path), and the file name.

28. Bib Extract By default, the new field that contains the item/mfhd information is 999. If you need a different tag, you’ll need to tell bibout what that is.

29. MeLCat extract sequences Voyager bibout.pl bib extract patout.pl today’s extract yesterday’s extract patdiff.pl Pmarcexport patron extract to MeLCat

30. Patron Extract, Usage patout.pl patron_extract_file patron_extract_file is the double quoted, comma separated value format patron data retrieved from Voyager

31. Patron Extract – What does it do? Similar to bibout, the patron extract program (patout.pl) also looks back so many days to get patron data. Voyager is queried and the patron extract data for MeLCat is put in CSV format.

32. Patron Extract, Output last name, first name, middle name, patron barcode, patron group, patron exp date, WIN (WMU ID number), patron ID (Voyager internal number), delete indicator (D=delete, blank to retain), blocked (Y/N), email address, email exp date patron exp date is 2382.12.31, if indefinite

33. Patron Extract, blocking Patron is blocked when: patron record expired before today lost items count exceeds lost limit defined in the extract recalls count exceeds recall limit defined in the extract fees due amount exceeds the maximum balance for the patron group used in the extract email address record expired before today

34. Patron Extract, assumptions The extract uses the patron’s most recent active patron group as the patron group of record.

35. Patron Extract, database access The patron extract gets its Voyager database access information from the same file mentioned earlier in the bib extract part of this presentation… Database Access Put this information in the file, all on one line: your.machine.name.edu/namedb/db_login/sid (namedb may be username in your situation) The slashes, “/”, belong in there, and the bib extract program will read this data so it can access Voyager. … then we need to tell patout where this file is, by specifying its path and the file name.

36. Patron Extract Again, like with bibout, we need to tell patout how many days back to look to get patron data. This value will probably depend on how often your patron data is updated.

37. Patron Extract We discovered setting up limits by patron group to be pretty much impossible. This is because a patron group can be in multiple circ groups, with each circ group able to have different limits. So we went with across-the-board limits, and you’ll have to use a global value for the lost-limit and the recall-limit.

38. Patron Extract, patron groups There will probably be some patron groups that should not be in MeLCat. You’ll need to tell patout which patron groups are allowed. This is done via the patron group ID. Here’s how to get these ID numbers:

39. Patron Extract, patron groups Now you have a listing of patron group IDs and names. Note that fine/fee limits can be found in the Systems Administration client by clicking on Circulation , then on Policy Definitions . Record your patron group limits at each corresponding patron group ID. Remember that in Voyager, money values (U.S.) are stored in cents, so the values you enter here, divided by 100, equal the dollar amounts.

40. Patron Extract We also need a representative circ group for patout to do its thing. From your Sys…Admin… client, pick a circulation policy group with the most appropriate lending attributes. Now run a query to get its ID number: and let patout know what it is.

41. Patron Extract Finally, patout needs to know where to write the output. Specify the file path and file name. The filename will automatically have a date stamp appended. For example, if you’re using a file name of “patron.out”, you’ll end up with file names like patron.out.20060825 patron.out.20060826 etc.

42. MeLCat extract sequences Voyager bibout.pl bib extract patout.pl today’s extract yesterday’s extract patdiff.pl Pmarcexport patron extract to MeLCat

43. Patron Differences Extract patdiff.pl looks at the current and the previous patron extract and puts the differences into the patron extract to be sent to MeLCat. The extract comparison is based on patron ID.

44. Patron Differences Extract The extract gets: new records records no longer showing up; they are marked for deletion the current version of any records with differences in them

45. Patron Differences Extract patdiff needs to know where to write its output file. Specify the file path and the file name.

46. Extracts ready! Your bib and patron extracts are now ready to be sent to MeLCat.

47. Thanks for listening. Questions? [email_address] 269.387.3885 Picture © 2005 by Roy Zimmer