More Related Content
Similar to Show104 buried treasure
Similar to Show104 buried treasure (20)
More from Mark Myers (12)
Show104 buried treasure
- 1. SHOW104 Buried treasure: Finding
the Hidden Gold in Lotus Notes Data
Mark Myers | London Developer Coop
Julian Robichaux | panagenda
© 2013 IBM Corporation
- 3. Who Are Mark & Julian?
Julian
– Java/Eclipse application developer at panagenda (panagenda.com)
– Developer since before Justin Bieber was born
– Notes/Domino since version 4.1
– Speaker at 7x Lotuspheres, various LUGs and View conferences
– Twitter: @jrobichaux, Blog: nsftools.com
Mark
– Member of the London Developer Co-op (londc.com)
– Developer from a support background
– 12+ years on Domino, 15+ years in IT
– Speaker at 2x Lotuspheres, 3x UKLUGs, 1x ILUG
– Twitter: @stickfight, Skype: Stickfight, Blog: stickfight.co.uk
© 2013 IBM Corporation
- 4. Why Are We Here?
You have years and years of data stored in your Notes databases
– Valuable
– Critical?
– Historically important
Often, that data needs to be accessed
from other places
– Reports
– Intranets/portals
– Non-IBM systems
What are your options?
As it turns out, you have a lot of options…
© 2013 IBM Corporation
- 5. Tools We Used
IBM® Lotus® Notes® version 8.5.3
IBM Lotus Domino® version 8.5.3
Microsoft® Windows® 7
IBM Connections V4.0
Various other software tools mentioned
throughout this presentation
Most code and techniques we talk about should be applicable to other versions of
Notes/Domino/Windows too
© 2013 IBM Corporation
- 6. Old and New
This presentation is an eclectic mix of Old and New, borne out of many many
years of Domino programming experience
Solutions are based on Ease, Speed and Reliability rather than the latest coding
fashion
If a solution seems old (more than 2 year old technology) it is because:
– Some implementations we see are still being done wrong (or not done at all)
– Some people are still on old versions of software
– Sometimes the “old” way is still the best way to do things
– Everyone is at a different experience level; maybe you’ve never seen some of this before
– This stuff still works
© 2013 IBM Corporation
- 7. About the Sample Database
This year’s sample dataset is… Sea Ducks!
The Atlantic Flyway Sea Duck Survey, conducted by the
U.S. Fish and Wildlife Service, was established in 1991
to record sea duck numbers using near shore (within 700
m of shore) habitats from Cape Breton, Nova Scotia to
Jacksonville, Florida.
https://migbirdapps.fws.gov/mbdc/databases/afsos/disclaimerafsos.html
© 2013 IBM Corporation
- 8. About the Sample Database
Public domain data and photos from the US Fish and Wildlife service
– See the “About this Database” page in the sample database for information, disclaimers,
and links
A little over 10,000 lines of data (i.e. – individual documents in a view)
Various means of grouping and data retrieval
– By year
– By state
– By duck type
Some documents also have image files stored in
rich text fields
So… potentially similar to data you already have
© 2013 IBM Corporation
- 9. Agenda
Overview
Working with Feeds
Integrating with IBM Connections
ODBC, JDBC, and Reporting
Generating Files on the Domino Server
Caveats, Considerations, and Data Scrubbing
Where to Get More Information
© 2013 IBM Corporation
- 10. What’s a Feed?
Structured data
– 0-to-N items
Parseable format
Retrievable via a link or some other automated method
Ideally (but not necessarily) semantic
– Semantic markup assigns “meaning” to data
http://en.wikipedia.org/wiki/Microformat
© 2013 IBM Corporation
- 13. ReadViewEntries
Some optional parameters:
– CollapseView / ExpandView
– NavigateReverse=1 – lists documents in reverse order, starting with the last document
– Start=n – indicates which document number to start with
– StartKey= / EndKey= – indicates which key in a sorted view to start and/or end with
– RestrictToCategory= – indicates which category to display in a single-category view
Count=nn
– Very important if you want to get ALL the docs in a view
– Default is 30, or whatever is specified by “Default lines per view page” on the Domino
Web Engine tab of the server (or web site) document
– Default maximum is 1000, or whatever is specified by “Maximum lines per view page”
OutputFormat=JSON
– Displays data as JSON instead of XML
A few other parameters too: check Domino Designer help
© 2013 IBM Corporation
- 16. Notes on Using ReadViewEntries
Maximum View Count settings on the server doc might prevent you from seeing all
the data in the view
You can save the query and refresh the data on demand
Categorized views get messy, flat views are better
If you need to login, make sure the server is using Basic Authentication (at least
for that URL)
– Not strictly necessary, but much easier from a programmatic standpoint
© 2013 IBM Corporation
- 17. Overriding Session Authentication
If your Domino server uses session authentication (most do), there is an option to
override this and use basic authentication for specific URLs
Since Domino 7
– http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/topic/com.ibm.help.domino.admin.
doc/DOC/H_OVERRIDING_SESSION_AUTHENTICATION_8847_STEPS.html
Must be using Web Site documents for web server access
– “Load Internet configurations from ServerInternet Sites documents” enabled on Basics
tab of server document
– http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/topic/com.ibm.help.domino.admin.
doc/DOC/H_R5_WEB_SITE_TO_RNEXT_OVER.html
© 2013 IBM Corporation
- 19. Overriding Session Authentication
• The URL pattern can use *
for wildcards (? is treated as
a literal character).
• If you don’t a wildcard at the
beginning and end, the rule
might not work the way you
expect.
© 2013 IBM Corporation
- 20. Overriding Session Authentication
Optional parameter on the Web Site document to auto-generate a session cookie
when Basic Authentication is used in an override rule
© 2013 IBM Corporation
- 21. Classic Feed: ICalendar
Still the best format for dealing with event based data
Near universally supported
Can be done without 3rd party libraries
© 2013 IBM Corporation
- 23. Classic feed: ICalendar
Most ICalendar clients tend
to be picky on carriage
returns so you need to be
careful
© 2013 IBM Corporation
- 25. Classic feed: ICalendar
Then insert the view in a form
with the text that wraps the
ICalendar Vevents, this text
details time zones and such
for the calendar as a whole
© 2013 IBM Corporation
- 26. Classic feed: Web Services
The De facto standard for inter-system communication
Still seen as a silver bullet by many managers
See a new web service as an opportunity to re-define how your data is seen
Complex if done badly, simple with a few tricks or the right software
© 2013 IBM Corporation
- 27. Classic Feeds: Web Services
Best Tools
– Lotus Domino server!
– Liquid XML Studio ( http://www.liquid-technologies.com/xml-studio.aspx )
• Brill visual designer
• Powerful but easy to use
• Costs a lot from a stingy developer’s point a lot (but worth it)
– Soap UI ( http://www.soapui.org )
• The simplest way for humans to interact with and test raw web services
• Free for the Lite version
© 2013 IBM Corporation
- 32. Custom Feeds: Decisions to Make
Data format
– XML, JSON, CSV, something else…
Data structure
– RSS, Atom, something else…
– Standard structures are widely supported
– Custom structures allow more flexibility
Static or dynamic
– Caching and performance issues
– User-specific information?
Do you control the clients?
– If not, it might be hard to make changes later
© 2013 IBM Corporation
- 33. Atom and RSS
Standard XML formats for document-based data
RSS is simpler to produce and parse
– Fewer options
– Easy to understand
– Great for syndication of news, blogs, and similar
Atom is more flexible
– Easily extensible for data customization
– Read AND write data, if you need to publish
– Still primarily for documents, rather than raw spreadsheet-type data
Almost any software that calls itself a Feed Reader can consume either type of
feed with no problem
© 2013 IBM Corporation
- 34. Feed Library vs. Build-Your-Own
XML and RSS/Atom is easy to break
– Character encoding issues
– Illegal characters
– Unclosed tags
– Improperly formatted dates
– Missing required elements
If you write “raw” XML to a text string for output,
test often and add plenty of checks for poorly formatted data
Feed libraries take care of the heavy lifting for you
– Built-in validation
– No need to study the specification documents
– Probably slower and more memory-intensive
– Usually several megabytes of required code libraries either stored in the database or
copied to the server
© 2013 IBM Corporation
- 35. Example: RSS Feed Generator Database
Template included with Domino 7
Easy to use for quick feeds:
– Create a new database on the server using the “RSS Generator” template
– Fill out a form to point to a view
– Point the users to the database URL
Reuse the code in your own applications
© 2013 IBM Corporation
- 38. Example: RSS Feed Generator Database
Unfortunately, the <enclosure> item has a hard-coded data type, so it’s only good
for docs where all the attachments are the same type
© 2013 IBM Corporation
- 40. Example: Java Agent Atom Feed
A few different Java feed parsing libraries to choose from:
– ROME – http://rometools.org
– Apache Abdera – http://abdera.apache.org
– Apache Wink – http://incubator.apache.org/wink
All require several additional libraries
– Apache Commons libraries
– Java StAX support
– Etc.
Handle HTTP connections as well as parsing
– Authentication too
An excellent test client:
– DEV HTTP Client (Chrome Plugin)
https://chrome.google.com/webstore/detail/aejoelao
ggembcahagimdiliamlcdmfm
© 2013 IBM Corporation
- 42. Example: Java Agent Submit Back to Atom Feed
Most Atom services (particularly IBM) give examples of what XML they want rather
than examples of the code you should use, so we will do the same
This is a IBM Connections Blog entry in raw XML:
<?xml version="1.0" encoding="utf-8"?>
<entry xmlns="http://www.w3.org/2005/Atom">
<author><name>I Am Funny</name></author>
<title type="text">Great Duck Joke</title>
<content type="text">Q: What has webbed feet and fangs? A: Count Duckula</content>
</entry>
© 2013 IBM Corporation
- 43. Example: Java Agent Submit Back to Atom Feed
An amazingly useful line when you are debugging or
testing as it outputs the XML so you can check it against
what you should be sending
© 2013 IBM Corporation
- 44. Example: Java Agent Submit Back to Atom Feed
Getting this URL is often the hardest part of dealing with
Atom
© 2013 IBM Corporation
- 45. Example: Useful Extra Lines
Add “registerTrustManager”
and you can connect to
SSL Ports
Nice simple login, you just need the user name, password
and the root domain you are logging on to
© 2013 IBM Corporation
- 46. Agents vs. XAgents
Java agents are not efficient when using large JAR files attached to agents or
script libraries
– They are reloaded into memory each time the agent runs
– Beware of memory problems
Copying the JAR files to the jvm/lib/ext directory on the Domino server keeps them
in memory
– If your admin allows it
– Have to do this with every server the code might run on
An alternative is XAgents: XPages written so they run as agents
– No UI, except for print statements (“write”, actually)
– JARs and Java code attached directly to the database
– Much better caching of JAR files
– http://www.wissel.net/blog/d6plinks/shwl-7mgfbn
– http://www.mindoo.com/web/blog.nsf/dx/17.07.2011101855KLEBRW.htm
© 2013 IBM Corporation
- 47. Caveats
Malformed JSON and XML
Character encoding issues
Multi-value fields can be tricky
Memory issues when using large Java libraries
– Or poorly written Java code
Make sure you use the right Content-Type HTTP header when you send data
– JSON should be “application/json”
– XML should be “application/xml” (preferred) or “text/xml”
© 2013 IBM Corporation
- 49. Integrating with IBM connections
2 basic forms of integration
– Submitting existing data directly into Connections (as demonstrated by the previous
Atom examples)
– Read data from Domino and displaying it in Connections via a widget
© 2013 IBM Corporation
- 50. Connections Caveats
This is not a Connections session
The methods shown in this session are one of many methods of integration rather
than specific best practices
– Niklas Heidloff (http://heidloff.net) and Mark Leusink (http://linqed.eu) have some good
examples
– These methods use as few toolkits such as Social Business toolkit/Social business SDK,
social enabler as possible.
Real life has taught that the best integration environment to build is one where you
can get all data if required (Gather all data using service accounts) then compile a
result and display to the user.
See the related sessions at the end of this presentations for more details and the
best practice sessions.
© 2013 IBM Corporation
- 51. Integrating with IBM Connections
Before we start we need to get a
database ready to call Java from
XPages
Move to Package Explorer
© 2013 IBM Corporation
- 52. Integrating with IBM Connections
In The WEB-INF directory
Right click and create a new
folder
© 2013 IBM Corporation
- 54. Integrating with IBM Connections
Next add this folder
to the projects build
path
Right click on the
root of the project
and select
“Properties”
© 2013 IBM Corporation
- 55. Integrating with IBM Connections
On the “Source” tab
click “Add folder”
and add the folder
you have created
© 2013 IBM Corporation
- 56. Integrating with IBM Connections
You will see the
Folder has
moved
Right click and
select “New” →
“Other”
Then Select
“Java” →
“Class”
© 2013 IBM Corporation
- 58. Integrating with IBM Connections
OK, Java class created, lets get it
some libraries
© 2013 IBM Corporation
- 59. Integrating with IBM Connections
If you are not allowed
to use the EXT
directory on the server,
the you will need to
create a new “lib” folder
© 2013 IBM Corporation
- 61. Integrating with IBM Connections
Update (or ask your administrator to) the java.policy file on your server to contain
the following
grant {
permission java.lang.RuntimePermission "getClassLoader";
};
© 2013 IBM Corporation
- 62. Now we will add the
Integrating with IBM Connections Atom client code
function
You need to pass the
session into this
function as it can’t get
its own
If you need to you can
login here (as
described previously)
© 2013 IBM Corporation
- 64. Integrating with IBM Connections
Set it as a Normal “Server JavaScript” Library
© 2013 IBM Corporation
- 65. Integrating with IBM Connections
Create a SSJS function to call your Java and pass the results back
Pass the “sessionAsSigner” here to give yourself administrator rights
© 2013 IBM Corporation
- 66. Integrating with IBM Connections – Pulling it together
Create a Custom
Control
© 2013 IBM Corporation
- 68. Integrating with IBM Connections
Now you can add a repeat control that calls the SSJS function, which in turn
calls the Java
The Rest of the surrounding boring code is in the demonstration db
© 2013 IBM Corporation
- 69. Integrating with IBM Connections – Pulling it together
Create a New
Xpage Library
© 2013 IBM Corporation
- 70. Integrating with IBM Connections – Pulling it together
Add the
duckstates_cc
custom control
© 2013 IBM Corporation
- 72. Integrating with IBM Connections
Now that you have an
XPage, you need to
present it to
Connections as a
Open Social widget
the easiest way of
doing this is with a
simple domino page.
© 2013 IBM Corporation
- 73. Integrating with IBM Connections
Features like
“dynamic-height” are
harder to get working
using this method
(rather than a
embedding the widget
in connections) but if
you have a static sized
widget it is the
simplest.
© 2013 IBM Corporation
- 74. Integrating with IBM Connections
And add it to your
Connections
installation
© 2013 IBM Corporation
- 77. SQL Queries of your Notes Data?
Lotus Notes is NOT a relational database
– You knew that, right?
IBM has a NotesSQL driver
– Free download, easy install
– Treats Notes Views like SQL tables
– Has been updated for Notes 8.5
© 2013 IBM Corporation
- 78. From the Notes Client: NotesSQL
Free tool from IBM
Windows-only (make sure you have the latest version on Windows 7)
Set up on each user’s machine (install plus DSNs)
Notes client must be installed (version 6.0 or higher)
Easy integration with reporting tools like Crystal Reports, MS Excel, and MS
Access
© 2013 IBM Corporation
- 79. Setting up NotesSQL
Download from
http://www.ibm.com/developerworks/lotus/
products/notesdomino/notessql
– Help files are in C:NotesSQLdocs after
install
After install, the NotesSQL directory and
the Notes program directory MUST be in
the Windows PATH
– You get vague error messages if they’re not
Create an ODBC DSN to point to a
database
© 2013 IBM Corporation
- 85. Optimizing Your Data
You will almost certainly want to create one or more “reporting” views
– Many columns of data
– No categorization
– Minimal sorting for better lookup performance
– Reasonable column names (check the “Programmatic Access” name for computed
column data)
If you’re reporting on rich text data, modify the DSN to allow more than 512
characters for rich text fields
Read the help docs on performance
© 2013 IBM Corporation
- 86. NotesSQL Caveats
Problems with MS Excel 2003
– Sometimes hangs while accessing or saving data
– Access 2003 works fine though
– Excel 2007+ also works fine
Problems with OpenOffice 2.x
– View columns with text values are blank
– OpenOffice 3 works, but has problems with very large views
Make sure users are using the correct views for lookups
– View names are not always obvious in existing databases with lots of views
If users are sharing reports/spreadsheets, the NotesSQL DSN must be set up
exactly the same for each user
© 2013 IBM Corporation
- 87. From the Server: Domino JDBC
OpenNTF project written by Philippe Riand
No Notes client required!
Access Notes view data directly from the Domino server using JDBC
© 2013 IBM Corporation
- 88. How It Works
Runs as a server task
– Uses DOTS (another OpenNTF project) to allow Java to run as a server task
– Domino 8.5.3 or higher
Define which views to expose as tables using an XML file in the database
Creates a SQLLite “virtual table” for each view
JDBC driver on the client reads the data from SQLLite tables on the server
Notes SQLLite JDBC
Views Tables Driver
© 2013 IBM Corporation
- 89. Setup: Install DOTS (Windows example)
Download from OpenNTF
http://openntf.org/p/OSGI+Tasklet+Service+for+IBM+Lotus+Domino
Unzip to c:dots
Copy these files from “c:dotsbuildwin32" to the Domino program directory:
dotsExtMgr.dll, dotsNSFHook.dll, ndots.exe
– Or buildwin64, or buildlinux64, or whatever
Create the following folders in the Domino program directory: osgi-
dots/shared/eclipse/plugins and osgi-dots/rcp/eclipse/plugins
Copy these files from “c:dotsbuild" to the Dominoosgi-dots directory: launcher.jar,
com.ibm.notes.java.api_XX.jar, and dotssec.jar
Copy c:buildcom.ibm.dots_XX.jar to osgi-dots/shared/eclipse/plugins
© 2013 IBM Corporation
- 90. Setup: Install DOTS
Download Eclipse 3.7.x (Indigo) from
http://www.eclipse.org/downloads/packages/release/indigo/sr2
– Get the “Report Developers” version, because we’ll be using BIRT later
Unzip Eclipse
– Use 7-Zip, not the native Windows unzip program
Copy required org.eclipse.* files from Eclipse to osgi-dots/rcp/eclipse/plugins
– These MUST be from Eclipse 3.6.x or 3.7.x. Other versions of Eclipse might not work,
and the versions already in the osgi directory on the Domino server definitely will not
work
org.eclipse.update.configurator org.eclipse.equinox.common
org.eclipse.osgi org.eclipse.core.runtime.compatibility.auth
org.eclipse.equinox.preferences org.eclipse.core.jobs
org.eclipse.equinox.registry org.eclipse.core.runtime
org.eclipse.equinox.app org.eclipse.core.contenttype
© 2013 IBM Corporation
- 91. Setup: Install DOTS
Add the following lines to notes.ini on the Domino server:
– NSF_HOOKS=dotsNSFHook
– EXTMGR_ADDINS=dotsExtMgr
Start the Domino server
Type "load dots" at the server
console
You should get a "Domino OSGi
Tasklet Container started"
message
– If you copied the samples.jar file,
you will also get lots of messages
every 60 seconds
To stop the DOTS service, type "tell dots quit"
© 2013 IBM Corporation
- 92. Setup: Install JDBC
Download from OpenNTF:
http://openntf.org/p/JDBC+Access+for+IBM+Lotus+Domino
Unzip to c:DomJDBC
Copy the following folders from {domino}/osgi/shared/eclipse/plugins to
{domino}/osgi-dots/shared/eclipse/plugins :
– com.ibm.commons.jdbc_XX
– com.ibm.commons_XX
Copy the following file from {domino}/osgi/rcp/eclipse/plugins to
{domino}osgi-dots/shared/eclipse/plugins :
– com.ibm.notes.java.api_XX.jar
– If you already have the java.api file from the DOTS install, just remove the existing one
and replace it with the one from Domino
© 2013 IBM Corporation
- 93. Setup: Install JDBC
Copy all the plugins from c:DomJDBCupdateSite-dots.zip to
{domino}/osgi-dots/shared/eclipse/plugins
Copy com.ibm.xsp.extlib.relational.domsql_1.0.0.jar from c:DomJDBCupdateSite-
xpages.zip to {domino}/osgi-dots/shared/eclipse/plugins
Restart DOTS
You should get a "Initializing DomSQL RMI server for Domino" message
© 2013 IBM Corporation
- 94. Setup: Install JDBC
The default JDBC port is 8089, and you need to make sure your firewall allows
ndots.exe to use that port
– You can change the port number by adding DomSQL_Port=XXXX in notes.ini
© 2013 IBM Corporation
- 95. Configuring Your Databases
Notes databases on the Domino server must be explicitly set up to allow JDBC
access to views
– Each database set up individually
– You can set up a single view, or all views, or a group of views
Security
– Only expose the views you want to expose
– Database ACL is respected
As with ODBC, you might want to create special “reporting” views
© 2013 IBM Corporation
- 96. Configuring Your Databases
Open the database in Domino Designer
Add the "Package Explorer" view if you don't
already have it (Window - Show Eclipse
Views - Other - Java - Package Explorer)
Expand the database in Package Explorer
and go to Web Content - WEB-INF
© 2013 IBM Corporation
- 97. Configuring Your Databases
Create a new folder under WEB-INF
called "jdbc“
– Right-click - New - Other - General - Folder
Create a new file in the new "jdbc" folder
called "views.domsql“
– Right-click - New - Other - General – File
– It doesn’t specifically need to be called
“views”, it can be anything with a .domsql
extension
– You can have multiple .domsql files
© 2013 IBM Corporation
- 98. Configuring Your Databases
To expose all views via JDBC, just use this in your .domsql file:
To expose specific views, rename views, override column properties:
© 2013 IBM Corporation
- 99. Configuring Your Databases
NOTE: DomSQL uses the "Programmatic Use" name for columns in the view. You
might want to override this either in the view itself or using a <column> definition in
the .domsql file.
© 2013 IBM Corporation
- 100. Example: Connect using SquirrelSQL
Many programs are available to access data via JDBC
One nice (free) one is SquirrelSQL
– Java, LGPL
– Plugins
– SQL code completion
– Export SQL query results to CSV, Excel,
XML format
– http://squirrel-sql.sourceforge.net
© 2013 IBM Corporation
- 101. Example: Connect using SquirrelSQL
Download and Install SquirrelSQL
Copy the com.ibm.domino.domsql.driver_XX.jar file to SquirrelSQL/lib
Start SquirrelSQL
Add a new driver with the following properties:
– Name: Domino JDBC Driver
– Example URL: jdbc:mysql://<hostname>[<:8089>]/<dbname.nsf>/<.domsqlname>[?
<user>=<value1>][&<password>=<value2>]
– Class Name: com.ibm.domino.domsql.DomSQLDriver
© 2013 IBM Corporation
- 103. Example: Connect using SquirrelSQL
Choose the menu option Aliases - New Alias. Create an alias with the following
properties:
– Name: LS13 Test
– Driver: Domino JDBC Driver
– URL: jdbc:domsql://localhost/ls13/SHOW104DuckData.nsf/Views
(“Views” is the name of the .domsql file)
© 2013 IBM Corporation
- 105. Example: Connect using SquirrelSQL
Use the Content, Row Count, etc.
tabs to view information
– Note that column data types are
automatically determined
Go to the SQL tab to run queries
Queries can be exported as CSV,
Excel, or XML
– Use the “Store Result of SQL in File”
toolbar button
© 2013 IBM Corporation
- 106. Example: Create a BIRT Report
BIRT: Business Intelligence and Reporting Tools
– http://www.eclipse.org/birt
Eclipse-based open-source reporting system
– Originally developed by Actuate
– Now a top-level Eclipse project
Can install stand-alone, as part of an Eclipse installation,
or as a runtime for J2EE integration
Reports can be published to a reporting server, or exported as PDF, HTML, Excel,
and other formats
© 2013 IBM Corporation
- 107. Example: Create a BIRT Report
Since we already downloaded Eclipse 3.7 for Report Developers (which includes
BIRT) as part of the DOTS setup, we will use that
Launch Eclipse
Create a new
workspace
Go to the workbench
© 2013 IBM Corporation
- 108. Example: Create a BIRT Report
File – New – Other –
Business Intelligence
and Reporting Tools –
Report Project
© 2013 IBM Corporation
- 109. Example: Create a BIRT Report
Give the project a name
Click “Finish”
Switch to the Reporting
Perspective when prompted
© 2013 IBM Corporation
- 111. Example: Create a BIRT Report
Give the report a name
Choose the “Chart & Listing”
template
Make sure “Show Report
Creation Cheat Sheet” is
checked
© 2013 IBM Corporation
- 113. Example: Create a BIRT Report
Create a new JDBC Data Source
© 2013 IBM Corporation
- 114. Example: Create a BIRT Report
Use “Manage Drivers” to use the Domino JDBC driver
© 2013 IBM Corporation
- 115. Example: Create a BIRT Report
Click “Add” to specify the location of the DomSQL driver
© 2013 IBM Corporation
- 116. Example: Create a BIRT Report
DomSQLDriver should now appear in the Driver Class list
© 2013 IBM Corporation
- 117. Example: Create a BIRT Report
Create a new Data Set using the Data Source
© 2013 IBM Corporation
- 118. Example: Create a BIRT Report
Create a SQL Query to define the Data Set
© 2013 IBM Corporation
- 119. Example: Create a BIRT Report
Right-click the table on the
report layout view and choose
"Edit Data Binding"
© 2013 IBM Corporation
- 120. Example: Create a BIRT Report
Select the data set you just created and click OK
© 2013 IBM Corporation
- 121. Example: Create a BIRT Report
You can now drag and drop fields from the data set onto the report and chart
© 2013 IBM Corporation
- 122. Example: Create a BIRT Report
Many formatting options for the report itself
– Columns
– Headers/Footers
– Grouping
– Page breaks
– Charts
Many output options for the final report
– PDF
– MS Office (DOC, XLS, PPT)
– OpenOffice (ODP, ODS, ODT)
– HTML
– Also send to a BIRT reporting server
© 2013 IBM Corporation
- 124. JDBC Optimization and Tuning
SQLLite tables can be created as temporary, memory, or file
Sorted columns can be used as indexes
Use indexed fields for JOINs
Avoid using COUNT in your SQL queries (let the reporting tool do that work)
Use the Debug page in the example database for testing
– And the NATIVE_TRACE_PERFORMANCE_HINTS option
You can create a .jdbc file if you want to add authentication info
Read the PDF that comes with the project for more tips
© 2013 IBM Corporation
- 125. JDBC Caveats
Windows-only right now
You really should set this up on a separate reporting server, rather than a normal
production server
– There can be memory issues
– The RMI task doesn't shut down properly unless you shut down the entire server.
Using default views along with the <view> definitions will result in duplicate tables
shown by the reporting client
If you have "Connection refused" errors, make sure DOTS is running!
– You can load DOTS/JDBC when the server starts up by adding "dots" to the ServerTasks
line
No connection is info shown on the Domino console
If you change your .domsql files, you have to restart the Domino server
© 2013 IBM Corporation
- 126. Contrast and Compare: NotesSQL vs. Domino JDBC
Notes SQL Domino JDBC
– Notes client required – No Notes client required
– No server configuration required – Server configuration required
– No database configuration required – Minor database configuration required
– Server or local databases – Server databases only
– XPages integration NOT recommended – XPages integration
© 2013 IBM Corporation
- 128. Issues Creating Files on the Domino Server
Sometimes you need to create a file directly on the Domino server
– Dynamic file generation for web page
– Create a file and attach it to an email
– A present for your Domino Administrator
With agents, you need to have:
– Unrestricted Access in the Security section of the server doc
– Restricted operations allowed in Agent security
With XPages, you need to have:
– Permissions set in the /jvm/lib/security/java.policy file
There are creative ways around this in certain situations…
© 2013 IBM Corporation
- 129. Example #1: Streaming a File from an XPage
In the first example, we want to generate an Excel spreadsheet on-the-fly and
send it directly to a web user
From a process perspective, it will be:
– The user clicks a link that says “Download Spreadsheet”
– The server creates a spreadsheet based on a view or custom parameters or whatever
– The user gets the “Where would you like to save this file” prompt
There are ways to trick Excel into thinking an HTML table is actually a spreadsheet
file
– See David Leedy’s http://xpagescheatsheet.com for an example
We want to generate an actual binary file and send it for this example
© 2013 IBM Corporation
- 130. Apache POI for Writing Native Excel Spreadsheets
http://poi.apache.org
Java only, no DLLs or MS Office installations required
You only need to make a single JAR file (poi-XX.jar) available to your code
Creates an actual binary Excel file, not a text file that can be opened with Excel
– Can read as well as write
– Formats: XLS, XLSX, DOC, DOCX, PPT, PPTX
– Also additional libraries for Visio, Publisher, and TNEF
Interesting examples in the HSSF (Office) and XSSF (Office XML) samples file
– Multi-sheet calendar
– Formatting, calculations, and freeze-panes
– Convert DOC or XLS to HTML for web display
© 2013 IBM Corporation
- 131. Attach the POI JAR File to the Notes Database
We will attach the Apache
POI JAR file directly to the
Notes database so it can be
used by the XPage
Open the database with the
Package Explorer view
– Like we did in the JDBC
example
Navigate to the
WebContent/WEB-INF folder
Right-click WEB-INF and
choose New – Other…
© 2013 IBM Corporation
- 132. Create a New Folder in WEB-INF called “lib”
© 2013 IBM Corporation
- 133. Copy the POI JAR File to the New “lib” Folder
Right-click the new “lib” folder
and choose the menu option
“Import”
© 2013 IBM Corporation
- 134. Find the POI JAR File and Attach It
© 2013 IBM Corporation
- 135. Add the “lib” Folder as a Source Folder
© 2013 IBM Corporation
- 136. Add the POI JAR File to the Build Path
This allows the Java file we will write to use the JAR for compilation
© 2013 IBM Corporation
- 137. Add the POI JAR File to the Build Path
© 2013 IBM Corporation
- 138. Create A Java Class
Next we need to create a Java
class that can be used by an
XPage
Java design element introduced
in Notes 8.5.3
For older versions, you can
create a class directly in the
“lib” folder in Package Explorer
© 2013 IBM Corporation
- 139. Create A Java Class
Needs a package name and a
class name
© 2013 IBM Corporation
- 140. Create A Java Class
Add code to generate an Excel file using POI
© 2013 IBM Corporation
- 141. Create A Java Class
The code writes to an OutputStream provided by the caller (SSJS from an XPage)
© 2013 IBM Corporation
- 142. Create an XAgent XPage
Create an XPage with a catchy name
© 2013 IBM Corporation
- 143. Create an XAgent XPage
We need to modify the source of the XPage to turn this into an XAgent
– http://www.wissel.net/blog/d6plinks/shwl-7mgfbn
This allows us to set the
HTTP response headers,
get a handle to the XPage
OutputStream, and call
the Java code we just
wrote
© 2013 IBM Corporation
- 146. What Just Happened?
The XPage used HTTP headers that indicated that this was an Excel file, and it
was a file download
The XPage called the Java class, passing it a reference to a Notes View and the
XPage OutputStream
The Java code generated a spreadsheet in memory and wrote the bytes directly to
the XPage OutputStream
The browser client received a file without ever having the file written to disc
© 2013 IBM Corporation
- 147. Example #2: Generating and Emailing a PDF
In the second example, we will generate a PDF on-the-fly and send it as an email
attachment
We will use an agent, so you have plenty of different ways to call it
Options for accessing a JAR file from an agent:
– Attach directly to an agent
– Attach to a script library
– Add to the jvm/lib/ext folder on the server
The trick here will be attaching to an email (or any Notes document really) without
creating a file on the server
© 2013 IBM Corporation
- 148. Java PDF Libraries
Several to choose from:
– Apache FOP
– Apache PDFBox
– PDFJet
– iText
– Many others…
Always check the license
Lowagie iText is an excellent option
– Initial release in 1999
– Currently on version 5.x (although it jumped from 2.1.7 to 5.0.0 in 2009)
– Open-source licensed as AGPL in version 5.0, dual-licensed as MPL/LGPL for version
2.x
– http://itextpdf.com
© 2013 IBM Corporation
- 149. Options for Creating a New PDF in iText
Generate the PDF from scratch
– Create a PDF doc in memory
– Add formatting, paragraphs, images, etc.
– Save to a file
Use an existing PDF as a template
– Interesting technique described by Jake Howlett (codestore.net)
– Create a template doc in Symphony or Word or whatever, and save to a PDF
– Open the PDF template file in iText
– Add paragraphs, images, etc. as above
– Save to a file
© 2013 IBM Corporation
- 150. Creating a Template
To generate this stunning template:
– I created a document in OpenOffice
– Added a headline and a graphic
– Used the “Export as PDF” option
© 2013 IBM Corporation
- 151. Creating the Java Agent
We need to create a Java agent and attach both the PDF template and the iText
JAR file to it
© 2013 IBM Corporation
- 152. Add the PDF Template to the Agent
© 2013 IBM Corporation
- 153. Add the JAR File to the Agent
© 2013 IBM Corporation
- 158. Agent Code (Attach the File)
The PDF generation method returns a byte array. Now we need a way to write a
byte array to a RichTextField.
© 2013 IBM Corporation
- 160. The Final Result
You can certainly make it
look prettier than this
iText has a lot of options
– Text styles
– Tables
– Images
– Barcodes
© 2013 IBM Corporation
- 162. Understand the Requirements
The scenario: someone says “I need that data”
Questions you should ask:
– Is this an actual requirement? (sometimes it's not)
– What are you doing with the data?
– How often?
– What else will you need?
Understand sorting, grouping, categorization
Is it for reporting, app integration, or migration?
Don’t set yourself up for failure
Reliability Reliability Reliability
© 2013 IBM Corporation
- 163. Understand the Users
Is this for a person, a department, or a committee?
Are you working with another developer, or a lesser being?
Does the user have a specific requirement or is this a fishing
expedition?
Have them walk you through the entire story of what they need
– Maybe they really need something different than what they're asking for
– Users don't generally know what their options are
If the user fails, you have failed
Data feeds loose their WOW factor very quickly and become utilities just like
electricity: under-appreciated but completely indispensable
© 2013 IBM Corporation
- 164. How Much Code Should You Write?
As little as possible with out sacrificing reliability
Always consider the "no-code" option first
– View export or copy-as-table
– NotesSQL
– ReadViewEntries (if it's another dev who needs it)
– No-Code means there's less to break
How often do you think the requirements will change?
– They will always change at least once
– Frequent changes == spend time upfront writing flexible code
Are you dealing with rich text or attachments?
The older the data, the more messed up it will be
© 2013 IBM Corporation
- 165. Try Not to Become a Report Writer
Try to make your code run without maintenance or constant updating
Try very hard to keep a separation between data and reports
– You are in charge of data
– Someone else is in charge of reports
– This is even true for migrations
Business users probably know the data better than you anyway
Writing reports is a slippery slope
– "Can I have another one sorta like that, but...?"
– "Just like this, but weekly instead of monthly."
– And just one more...
– And just one more...
You gotta do what you gotta do, just watch out
– Data is plenty of work all by itself
© 2013 IBM Corporation
- 166. Normalizing and Scrubbing Data
Consider which fields could be categories
– That data needs to be consistent
Are numbers and dates really numbers and dates?
– You'll need to run pre-exports
Fix it in the database or in the export?
© 2013 IBM Corporation
- 167. Use a Dedicated Report Server when Possible
Should it be replicated or copied data?
Make sure replicated data is one-way unless it specifically needs to be otherwise
Sometimes copies are better
– No risk of deleting/changing production data
– Re-run processes over and over and always expect the same data
Much less concern over slow processes, hangs, crashes
Admins more willing to "loosen" security
– Agent permissions
– Java security file
– Installing external JARs, DLLs, etc.
© 2013 IBM Corporation
- 168. Agenda
Overview
Working with Feeds
Integrating with IBM Connections
ODBC, JDBC, and Reporting
Generating Files on the Domino Server
Caveats, Considerations, and Data Scrubbing
Where to Get More Information
© 2013 IBM Corporation
- 169. Links and References
Sample database and slides
– http://londc.com/ldc.nsf/pages/goodies
Overriding Session Authorization on Domino:
– http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/topic/com.ibm.help.domino.admin.
doc/DOC/H_OVERRIDING_SESSION_AUTHENTICATION_8847_STEPS.html
Liquid XML Studio
– http://www.liquid-technologies.com/xml-studio.aspx
Soap UI
– http://www.soapui.org
Java Atom/RSS Feed Parsing:
– ROME – http://rometools.org
– Apache Abdera – http://abdera.apache.org
– Apache Wink – http://incubator.apache.org/wink
© 2013 IBM Corporation
- 170. Links and References
XAgents
– http://www.wissel.net/blog/d6plinks/shwl-7mgfbn
– http://www.mindoo.com/web/blog.nsf/dx/17.07.2011101855KLEBRW.htm
NotesSQL
– http://www.ibm.com/developerworks/lotus/products/notesdomino/notessql
Domino JDBC
– http://openntf.org/p/JDBC+Access+for+IBM+Lotus+Domino
SquirrelSQL
– http://squirrel-sql.sourceforge.net
BIRT
– http://www.eclipse.org/birt
Apache POI
– http://poi.apache.org
iText (Java PDF generation)
– http://itextpdf.com
© 2013 IBM Corporation
- 171. Related Sessions
AD206 : IBM Lotus Domino XPages: Embrace, Extend, Integrate
– Niklas Heidloff, IBM; Padraic Edwards, IBM
AD208 : IBM Lotus Domino XPages Performance in a Nutshell
– Maire Kehoe, IBM; Tony McGuckin, IBM
AD204 : How To Develop Great Applications Using XPages Design Patterns
– Tony McGuckin, IBM; Martin Donnelly, IBM
AD202 : Debug Server Side Javascript, Java, and XPages Apps Using the SSJS Debugger
– Dan O'Connor, IBM; Michael Blout, IBM
ID110 : Deep Dive into IBM Lotus Notes Calendaring and Scheduling and Related Systems
– Bruce Kahn, IBM
AD210 : It's Here! Calendar APIs And REST Services
– Dave Delay, IBM; Nathan Barry, IBM
BP208 : XPages Blast
– Tim Clark, TC Soft Consulting Limited; Matt White, Fynn Consulting Ltd
© 2013 IBM Corporation
- 172. Legal disclaimer
• © IBM Corporation 2013. All Rights Reserved.
• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is
provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not
be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any
warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.
• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this
presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing
contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.
• Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon
many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can
be given that an individual user will achieve results similar to those stated here.
• Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.
• Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
• Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.
• Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.
• UNIX is a registered trademark of The Open Group in the United States and other countries.
• Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.
• All references to [insert fictitious company name] refer to a fictitious company and are used for illustration purposes only.
172 © 2013 IBM Corporation
- 173. THANK YOU!
Mark Myers Julian Robichaux
London Developer Coop panagenda
mark@energywins.co.uk jrobichaux@panagenda.com
Twitter: @stickfight Twitter: @jrobichaux
© 2013 IBM Corporation