More Related Content Similar to OpenNTF Webinar Series: DQL with John Curtis September 2020 (20) OpenNTF Webinar Series: DQL with John Curtis September 20203. ASKING QUESTIONS
• First Question – Will this be recorded?
• Yes, view on YouTube!!!
• https://www.youtube.com/user/OpenNTF
• Use the Questions Pane in GoToWebinar
• We will get to your questions at the end of
the webinar
• The speakers will respond to your questions
verbally
• (not in the Questions pane)
• Please keep all questions related to the
topics that our speakers are discussing!!!
• Unrelated Question => post at:
• http://openntf.slack.com/
4. THANKS TO THE OPENNTF SPONSORS
• HCL made a significant contribution to help our
organization
• Funds these webinars!
• Contests like Hackathons
• Running the organization
• Prominic donates all IT related services
• Cloud Hosting for OpenNTF
• Infrastructure management for HCL Domino and Atlassian
Servers
• System Administration for day-to-day operation
5. THIS IS OUR COMMUNITY
• Join us and get involved!
• We are all volunteers
• No effort is too small
• If your idea is bigger than you can do on your own, we
can connect you to a team to work on it
• Test or help or modify an existing project
• Write guides or documentation
• Add reviews on projects / stars on Snippets
6. NEXT WEBINAR
• October 22, 2020
• Michael Smith – A Flexible View Control for XPages
• Signup at https://openntf.org/webinars
7. UPCOMING EVENTS
• Let’s Connect Virtual Event – September 29-30
• https://letsconnect.world/agenda/
• Collabsphere – October 27-29
• https://collabsphere.org/ug/cs2020.nsf/index.html
• DNUG – monthly online events
• https://dnug.de/en/dnug47online-2/
8. 8 | Copyright © 2019 HCL Technologies Limited | www.hcltech.com8 | Technology for the Next Decade, Today Copyright © 2019 HCL Technologies Limited | www.hcltechsw.com
What's New with Domino Query
Language (and Domino Appdev)
John Curtis
Software Architect, Project Lead, Domino Development
HCL Digital Solutions
@john_d_curtis_
http://www.jdcurtis.blog
john_curtis@pnp-hcl.com
9. Copyright © 2020 HCL Technologies Limited | www.hcltechsw.com
DQL Evolution
October 2018
December 2018
1H-2021
March 2020
• Consolidation of Domino search and
results processing
• Constant improvement in
• Performance
• Capability
• Multiple venues and
programming models
December 2019
DQL V1
FT Support
Inboard catalog
FT Support completed
Statistics
Performance
Results processing (sort, federation, joins)
Formula Language inclusion
10. v11.0.1 DQL Features
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• Complete full text index utilization – enable numeric
and date terms to take advantage of FT indexes
▪ 5-200X improvement in speed
▪ DQL plan and costing executes FT terms first, then views,
then NSF scans
▪ Seamless – existing DQL syntax will work automatically on
terms and across Boolean operators
▪ Caveats/limitations:
➢ Single-occurrence fields only (Domino does NOT FT index multiple
field values of numbers and dates)
➢ No TIMEs in the datetime values, only dates
11. v11.0.1 DQL Features
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• DQL search strategy by term type
Term type Full Text Search View Search NSF Scan
field = ‘text’? No Yes Yes
field contains (‘Text’) Yes No No
field = 123122 Yes * Yes Yes
field > ‘text’ No Yes Yes
field < 123122 Yes * Yes Yes
field = @dt(‘2020-09-17’) Yes * Yes Yes
field > @dt(‘2020-09-17’) Yes * Yes Yes
field = @dt(‘2020-09-17T11:00:00.00’) No Yes Yes
* = single occurrence only (FT search does NOT handle ranges or lists)
12. v11.0.1 DQL Features (continued)
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• DQL statistics
▪ Avg, max, min by term type
and overall
▪ “show stat query*”
• Enhanced Boolean
optimization
▪ Deeper results injection
▪ Reduces view and especially
NSF scan processing for
much faster query times
13. v12 Early Access September 2020 drop DQL Features
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• DQL client/server
▪ DominoQuery working against remote databases (not
supported till now)
▪ Transaction appears in the “show trans” list from the
Domino console
▪ Full support for substitution variables, entire DQL syntax
14. v12 Early Access September 2020 drop DQL Features (continued)
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• Embedded Formula Language
▪ Syntax
▪ @FL or @formula (case insensitive)
▪ Embedded syntax enclosed in single quotes
@fl(‘@length > 2320)
@FORMULA(‘@left(fld1;3) = "abl" or @left(fld2;2) = "ab“’)
▪ Provides
▪ Intra-document terms (between different fields)
▪ Full expression support
▪ HUGE library of existing functionality
15. v12 Early Access September 2020 drop DQL Features (continued)
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• Embedded Formula Language (continued)
▪ Support for substitution variables
@fl(‘@length’) > ?lengthvar
@formula(‘@left(fld1;3)’) = ?fld1var or @FL(‘@left(fld2;2)’) = ?fld2var
▪ Complete compatibility with the rest of DQL
▪ All formula language terms satisfied via NSF scanning
▪ All ANDed/ORed terms satisfied with a single pass
▪ Current implementation
▪ 256-byte maximum Formula Language syntax allowed
▪ No substitution variable support for an entire FL string
17. v12 Work in progress
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• Results processing
▪ Extensive handling of document collections
➢ NOT just DQL – ALL document collections
▪ Across multiple databases and database schemas
▪ Flexible computed values everywhere
▪ Sorting (including categorization)
▪ Combine rules (joins, lookups, etc.)
▪ Aggregate function support
▪ Impossible to avoid complexity
18. v12 Work in progress
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• Results processing (continued) – the details
▪ (Notes)QueryResultProcessor
➢ New backend (Java/Lotusscript) class
➢ Supplied lists of data and other classes working together
➢ Builds on known classes never combined or used this way before
➢ Extensive capabilities, multiple output formats and objects
➢ Low code? Not so much, but let’s build one to see
19. v12 Work in progress
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• (Notes)QueryResultProcessor (continued) – methods
▪ AddSortColumn(programmatic name, column title, sort order,
hidden, categorized)
▪ Akin to creating a view column on the fly
▪ In fact, one of the output types IS a view
▪ May add additional attributes but those shown should be
familiar from use in Designer
▪ REMEMBER the programmatic name – it will appear again in
other calls
▪ If NOT overridden, will be used as a field name to fetch
20. v12 Work in progress
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• (Notes)QueryResultProcessor (continued) – methods
▪ AddDocumentCollection((Notes)DocumentCollection doccol,
collection name)
… or …
▪ AddDominoQuery ((Notes)DominoQuery, collection name)
▪ (Notes)DocumentCollection can be created against any
database using any means (db.search, ftsearch,
getalldocumentsbykey or DQL)
▪ DominoQuery option will also execute DQL for you
▪ Remember collection name – will appear later
21. v12 Work in progress
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• (Notes)QueryResultProcessor (continued) – methods
▪ AddFormula(document collection name, programmatic column
name, formula)
➢ Used to override the programmatic name with a formula to
compute and fetch values for processing by the results engine
➢ Can use one call to override the formula for multiple document
collections using wildcards
Addsortcolumn (“name”, “Name Column Title”, “ascending” … ) …
AddDocumentCollection(doccol, “doccol1”) .. e.g. from db.search - database 1
AddDocumentCollection(doccol2, “doccol2) .. e.g. from ftsearch - database 2
AddFormula(“doccol*”, “name”, “@Propercase(NameField)”)
22. v12 Work in progress
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• (Notes)QueryResultProcessor (continued) – methods
▪ AddCombineRule (left programmatic name, rule, right
programmatic name)
➢ Prior to sort or view create, combine data using the sort columns
(with applied formulas) using the specified rule (or method)
➢ “Join” is the most familiar rule type, also keyword lookup ++
▪ SetOutputType (type, viewname, streamtype)
➢ View (with name) or stream (with streamtype)
▪ Execute (run the result processor)
23. v12 Work in progress
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• (Notes)QueryResultProcessor how it works
Sort
doccol1
DQL1
ftdoccol
Dbsearch
doccol
DQL2
Combine Fetch
View
Specified
output
Input
(Notes)DocumentCollections
Optional join/lookup
Quick fetch
including DQL executes
(uses formula overrides
Domino sort service
(with streaming optimization)
or
24. v12 Work in progress
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• (Notes)QueryResultProcessor additional information
▪ NEW - Sort package is decoupled from view building
➢ In stream mode, we don’t need to wait for a completed view
build
▪ Separate column formulas for each DocumentCollection
▪ Views created can be used and discarded or retained
➢ Given input DocumentCollections, views cannot be refreshed
➢ Document security problem (multi-db documents have multiple
readers and authors lists)
▪ Join strategies – sort/merge and nested loop using views or
FT indexes
25. Early Access September 2020 drop non-DQL Feature
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• Domino Transaction Support in Lotusscript/Java
▪ (Notes)Database object has 3 new methods
➢ TransactionBegin – starts a unit of work
➢ TransactionCommit – completes a unit of work
➢ TransactionRollback – aborts everything since TransactionBegin
▪ Automatic rollback happens when
➢ Database object is closed or goes out of scope
➢ Agent is terminated
▪ Large, uncommitted transactions can cause log file full (crash)
▪ Transactions cannot nest (one db, one level, one at a time)
26. v12 Work in progress – non-DQL
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• Field length limit relief
▪ Aha idea - https://domino-ideas.hcltechsw.com/ideas/DDXP-I-5
▪ Traditional limits:
➢ 32k limitation on field size
➢ 62k limitation on document summaries
• Other problems:
➢ Better error messages when limits are hit
➢ Support in views and folders for larger fields
➢ Designer support (code size)
➢ Other Domino limitations – like 32k Lotusscript arrays
27. v12 Work in progress – non-DQL
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• Field length limit relief (continued)
• (load) compact <db> -ls (or -largesummary) on
➢ First delivered ~ 9.01FP8
➢ New max sizes:
✓ All fields (text lists or number or time ranges) – 64K
✓ Document summary size - 16MB (had been 62k)
➢ Requires compact to ODS 52
• Have you tried this?
28. v12 Work in progress – non-DQL
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• Field length limit relief (continued)
▪ Multiple core API and persistent data structure changes
▪ Will require a new ODS and large summaries enabled
▪ Only the max summary size (16MB) will remain –
➢ That is, the SUM of the lengths of all summary fields
➢ No size limit within the 16MB for individual summary fields
▪ Nonsummary fields can be up to 4GB (not a typo)
▪ Retain a hard limit on view-indexable fields (64K)
29. v12 Work in progress – non-DQL
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• Field length limit relief (continued)
• “Field is too large (32K) or View's column & selection formulas
are too large”
➢ Catch-all, multiple venues and error states throw the error
➢ Often is the SUM of field lengths generating the error – no one
field is responsible
• Effort for v12 – separate the states into 2 types and improve –
➢ Overall summary size – provide a document summary “map” to
show sizes only
➢ Individual fields – include field name in the message