This document discusses techniques for compacting, compressing, and de-duplicating data in Domino applications to reduce storage usage and improve performance. It covers compacting application design elements and documents, compressing document bodies, recompressing attachments using LZ1, planning DAOS rollout with the DAOS Estimator tool, configuring and implementing DAOS to store attachments externally, and resolving issues that can occur with DAOS. The goal is to leverage existing storage, reduce application sizes, and avoid redundant attachments.

1. Compact, Compress, De-Duplicate
Ulrich Krause
ILUG, 10. – 12. November 2010,
Belfast, Ireland

2. www.eknori.de
About: Ulrich Krause
Lotus Notes / Domino Administrator & Developer since 1993
OpenNTF project chef
– !!HELP!!
– Cebulon
– TriggerHappy
German Notes Forum (http://www.atnotes.de)
Blog http://www.eknori.de

3. www.eknori.de
The situation
More and bigger documents and attachments
More expensive storage is needed
The average size of an application increases
Duplicate attachments ( n times )
Duration of standard tasks increases
More time for Backup
Network traffic increases
I / O and performance issues

4. www.eknori.de
The aim
Leverage existing storage
Reduce size of applications
Avoid attachment redundancies
Speed-up standard tasks
Reduce backup time
Reduce network traffic
Reduce disc I / O

5. www.eknori.de
Application model ( simplified )
Design
Documents
– Text
– Attachments
– Embedded objects
Design
Text
Attachments

6. www.eknori.de
Compact Secrets
DEBUG_ENABLE_COMPACT_8_5=1
– W nn - Only compact databases which haven't been compacted
successfully in the last nn days.
– w - Exclude system databases (e.g., log.nsf, names.nsf).
– X nn - Limit each database compaction to no more than nn minutes.
(Applies to in-place compaction only)
– x nn - Limit total compaction time to nn minutes.
– # nn - Execute up to nn compactions in parallel using different threads.
• Don‟t use more than 8 threads in parallel
– ODS performs a copy-style compact only on databases with an ODS
smaller than the ODS provided with this parameter.

7. www.eknori.de
Compact Secrets (Sample)
Load compact -# 8 mail
Compacts all applications in the mail subdir in 8 parallel
Threads

8. www.eknori.de
Compact via application properties
Compacting a database does not
necessarily reduce its size
If the Notes database requires
structural changes (typically a change of
ODS [on-disk structure]). then a
copy-style compaction will be performed (equivalent to -L).
If the Notes database is logged (Transactional Logging
enabled), then a simple in-place compaction will be
performed (equivalent to -b).
If the Notes database is not logged (Transactional Logging
disabled), an in-place with file size reduction compaction will
be performed (equivalent to -B).

9. www.eknori.de
Copy Style Compaction
The compact -C (copy-style compaction) (1) creates a new
empty application container on the same server with the
same replica ID and a .tmp suffix, (2) copies all of the
documents from the original application to the new application
container.
After all documents have been populated in the new
application, (3) the old .nsf application is deleted and the (4)
.tmp file is renamed to the original filename.

10. www.eknori.de
Defrag
File fragmentation saps Domino server I/O performance
Even a slightly fragmented file can cause a 50% decrease in
non-cached read performance
Load compact –C does not allocate space needed before
compacting the application
Applications are fragmented during copy-style compaction
Full Text Indexes are fragmented too
Even NLO files in DAOS repository are fragmented over the
time
Use a tool to defrag your files speed up your Domino server !

11. www.eknori.de
Defrag
DefragNSF
• Adam Osborne
• http://www.preemptive.com.au/defrag
Domino Defrag
• Andrew Luder, OpenNTF
• http://www.openntf.org/Projects/pmt.nsf/Pr
ojectLookup/DominoDefrag
Both products use
the Microsoft
defragmentation
API

12. www.eknori.de
Defrag
Sample: Lotus Software KnowledgeBase ( 5 * compact –c )
Before
– 68 fragments
– 7550 ms
After
– 1 fragments
– 125 ms

13. www.eknori.de
Defrag
Domino Defrag uses IBM Lotus Notes C API
Open Source
FREE
Well documented
32/64 Bit support
New in Version 3: Free Space Defragmentation
– Uses jkdefragcmd.exe (http://kessels.com/jkdefrag/)
There is also a professional support plan available

14. www.eknori.de
Design Note Compression

15. www.eknori.de
Design Note Compression
New feature as of Domino 8
Needs ODS 48 or above
– Create_R8_Databases=1
– or Create_R85_Databases=1
– load compact -c
– load compact -c –n
Check “Compress database design”
Reduces size of design by 55 – 60 %
Reduces disc I / O

16. www.eknori.de
Design Note Compression
Design
Text
Attachments
Design
Text
Attachments

17. www.eknori.de
Document Body Compression

18. www.eknori.de
Document Body Compression
New feature as of Domino 8.0.1
Reduces size of Rich Text up to 60%
Needs ODS 48 or above
load compact -c –v
Check „Compress document data“

19. www.eknori.de
Document Body Compression
Design
Text
Attachments
Design
Text
Attachments

20. www.eknori.de
LZ1 Attachment Compression

21. www.eknori.de
LZ1 Attachment Compression
As of release 6
Replaces Huffmann algorithm
NOT active when creating a new
applications.
Recompress using
• load compact -c –ZU
• In 6.5.6 also set
DEBUG_ENABLE_LZ1_REPAIR=1
To avoid problems with JAVA applets
in <= 8.5 set
• DEBUG_ENABLE_LZ1_HOST_TYPES=0
• fixed in 8.5.1

22. www.eknori.de
LZ1 Attachment Compression
There have been problems reported that the fulltext search
does not work as expected using LZ1 compression.
To revert back to Huffmann use
– Load compact –c –ZD
– In 6.5.6 also set DEBUG_ENABLE_LZ1_REPAIR=1

23. www.eknori.de
LZ1 Attachment Compression
Design
Text
Attachments
Design
Text
Attachments

24. www.eknori.de
DAOS Estimator

25. www.eknori.de
DAOS Estimator
Free tool to plan your DAOS rollout
• Download: http://www-
01.ibm.com/support/docview.wss?rs=463&uid=swg24021920

26. www.eknori.de
DAOS Estimator (notes.ini)
DAOS Estimator by default has a large gap in attachment
size evaluation between 64k and 1 MB
– DAOSEST_BUCKETS=16,64,128,256,512,768,1024,2048,3072,4096
– The above line sets to:
– 16K, 64K, 128K, 256K, 512K, 768K, 1MB, 2MB, 3MB and 4MB
Source: Lotus Software Knowledge Base
• http://www-
01.ibm.com/support/docview.wss?rs=0&q1=1418102&uid=swg21418102

27. www.eknori.de
LZ1 != HUFFMAN
Db1 – LZ1 enabled
Db2 – LZ1 disabled
 Result is 2 .NLO Files !!

28. www.eknori.de
D A O S

29. www.eknori.de
DAOS
Domino Attachment Object Service
Attachments are no longer stored in application
Attachments are stored as NLO (Notes Large Object )
Document contains only a ticket
Separates static and dynamic data
Only one copy of a bit-like attachment
Transparent to users and 3rd-Party Products

30. www.eknori.de
Prerequisites
Domino 8.5 ( ODS 51)
SHARED MAIL OFF
Transaction protocol
Create DAOS Repository
Activate DAOS on server
Activate DAOS on application

31. www.eknori.de
Switch OFF Shared Mail
Serverdocument – Shared Mail
Field Shared Mail = None
„Load object unlink shared.nsf“

32. www.eknori.de
Transaction Protocol
Server document – Transactional Logging
Required for DAOS – no discussion!!
Use a separate disk drive.
Create log file folder
Choose method
• Circular
» 4GB limit
• Archive
» No limit, backup will delete logs
• Linear
» Like circular, NO size limit
CREATE_R85_LOG=1
Restart server

33. www.eknori.de
DAOS Repository
Use separate disk drive
Can be SAN, NAS
Each server has it„s own repository
– Max.1.000 subdirectories
– Max 40.000 .NLO / subdirectory
– = 40.000.000 unique .NLOs
– Daos.cfg ( in Domino Data Directory)
– Daoscat.nsf
.NLO files are encrypted with server.id
– DAOS_ENCRYPT_NLO=0
Include repository in backup!!

34. www.eknori.de
DAOS Repository
How do you change the number of DAOS NLO's per
directory?
DAOS_MAX_FILES_PER_SUBCONTAINER=40000
Requires a server restart
Will only affect the current DAOS subdirectory (older
subdirectories are not changed).
This will also affect the maximum number of files in the DAOS
store which is currently this values multiplied by 1000 (this
value is not configurable currently).
• 1.000 * DAOS_MAX_FILES_PER_SUBCONTAINER = MAX_NLO

35. www.eknori.de
Activate DAOS on server
Goto Serverdocument – DAOS
Set Minimum size of an object
• recommended 64kB (= DEFAULT in 8.5.1 )
• Evaluate with DAOS Estimator
Set DAOS base path
• Put is outside the data path!
Set Delete objects in repository
• Depends on your backup strategy
• 0 Days = immediate delete after the refcount = 0
• Manually from console: TELL DAOSMGR PRUNE 0
 Restart the server
 show server

36. www.eknori.de
Activate DAOS on applications
Application  properties  „propeller
hat“ for single application
Use Admin client for multiple
applications
Valid for new attachments only!
To move existing attachments into the
DAOS repository
– Load compact –C –DAOS ON <PathNSFs>

37. www.eknori.de
Bug Alert !!!
load compact -c –DAOS ON mail (w/o trailing backslash)
enables DAOS on ALL applications, not only on those in mail
subdirectory !!
Documented Lotus Software Knowledge Base #1410926
SPR AHOE7XHM9J (fixed in 8.5.2)

38. www.eknori.de
The Result
Design
Text
Tickets
Design
Text
Attachments
DAOS Repository

39. www.eknori.de
Transaction Protocol Trap
load compact –c –DAOS ON
NO attachment in DAOS repository
tell daosmgr dbsummary count = 0
load compact -C -T -DAOS on

40. www.eknori.de
DAOS and replication
DAOS is a server feature!
On a client replica, all attachments are stored in the
application.
Replicas on a NON-DAOS server store attachments in the
application
Each server in a cluster has it„s own DAOS repository.

41. www.eknori.de
DAOS and replication (cont.)
 Client  Server (as of 8.5.1)
– „known” attachments are no longer transferred to server = „smart
replication“
• Forward mail
• Reply with history
Server  Server (as of 8.5.1)
– “known” attachments are no longer replicated to target server
8 new statistics
– show stat DAOS

42. www.eknori.de
DAOS and replication (cont.)
Domino 8.5
Domino 8.5.1
WARNING!!: ADMINP_ACCELERATED_REPLICA_OVERRIDE=4 !!

43. www.eknori.de
DAOS and replication (last slide )

44. www.eknori.de
DAOS and mail.box
Remove RM_NO_LOG_OBJECTS_IN_MAILBOX from
notes.ini.
Enable DAOS on all mail.box files
Enable DAOS on all mail files
Increases performance

45. www.eknori.de
DAOS and Encryption
Increases number of NLO files!
– User A writes an encrypted mail with attachment to User B
– User B writes an encrypted mail with same attachment to User A
– Results in 4 NLO Files in repository
Does not save space but separates static from dynamic data

46. www.eknori.de
DAOS and Quotas
Technote: 1405456
Depends on the quota enforcement setting that is in place
Check space used in file when adding a note
– Acts on the physical size of the application (without attachments)
– Possibly a problem when switching DAOS off!!
• 64GB limit
Check file size when extending the file
Check file size when adding a note
– Both acts on the logical size of the application (with attachments)

47. www.eknori.de
Keep daoscat.nsf in sync
DON„T use DDM !!
Use program document
Tell daosmgr resync
SPR# DROO7ZNPZ8
Added a new DAOSMGR resync switch
to perform a quick resync.
This new switch updates the DIT and
DOI but does not run through the
applications or updates the NLO
refcounts. ( as of 8.5.1 IF1 )

48. www.eknori.de
„Needs Resync“
DDM points to the first application that causes the DAOS
catalog to go into „Needs Resync“ state.
If you cannot find a cause for the issue set
debug_threadid=1
console_log_enabled=1
daos_logging=<path/daos.txt>STATE_CHANGE
Many improvements in 8.5.2 that reduces the number of state
changes (SPR# RCOR83NLQK)

49. www.eknori.de
DAOS resync time window feature
The following two notes.ini parameters control when resync
will be allowed to run
– DAOS_RESYNC_START_TIME=12:00:00 AM
– DAOS_RESYNC_STOP_TIME=04:00:00 AM
Note that these parameters do not launch resync
automatically; they just control whether or not resync may run
at a given time, and also tell resync when to stop.
A program document could be created to launch resync just
after midnight. If the catalog is SYNCHRONIZED, then the
resync will not take place. However, if a resync is needed,
then the resync process will begin. When the end time is
reached, the resync threads will finish up the databases they
are processing and exit.

50. www.eknori.de
DAOS OFF
What happens when DAOS is disabled / switched OFF?
In application properties
– No documents with attachments in the application
– Application has documents with attachments
• Status in Admin Client changes to „Read Only“
• DAOS still works for already stored attachments but no new attachments are
written to the repository for the specific application.
In server document
– Same as 1

51. www.eknori.de
DAOS and missing NLO
NLO has been deleted on the OS level
TELL DAOSMGR LISTNLO MISSING <application.nsf>
Output to <datadir> by default

52. www.eknori.de
Daosmgr in Domino 8.5.2
daosmgr options have been enhanced in Domino 8.5.2
You can now do a search for missing NLO on all applications
in a subdirectory
– tell daosmgr listnlo missing mail

53. www.eknori.de
DAOS and missing application
NSF has been deleted on the OS level
 Message on server startup
• Informational - The DAOS catalog is not synchronized. Deletions will be
postponed. Please run 'tell daosmgr resync' at the next convenient
opportunity to re-synchronize.
Has the application been deleted by accident?
– NO
• TELL DAOSMGR RESYNC oder RESYNC FORCE
– YES
• Restore from backup
• Create a new replica from a cluster mate
 Same behaviour when moving an application (on OS level)

54. www.eknori.de
Can we save another 50%
If we disable encryption, then we could put NLO for all
servers into a single repository …
Yes, in theory
It„s only experimental
Does not work because
each server has its own
daoscat.nsf.
Would cause problems
with pruning

55. www.eknori.de
DAOS and Nagios ( Statistics )
Catalog.Status
Repository.Size
 Plugin

56. www.eknori.de
DAOS amd Statistics
8 new statistics (8.5.1)
– show stat DAOS
8.5.1
Debug_NSF_Show_Allstats=1

57. www.eknori.de
DAOS and Statistics
Windows:
@echo off
setlocal
for /f "tokens=*" %%a in ('dir/w/s %1 ^|findstr "Datei(en)"') do (
set info=%%a
)
echo %info%
Linux:
#!/bin/sh
REPOSITORY=/local/daos
OUTFILE=/local/daos.txt
fCount=`find $REPOSITORY -type f | wc –l`
fSize=`du -ksb $REPOSITORY`
IFS=”/”
array=($fSize)
fDate=$(date +”%d-%m-%Y”)
echo $fDate / $fCount / ${array[0]}>> $OUTFILE

58. Use XML

59. www.eknori.de
DAOS and daos.cfg
Contains information on file count and file size of .NLO files
Created on server startup when it has been deleted
Updated on server shutdown
Do not backup or restore!!

60. www.eknori.de
DAOS and daoscat.nsf
Contains index and refcount of NLO
Contains index of all applications using DAOS
Does NOT contain any view
Re-Created on re-sync
Created on server startup when it has been deleted
Do not backup or restore!!

61. www.eknori.de
Admin Client ( Files tab )
New columns regarding DAOS
– Logical size
– Physical size
– DAOS status
– DAOS count
– DAOS size
New columns regarding compression

62. www.eknori.de
Backup & Restore
No change in backing up NSF files.
Additional backup job for NLO files
NLO files do not require any special agent for backup
IMPORTANT: the backup sequence
– server is up during backup
• NSF before NLO
– server is down during backup
• it‟s your choice
IBM Technote # 1358548 - DAOS Backup and Restore

63. www.eknori.de
Resources
DAOS Wiki
• http://www-
10.lotus.com/ldd/dominowiki.nsf/archive?openview&title=DAOS&type=cat&c
at=null&tag=DAOS
DAOS Backup and Restore
• http://www-10.lotus.com/ldd/dominowiki.nsf/dx/daosbackup-and-restore
DAOS Estimator
• http://www.ibm.com/support/docview.wss?rs=463&uid=swg24021920
DAOS Technical Overview
• http://www.nelotus.org/A55CBA/nelotus.nsf/87fde291d7608ada852564c9006
eeba5/b6523e7f4b15585b862575b5004d3488/$FILE/DAOSTechnicalOvervi
ew_NELotus.pdf

64. www.eknori.de
This is it!
Contact me at:
ulrich.krause@eknori.de
Thank you
dank u wel
merci
Danke