Session from NCUG. Stockholm 12.06.2019. Basic Domino Performance Tuning. Ideas how to improve performance, statistics how to get information that we have issues and how to fix them

1. Domino/Notes
Performance tuning
Episode 1. Basic Tuning
Vladislav Tatarincev, vlaad@cyone.eu
CEO, CYONE
www.cyone.eu

2. • Domino experience since 1998.
• IBM Premier Business Partner
• Professional team:
- Certified IBM Lotus Domino Administrators (4.x,5.x,6.x,6.5,7.x,8.x,8.5.x,9.0)
- Tivoli Certified Professional (IBM Security/Tivoli Directory Integrator)
- IBM DB2, Oracle DBA
- Designed 2FA solution for IBM Verse/Domino iNotes/XPAGES www.cyone.eu/2FA
- IBM Domino Monitoring www.cyone.ru/CYMON/
- Cо 3 IBM Redbooks.
- Optimizing Domino Administration
- Domino 8.5 Deployment Guide
- Lotus Traveler Deployment Guide
- Panagenda IONET SOFTWARE
- EXTRACOMM RedHat Advanced Partner
- MobileIron Partner
- Barracuda Crossware
Company “CYONE”

3. Vladislav Tatarincev (vlaad@cyone.eu )
Blog www.vlaad.lv
• Domino/Notes Experience from 1998
• Certified IBM Domino Administrator (4.x,5.x,6.x,6.5,7.x,8.x,8.5,9.0)
• Certified IBM Domino Developer (5.x,6.x,6.5,7.x,8.x,8.5,9.0)
• Сertified IBM Instructor (6.x,6.5,7.x,8.x,8.5,9.0)
- CoAuthor 3 IBM Redbooks
- Optimizing Domino Administration
- Domino 8.5 Deployment Guide
- Lotus Traveler Deployment Guide
- Speaker on Domino/Notes usergroups:
- IBM Domino Performance Tuning
- IBM Connections Adoption - Kaizen

4. Agenda
• How we can get info that we have performance issues
• How to speed up Domino
• How to speed up Notes
• How to improve performance of Domino databases
• Common problems and how to solve them

5. What affects Domino/Notes performance?
• Network
• Hardware
• CPU (agents, Indexes, LotusScript/Java)
• HDD (read / write)
• RAM (Swap/Virtual Ram)
• Server OS
• Domino Configuration
• Database / ODS/ Advanced Properties
• Notes OS
• Notes OS version, configuration, antivirus

6. How to get info that there are problems with
performance?
• Domino Statistics
• Monitoring (Domino server reply)
• User Complains?
• Developent tools/LotusScript/Profiler

7. Network
• Encryptions slows down Notes
• Encryption works if enabled on Domino OR Notes
• Encryption of Databases (Medium/Strong)
• Medium old/Medium new (9.0.1.x))
• ENABLE_MEDIUM_ENCRYPTION_FIX= FFFFFFFB
• https://maironmendes.files.wordpress.com/2014/02/id104.pdf

8. Network
• Compression may speed up Notes up to 2 times
• Compression works ONLY IF,enabled on client and server
• Not for Citrix

9. User complains should not be main source of
information
• User complains ar subjective
• Hard to measure performance problems
• May depend on mood or missed deadlines
• When we improve something, after few days people
does not notice improvement

10. Server statistics
Good news:
• Server tells what is wrong , where is a bottleneck
You must know which parameter means what.
There are 3000-4000 statistics, you should have knowledge to interpret
them, what are border values, when you need to trigger alarm
Tools:
SHOW STAT
Domino Administrator
3rd Party tools Li

11. IBM Domino Administrator Monitoring / DDM
oThere are too many statistics, you must react on values 24x7
oDDM works only with numeric values 
oYou may skip important event
oDDM works only when server is up (you may ping it from other servers
with DDM probes)

12. DDM.NSF / Events4.nsf – existing Performance
Probes

13. DDM.NSF/Events4.nsf

14. • Domino ˜400 metrics and stats checked
24/7
• Many Dashboards for different purposes
• Users
• Mail
• Network
• Cluster server stats
• Custom databases/3rd party
• Performance
• Traveler/IMSMO/Verse
• You can create your own
dashboards
• www.cyone.eu/CYMON/

15. • Ability to monitor custom / 3rd
party applications:
• Number of documents in queues
• Number of document @Formula
• Database open time
• Database search time
• CYMON can solve found issues
according to scenario, run script,
command, agent, etc.

16. Disk load
• Platform.LogicalDisk.1.*
• AvgQueLen.AVG – Average (important)
• AvgQueLen.Peak – Max (less important, since spikes may occur)
• Platform.LogicalDisk.1.AvgQueLen - Current disk load
• If value is more than 1, then disk system is overloaded, if >2 then
real problem.
• No users in the night, load is less. Average = Load in evening, Early
morning + working day.
• If Average load is 2, it means 16 hours outside working hours, 8
working hours
(Morning load+WorkingDayLoad+EveningLoad)/3=2
• 3x2=6 (6=1+4+1 или 1,5+3+1,5)

17. The View is being updated (ask for DomLogSlim.ntf)

18. Notes.ini Updaters=1 or 7. Number of Queues 1 or
7

19. Notes.ini Updaters=6 (default 1)
• Must not exceed CPU
• In this example, 4 tasks, each has MAX=11-12, then 4х12=48 (if 1)
• If Max is high, then user waits for system

20. Full Text index
• CPU Spikes on servers, most common reason - FT
• Default value – Immediate (Must be changed )
• Free App from CYONE – Finds all FT and shows triggers, or fixes them
(Immediate->Hourly, Hourly->Daily)
https://cyone.box.com/s/ps0nhyg54uy4ttve5p1mpq8oagbyp3u9
• V10 Feature (<200 docs preindexed on Search)

21. Domino Statistics - your best friends
• If you compare TimeMS and server UpTime you can calculate % server
spends on FT from start time
• Sometimes FT works 90% of time
which means huge load on CPU/HDD

22. Catalog.nsf – You can find FT Sizes in Catalog.nsf
Notes.ini CATALOG_DISK_USAGE=1 will show Whitespace in databases
All Databases appear in catalog.nsf, even if «list in catalog” not set
https://www.slideshare.net/gabturtle/domino-adminblast

23. Separate Views fromFullText (a-la VIP PASS)
• UPDATE_FULLTEXT_THREAD=1

24. Inline View (9.0.1 FP9) and V10 Hot Views
• View rebuild priority VIP databases
• Notes.ini INLINE_VIEW_INDEX=sales.nsf,marketing.nsf
• Or enable per database or per view
load updall disc9.nsf -T "By Author" -inline on
• For all views in database:
load updall disc9.nsf -inline on
Domino v10
NIF_VIEW_USAGE_ENABLED=1
Will find frequent views and precache/build it. (Turned off by
Default, must be enabled by notes.ini)

25. OS Tuning
• Why Windows Server need to index files (word,xls) on Domino (nsf). Turn it
off!
• Print Spooler? Visual features?
Check for Linux/Windows Performance tuning guide
• Windows 7->8.1->10 Same for server, gives 10-20% on same HW
Linux
• http://blog.nashcom.de/nashcomblog.nsf/dx/linux-io-performance-tweek.htm
• Special Tuning Guide from RedHat
https://access.redhat.com/documentation/en-
us/red_hat_enterprise_linux/7/html/performance_tuning_guide/index

26. Linux Tuning
• CFQ->NOOP
• Credit -> Daniel Nashed (http://blog.nashcom.de )
http://blog.nashcom.de/nashcomblog.nsf/dx/linux-io-performance-tweek.htm
http://blog.nashcom.de/nashcomblog.nsf/dx/runfaster1-for-domino-on-linux.htm

27. ODS - On Disk Structure
• ODS database format
• Higher ODS, more features. More stability,
more speed
• ODS is like building where people live, better
building, more safe and better people live
there.
• ODS version can be seen
File->Application -> Properties
• In Average new ODS gives 20% performance
improvement
• 20 ODS – Notes 4.x
• 41 ODS – Notes 5.x
• 43 ODS – 6.x/7.x
• 48 ODS – 8.0
• 51 ODS – 8.5
• 52 ODS – 9.x
• 53 ODS – 10.x

28. ODS - Upgrade
• From 4.x -> 7.x ODS automatically updated when “COMPACT –c”
performed.
• Starting Domino 8, 8.5 and 9.0 notes.ini enables new ODS on client
and server
• CREATE_R8_DATABASES=1 (enables 8.x ODS 48)
• CREATE_R85_DATABASES=1 (enables 8.5.x ODS 51)
• CREATE_R9_DATABASES=1 (enables 9.x ODS 52)
• CREATE_R10_DATABASES=1 (enables v10 ODS53)
• Only one parameter must exist, to avoid confusion
• By default Domino 9.x will use ODS 43 (6.x) and
V10 will use ODS 52 (Domino 9)
• Use Compact –c -* (to upgrade also templates to new ODS)

29. Upgrade ODS 43->52 will speed up performance of client 20-40%
On server (Create_R9_DATABASES=1 or Create_R10_DATABASES=1) and Compact –c
(Copy Style)
On Clients Create_R9_DATABASES=1 and (NSF_UpdateODS=1 or GUI checkmark in policy)

30. IBM Notes 9.0.1 and 9.0.1FP10IF2 two different
versions with 6-7 years difference and 3000
improvements
• Thousands of improvements
• Many fixes
• Updated client will work around 30% faster on same hardware
• Better security
• Version Zoo is bad, different problems, compatibility, stability
functionality

31. Fine tuning of jvm.properties
• Changing the maximum heap size in the JVM to increase Notes performance
http://www-01.ibm.com/support/docview.wss?uid=swg21456903
https://www.ibm.com/developerworks/community/blogs/storage_redbooks/entry
/lotus_notes_thanks_for_the_memory?lang=en
• Windows Notesframeworkrcpdeployjvm.properties
Mac Applications/IBM Notes.app/Contents/MacOS/rcp/Deploy/jvm.properties
• Do not exceed 2 GB – it will be bad for Notes (Notes is still 32Bit)
• vmarg.Xmx=-Xmx256m -> -Xmx1024
vmarg.Xms=-Xms128m -> -Xms512m
vmarg.Xmca=-Xmca32k -> -Xmca512k
vmarg.Xmn=-Xmn96m
• Parameters are adjusted by HCL/IBM in 9.0.1FP8/FP9.
You can change them in installation, or by script or Panagenda MarvelClient

32. Antivirus is evil, if works by default
• Some exclusions MUST be made on IBM Domino сервер
• http://www-01.ibm.com/support/docview.wss?uid=swg21385302
• .nsf, .ndx .ntf, .ndk, .bib, .njf, .log, .nbf, .adr, .ndx, .dtf, .txn, .properties, .xml, .jar, and any files in
.FT folders. As well .nlo for DAOS attachments
• https://www.slideshare.net/panagenda/dev1185-ibm-notes-performance-boost-reloaded-ibm-
connect-2017
• If Antiviruss admin says it is ok, make new PC, without antivirus and compare performance
• Then configure security and antivirus for users
• Windows Defender

33. Copy right https://demotivatorium.ru/demotivators/d/84974/
A bit overloaded …

34. Disks and disks partitions - how many disks we need
1. Domino bin + Data
2. Domino Bin, Data, Trans.Logs
3. D.Bin, Data, (View Rebuild Dir, FT Dir), TransLogs.
4. Domino 8.5 Bin, Data, DAOS, (View Rebuild Dir, FT Dir), TransLogs.
5. + Domino 9.0.1FP10IF4 View Index(NDX) – moving View indexes
outside db (introduced in FP8)

35. HDD/RAID
• OS – RAID 1/5
• DATA – RAID 1/5/10
• Transactional Logs – RAID 1 (ONLY) (on RAID 5 system will work slower )
• It is better to have 4 average server, than 1 mega server
• If using virtualization, separate disks , not one mega disk. Vritualization works better with few
files, than with one huge file.
• https://www.ibm.com/support/knowledgecenter/en/SSKTMJ_9.0.1/admin/tune_improvingunixs
erverperformance_c.html

36. Unread Marks
• For every user Domino creates a table, where we mark which document
was read by user. If programmer does not use it in views, then must
select «Dont Maintain Unread Marks»
• 1000 users, database 30 000 documents =
then table of 30 000 000 records is maintained just for unread table.
• These are milliseconds than end with minutes and hours

37. DB Properties – improve perf

38. Idea of Transactional Logs
• Write data sequentally into a file
• After read it sequentally and flush data to databases

39. How Transactional Logs work?

40. How to enable Transactional Logs
• Standard – Starting poing, but not most efficient
• Favor Runtime – best performance (+5/15% comparing to Standard)
• Favor Recovery –
Good for development , quickly brings server up, but not for stable
production
• Create_R85_Logs=1
(Enables new Tr.Logs format)
• Even for Domino 9/v10
notes.ini Create_R9_Logs=1
Does not exist!!!

41. DAOS – Attachment deduplication system
• Requirement –Transactional Logging and ODS 51 , 52, 53
• Enable on server
• Enable LZ1 compression for files (copress them)
• Enable DAOS on database and perform Compact Copy Style
• DAOS Estimator
• 10 000 will get 10 MB file, on server HDD will take
10MB, after LZ1 5-7MB. Without DAOS 10 GB

42. There is always space for improvement
• If you have DAOS, you may improve performance of server
• Enable DAOS on mail.box
Christoph Adler (panagenda)
Luis Guirigay (IBM)
• https://engage.ug/engage.nsf/pages/2018_Slides_b/$file/Engage2018_IBM-
NotesDomino-Performance-Boost-Reloaded.pdf
• Remove RM_NO_LOG_OBJECTS_IN_MAILBOX=1 from Domino 8.5 and above!
• https://www-10.lotus.com/ldd/dominowiki.nsf/dx/rm_no_log_objects_in_mailbox-
disables-mail.box-transaction-logging
• Completely remove this setting from Domino servers, versions 8.5 and higher.
Significant performance improvement work has been included in
D8.5 such that this option is no longer a high leverage configuration.
The D8.5 DAOS feature works more efficiently with mail.box transaction logging enabled.

43. Notes.ini
• MailLeaveSessionOpen=2 (TCP)
• Updaters=6 (less or equal СPU count) (will rebuild views faster)
• DominoGzipEnabled=1 (HTTP)
• FT_FLY_INDEX_OFF=1 (disbales FT rebuidl on the fly if FT not enabled)
https://www-10.lotus.com/ldd/dominowiki.nsf/dx/disable-on-the-fly-full-text-indexes
• Log_DisableTXNLogging=1 Disables TR.Log on Log.nsf (enable archiving on log.nsf to avoid
slow recovery)
• FTBasePath=E:Fast_Disk_for_FullText_rebuild
• VIEW_REBUILD_DIR= F:Fast_Disk_for_View_Rebuilds
• Moving FT and VIEW outside data, will reduce size/time of backup&recovery

44. DBMT related server Notes.ini
DBMT_PREFORMAT_PERCENT =120
Allocates additional 20% of space to avoid data fragmentation

45. Moving Views outside (only on 901FP10IF4 to
avoid corruptions)
• NIFBasePath=/local/NIFBase
NIFNSFEnable=1
CREATE_NIFNSF_DATABASES=1
• https://www.ibm.com/support/knowledgecenter/en/SSKTMJ_9.0.1/
admin/admn_moving_views_out_of_databases.html

46. Panagenda MarvelClient analysis of Client start
•

47. Cache or not to Cache

48. Managed Replica – Smart Replica
@GetMachineInfo([IsLaptop]);
Configuring managed replicas using the Desktop Settings document
http://www-01.ibm.com/support/docview.wss?uid=swg21437957

49. Tips for programmers or how (not) to kill
your server

50. Deep Dive: Client_Clock and Server_Clock
• Console_Log_Enabled=1
• CLIENT_CLOCK=30
• logstatusbar=1
• Debug_Outfile=c:tempdebug.txt
• Debug_Console=1
• Great tool by Andrew Magerman
• https://www.openntf.org/main.nsf/project.xsp?r=project/Notes%20RPC%20Parser

52. Notepad++
• d{3}([0-9])( ms.)
Search for milliseconds that has 4 digits and space and «ms.»
d{x} - where X – symbol count
[0-9] - digits
( ms.) “string at the end”
• Tip from my colleague Aleksandr Korolkov

53. NRPC Parser – Andrew Magerman
• https://www.openntf.org/main.nsf/project.xsp?r=project/Notes%20RPC%20Parser

54. LotusProfiler / LotusScript
• There are some DONT:
Dont use GetNthDocument(n), but use GetNextDocument(doc).
GetNthDocument(n) – is really slow, this is well documented
• If you work with array, increase array in portions (e.g. in 10 or 100
increments, not by 1 element)

55. HTTP GoSuperFast=1
• Network Listen Queue = 512
• Change to 1024 or above to get a HTTP Rocket.
• Do not exceed 2048

56. Agents LN/HTTP/WWW
• DominoAsynchronizeAgents=1 (One agent can run in parallel)
• Whout this second request will wait while first ends
• Max % Busy
before delay
• Max concurrent
agents (def) =1

57. Agent Profiling

58. Profiling Results
• Db.isClusterReplication

59. Why databases are slow
• Lack of knowledge (20%)
• Lazy/Never thought about it/Was not aware it will work this way? – 20%
• Can you code it next week? So we fit budget/deadline? (60%)
• Databases often coded, tested, on different amount of documents in test in
production

60. Do it right= HCL/IBM Development Guidelines
• Small number of views 50( 50 is not 500)
• Common scenario 300 views /View X 30 columns per view Х Sorting-
2-way + FullText Index =Immediate + db has 300 000 documentes +
Unread Marks + complex formulas in view
• This databases is designed=doomed to be slow
• View index size must not exceed 20-30%, there are cases View Index
=90% db.size
• @IF inside view is bad, some people but Loops in view While/For

61. Unoptimized Views kills
server/hdd/network/client/impression on Notes

62. Lazzy: Face of programmes when somebody tells their code is not optimized
How to calculate document size in MB? DocumentSize / 1024/1024
Or while writing a code calculate once 1024х1024=1048576
And perform 1 math operation instead of 2 (on 300 000 documents ON every refresh on 10 000
Users)
In case of 100 000 документов this is 100 000 additional/unneeded operations
@If(JobStatus=”Draft”;30; JobStatus=”Started”;56; JobStatus=”Canceled”;81;0)
Sometimes there are 20 statuses, 100 000 documents and when Index is outdated, server have to
rebuild it/recalculate

63. Sorting in both directions on 3 columns – Kills index
• 2 x N = options View =

64. All Statuses, all icons MUST be calculated on document level in onSave event
• All view info must be calculated on document level, if performance is important and amount of
document is high

65. Notes Document and cache (for better
performance)
• https://www.ibm.com/support/knowledgecenter/en/SSVRGU_9.0.1
/basic/H_NOTESDOCUMENT_CLASS.html

66. Developement tips
• Enable LZ1 file compression (if documents are not Signed), if Signed
enable for New documents only
• Enable text compression (if documents are not Signed), if Signed enable
for New documents only
• Enable DAOS
• Dont implement your own DAOS – IBM did it already!
• Disable Unread Marks
• Enable design compression
• Try to avoid using @Time in View

67. Replace @Today/Time functions
• After midnight in document or in profile store todays date
• Instead of comparing with @Today compare with MyFieldToday in
profile or in current document (StampAll)
• Since MyFieldToday will be already calculated, server will get this value
faster
• @Today/@Now will be outdated after 1 second, and server is doomed
to rebuild it again and again

68. Usefull links
• http://www.nsftools.com/tips/PerfTips.htm
• Domino Server Performance Troubleshooting Cookbook
http://www-1.ibm.com/support/docview.wss?rs=463&uid=swg21234550&loc=en_US&cs=utf-
8&lang=en
• https://www.slideshare.net/panagenda/dev1185-ibm-notes-performance-boost-reloaded-ibm-
connect-2017
• https://www.nashcom.de/nsh/web.nsf/ff5ce882e73ab026c1256942003bdf10/6084a81e9f2c2b
00c1256cc30030c6c1/$FILE/bp106_final.pdf