This document provides information on monitoring and managing Domino server health. It discusses analyzing and maintaining Domino server logs, using log filters, and analyzing log results. It also covers monitoring message tracking, mail probes, statistics, events, activity trends, and configuring the New Relic reporting tool. The document discusses database maintenance tasks like compacting and fixing up databases. It also discusses using the Domino Configuration Tuner tool and leveraging cluster symmetry and automatic database repairs.

1. Gabriella Davis - IBM Lifetime Champion
Technical Director - The Turtle Partnership
Brussels May 14th 2019
DOMINO SERVER HEALTH -
MONITORING AND MANAGING

2. • Admin of all things and especially quite
complicated things where the fun is
• Working with the design, deployment and security
of IBM technologies within global infrastructures
• working with the real world security and privacy
aspects of expanding data ecosystems
• Stubborn and relentless problem solver
• http://turtleblog.info
• https://www.turtlepartnership.com
• IBM Lifetime Champion

3. WORKING WITH LOGS

4. DOMINO SERVER LOGS
• Analysing
• Miscellaneous Events
• Mail Routing Events
• Security Events

5. NOTES CLIENT LOGS
• It’s not a race to read messages on your status bar when you can look in your local log.nsf for
those messages in more detail
• Security events show you issues with accessing the ID Vault for instance
• Miscellaneous events will show you network failures amongst other things
• Client_Clock=1 in local notes.ini for RPC activity from your Notes client
• Console_Log_Enabled=1

6. DOMINO SERVER LOGS
• Log Analysis

7. LOG ANALYSIS RESULTS
• Does not require your log.nsf to be full text indexed

8. DOMINO SERVER LOGS
• Maintaining logs at a manageable size is always an issue especially on busy servers
• Log=log.nsf,1,0,7,40000 - maintains 7 days of data with 40k document sizes
• Archiving
• You can archive a log.nsf the same as you would any other database
• Create the archive settings under database property
• Create a program document to compact log.nsf with -a option for “archive”
• Don’t maintain unread marks - why would you?
• Never Full Text Index
• Console_log_Enabled

9. LOG FILTERS
• Configured in events4.nsf
• Choose to only log specific entry types and error levels either for the whole server or for
individual databases

10. LOG FILTERS - CONSOLE

11. NSD
• If the server or client crashes it will generate a NSD file in the directory
IBM_TECHNICAL_SUPPORT
• Simple (very) Analysis
• open the file in a text editor
• search for “fatal” and find the thread and task that caused the crash
• search for the thread ID to find the database / document / agent

12. FAULT REPORTING DATABASE
• Configure in the server configuration document
• Can be configured for clients in desktop policies
• The NSD will be mailed in to the assigned fault recovery database
• from the Log Analysis view in Domino Administrator choose “open fault reports” and the database
will be created

13. FAULT REPORTING DATABASE
• Disposition view
• all faults when analyzed have a disposition value that categorises as
• Problem
• Possible Problem (possibly actionable )
• Possible Problem (likely NOT actionable )
• Informational
• Unknown (investigate)

14. POSSIBLE PROBLEM - ACTIONABLE
• Out Of Memory: Represents a crash in which the Java virtual machine (JVM) ran out of a
memory resource such as heap space.
• Launched Notes multiple times: Indicates that the user quickly launched multiple instances of
the Notes client
• Possible hang: Indicates that the Notes client was manually terminated while it appeared to be
doing useful work.
• User Kill: Indicates that the user manually terminated the client while it appeared to be waiting
for input or network timeout

15. DELETION LOGS
• Individual databases can be enabled to log deletions
• The detail of who deleted a document and the content of that document is written to a
delete.log file
• One for all databases that are enabled
• Enabling is done via a compact option -dl that also specifies the fields in the document to log
e.g.
• load compact mailgabdavis.nsf -dl on subject,posteddate,sendto,recipient
• enable log deletions on gabdavis.nsf and track the subject, post-date, sender and
recipient
• different databases will want to use different fields in the log
• any Text, Text_List, RFC822_Text, or Time field can be used

16. REVIEWING A DELETION LOG ENTRY
• “20190408T211516,06+01",
• “Mailgabdavis.nsf”,
• “80256487:00352154",
• “nserver","CN=DEWO=Turtle",
• “SOFT",
• “0001","72C0E3F8:44B53FB5DC4EDBF8:A785466D",
• "from","""New Relic” “<marketing@newrelic.com>",
• “sendto","gabriella@turtlepartnership.com",

17. ALL THINGS MAIL

18. MESSAGE TRACKING
• Configure message tracking via the
server configuration document
• it eats up resources so don’t over
configure it
• it’s not designed for auditing
• consider configuring it purely on the
outbound SMTP server or inbound
SMTP server

19. USING MESSAGE TRACKING

20. DDM MAIL PROBES
• Transfer queue check
• is mail backlogged to or from your servers
• In this example I’m monitoring if the SMTP mail that sends to messagelabs via our hub SMTP
server is being delayed

21. DDM MAIL PROBES
• Mail Reflector
• validate that a specific email address can be reached. Invaluable for a very specific use case
where routing of certain messages is business critical
• Mail Flow Statistic Check
• Monitoring the % of pending mail on each server to alert any potential delays in mail delivery
• Slack % = Mail.TotalPending - Mail.Dead - Mail.Held - Mail.Waiting) / Mail.Waiting

22. DATABASE MAINTENANCE

23. MAINTENANCE TASKS
• Compact
• using -B or -c as a copy style compact creates a new DBID which backup software will
recognise as an entirely new db
• -REPLICA creates a new replica instance then deletes the in use instance and renames the
copy which is useful for a database that is continually in case
• Fixup
• don’t use it too much, I like to use it monthly at most. It can be a hammer to crack a nut.
• Updall
• Updates view and full text indexes
• notes.ini Update_FullText_Thread separates the full text indexing thread from the view
indexing threads for efficiency
• Load updall - nodbmt tells updall to run but not perform the functions that DMBT already
does

24. DATABASE MANAGEMENT TOOLS
• DBMT Server Command - run weekly
• runs copy-style compact operations
• purges deletion stubs
• expires soft deleted entries
• updates views
• reorganises folders
• merges full-text indexes
• updates unread lists
• ensures that critical views are created for
failover
• Replaces Updall

25. DBMT PARAMETERS
• -compactThreads
• -updallThreads
• -ftiThreads
• -timeLimit refers to compact timeout for
DBMT
• -range starttime stoptime
• compactNdays (run Compact every x
days)
• ftiNdays (run FT Index every x days)
• force d (day Sunday =1) fixup if compact
fails for consecutive day

26. STATISTICS AND EVENTS

27. STATISTICS
• View all current statistics in Domino
Administrator, identify what you might
want to monitor
• Statistics Event Generators
• Set alerts / notifications / log entries
when statistic thresholds are met
• configured in events4.nsf

28. ENABLING STATISTICS COLLECTION
• Configured in events4.nsf - statistic collection
• Server task “Collect” gathers the statistics and writes them to the specified reporting database
• tell collector collect

29. STATISTIC MONITORING
• Events4.nsf

30. STATISTIC MONITORING

31. DATABASE EVENT GENERATOR
• Monitor ACL changes
• Monitor replication
• Monitor unused space *
• Monitor for user inactivity *
• * These can also be done by the DDM Database Probe both are configured in events4.nsf

32. DDM PROBES
• Set collection servers to group similar server types together to get the maximum context for
logs or if you don’t have many servers have them all collected together
• ddm.nsf holds errors written to the console as well as errors generated by enabled probes
• Don’t enable too many or for too long but use the probes to monitor bottlenecks, business
critical services or identify resource issues such as disk or memory performance

33. APPLICATION CODE

34. DIRECTORY PROBES

35. SECURITY PROBES

36. TRACKING SERVER ACTIVITY

37. CONFIGURING ACTIVITY TRENDS
• In the server configuration document - configure both Activity Logging and Activity Trends

38. ACTIVITY TRENDS
• Once activity tracking and trends are enabled via the Server Configuration document you can
review activity trends via Domino Administrator - Performance

39. HISTORICAL ACTIVITY TRENDS

40. NEWRELIC REPORTING
• By default servers report into a Domino database that can get huge and has no built in analysis
• We can now utilise cloud services that are designed specifically to analyse statistics and can
consolidate reporting from multiple servers in multiple domains
• Newrelic is only one cloud service for publishing statistics (but it’s free)
• Configuring statistics reporting is simply a case of adding one line to the notes.ini on the server
• NewRelic_License_Key=<key> (found under your NewRelica account settings)

43. DOMINO CONFIGURATION TUNER

44. DOMINO CONFIGURATION TUNER
• Domino Configuration Tuner is an analysis tool based on a set of pre-configured best practice/
worst practice rules
• The Rules are shipped by with the Domino installs inside dct.ntf and are updated via a public
update site
• although not since 2014
• Makes recommendations on configuration changes to enhance performance and security and
reduce TCO

45. HOW DOES IT WORK?
• Run and installed via the Domino Configuration Tuner database
• Updated by online template updates and rule updates
• Scans
• Server documents
• notes.ini settings
• advanced database properties
• Intended to scan servers within a single domain
• It’s a good baseline to begin a healthcheck for instance or post any significant upgrade

46. HOW DOES IT WORK?
• Creates reports on each scanned server based on the rules you select
• Each report contains
• Issues
• recommendations for adjustments
• links to supporting documentation

47. PRE-REQUISITES
• v8 Notes client (standard or basic) or administrator
• dct.nsf database and dct.ntf template
• servers 7.x or higher

48. SETUP
• DCT.NSF
• StdDominoConfigTuner Template (dct.ntf)
• ID must have reader access to names.nsf
• ID must have ‘View Administrator’ rights
• Requires no server or domain changes

49. RUNNING THE TUNER
• You can manually type in the full hierarchical names of any other servers you want to scan as
part of this analysis
• Separate multiple server names with commas, semi colons or new lines
• You can only scan servers you can reach so you need a connection document to any you list
• or the server needs to be available via your passthru server in your location

50. UNDERSTANDING THE RESULTS
• Summary results
• Issues by criticality

51. UNDERSTANDING THE RESULTS
• Filter results to make analysis easier
• by server
• by specific rules
• by severity

52. UNDERSTANDING THE RESULTS
• Categorised results of recommendations, sorted by criticality and then by server name
• Each recommendation comes with an explanation so you can evaluate on a result by result
basis if you want to make the change

53. AUTOMATIC DATABASE REPAIRS

54. DATABASE REPAIR & SYMMETRICAL CLUSTERS
• Missing databases across cluster mates
• Missing or corrupt NLOs
• Fast distribution of databases
• Corrupt databases
• I have built clusters by using cluster symmetry to populate the new server faster than
accelerated replicas

55. CLUSTER SYMMETRY
• The servers listed on the cluster configuration document are instructed to probe other servers
in the cluster if they need to populate or replace a document
• Each server uses its own cldbdir.nsf to probe a server that it knows contains the file it needs
and pull that file
• Files must have
• the same path and filename on each cluster mate including case sensitivity relative to the
data directory
• have the same DAOS configuration for minimum attachment size
• be ODS52 or higher
• The server to be populated must be running the AutoRepair and RepairCleanup tasks (put
them in the servertasks= line)

56. HOW DOES IT WORK?
Folders to maintain
Fix or list missing files - to
be safe choose “list”
initially

57. REPAIR LIST FILES CONSOLE COMMAND
• Mailturtle.nsf, 8055555:00372848 [Present]
TurtleRFC.NSF, 8055555:007ABEBD [Present]
Turtledbdesign.nsf, 858055555:0076FBF6 [Present]
Mailghedley.nsf, 802568E4:8055555 [Present]
Missing files for server CN=Clouds/O=Turtle
archivea_turtle.nsf, 80256A24:0053E4CF [Missing]
archivel_turtle.nsf, 80256A24:0053F569 [Missing]
• Repair List Missing to show only “missing” files

58. REPAIR ALL CONSOLE COMMAND
• “ archivea_turtle.nsf needs to be repaired but Auto Repair is configured to list only”
• With the configuration document set to “List” instead of “Fix” the server will not fix the
database

59. REPAIR ALL CONSOLE COMMAND
• Repair end copy C:IBMDominodataarchivel_turtle.nsf : Invalid NSF version
• Requires ODS52 minimum (9.0.1) database format
Repair File <filename> (case sensitive
repair File Mailturtle.nsf
Remote console command issued by Gabriella Davis/Turtle: repair File Mailturtle.nsf
File Mailturtle.nsf already exists, no repair necessary.
REPAIR: 0 file(s) queued for repair

60. • REPAIR: Job-00007.1 *Copying* - Repair copying C:IBMDominodataarchivea_timd.nsf
from donor CN=Oceanic/O=Turtle

61. • Domino’s focus on self monitoring and reporting continue to reduce its TCO
• Use events to monitor business critical activity
• Use probes to troubleshoot problems or identify bottlenecks such as when
launching a new application
• Don’t monitor more than you can regularly review
• plan to review fatal and failure events daily and the rest weekly
• have a plan for both weekly and monthly maintenance tasks, monitor database
health and leverage the automatic database repair feature to get ahead of
corruptions

62. QUESTIONS?
GABRIELLA@TURTLEPARTNERSHIP.COM
@GABTURTLE
SKYPE: GABRIELLADAVIS