IBM ConnectED 2015 BP110: Mastering Your Logs, Everything You Should Know about Logging in IBM Domino

BP110: Mastering Your
Logs
Everything You Should
Know about Logging in
IBM Domino
Ben Menesi, Ytria

 Head of Product at Ytria
 IBM Notes Administration & Development for
various enterprise IBM customers
 Certified Advanced Domino Developer & Admin
and Security Professional v7+
 Speaker at ICS / ESS technology conferences around the globe
 IBM Champion for Collaboration Solutions 2014 & 2015
ca.linkedin.com/in/benedekmenesi
Speaker: Ben Menesi
@BenMenesi

 Founder, CEO & CTO at Ytria
 15+ years IBM Notes & API development
 Speaker at ICS conferences & published author
 IBM Design Partner
ca.linkedin.com/pub/eric-houvenaghel/0/37/941
Co-Author
Eric Houvenaghel

1.
• Events take place
2.
• Events are logged
3.
• Some events captured
4
• Few events addressed
5
• Lots of events missed, angry users on the
phone

 Everything about LOG.NSF
– Understanding different log events
– Extracting additional data
– Log Retention, Searching the log and more
 User Activity Logging
– Understanding what user activities are logged
– Utilizing Recorded user activity
 Replication Logs
– Replication History tips & tricks
– Understanding what / where / why
– Replication Triangulation
Agenda

LOG.NSF
What is (or isn’t) logged in log.nsf?
How can we extract more data?
Best Practices for retention, searching and more

 Almost everything from your Domino server (HTTP Web Server is
another topic!)
– Generally 90%+ of what’s in your Domino console will be in here
• Except for Traveler and some DAOS command logs
Log.nsf
What is being logged?
Log.nsf
replica
amgr
router

 MAIN Log.nsf setting in your server’s Notes.ini file
 Logfilename: typically log.nsf
 Log_option: 1 = Log to the console, 2 = Force DB fixup when opening
log file, 4 = Full document scan
 Days: # of days logs will be kept*
 Size: Size of log text in event documents*
 Days2: Optional # of days to keep activity trends data documents
Log.nsf
Log file structure
LOG=logfilename, log_option, cant_touch_this, days, size, days2

 Understanding your log views
Log.nsf
Log file structure
 SELECT Form=“Events”
 SELECT Form=“Mail Routing”
 SELECT Form=“Replication”
 SELECT Form=“Security”
 SELECT Form=“NNTPEvent”
 SELECT Form=“Passthru
Connection”

 Server tasks simultaneously log into various log.nsf documents
 Miscellaneous Events (form=Events)
– Size & content controlled by the Log= server notes.ini parameter (later)
 Mail Routing Events (form=Mail Routing)
– Mainly updated by the SMTP Server and router tasks
 Security Events (form=Security)
– E.g. ID Vault messages & errors
 Replication Events (form=Replication)
– Replica & CLREPL tasks
Log.nsf
Log file structure

 Understanding your log views
Log.nsf
Log file structure
 Select Form=“Session”
 Select Form=“Session”
 SELECT Form=“Activity”
 SELECT Form=“Activity” |
Form=“Size”

 User Session Activity (form=Session)
– Sessions this server had with users or other servers
 Activity Data (form=Activity)
– Populated by the nightly Statistics Log task
Log.nsf
Log file structure

 Server tasks can log events to multiple types of documents!
– But no duplicate logs: different information!
– Example: Server 1 replicator opens session to Server 2 as per
Connection Document settings
Log.nsf
Log file structure
Misc.
Replication

 Miscellaneous events: what do we see?
– Some high level info (Server, Start & Finish Time) and
– UI only shows ONE text type item with limited information
Log.nsf
1.) Miscellaneous events

 What the EventsR6 item tells us us
– Used to be a Rich Text item Prior to R6 (whew!)
– @If(@IsAvailable(Eventlist);EventList;@IsAvailable(Eventslist);Events
List;Events)
– EventsList item: only contains the Event time & description for each
event
 What is an event?
Log.nsf
Time Status Code Severity Type
Addin Name Target Server Target DB Target User

 Miscellaneous events – event attributes logged in multi-value Text
items
Log.nsf
EventList
Time &
Description
EventTime
EventSeverity
EventStatus
EventType
Event 1
Time
Severity
HEX Status
Type
Time &
Description
Event 2
Time
Severity
HEX Status
Type
Time &
Description
Event 3
Time
Severity
HEX Status
Type
Time &
Description
Event x
Time
Severity
HEX Status
Type

 Miscellaneous events – EventSeverity
– Severities are recorded via numbers 0 – 5 where
• 0: Unknown
• 1: Fatal
• 2: Failure
• 3: Warning High
• 4: Warning Low
• 5: Normal
Log.nsf

 Miscellaneous events – EventType
– Multiple type identifiers in use
• Best not to mess with this stuff
Log.nsf

 Can we get more data?
– We can use formulas and the list of Severities, Events & Status Codes
to get a better idea of each Misc. log document
 Total # of events / document [Eventlist item]
– @Elements(Eventlist)
 Total # of unique events [EventStatus item]
– @Elements(@Unique(EventStatus))
 Highest severity / document [EventSeverity item]
– _sev:=@If(@Sort(@Unique(@Text(EventSeverity));[Ascending])[1]="0
";@Sort(@Unique(@Text(EventSeverity));[Ascending])[2];@Sort(@Un
ique(@Text(EventSeverity));[Ascending])[1]);
Log.nsf

 Customization options (Cont’d)
 Once we have the _sev variable, use icons to display (from DDM
resources)
– @If(_sev="0";"iconNormal";_sev="1";"iconFatal";_sev="2";"iconFailure
";_sev="3";"iconHigh";_sev="4";"iconLow";_sev="5";"iconNormal";"")+"
.gif“
 # of Fatal severities / document [EventSeverity item]
– @Elements(@Trim(@Replace(@Text(EventSeverity);"5":"4":"3":"2":"0
";"")))
 # of Failure severities [EventSeverity item]
– @Elements(@Trim(@Replace(@Text(EventSeverity);"5":"4":"3":“1":"0
";"")))
Log.nsf

 # of High Warning severities / document
";"")))
 # of Low Warning severities / document
";"")))
 # of Normal severities / document
";"")))
 # of Unknown severities / document
";"")))
Log.nsf

 Customization options: this is what you’ll get
– Tip: we can make the Highest and severity columns sortable but
careful with your indexes!
Log.nsf

 Mail Routing Events
– Same Structure as Miscellaneous events
– Incrementally fills documents (considering 40KB / Log= limit)
Log.nsf
2.) Mail Routing Events
Time EventList EventStatus EventSeverity

 Is there any additional data we can mine?
– Use same custom views (Severities, unique events, etc…)
• Tip: Messages can be traced based on the last 8 characters of their
Universal ID
• Extract list of Messages transferred per Mail Routing Log, or their
(unique) numbers.
Log.nsf

 Unique # of Messages transferred [EventList item]
– Could be used as a sortable column formula
• @Elements(@Unique(@Explode(@Implode(@Trim(@Word(@Replac
eSubstring(EventList;"Router: Message ":"Router: No messages":"
transferred to ":"NOT transferred ":" delivered to
";"§§§":"§§§":"§§§":"§§§":"§§§");"§§§";2));", ");", ")))
Log.nsf

 List of unique message codes [EventList item]
– Could be used as a categorized column formula – help finding emails
• @Unique(@Explode(@Implode(@Trim(@Word(@ReplaceSubstring(
EventList;"Router: Message ":"Router: No messages":" transferred to
":"NOT transferred ":" delivered to
";"§§§":"§§§":"§§§":"§§§":"§§§");"§§§";2));", ");", "))
Log.nsf

 Security Event documents
– Same Structure as Miscellaneous events
– Incrementally fills documents (considering 40KB / Log= limit)
– Typically you won’t even need a special view to understand unless
you have a very high volume.
Log.nsf
3.) Security Events

 Replication Events <> Miscellaneous events
– Important: Logs documents PER SESSION
– Only Replication Sessions where the current server did the work!
• Search all participating server log files to get the big picture!
– What is stored in those replication events?
Log.nsf
4.) Replication Events
InitiatedBy Server SourceServer
Body BytesIn BytesOut Pathname
EventList EventSeverity EventStatus EventType

 What additional data can we extract and use?
– EventList & EventSeverity items ONLY created on error
– # of entries in the Pathname item tells us how many events
– Body item contains: Access, Additions, Updates, Deletes, Kbytes sent
& received for each DB
• Body item is NOT multi-value (can’t really work with formulas)
• Body item is NOT part of the summary if more than 1 event is logged
= can NOT be displayed in views
• Cluster Replication events are NOT properly logged unless you use
the RTR_Logging parameter
• http://www-01.ibm.com/support/docview.wss?uid=swg21214739
Log.nsf

 Interesting stuff: even when DB1 only has to SEND data to DB2,
we can see data being received by DB1
– BytesIN & BytesOUT: NOT reliable:
Log.nsf

 Creating a custom replication view
 Replicator / Cluster Replicator? [Initiatedby item]
 Number of events [Pathname item]
– @Elements(Pathname)
 Any Errors? [EventSeverity item]
– @If(@Elements(EventSeverity)>0;150;"")
 Direction [Body item]
– _bl1:=@Contains(Body;"PULL");
– _bl2:=@Contains(Body;"PUSH");
– @If(_bl1=1&_bl2=1;"PULL-
PUSH";_bl1=1&_bl2=0;"PULL";_bl2=1&_bl1=0;"PUSH";"-")
Log.nsf

 Creating a custom replication view (Cont’d)
Log.nsf

 Usage Session logs – what do we (not) see?
– IMPORTANT: Logging on a /USER/SESSION base!
– Contains a TON of raw data
Log.nsf
5.) Usage Session Documents

 What is an event?
Log.nsf
Database Reads Writes Transactions
Bytes Read Bytes Written DB Open time
UserName Pathname Reads Writes
Transactions SentFromServer SentToServer Body

 What additional data can we extract and use?
– PathName item (multi-value) contains list of databases accessed
• @Elements(PathName) gives us the # of DBs accessed for each
session
– Rearrange columns in the Usage By User view
Log.nsf

 Searching your log file is painful
– Reason: you’ll be presented with any matching documents but you
won’t be able to read between the lines
– Looks familiar?
– … and the sad part: 6 results would actually make us happy (RIGHT?)
Log.nsf
How to search?

 Option #1: search using the Admin client
– TONS of parameters to specify – slow setup. Better for periodical
search requirements
Log.nsf
How to search?

 Option #1: search using the Admin client (Cont’d)
– Tip: do NOT use the Event Type tab!
• You can miss results because events are logged across types
Log.nsf
How to search?

 Option #1: search using the Admin client (Cont’d)
– Tip: You can save queries in domadmin.nsf to reuse later
– BEWARE: Search results populate new documents!
• Avg. 180 events / document: single search with 1800 matches creates
10 documents
• Results can not be analyzed nor copied and are stored across multiple
documents
– Conclusion: Useful for very specific queries that you may need
periodically
Log.nsf
How to search?

 Option #2: A quicker DIY solution (less customizable)
– Advantages: reads matches from multiple log documents and saves &
displays them in one text file
– http://searchdomino.techtarget.com/tip/Easily-find-a-string-in-a-Lotus-
Domino-server-log
Log.nsf
How to search?

 Option #3: Pro tip for data-range search (Kudos to Kim Greene)
– Select log documents and Actions Forward
• Aggregates EventList item contents in one document
• Use CTRL+F to search
Log.nsf
How to search?

 Option #4 Great open source log parser via OpenNTF from Jakob
Majkilde
– Installed on server, access to Console Logs, Trace Logs, Log.nsf and
more with a nice Xpages interface
– http://openntf.org/main.nsf/project.xsp?r=project/XPages%20Log%20
File%20Reader
Log.nsf
How to search?

 Option #5+ 3rd Party tools
– Use a tool like Ytria consoleEZ
Log.nsf
How to search?

 Log deletions managed by parameter defined in log= server
notes.ini parameter
– Deletions made at once, all entries older than 7 days removed
– Add to that: Deletion Stub Purge interval
• Default 90 days could result in =< 120 days old deletion stubs
• Deletions happen every 1/3rd of the time defined here
Log.nsf
Log file retention

 Known problem in previous Domino versions: log.nsf continues to
grow and ignores log= settings
– Workaround: use “Remove documents not modified in the last (days)
• Beware: this field also defines Deletion Stub Purging time
• Tip: Deleted documents are removed WITHOUT creation of deletion
stubs
Log.nsf
Log file retention

 Recommendation
– Use the “Remove documents note modified in the last (days)” setting
– Use console logs to keep old log data
Log.nsf
Log file retention

 Log_AgentManager
– 0 for no logging, 1 for partial & successful agent execution events, 2
for successful events only
 Log_Replication
– 1 for logging when DB replicates, 2 for summary info about each DB,
3 for detailed info about each replicated note
• Great for debugging!
 RTR_Logging
– 1 (Default), 4 for Logging replications (attempted & performed)
• Cluster replication will be logged
Log.nsf
A few notes.ini parameters in the back pocket

 Mail_Log_To_MiscEvents
– 0/1 whether to log mail events into Misc. event documents
• Tip: MailLogToEventsOnly=1 to NOT show router messages in the
Domino Console
 SMTPClientDebug=1
– Use temporarily to debug outbound mail problems
• Tip: don’t get confused by client, this one’s for your server!
 HTTPLogUnauthorized=1
– Logs 401 HTTP errors in server console AND Misc. log events
• HTTP Users attempting to access resources that aren’t available and
failed user authentication requests
• This is GREAT: brings a bit of web server logs into your log.nsf
Log.nsf

 Log_Console=2
– Logs all console commands even if prefixed with ! (By default !sh task
won’t be logged!)
 LOG_DisableTXNLogging=1
– 0/1 Take your Log.nsf out of DBs to be transaction logged
• Also clubusy.nsf and mail.box!
 No_Force_Activity_Logging
– 0/1 Controls whether the statlog task automatically enables activity
recording for all DBs (Default = 0)
• Tip even if disabled activity is being recorded in the Log.nsf Usage
views
Log.nsf

USAGE ACTIVITY
What are our most used databases?
How do we know what a given user has been up to?
How do we get user activity for multiple databases?

 User Activity can be a *VERY* rich source of data
– Enable in the database properties window
• Tip: No_Force_Activity_Logging=0 (Default value) allows on ALL dbs
• Tip: Set to confidential to prevent users with lower than Designer
access from accessing it
Database User Activity
Recorded User Activity

 Did you upgrade your database On Disk Structures?
– I’m not proud of it
– Tip: use Compact –REPLICA for upgrading system database ODS’s
without downtime!
Before we go any further

 Prior to ODS 48
– Entry size allocated 44 bytes, Object size 61600 bytes (max 1400 entries)
 ODS 48+ (undocumented improvement)
– Entry size allocated 92 bytes, Object size 128800 bytes (max 1400 entries)
 Curious: only two additional counters added, yet ODS 48+ entry is over
2x the size of those prior to ODS48: Why?
User Activity – What is really captured?
Date &
Time
Reads Adds Updates Deletes User
Date &
Time
Reads Writes User

 Does Notes only capture as much as it displays?
– Nope
User Activity – What is really captured?
Reads Adds Updates Deletes
Data
Non
Data
Σ
Data
Non
Data
Σ
Data
Non
Data
Σ
Data
Non
Data
Σ

 Differentiating between Data (as in: Document class notes) and
Non-Data (as in: everything else such as Designs, ACLs, etc…)
can help us look for more specific scenarios:
– Databases where no DOCUMENT class notes were read for some
time
– Databases where no DOCUMENT class notes were added or updated
since awhile
– Users who’ve deleted Designs or ACLs
– And more.
User Activity – Data & Non-Data

 Makes no difference between users and servers
– Workaround: use a tool to parse the data and input exceptions
 Only maintains 1400 entries
– Wait, this doesn’t need to be a problem!
 Still answers questions like “What databases aren’t used on a
regular basis?”
– Databases that contain 1400 entries are properly used apps
• @Date(FirstEntry) - @Date(LastEntry) tells us how frequently used
– Databases with lower than 1400 entries indicate they aren’t used that
much
User Activity - Downside

 How do we get the user activity for multiple databases
– Third party tools like http://www.agecom.com.au/useractivity
– …or Ytria’s databaseEZ
– …or we can implement it ourselves using the NotesUserActivity class:
• Uses the W32_NSFDbGetUserActivity API call (restrictive: no Data vs.
Non-Data)
http://www.bananahome.com/ldd/sandbox.nsf/ByDate/c12a2fd2142758b
68525688d00708397?OpenDocument
User Activity – How to get it?

REPLICATION
LOGS
How is replication logged?
What can we read out?
What to watch out for?

 Set via Log_Replication=value
– 0 Do not log replication events
– 1 Log that DB is replicating (Default value)
– 2 Log summary info about each DB
– 3 Log info about each replicated document (both design and doc class)
– 4 Log info about each replicated field
 Note: only impacts logging of replication events performed by the
current server!
Replication Logs
Replication verbosity logs

 Keeps track of what the current database replicated with
– Tip1: Local <> Server replication leaves NO TRACE in server replica’s
replication history!
– Tip2: When there’s nothing to replicate, no replication history entry
created
Replication Logs
Replication History

 Feature aiming to optimize performance introduced in Domino 7
– Result: Replication History Entries indicating replication between A & C
which never happened
Replication Logs
Replication Triangulation
A
B
C
A – B
A – B
B – C
B – C
A – C

 Idea: to prevent complete recalculation when replicating with an
unknown server
– Results show 400% CPU increase in some cases (1700 servers)
– Disable (server side):
• NSF_REPLHIST_NO_TRI=1
• REPL_NO_WS_TRI_HIST=1
• REPL_NO_REMOTE_TRI_HIST=1
– Gotcha: need to clear replication history entries for all impacted DBs!
– Disable (client side):
• NSF_REPLHIST_NO_TRI=1 [No triangulated entries read]
• REPL_NO_WS_TRI_HIST=1 [No triangulated entries written]
Replication Logs
Replication Triangulation

 Purge Interval Replication Control
– New(ish) awesome option for preventing old documents from coming
back (Introduced in Domino 8.5.3)
• Set via Replication Options > Space Savers
– DEBUG_REPL_PIRC=1 gives you date and summary info about
documents not allowed to replicate
– DEBUG_REPL_PIRC=2 or greater provides more details about
documents blocked by PIRC
Replication Logs
PIRC logging

 Please fill out your evaluation forms
– Your feedback is very important!
 Feel free to get in touch!
– Contact - Ben Menesi
Thanks
Thank you for attending!
ca.linkedin.com/in/benedekmenesi
@BenMenesi

Engage Online
 SocialBiz User Group socialbizug.org
– Join the epicenter of Notes and Collaboration user groups
 Social Business Insights blog ibm.com/blogs/socialbusiness
– Read and engage with our bloggers
 Follow us on Twitter
– @IBMConnect and @IBMSocialBiz
 LinkedIn http://bit.ly/SBComm
– Participate in the IBM Social Business group on LinkedIn
 Facebook https://www.facebook.com/IBMConnected
– Like IBM Social Business on Facebook

Copyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from
IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial
publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS"
WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION,
INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted
according to the terms and conditions of the agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have
used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which
IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and
discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific
situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any
relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide
legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested
those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-
party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual
property right.
IBM, the IBM logo, ibm.com, BrassRing®, Connections™, Domino®, Global Business Services®, Global Technology Services®, SmartCloud®, Social Business®, Kenexa®, Notes®,
PartnerWorld®, Prove It!®, PureSystems®, Sametime®, Verse™, Watson™, WebSphere®, Worklight®, are trademarks of International Business Machines Corporation, registered in
many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright
and trademark information" at: www.ibm.com/legal/copytrade.shtml.
Notices and Disclaimers

IBM ConnectED 2015 BP110: Mastering Your Logs, Everything You Should Know about Logging in IBM Domino

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (12)

Ähnlich wie IBM ConnectED 2015 BP110: Mastering Your Logs, Everything You Should Know about Logging in IBM Domino

Ähnlich wie IBM ConnectED 2015 BP110: Mastering Your Logs, Everything You Should Know about Logging in IBM Domino (20)

Mehr von Benedek Menesi

Mehr von Benedek Menesi (11)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

IBM ConnectED 2015 BP110: Mastering Your Logs, Everything You Should Know about Logging in IBM Domino