Are you concerned about your infrastructure being configured correctly? Do you have problems happen that you don’t know how to prevent? Do you think your servers might have room for improvement? Wonder no more. This session will show you what you need to be looking at to ensure your server environment is running as cleanly and efficiently as possible. You will learn what you need to be looking for in your server configuration, problems found at numerous customer environments and what steps should be taken to remedy the various situations covered in this session. Be preventative, not reactive! Performing a health check is one of the most economical ways to ensure your social and collaboration environments are running properly.

1. Adm07:
The Health Check
Extravaganza for Social and
Collaboration Environments
Kim Greene, Kim Greene Consulting, Inc.
Luis Guirigay, IBM
1#engageug

2. 2#engageug
Kim Greene - Introduction
• Owner of Kim Greene Consulting, Inc.
• 15+ years experience with Domino and
Sametime and 20+ years of experience with
IBM i (AS/400, iSeries)
• Services include System & Application
performance optimization, Administration,
upgrades, health, performance, security etc.
checks, migrations, custom development,
enterprise integration
• IBM Champion
• Blog: www.dominodiva.com
@iSeriesDomino http://www.kimgreene.com

3. Luis Guirigay - Introduction
3#engageug
• WW Executive IT Specialist
• Global Technical Ambassador at IBM
• Published Author. IBM Redbooks and
developerWorks (Domino, DB2, iSeries,
Connections, Sametime)
• IBM Certification Exams for ICS Products (writer
and reviewer)
• WW Program Manager for Project Hawthorn
(mail support for MS Outlook)
• SME – Social, Collaboration, Cloud, Verse and
Messaging
Follow me @lguiriga or http://about.me/lguiriga

4. Agenda
4#engageug
• Proper maintenance
• Configuration & best practices
• Keeping current a.k.a patching
• Monitoring
• Security
• SmartCloud Notes Hybrid
• IBM Mail Support for MS Outlook

5. PROPER
MAINTENANCE
5#engageug

6. Many Moving Parts – What to do?
6#engageug
• Modern Collaborative Systems have many moving
parts but which ones are most in need of
maintenance and how?
• Let’s look at some of the Systems and all their
Moving Parts

7. The Bits and Pieces of IBM Domino
7#engageug
These are the individual moving parts that make up your
IBM Domino environment:
Main Components:
● Servers (the OS)
● Server function
(application, mail, traveler,
etc.)
● Domino NSFs
Possible Additions:
● Transaction logs
● .NLOs
● DB2
● Third party products /
applications

8. .NSF Maintenance
8#engageug
Updall
• Updates view indexes
• Runs at 2AM by default
Fixup
• Check integrity of Domino databases
• Resolve corruption problems
• Especially important if not using transaction
logging

9. DBCapture Tool
9#engageug
Automatic identification and collection (i.e. taking them off-line)
of corrupt databases without bringing down Domino server
• Files renamed to .cor and moved to IBM_Technical_Support folder
• Can still run fixup / compact / updall on them!
Enable using server notes.ini:
• DATABASE_CAPTURE_ENABLED=1

10. DBCapture Tool
10#engageug
Tips:
Can invoke manually; ignores Status but respects Capture and Size limits
• load dbcapture dbnames.nsf
DATABASE_CAPTURE_SIZE_LIMIT (in mbs) sets size of all collected
databases
• Default: 100 / set to 0 for no limit
DATABASE_CAPTURE_LIMIT sets maximum # corrupt DBs to be collected
• Default: 10 / set to 0 for no limit
Gotcha
• DATABASE_CAPTURE_ENABLED value resets every time capture is done, and is enabled again
when server is restarted! (i.e. does not run continuously)

11. .NSF Maintenance
11#engageug
• Compact
• Equivalent to a “Defragment” for a Domino database
• Rearrange database or reduce file size
• Run with multiple threads via notes.ini
• debug_enable_compact_8_5=1
OR
• Use DBMT
• Recent customer example
Compacted databases after upgrade, recovered 418 GB of disk space, a 42%
reduction!!

12. Compact Tips
12#engageug
compact_filter=dbname.nsf
• Prevents compact running on specific databases
• Ex: compact_filter=log.nsf, names.nsf, admin4.nsf
> load compact -c mail/ladmin.nsf
Database 'mail/ladmin.nsf' is not present in the ini
parameter 'COMPACT_FILTER'. Proceeding with compact.
• Compact –ODS
• Copy style compact if current ODS is less than desired level
• 95% Space Utilization is a good thing

13. Compact Replication
13#engageug
• Use –REPLICA switch on Compact command
• Creates replica of database under the covers while source database remains
accessible
• Use to remedy “Insufficient memory” or “Unable to extend an
ID table – insufficient memory” errors caused by frequent
additions and deletions in a database
• Internally reorganizes IDs in new replica
• Avoids ID table fragmentation leading to above errors
• Preventative maintenance to avoid fragmentation causing database to become
inaccessible
• Maintains Views and Unread Marks between old and new replica

14. Derby Database Maintenance
14#engageug
• Over time Traveler performance can deteriorate, defrag to
restore performance
• Steps to start a Defrag manually:
• Tell traveler shutdown
• Tell http quit
• Load traveler -degrag
• Notes.ini variables
• NTS_DEFRAG_INTERVAL_DAYS=<# of days>
• NTS_LAST_DEFRAG=<timestamp of last defrag>

15. Traveler Database Maintenance
15#engageug
Defragging changes in 9.0.1.8 and later versions of Traveler, use DBMAINT now
• Tell traveler dbmaint set interval 7 **
• 11/19/2015 09:37:02 Traveler: DB maintenance will be performed every 7 days.
• Tell traveler dbmaint set time 23:00
• 11/19/2015 09:39:58 Traveler: Time of day for DB maintenance has been set to
23:00
• Tell traveler dbmaint set day Sunday
• 11/19/2015 09:51:40 Traveler: Day is now configured to Sunday.
• Tell traveler dbmaint set auto on **
• 11/19/2015 10:12:27 Traveler: Automatic maintenance of your database has been
set.
• 11/19/2015 10:12:27 Traveler: The next maintenance is scheduled for 2015-11-22
23:00.
• 11/19/2015 10:12:27 Traveler: Maintenance will be performed every 7 days at
23:00.
• ** Only options available for Derby database

16. The Bits and Pieces of IBM Connections
16#engageug
These are the individual moving parts that make up your IBM
Connections environment:
Main Components:
● Servers (the OS)
● WebSphere
● DB2
● LDAP
● IHS
● TDI
Possible Additions:
● Cognos
● IBM Docs / IBM FileViewer
● IBM Forms/Surveys
● Third Party Products
● ICMail
● Shared File Space (NAS/NFS,
etc.)

17. The Bits and Pieces of IBM Sametime
17#engageug
These are the individual moving parts that make up your IBM
Sametime environment:
Main Components:
● Servers (the OS)
● WebSphere
● DB2
● LDAP
● Domino (Community
Server only)
Possible Additions:
● Proxy Servers
● Integrations with Voice/Video
devices
● Integrate Sametime with other
systems (awareness, meetings, etc.)
● Third Party Products – IM Queue
Managers, etc.

18. VPUserInfo.nsf - A Contact List Tune-Up
18#engageug
• Vpuserinfo.nsf can grow very large and make Sametime
very slow to login and respond to searches.
• Use a custom agent to look for users no longer registered in the Domino
Directory and remove all contact lists for those users.
• If users are seeing partial empty lists:
• load fixup vpuserinfo.nsf
• load updall -r vpuserinfo.nsf
• load compact vpuserinfo.nsf

19. DB2 - Three “Rs” rule
19#engageug
• Reorganisation
• Recommended after large amounts of data get added
• Runstats
• Run often to make sure queries are being executed optimally
• Rebind
• Recommended after applying a fix pack or similar

20. CONFIGURATION
& BEST PRACTICES
20#engageug

21. The Tyranny of the “Default”
21#engageug
• Everyone gets an “average” server if they do
nothing at all
• It will run, but will it run well?
• Is this acceptable to you?

22. Connection Documents
22#engageug
• Key for properly controlling replication
• What to replicate
Replication type and files / directories to replicate, and avoid
• Tip: Are you replicating names.nsf, admin4.nsf, events4.nsf and dir
assist db throughout the domain??
• Replication time limit
• Tip: Set to less than the repeat interval

23. Connection Document Settings
23#engageug
• Critical to watch connection document settings
• Customer example
• 09/26/2014 11:52:33 AM {User ABC/ACME} DBStore::GetDB:
Unable to open CN=DomMail/O=ACME!!mailuabc.nsf (Connection
denied. The server you connected to has a different name from the
…
• Connection document was culprit
IP of Source server, not Target!

24. Notes.ini Files
24#engageug
• Is there lurking debug still enabled?
• Did you really check??
• Consumes valuable resources
• Make sure your notes.ini doesn’t look like this
• Debug_threadid=1
• Log_AgentManager=1
• Debug_sem_timeout=10000
• Log_update=2
• NSF_DocCache_Thread=1
• debug_nif=0
• Debug_nif_update=1
• FT_LIMIT_HIGHLIGHT_FILTER=1
• LDAPDEBUG=1
• SMTPDebug=3

25. Notes.ini Files
25#engageug
• Don’t forget about the configuration document!!

26. Notes.ini Files
26#engageug
• Recommended to be enabled at all times:
• CONSOLE_LOG_ENABLED=1
• Captures server console data and logs to console.log file
• CONSOLE_LOG_MAX_KBYTES=204800
• Restricts the console Log size to 200MB and then overwrites
oldest entries
• DEBUG_THREADID=1
• Stamps server threads and logs to console.log file
• DEBUG_CAPTURE_TIMEOUT=1
• Captures semaphore time stamp and logs to semdebug.txt
• DEBUG_SHOW_TIMEOUT=1
• Captures semaphore information and logs to semdebug.txt

27. Transactional Logging
27#engageug
• Has been around for years!
• Sequential writing into a log file
1 2 3 4 5 6 7 8 9 …
Remove unread mark
Remove unread mark
vs.
• Allows Incremental Backup/Restores
• TXN Logs stored on a separate disk controller for best performance
(depending on your platform)

28. Traveler HTTP Threads and Sizing
28#engageug
• Tell Traveler stat show push.devices.total
Push.Devices.Total = 225
• This indicates that 225 devices are registered for
synchronization with the Notes Traveler server and
that at least 270 HTTP threads are needed (1.2 x 225 =
270).
Tip: The number of active HTTP threads needed for Traveler
is calculated this way: 1.2 x Number of registered devices =
Number of needed active HTTP threads

29. Sametime MUX
29#engageug
• Geographic
• Go from 20,000 users to 100,000 per Community

30. IBM Connections
30#engageug
• DB2: 64 Bits, 8GB -128GB
• Dedicated Storage or high
performance disk
• Use a Caching Proxy Server
https://ibm.biz/BdHCUh
• DB2 Pool Size
• Content Compression
• More Tuning Tips
https://ibm.biz/BdHC5j

31. KEEPING CURRENT
A.K.A PATCHING
31#engageug

32. On Disk Structure
32#engageug
• Don’t forget to upgrade databases to latest ODS level
when upgrading servers
• What is the ODS about?
• Newest internal structure enables database to benefit from
newest features
• Examples of benefits
• R5.0 (ODS41) = participate in transaction logging
• R6.0 (ODS43) = LZ1 compression and shared templates
• R8.0 (ODS48) = design and document compression
• R8.5 (ODS51) = DAOS
• R9.0.1 (ODS52) = Performance improvements, better handling of huge
(2GB+) attachments

33. How to Upgrade On Disk Structure
33#engageug
• For server
• Copy style compact (compact –c)
• Remember compact -ODS
• For client
• Use policies to update local ODS levels
• Push to clients via dynamic policies / or organizational policies
• Desktop Settings policy document: Mail tab > “Enable upgrade for all local
NSFs to latest ODS version”
• Gotcha: requires the 8.5.2 Domino Directory on server
• CREATE_R(85/R9)_DATABASES=1
• Even better: NSF_UpdateODS=1 (Will keep updating ODS levels
as new versions are released)
• Tip: Although it’s said to be both server & client side, it only works on the
client side!

34. Preventing ServerTasksAt Updates
34#engageug
• Tired of losing your ServerTasksAt customizations when
upgrading?
• SetupLeaveServerTasks to the rescue
• Add SetupLeaveServerTasks=1 to server’s notes.ini
• Disables automatic updating of ServerTasksAt#= lines during a Domino
Server upgrade

35. MONITORING
35#engageug

36. Key Items To Keep In Mind When
Monitoring
36#engageug
All systems require you to cover the basics for all servers involved:
CPU, Memory, Disk, Network
When monitoring:
• Make it actionable
• Know your baseline
• Know what your results mean
• Investigate!

37. Monitoring for Domino
37#engageug
• Pay attention to console messages, don’t ignore them!
• admin4.nsf has not replicated (PUSH) with ANY server since
MM/DD/YYYY HH:MM:SS (1681 hours ago)
• Error validating execution rights for agent 'Notify' in database
‘subdir/dbname.nsf'. Agent signer ‘XXX01/YYY', effective user
‘XXX01/YYY'. Agent signer.
• RnRMgr: The design of Rooms.nsf is not one supportable by
RnRMgr. Autoprocessing is being disabled for this DB.
• Directory Cataloger finished processing DirectoryCatalog.nsf: File
does not exist
• Agent Manager: Full text operations on database ‘mail/myfile.nsf’
which is not full text indexed. This is extremely inefficient.

38. Monitoring for Domino
38#engageug
• Health Monitor
• Easy to use and provides 24/7 monitoring
• Enabled via Administration Preferences

39. Monitoring for Domino
39#engageug
• Health Monitor
• Watch servers on single screen
• Monitor servers and/or tasks needing attention

40. Monitoring for Domino
40#engageug
• tell traveler status
Example Yellow status
• Example Green status

41. IBM Connections
41#engageug
• CPU Utilization on WAS
• If > 70% for 5 minutes or longer = too high
• CPU Utilization on DB2
• If >50% for 5 minutes or longer = too high
• Look for these words in SystemOut log
• “Hung”
• “Starvation”

42. SECURITY
42#engageug

43. Security and Collaboration Systems
43#engageug
IBM Connections, Sametime and Domino are made up of individual
components that all have separate security concerns and (potential)
vulnerabilities.
No system will be 100% secure. If Your Domino/Connections/Sametime
environment were your home, what you would look for:
1. Every door of your house has a lock and a deadbolt and every
window can be closed and locked.
2. You would not leave a key under the front mat or in the flower pot
next to the door.
3. No Notes sticking on the front door detailing which flowerpot to look
under for the key.
4. You would have a security light or two and maybe a warning sign of
the dangerous attack Chihuahua dog that lives in your house . . .

44. Security: Common Sense Questions to Ponder
44#engageug
1. Do you really want to use the same system/generic account for each
function?
2. Do you really need the “One Admin Account to Rule Them All”?
3. Do you have so many admins that creating individual admin accounts for
them is a great administrative overhead?
4. When assigning rights, are you thinking of “person” or of “job function”?
5. Do you have more than one “person” or “admin type” for each function so
you have continuity?
6. Is your brilliant administration scheme actually documented someplace?
7. If you use hierarchical directories (LDAP …, it’s hierarchical) are you taking
advantage of it?

45. Domino – Protected Groups
45#engageug
• Prevents accidental deletion of designated “critical” groups
• Configured in Directory Profile of the Domino Directory
• Tip: You must edit and save once to become operational
• Requires Domino directory to have 9 design
• Defaults to LocalDomainAdmins, LocalDomainServers, and
OtherDomainServers

46. Domino – Protected Groups
46#engageug
• Open Domino Directory→Actions→Edit Directory Profile

47. Domino – Protected Groups
47#engageug
• Prevent deletion of these groups

48. Internet Access to Domino
48#engageug
• Oldie but goodie.....PASSTHRU SERVERS!!!
• Separate Domino Domain
• Configuration Only Names.nsf

49. Lock Down Ports
49#engageug
• Lock down ports not using
• Number one step for outside attacks
• Nmap is great tool for testing open ports

50. Lock Down Ports
50#engageug
• Ports commonly seen open
PortPortPortPort FunctionFunctionFunctionFunction
25252525 SMTP
80808080 HTTP
85858585
110110110110 POP3
113113113113 Authentication service
143143143143 IMAP
179179179179 Border gateway protocol
389389389389 LDAP
443443443443 HTTP SSL
465465465465 SMTP SSL
541541541541 uucp-rlogin
Fortimanager and Fortigate server
587587587587 Alternate outgoing SMTP
993993993993 IMAP SSL
995995995995 POP3 SSL
1352135213521352 Notes remote procedure call
2050205020502050 Java server console
1503150315031503 Sametime meeting server listen
1533153315331533 Sametime community server listen
8081808180818081 Alternate HTTP port
60000600006000060000 DIIOP
63148631486314863148 Remote debug manager

51. Lock Down Ports
51#engageug
• Lock down at firewall level
• To prevent getting to server
• Lock down at server level
• In case firewall is not secured properly
• Is LDAP, POP3, IMAP, DIIOP, etc. in use?
• Enabled by default

52. ID Vault
52#engageug
• It’s a vault with a secured/encrypted copy of all user ids
• You can have multiple vaults
• Important: Do not use standard replication for ID Vault
replicas
• Some of the benefits are:
• Lost or forgotten user passwords can be recovered or
reset easily
• User renames and key rollovers are automated
• User IDs are synchronized across machines
• No need to carry ID files for new installs
• Corrupted IDs are replaced automatically

53. IBM MAIL SUPPORT FOR
MS OUTLOOK
(HAWTHORN)
53#engageug

54. Functionality Today
54#engageug
• Primary Domino communication via HTTP
• Exchange ActiveSync synchronizes all data
• Mail, calendar, contacts, folders
• REST services:
• Out of office
• Encryption
• Room finder
• Quota management
• Delegate management
• Address book search via LDAP
• Native Outlook capability
• Any LDAP will work (not just Domino)

55. Architecture Guidelines
55#engageug
• Outlook users must have a replica on the IMSMO servers
• Second IMSMO Server required for HA via Outlook
• Load Balancer is also required. Outlook is dumb!
• You can build a cluster with IMSMO and non-IMSMO servers
• You can use the same DB2 server to host multiple DB2
instances
• Think one DB2 server for multiple IMSMO clusters
• You must use a proper SSL certificate

56. SMARTCLOUD NOTES
HYBRID
56#engageug

57. Architecture
57#engageug

58. Questions
58#engageug
Thank You!!