Debugging IBM Connections for the Impatient Admin - Social Connections VII

Debugging IBM Connections
for the Impatient Admin
Martin Leyrer

The Team
Stuart McIntyre Brian O’Neill
Maria EnderstamLars Samuelsson
Martin JinochJan ValdmanWannes Rams

Knowing Me, Knowing You
● Plan Ahead
– Tools
– Software Planning
– Deployment Planning
– “Security” Tools
● Network & Infrastructure
● Fileshares
● DB2
● LDAP
● Websphere Application
Server
● IBM HTTP Server
● IBM Security Directory
Integrator
● How to Talk With IBM
Support
● Q & A

Plan Ahead – Tools – Editors
<

Plan Ahead – Log Viewers/Linux

Plan Ahead – Log Viewers/Windows
Get-Content
SystemOut.log
-wait | where
{ $_ -match “
E ” }

Fiddler
Fiddler helps you recording all the HTTP and
HTTPS traffic that passes between your computer
and the Connections Server
http://www.telerik.com/fiddler

Burpsuite
Burp Suite contains an intercepting Proxy, which
lets you inspect and modify traffic between your
browser and the target application.

Network Tools
● ssh
● ping
● dig or nslookup
● telnet
– To install Telnet Client on Windows Server 2008 or
later:
pkgmgr /iu:"TelnetClient"

Plan Ahead – SW Planning –
System Requirements
https://ibm.biz/BdRWHg

Plan Ahead – SW Planning –
Updates & Hotfixes
http://www.ibm.com/support/fixcentral/

BP: Recheck System Requirements
● Verify BEFORE EVERY
install
● Do not rely on the
Installation Manager
● Do not forget DB2
● Do not forget TDI
● Check for updates via
Cummulative Refreshs
(CRs)

BP: Install Only Supported WAS
Fixpacks

Plan Ahead – Deployment Planning
https://ibm.biz/Bdxhei

BP: Passwords
● Avoid “special” characters
● Especially: @ ! < > { }
● No national language characters
● Stick to ASCII and 0-9 if possible
● No longer than 128 characters
● Watch out for differences in password rules
between OS, LDAP, WAS and DB2
– WAS: http://www.ibm.com/support/knowledgecenter/SSHRKX_8.0.0/plan/sec_chars.dita
– DB2: https://ibm.biz/BdR7Jg

BP: Filehandling
● Make sure you have
“enough” disk space
available to install all
components
● Make sure %TEMP%
has at least 4 GB
diskspace available
● Always transmit install
files in tar/zip form
● Prepare installfiles in a
reproducible and
understandable form

BP: “Security”
● Disable Virus Scanners,
… on the server during
install
● Disable Firewalls
between Servers during
install
● Disable tools that “delete
unknown directories in
the root level of a drive”
and similar

Voulez-Vous
● Plan Ahead
– Hostnames
– BP: DNS
– BP: Port Check
– BP: Load Balancers
● Fileshares
● DB2
● LDAP
Server
● IBM HTTP Server
Integrator
Support
● Q & A

Network - Hostnames
● Define all hostnames up front
– Only fully quallified hostnames (FQHN, eg.
connections.example.com)
– No “flat” names or WINS name resolution
● Install with names for each service, even when on same
machine
– Especially DB2, LDAP and Fileserver
● Verify, that all hostnames resolve
– on all servers
– on all test clients
– nslookup, dig and ping are your friend

Network – BP DNS
If the DNS is not
working,
if you can't resolve
hostnames properly,
STOP!

Network – BP Port Check
● Of course, it is never
the network, but …
● telnet is your friend
● Test connectivity to
LDAP, SMTP, DB2,
IHS, …
● Available everywhere
telnet mail.example.com 25
220 mail.example.com ESMTP
Service (IBM Domino Release
9.0.1 HF402) ready at Thu, 12
Jun 2014 12:36:41 -0500
telnet dominoldap.example.com
389

Network – BP
Load Balancers & Reverse Proxies
● Be wary of Round Robin DNS
● Install and test Connections first
● Introduce Load Balancers, Reverse Proxies
once Connections works fine
● Be very wary regarding Network Components
SSL/TLS configuration

The Winner Takes It All
● Plan Ahead
● Fileshares
– BP: Shared content store
– BP: Username &
Password
● DB2
● LDAP
Server
● IBM HTTP Server
Integrator
Support
● Q & A

Network – BP Shared Content Store
● Always reference the “Shared Content Store” via
Universal Naming Convention (UNC) paths
● Do not forget to run WAS as the Domain “fileshare-
user”
● Always use a dedicated hostname for the “Shared
Content Store”--share
– even if on the same machine
– a CNAME is sufficient
– helps with a later migration/separation of servers (CCM)

Fileshare – Know Your Credentials
● Dedicated Domain User
aka. “fileshare-user”
● Password not allowed to
expire!
● Needs “Log on as a
service” right (GPOs)
● Require username and
password before installing
● Test in the GUI by logging
in as the “fileshare-user”
and access the Fileshare
● Test via net use
net use X:
ic-share01.example.comic-
share passw0rd /USER:fsuser
/PERSISTENT:NO

Gimme! Gimme! Gimme!
● Plan Ahead
● Fileshares
● DB2
– BP: Install & Updates
– BP: SQL GUI Clients
– BP: Backup & Restore
● LDAP
Server
● IBM HTTP Server
Integrator
Support
● Q & A

DB2 – BP Connectivity
● If DB2 went down,
restart
Connections
● Test via IBM Data
Studio or db2
command line
● Test via Telnet
telnet db2.example.com
50000
● Test via Websphere ICS
●

DB2 – Fixpacks
https://ibm.biz/BdRWHJ

DB2 BP: Connections DB Scripts
● ALWAYS run the DB2 database creation scripts or
Wizards as db2admin and not as Administrator

Database - Versions
Mark Myers:
Connections Db Schema Versions
Connections App: Profiles
DB: PEOPLEDB
Table: SNPROF_SCHEMA
Connections v4.0 BASE: 33
Connections v4.0 CR1: 33
Connections v4.5 BASE: 36
...

DB2 – IBM Data Studio
● IBM Data Studio Client “replaces” DB2 Control
Center
● Eclipse based
● Installs via Installation Manager
● Make sure you download the full product
images

DB2 - Squirrel
● Recommended by Mark Myers
– http://squirrel-sql.sourceforge.net/
– Java client So experiences the same issues as your
code
– Uses IBM’s own jar files.
– hellishly powerful
– FREE

DB2 Backup (and Restore)
● Valdemar Lemche: DB2 backup scripts
...
DB2 BACKUP DATABASE HOMEPAGE TO "D:BackupDB2" WITH 2
BUFFERS BUFFER 1024 PARALLELISM 1 WITHOUT PROMPTING
...
db2 restore database HOMEPAGE from "D:BackupDB2" REPLACE
EXISTING WITHOUT ROLLING FORWARD
https://ibm.biz/BdRWMT

Knowing Me, Knowing You
● Plan Ahead
● Fileshares
● DB2
● LDAP
– Prerequisites
– LDAP Browser
– BP: Do's And Don't's
– BP: Webseal
Server
● IBM HTTP Server
Integrator
Support
● Q & A

LDAP - Prerequisites
● You need a “bind” user
and a password, if no
anonymous access
● Password not allowed to
expire
● Know your BASE DN
● Know your Search filters
● Know your login fields
● Require these infos
before installing
Test connectivity from
every server
telnet dominoldap.example.com
389

LDAP - Browser
● Apache Directory Studio
● Softerra LDAP Browser
● LDAP Browser in ISC
● LDAP Browser inside
TDI
● Use Wireshark to read
LDAP
● (command line ldap)

LDAP – Domino And Complex
Filters
● CCM and Cognos
integration fails to look
up users in environments
using Domino LDAP with
complex LDAP search
filters
● Contact Domino Support
to obtain a Hotfix for
SPR CAHT959LQG for
your specific Domino
version

LDAP - Mismatch of realms
● com.ibm.websphere.security.auth.WSLoginFailedExce
ption: The user is from a foreign realm,
websealldap.example.com:389, and this foreign realm
is not trusted. Current realm is
defaultWIMFileBasedRealm
● Change the Realm Name from
“defaultWIMFileBasedRealm” to
“websealldap.example.com:389”

I Have a Dream
● Plan Ahead
● Fileshares
● DB2
● LDAP
Server
– BP: Location, Location, Location
– BP: Debugging/Tracing
– BP: Backup config
– BP: Houskeeping
● IBM HTTP Server
Integrator
Support
● Q & A

WAS – Location, Location
● app_server_root – WAS installation directory
– eg. F:IBMWebSphereAppServer
● profile_root – WAS profile/Deployment Manager profile
directory
– eg. F:IBMWebSphereAppServerprofilesprofile_name
● log_root – path under which to store log files for WAS profile
– eg.
F:IBMWebSphereAppServerprofilesprofile_nameserver_
name
● connections_root – IBM Connections installation directory
– eg. F:IBMConnections

WAS – Location, Location, Location
● SystemOut.log
● SystemErr.log
● trace.log
● StartServer.log
● stopServer.log
● XCluster_server.pid

WAS – SystemOut.log
[11/11/14 2:01:14:777 CST] 0000006d webapp I
com.ibm.ws.webcontainer.webapp.WebGroupImpl WebGroup SRVE0169I:
Loading Web Module: Extensions.
[11/11/14 2:01:15:433 CST] 0000006d Events I
com.ibm.lconn.events.internal.impl.Events init CLFWY0186I:
Synchronous event invocation is enabled
[11/11/14 2:01:15:448 CST] 0000006d VenturaConfig W
com.ibm.ventura.internal.config.VenturaConfigurationProviderImpl
<init> failed to initialize hystrix
[11/11/14 2:01:16:244 CST] 0000006d Events I
com.ibm.lconn.events.internal.impl.Events static CLFWY0181I:
Asynchronous event invocation is enabled and operational
[11/11/14 2:01:16:244 CST] 0000006d webcontainer I
com.ibm.ws.webcontainer.VirtualHostImpl addWebApplication
SRVE0250I: Web Module Extensions has been bound to
default_host[*:9080,*:80,*:9443,*:5060,*:5061,*:443,*:9081,*:9444,
*:9082,*:9445].

WAS – Tracing Levels and Targets
Trace option Output file
all trace.log
finest or debug trace.log
finer or entryExit trace.log
fine or event trace.log
detail SystemOut.log
config trace.log and SystemOut.log
info trace.log and SystemOut.log
audit trace.log and SystemOut.log
warning trace.log and SystemOut.log
severe or error trace.log and SystemOut.log
fatal trace.log and SystemOut.log
off trace.log and SystemOut.log

WAS – Debugging Connections
● Debugging/Trace strings are provided for all Connections
Components
● Part of the “Must Gather” aka. “Collect Data” Technotes
● Consist of lines like:
– Component Trace:
● *=info: com.ibm.lotus.connections.search.index.sand.*=all:
com.ibm.lotus.connections.search.admin.index.impl.*=all:
com.ibm.lotus.connections.search.*=all
– Search Only:
● *=info:com.ibm.lotus.connections.dashboard.search.parser.utils.*=fin
est:
com.ibm.lotus.connections.dashboard.search.parser.SeedlistIterator
=finest:
com.ibm.lotus.connections.dashboard.search.index.impl.*=finest

WAS – Collecting Data for
Connections 4/4.5/5
● Profiles: https://ibm.biz/BdRWHF
● Search: https://ibm.biz/BdRWHE
● Files: https://ibm.biz/BdRWHX
● Blogs: https://ibm.biz/BdRWHH
● Activities: https://ibm.biz/BdRWH4
● Forums: https://ibm.biz/BdRWHj
● Wikis: https://ibm.biz/BdRWHZ
● News: https://ibm.biz/BdRWH2
● Waltz,Sonata:
https://ibm.biz/BdRW8h
● Communities:
https://ibm.biz/BdRWHs
● Bookmarks:
https://ibm.biz/BdRWHi
● Homepage:
https://ibm.biz/BdRWHr
● Cognos/Metrics:
https://ibm.biz/BdRWHz
● CCM/Filenet:
https://ibm.biz/BdRWHY

WAS – How To Enable Debug/Trace
● Log in to the IBM
WebSphere Application
Server (WAS)
administration console
using an administrator ID
● Go to Troubleshooting →
Logs and Trace →
Logging and Tracing →
Server 1 (or the server the
Connections Application is
installed on) →
Diagnostic Trace Server →
Change Log Detail Levels

WAS – Backup Configuration
● Always backup before making configuration changes
● From the deployment manager bin directory run the
backupconfig(.sh)
– Backupconfig c:backupsgdbackup.zip –nostop
● The backup will be a zip file,
● The –nostop command prevents backupconfig from stopping the
deployment manager before running
– Restoreconfig c:backupsgdbackup.zip
● Restore once you have stopped the server
● See “Connect 2014 BP304: What We Wish We Had Known: Becoming an IBM
Connections Administrator” by Gabriella Davis and Paul Mooney for Details
https://ibm.biz/BdRWHq

WAS – BP: If Nothing Syncs ...
syncnode.bat is your friend in need
syncNode <deploymgr host> <deploymgr port> [options]
syncNode ssc.example.com 8879 -username wasadmin -password pass0rd

WAS – Ports Of Call
● All ports of all servers can be
looked up in the ISC
● Go to Servers → Server
Types → Websphere
Application Servers → Select
the server to check →
Communications → Ports
● Deployment manager ports
can be found in the ISC at
System administration →
Deployment manager →
Ports

WAS – BP: Modify Configuration
● NEVER (if possible) manually edit an XML
configuration file, always use wsadmin which
verifies the XML structure as it’s checked back
in
● wsadmin –lang jython –username wasadmin –password passw0rd
● execfile(“connectionsConfig.py”)
● Checkout:
LCConfigService.checkOutConfig(“F:/IBM/TMP”,AdminControl.getCe
ll())
● Checkin (and validation) after edit:
LCConfigService.checkInConfig()

WAS – Best Practices
● Install “plain” and get everything to work, then
integrate Reverse Proxies, SPNEGO, …
● Sometimes, you have to do things twice.
– As demonstrated by @socialshazza in
https://ibm.biz/BdRWHv
● “If in doubt at any point that something isn't working.
Restart EVERYTHING. Websphere does like a good
restart.”
– Gab Davis in https://ibm.biz/BdRWHm

Mamma Mia
● Plan Ahead
● Fileshares
● DB2
● LDAP
Server
● IBM HTTP Server
– BP: Rewrite Logging
– BP: Rotating Logs
– BP: plugin-cfg.xml
– BP: TLS certificates
Integrator
Support
● Q & A

IHS – Rewrite Logging
● Needed for Reverse Proxy setups but can be tricky
● Turn on Rewrite logging in the httpd.conf:
– RewriteLogLevel 3
– RewriteLog "/usr/local/var/apache/logs/rewrite.log"
[example.com/sid#80077333][rid#800b7a33/initial] (2) init rewrite engine
with requested uri /press/wp-comments-post.php
[example.com/sid#80077333][rid#800b7a33/initial] (2) rewrite /press/wp-
comments-post.php -> http://64.246.32.000/
[example.com/sid#80077333][rid#800b7a33/initial] (2) explicitly forcing
redirect with http://64.246.32.000/
[example.com/sid#80077333][rid#800b7a33/initial] (1) escaping
http://64.246.32.000/ for redirect
[example.com/sid#80077333][rid#800b7a33/initial] (1) redirect to
http://64.246.32.000/ [REDIRECT/301]

IHS – Rotating Logs
● rotatelogs works in conjunction with Apache's piped
logfile feature
● rotatelogs is part of IHS
● Supports rotation based on a time interval or
maximum size of the log.
● daily rotating logs via http.conf:
– CustomLog "|/opt/ibm/HTTPServer/bin/rotatelogs
/var/log/ihs/access_log.%Y.%m.%d 86400" common
– ErrorLog "|/opt/ibm/HTTPServer/bin/rotatelogs
/var/log/ihs/error_log.%Y.%m.%d 86400"

IHS – Plugin Configuration
● Make sure your plugin-cfg.xml is deployed to
the “correct” directory, used by IHS

IHS – Certificates
If you are using TLS certificates,
create a calender entry at least one week before
the certificate expires,
to remind you to renew it.

Super Trouper
● Plan Ahead
● Fileshares
● DB2
● LDAP
Server
● IBM HTTP Server
Integrator (tpfka TDI)
– Log Location
– Dry Run
– Keep Temp Files
– BP: Lock file
Support
● Q & A

TDI – Log File Location
● Depends on the
location of your
TDISOL directory
● eg.:
F:IBMWizardsTDIPopulation
winTDIlogs

TDI – Trace Strings
● TDISOLprofiles_tdi.properties
– source_ldap_debug=true
– tds_changelog_debug=true
– sync_updates_clean_temp_files= false
● TDISOLwinetclog4j.properties directory:
– look for
● log4j.rootCategory=INFO, Default
– change it to
● log4j.rootCategory=DEBUG, Default

TDI – Helpful Settings
● TDISOLprofiles_tdi.properties
– sync_updates_show_summary_only=true
“dry run”, only show changes, do not execute them
– sync_updates_clean_temp_files=false
keep working files, useful for debugging

TDI – BP: .lock File
● Gets created by sync_all_dns.bat
● Prevents starts while other sync scripts are still running
● Causes “Profiles do not get updated” tickets from users
– Backup shuts down server, scheduled sync_all_dns.bat
does not finish
– sync_all_dns.lck prevents script from running from then on
● Fix
– Schedule clearLock.bat accordingly (ugly hack)
– Monitor for existence of sync_all_dns.lck and fix cause

SOS
● Plan Ahead
● Fileshares
● DB2
● LDAP
Server
● IBM HTTP Server
Integrator
Support
– Language
– ESR, ECUREP
– BP for opening a PMR
● Q & A

IBM Support – General Information
● Whenever using IBM software, a customer should buy
„Subscription & Support“ aka „Maintenance“ (usually on a
yearly basis)
● S&S allows you to receive and use software updates and
fixes
● S&S allows you to contact IBM support (no limit on the
contacts) for
– Reporting software defects, asking for a workaround or fix
– Reporting enhancement requests
– Usage Support
● No extra (per call) costs for contacting support

IBM Support – What is a PMR?
● IBM tracks support calls in PMRs (Problem
Management Record)
● A PMR is basically a help desk ticket number
● You can report only a single problem per PMR
because a PMR is always assigned to a single
team
● With Connections, Support finds the right team
for you

IBM Support – Language
● Write PMR description in English
– Saves on translation via 1st
Level Support, faster
roundtrips
● Supply English logs, if possible
● Change Install/UI/Log language to English:
– IBM Installation Manager: https://ibm.biz/BdRWHG
– DB2: https://ibm.biz/BdRWHn
– TDI: https://ibm.biz/BdRWHe /HT @m0urs
– WAS: https://ibm.biz/BdRWHp

IBM Support – How To Open A PMR
● By phone
– Not recommended!
– mistyped email address or misunderstood phone number
– PMR may exist but IBM support can't contact you
● On the Web
– http://www.ibm.com/software/support
– Select Software product
– Give a description of the problem
– Select a Severity of the problem
● Sev 1 for highest prio to Sev 3 for minor problem with no urgency

IBM Support – Data Upload Via
ECUREP
http://www-05.ibm.com/de/support/ecurep/send.html

IBM Support – BP PMR Handling
● If you receive an email from support, make sure
to reply to the ticket system as well
– lotus_support@ecurep.ibm.com for ICS products
– A PMR has a status, if you don't reply this way, the
PMR will stay in status „waiting for customer
feedback“ and will not appear on a todo list and will
not be catched by superisors if your analyst is out
sick
● If you need assistance outside business hours,
update the PMR AND give IBM a phone call
asking for 7x24 assistance

Thank You for the Music
Martin Leyrer
IT-Specialist at an international IT Company
E-mail: leyrer@gmail.com
Twitter: http://www.twitter.com/leyrer
Facebook: https://www.facebook.com/leyrer
Blog: http://www.leyon.at
LinkedIn: http://at.linkedin.com/in/leyrer
Slideshare: http://www.slideshare.net/Martin.Leyrer

IBM Support – Link Collection
● IBM Support Handbook
http://www14.software.ibm.com/webapp/set2/sas/f/handbook/home.html
● Accelerated Value Program
http://www-01.ibm.com/software/support/acceleratedvalue/
● IBM Lotus Software Security Bulletins
http://www.ibm.com/developerworks/lotus/security/
● IBM Support: Fix Central
http://www-933.ibm.com/support/fixcentral/
● ECUREP data upload
http://www-05.ibm.com/de/support/ecurep/send.html

Picture references
● Lemur catta running
http://commons.wikimedia.org/wiki/File:Lemur_catta
● Alien head
http://commons.wikimedia.org/wiki/File:Alien_head.
● Tea Seave
http://commons.wikimedia.org/wiki/File:Tea_Sieve.j
● Search User Icon
http://www.iconhot.com/icon/bunch-cool-bluish-icon

Links
● Notepad++ https://ibm.biz/BdRWSd
● gVim https://ibm.biz/BdRWSx
● Baretail https://ibm.biz/BdRWSF
● Tail for Win32 (command line)
https://ibm.biz/BdRWSH
● TailMe https://ibm.biz/BdRWS4
● tail.exe @ Windows Server 2003 Resource Kit
Tools https://ibm.biz/BdRWSj
● Log Expert http://logexpert.codeplex.com/

Installation Manager – Location,
Location, Location
● Installation logs
– Windows Server 2008 (root)
● C:ProgramDataIBMInstallation
Managerlogs
– Windows Server 2008 (non-
root)
● C:Users<user>AppDataRoamin
gIBMInstallation Managerlogs
– Linux & Unix (root)
● /var/ibm/InstallationManager/logs
– Linux & Unix (non-root)
● /
<user>/var/ibm/InstallationManage
r
● AgentDataLocation
– data that is associated
with an application
– includes the state and
history of operations
– Paths as listed to the left
without the “logs” ;)

Installation Manager - Settings
● Disable Passport
Advantage
– File → Preferences
– Passport Advantage
– Uncheck option
● Keep installed packages
around

WAS - Location
● If in doubt, check the WAS Environment
variables for clues

Debugging IBM Connections for the Impatient Admin - Social Connections VII

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Debugging IBM Connections for the Impatient Admin - Social Connections VII

Ähnlich wie Debugging IBM Connections for the Impatient Admin - Social Connections VII (20)

Mehr von Martin Leyrer

Mehr von Martin Leyrer (15)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Debugging IBM Connections for the Impatient Admin - Social Connections VII

Hinweis der Redaktion