You don't want to do it like that

UKLUG 2012 – Cardiff, Wales September 2015
Presenter: Sharon Bellamy
Company: Cube Soft Consulting Ltd.
You don’t want to do it like
that …..
Lessons learnt
Best and Worse practises
of 6 years of working with
IBM Connections
@socialshazza

UKLUG 2012 – Cardiff, Wales
Who am I ?
•WebSphere Admin since 2003
•Connections Admin since 2009
•IBM Champion since 2011
•Trustee for iCosplay anti-bulling campaign
•Little bit of a Star Wars Fan
•Costumer & Mandalorian Merc
@socialshazza

Agenda
•There is no right way – but
plenty of wrong ones
•Knowing what you want to
achieve
•Planning is the key
•Test/Dev system is an
essential
•Pilots –a blessing or a
curse
•The default way isn’t
always best
•Tuning & Tweaking
•Overkill examples
•Make a lessons learned &
document EVERYTHING
•Any Questions ?
@socialshazza

There is no right way – but plenty of wrong ones
@socialshazza

Common sense ….
•Deploy with the supported OS / DB
familiar to your company /skills –
easier for debug / support –
Windows, Linux, AIX, iSeries – all are
supported
•Firewalls / DNS / Network – get these
sorted before you start.
•Dedicated *Admin* user for install
•Simpler to implement phases to
minimise risk – allows testing at each
stage
•Test without proxies – add once
working – Keep it simple to install
@socialshazza

Connections Upgrade / Migration
•V4 > V5
•Straight Connections very little
customization
•Customer required additional
features
• CCM
• Surveys
• Docs
• Cognos/Metrics
• Additional customizations
@socialshazza

What happens when you try to
do too much at the same time
•Issues with Cognos
•Issues with CCM
•Issues with Docs
•Issues with data migration
Hard to debug as too much going
on at any one time.
No quick wins, customer loses focus, many PMRS opened, hard to resolve issues
as multiple products with problems.
@socialshazza

How I would have done it
•Connections upgrade from 4 to 5
•Deployed FIB for surveys and add
the additional customization
•Deployed Cognos / Kudos for
metrics
•Connections Content Manager
•Then last but not least IBM Docs
Quick wins, customer has working V5 system, easy to debug any issues as only
with one of the components – may looks like it will take longer – often doesn’t
@socialshazza

Knowing what you want to achieve
@socialshazza

Want to use the
Domino/Notes Files
and Profiles
entitlements –
Where do we start?
Connections proof of
concept – lets try
before we buy – if we
like it we’ll make this
our live server!!
I need to do some
Connections
development and
need a dev system
We need a test / dev
system, upgrade
testing, UI
development,
integration test system
Our system was
installed by a BP /
IBM – now I need
to look after it,
where do I start? Need to upgrade to
Connections 5 – test a fix
pack, install a new
component and I don’t
want to break liveI am THE IT guy –
Never touched
WebSphere – HELP
!!!
@socialshazza

•Gather requirements – get them signed off
•Read the system reqs – ensure you have the
correct software
•Make a list of to-dos – check it .. at least twice
•Read the install /upgrade guide through and make
notes – then read it again ..
•Deploy in phases where possible – i.e Connections first - then
Cognos, CCM , Docs etc..
@socialshazza

Planning is the key – the P rule
@socialshazza

Install Planning:
•Check the system requirements regularly – they change
•CR’s often require additional WebSphere fixes
•Different OS’s have different Pre-Requisites – read these .. Do not skip any
steps.
•If an issue occurs fix it – don’t plough on regardless
LDAP:
•Better data in LDAP, better profiles
•For dev/test and POC systems use the live LDAP – if you can’t, ensure you
keep the same schema – you won’t be able to move data otherwise
•For Connections mail – there are dependencies for Exchange / Domino
@socialshazza

Sizing:
•CPU cores – rule of thumb 1 core per JVM – small
deployments of a few hundred users – 2 cores is
normally sufficient -can happily sit on one
reasonably sized machine
•Size your system for current and expected growth – this will affect the
topology required (see overkill)
•Minimum requirements may not be enough (4GB of memory is normally too
low) – Memory swapping kills all performance tuning
•Virtual / network storage is easier to extend than physical disk
•How many VM / Machines required?
•If medium deployment how many JVMS / WebSphere servers / Nodes?
@socialshazza

Migration/Upgrade:
•NEVER (seriously NEVER) do an in place migration
•Side by Side allows to test a clean system before running the data migration
•Use the same DB type (i.e DB2 to DB2)
•Install with example.com for mail notifications
•Run at least one test data migration
•Test all customisations with migrated data
•Document everything – record for go live and subsequent upgrades
•If an issue occurs fix it – don’t plough on regardless
•If you have to do an in place DB Migration make sure DBA understands the
steps and order things need to be run
@socialshazza

Migration/Upgrade
•Connections 4.5 to 5 upgrade with Oracle as backend DB on AIX –
no GUI – no problem (we thought)
•Issues with space on live DB server – Plan to build new live server
with a temp clean DB then in place DB migration when went live
•DBA didn’t read the pre-req’s or understand the creation scripts
•Wasted weeks of back and forth with emails,
calls etc.
•Eventually we installed using a windows oracle
DB and exported the scripts to send to the
DBA !!!!
@socialshazza

Migration/Upgrade
•Tested in a DEV environment first - worked
•Then we updated to CR1
•Rinse and repeat the calls, emails etc.to diagnose the issue
•Tested against our windows oracle DB which worked
•DBA missed some of the scripts to update the DB to CR1!!!
•Busy DBA’s can’t always / don’t always
understand what you need.
•Be as CLEAR and precise as you can
•Live migration went according to plan (W00T!)
@socialshazza

Test/Dev system is an essential
@socialshazza

•Doesn’t have to be the same size as
Live
•Use for testing, patching, config
changes, ui and customization
changes etc.
•Development can use it so live is not
at risk
•Plug into live LDAP for real users
•Import live data for testing
•Must be same OS, DB etc as Live
•Can be small deployment on one
machine
•Deploy in a day
•Use hosts entries to *trick* system
pieces are on different machines if
necessary
@socialshazza

•You may be told you don’t need a dev
/ test system
•Even small deployments need a
system to test CRs, fixes and
changes
•Fight for a test / dev system
•Can be small on one box – VM, or
even under a desk in the office.
•A few hours to spin up Don’t back down
Ensure you have a test
/ dev system
@socialshazza

Pilots … a blessing or a curse
@socialshazza

Pilots
•LDAP – use the live or ensure a test one has the same
users/schema if moving data
•Migrating Data - from pilot to live users must use pilot like a live
system or there will be much data clean up
•Decide before the pilot starts - move data to live or not,
stick to the decision
•Requirements may change – as a result of the pilot
@socialshazza

LDAP
Test LDAP for POCs – project manager wanted a central test LDAP / System for all customer POCs,
testing and demos – great idea if we trash the data – NOT use it for populating live
Test LDAP
Half way through the project decided all the data in the POC would be moved to their new live system –
UH OH !!
Live LDAP
TDI Sync hash – UID, EMAIL or GUID – one must match or users are inactivated / disabled
UID MAIL First Name Last Name GUID
Sharon Sharon@example.com Sharon Test1 F4DA4571-C0E9-4096-8B94-9D3FDFD3D8A6
Sam sam@example.com Sam Test2 5282844F-7DF3-4FEE-BDA5-1D876A50D3B5
Tim Tim@example.com Tim Test3 1BC9C1D9-F1A9-4BC0-AF6D-472B39833E8A
UID MAIL First Name Last Name GUID
SBellamy Sharon@cube-soft.co.uk Sharon Bellamy 3052DB8A-3B6F-44C9-84B2-FC6ACB881184
SSmith sam@customer.com Sam Smith 2AC55A42-2605-4D24-AFFC-2B94A7F910F1
TClark Tim@tc-soft.com Tim Clark EAEDD003-9190-47CA-9D08-780F4FF88FBC
@socialshazza

The default way isn’t always best
@socialshazza

•Many guides and zero to heroes
•Awesome for a first try on a test system not
necessarily for a real live one
•Split your Apps into meaningful clusters if medium
• I generally split into 4 jvms (websphere servers)
• Add additional apps (ccm, docs viewer) as a separate jvm/server
• Be prepared to tune and tweak once you deploy
•Build to allow for expansion
•The std 3 cluster out of the box isn’t always best
@socialshazza

Draw out some diagrams of how you see the system finished – helps to plan
Small deployment – 200 users, DB was oracle not
controlled by Connections admins – we had to do
an in place migration on the live DB
Medium deployment – 2,000 users, DB was DB2
controlled by Connections admins – HTTP server
was built internally then moved to DMZ once
configured
Domino LDAP
@socialshazza

Tuning and Tweaking
From this
To this
@socialshazza

Tuning and Tweaking
Read the performance tuning guides
4.0 –
http://www-10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connections_4.0_Performance_Tuning_Guide
4.5 Addendum -
http://www-10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connections_4.5_Performance_Tuning_Guide_Addendum
5.0 CR1
http://www-10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connection_V5_CR1_Tuning_guide
They are riveting bedtime reading !! Seriously they can help a lot
@socialshazza

Tuning and Tweaking – log language
Change the log language to English in WebSphere
Add the gerneric JVM argument
-Duser.language=en –Duser.region=GB
To each application server (Process definition – Java Virtual Machine)
Also to deployment manager and node agents
English is the preferred language for support – may speed up any issue
resolution
@socialshazza

Tuning and Tweaking – log language
Change the log language to English in TDI
Edit the ibmdisrv.bat|sh
In <TDI_ROOT>
Add the following to the LOG_4J variable
‐Duser.language=en –Duser.region=US
All TDI logs are now produced in English
@socialshazza

Tuning and Tweaking – Log Sizes
•Each JVM (server) has a set of
logs (SystemOut and SysErr)
•By default 1mb & overwrite –
useless for debugging / PMRs
•Change to a reasonable size
(10 – 20 mb)
•Keep a set of historical logs for
debug (a least 5 – 10)
•Each server has a set (& Dmgr/NA)
•Change under troubleshooting > logs
and trace > servername > jvm logs
•Or use the Community scripts
to set the log sizes
@socialshazza

Tuning and Tweaking – JVM tuning
•Default Java Heap Sizes on
Small / Medium Deployment is
2506 MB per application server
•Large Deployment depends on
application: 0.5 to 2.5 GB
•Memory swapping KILLS all
tuning efforts – DO NOT exceed
the physical memory
•Summing the JVM Heap sizes is
not enough – it can use more
than the max size – always
leave a pool of additional
memory for this.
@socialshazza

Tuning and Tweaking – JVM tuning
•Monitor regularly and re-tune if
necessary
•If you experience system slow
downs / hangs check logs for
JOOM errors
•Change for each server under
ServerTypes > Websphere Application Servers >
servername > Java Process Management > Process
definition > Java virtual machine
•Or use the Community Scripts
•In some cases set min and max
the same
@socialshazza

Tuning and Tweaking – Data source
•DataSource connectionPool
Sizes are set to Default min of
1 max of 10
•Configure under datasources > ds
name > connection pool properties
•Details of recommended tuning
in the guide
@socialshazza

Using Community Scripts for tuning
•See http://scripting101.org for
details and download
•Work on Windows, AIX and
Linux
•Hopefully soon on iSeries 
Things you should tune
•JVM heap sizes
•Data source connection pools
•Log sizes and roll over

•Once live about 2 weeks in we
started to hit performance
issues
•Out of memory problems,
system hanging – all the
symptoms of a GC issue
•We added more memory,
tweaked the JVMs – system
worked for a week then we
hit the same issues
Tuning – JVM OOM issues
•Upgraded Connections 4 to 4.5
CR2
•Approx. 3,000 users well
established environment
•Connections customer since V2
•Historically has issues with
migration as system heavily
customized
•Single instance – 16GB WAS
machine – 4 JVMs
@socialshazza

Tuning – JVM OOM issues
•IBM called in – CRITSIT!!
•Looked like a WebServer
thread issue
•We then started seeing Java
OOM issues again
•Discovered on boarding team
were adding batches of users
with out communicating it
Resolution:
•Monitored the JVM issues
•Added more memory to the
machine (32GB so doubled)
•Tuned JVMS, data sources and
HTTP server
•Communication when more
users being added to the
system
@socialshazza

Some awesome examples of over-kill
overkill
noun
noun: overkill
1.excessive use, treatment, or action.
“animators now face a dilemma of
technology overkill”
2. the amount by which destruction or
the capacity for destruction
exceeds what is necessary.
“the existing nuclear overkill”
@socialshazza

iSeries deployment
•IBM caved in and gave us iSeries
support – thank you IBM
•Assumed all iSeries customers may
want a large deployment topology
•Customer deployed Connections 4.5
in a large topology with one jvm for
every app – TOTAL overkill – for 180
users!!!
•HTTP server was having issues – not
enough threads set (iSeries default
is different to other OS)
• TDI won’t sync properly – only way
to get users in is to run a collect_dns
and populate from file – sync_all
deactivates all the accounts no
matter what the hash is set to
@socialshazza

iSeries deployment
Great fun to start and stop this lot
QWAS8 subsystem has MANY jobs
running under it
Machine can cope but its TOTAL
OVERKILL
@socialshazza

iSeries deployment
•New instance has 4 JVMs
•Web Server tuned correctly
•TDI now working as it should be
(see my blog post here - http://cube-soft.co.uk/latest-
ptfs-appear-to-cause-issues-with-tdi-on-iseries/)
•We have a Script to start and
stop the servers
•Working to get the community
scripts iSeries compatible
Resolution:
@socialshazza

Linux HA deployment
•Customer wanted a HA environment
•Linux based for Domino entitlements – Profiles and files
•Would start as small and roll out to all employees
• Approx 300 once complete
•TDI to sync with the LDAP server twice a day
•Customer installed following a zero to hero
•We were called in to help with some support issues
@socialshazza

Linux HA deployment
•System has 30 – yes you read
that right 30 users
•Just using the files and profiles
entitlement that comes with
Domino
•Even with 300 this is still
overkill
•HA still possible with a single
JVM per machine no need for 4
VMs when 2 would suffice
@socialshazza

Document EVERYTHING (yes really)
@socialshazza

Lessons learned and Doc everything
•Everyone (except me) hates
writing documentation
•BUT – make notes as you go, it
doesn’t need to be a full step by
step guide with screenshots
•Document all customizataions
•Any additional changes made
•Anything of note that deviates
from the guides
@socialshazza

Lessons learned and Doc everything
•Lessons learned from
installing, migrating and
updating
•Resolution to any PMRs
•Use the documentation scripts
to output WAS info, ports etc to
a file
•wsadmin -lang jython -f
ibmcnx/doc/Documentation.py
@socialshazza

Any Questions?
@socialshazza

How to contact me …..
socialshazza.com
@socialshazza
dilftechnical
cube-soft.co.uk
@socialshazza

Resources
Useful URLS
• http://www-01.ibm.com/support/docview.wss?uid=swg27012786 –
Connections system requirements
• http://www-10.lotus.com/ldd/lcwiki.nsf -
IBM Connections Wiki (Documentation)
• http://www-‐01.ibm.com/support/docview.wss?uid=swg21690640 –
Poodle Fix (Vulnerability in SSLv3)
•http://socialconnections.info
Connections Blogs
http://www.stoeps.de http://scripting101.org
http://martin.leyrer.priv.at/ http://kbild.ch
http://dilf.me.uk/socialshazza http://notesbusters.com
http://www.notesgoddess.net http://wannes.rams.be
http://ibmconnections.com http://wannes.rams.be
http://turtleblog.info http://portal2portal.blogspot.co.uk/
https://www.urspringer.de http://meisenzahl.org/
http://blog.robertfarstad.com http://www.curiousmitch.com
@socialshazza

Thank you for listening 
@socialshazza

You don't want to do it like that

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie You don't want to do it like that

Ähnlich wie You don't want to do it like that (20)

Mehr von Sharon James

Mehr von Sharon James (14)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

You don't want to do it like that