Curious on how to make your Connections PINK environment run smoothly while reducing support effort? Need help debugging and getting to the core of some Connections challenges? Join Nico to find out how to resolve common issues, learn troubleshooting basics and other useful knowledge to ensure an efficient Connections PINK on-premises environment. Level up your debugging skills while learning more about back-end topics such as IBM Cloud private, Kubernetes, Docker as well as Orient Me, Metrics and Connections Customizer. Walk away with Connections PINK best practice tips and tricks to help you provide steady and efficient social capabilities!
6. Be aware of the big picture
1. Get an overview
2. Define the involved components & services
3. Start debugging on a high level
4. Track down the root cause
6
7. Track down the root cause
• Reproducible and/or periodically?
– Scheduler?
• Sequence error?
– When I do this, that occurs…
• Client-side issue?
– Browser, Proxy, Location
• Or server-side issue?
– Different behavior on different Nodes
– Analyze involved components & services
• Last changes?
– Configuration, Frontend, Backend
– OS, Hardware, Network, Firewall
7
8. Get support
• Knowledge Center https://goo.gl/up6cxG
– Troubleshooting Section https://goo.gl/IaVinx
• IBM Connections Forum http://goo.gl/CVvQCU
• IBM Cloud private Slack channel https://slack-invite-ibm-cloud-
tech.mybluemix.net/
• Community Blogs and/or Chats (they have a new home!)
• Fix Central
• Support Case (PMR)
– include logs
– /opt/deployCfC/collectLogs.sh
8
13. Client-side issues
• Test with different Browsers & versions (Chrome, IE, FF)
– Policies, Settings?
– IE VMs are helpful
• https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/
– Do not use IE on Servers
• Related only to some locations (Proxies) or languages?
• Use Developer Tools (Browser Console, Network Tab)
13
14. NGINX Proxy
• Part of the PINK deployment
• Forwards all request to Connections Customizer
• Get logs
– /var/log/nginx/access.log
– /var/log/nginx/error.log
• Enable debugging
– Customize /etc/nginx/nginx.conf
– Global debugging (1)
– Based on IP address (2)
– systemctl reload nginx
14
15. Connections Customizer
• Containerized Node.js microservice
– mw-proxy
• Injects customizations and forwards
requests to IHS
• Get logs
– kubectl get pods -n connections |grep -i mw-proxy
– kubectl logs -n connections mw-proxy-*
• Debugging
– Enabled by default
– Downsize replicas or use kubetail
• kubectl patch deploy -n connections mw-proxy -p
'{"spec":{"replicas":1}}'
15
16. Make Your Data Work For You
Troubleshooting
Applications
17. Orient Me
• Based on four frontend microservices
– orient-web-client (1)
– middleware-graphql (1)
– itm-services (2)
– community-suggestions (3)
– and many more backend microservices
• Get logs
– kubectl get pods -n connections
– kubectl logs -n connections *
17
19. PINK Sanity checks
• New with 6.0.0.5
• Get port from services
– kubectl get services -n connections |grep -i sanity
19
20. Metrics UI
• UI and Event capturing still on WebSphere
• Backend based on Elasticsearch
– Cognos is not needed anymore
• Migration path from Metrics DB to Elasticsearch
• Enable tracing (WebSphere)
– com.ibm.connections.metrics.*
• UI
• Event tracker
• Elasticsearch
20
22. IBM Cloud private
• “Toolset” including Kubernetes, Registry, ELK Stack, Monitoring &
Management UI and many more
• Get logs
– kubectl logs –n kube-system <pod>
• ELK Stack for kube-system namespace (ICp 2.1+)
– docker logs
• kubelet, calico
– journalctl –u docker.service
• Enable debugging
– docker daemon –debug
– Install Calico CLI (calicoctl) to debug Cluster network issues
• Part of the installer (6.0.0.5)
22
23. kubectl
• Deploy it locally!
– User menu – Configure Client
– bx pr cluster-config
• kubectl Cheat Sheet
– https://goo.gl/pQ5ENv
• Change the default namespace to skip -n connections
– kubectl config set-context $(kubectl config current-context) --namespace=connections
• kubectl logs
– or kubetail
23
24. kubectl
• kubectl logs –p
– Print logs of the previous instance of the container
• kubectl describe pod
– Check Event section for pod creation issues
24
25. App Registry
• Based on two Node.js microservices
– appregistry-client
– appregistry-service
• Dependencies to Redis and MongoDB
• Get logs
– kubectl logs –n connections <pod>
• Enable debugging
– kubectl patch deploy <pod> -n connections -p
'{"spec":{"template":{"spec":{"containers":[{"env":[{"name":
"LOG_LEVEL","value":"debug"}],"name":”<pod>"}]}}}}’
25
26. Metrics backend
• Based on three Elasticsearch microservices (with 3 nodes each)
– es-client
– es-master
– es-data
• Authentication is based on a client certificate (Search Guard plugin)
– Event capturing still on WebSphere!
26
30. Solr
• Cluster based on three nodes
• Get logs
– kubectl exec -it -n connections solr-0 -- cat /home/solr/data/server/logs/solr.log
– kubectl logs does not display runtime logs
• Access Solr
– kubectl exec -it solr-0 -n connections -- curl --insecure -E /home/solr/solr-
6.3.0/certs/cert.pem --key /home/solr/solr-6.3.0/certs/key.pem <url>
• Get Cluster information
– https://localhost:8984/solr/admin/collections?action=clusterstatus&wt=json
• Enable debugging
– https://localhost:8984/solr/admin/info/logging --data-binary
'set=root:FINEST&wt=json'
30
31. Redis
• Cluster based on three nodes & Redis Sentinel
• BLUE is forwarding events
– Community creation, new user profile, …
– /connections/config/highway.main.settings.tiles
• c2.export.redis.host|port|pass
• Subscribe Events
– kubectl exec -it -n connections redis-server-0 -- redis-cli
-a <password> subscribe connections.events
• Use telnet to validate the connection
31
32. PINK authorization
1. User accesses Connections and will be redirected to Orient Me
2. User will be redirected to BLUE (/homepage/login) for authentication
3. User authenticates with BLUE (LDAP, SSO) and gets a LtpaToken and
JSESSIONID
4. BLUE requests a PINK token (/social/auth/token) and creates a JWS
token cookie afterwards
5. BLUE redirects the request back to PINK (/social)
6. PINK checks for the LtpaToken and JSESSIONID (if not present
7. à Step 2)
8. PINK authorizes the User after a last check against the Profiles API
32
34. Make Your Data Work For You
Troubleshootingata migrations
35. Orient Me – Profile Migration
• Microservice based on Node.js (people-migrate)
– Migrates Profiles, Report Chain, Network and other information
• Global configuration file /usr/src/app/migrationConfig
• Talks to Profiles & Communities
– Use curl to try to access
– Authentication is working?
• Talks to MongoDB
– Up and running?
• Logs (/usr/src/app/logs)
– failed_users.txt
– migration.log
– report.html
• Generate report
– npm run start report mailaddress
35
36. Metrics – Event Migration
• Events migration (Metrics DB to Elasticsearch) is done by wsadmin
– execfile('metricsEventCapture.py’)
• Get logs
– <was_profile>/logs/<server>/MetricsMigration_*.log
• Enable debugging
– com.ibm.connections.metrics.migrate.*
• Try to connect from WebSphere host
– openssl pkcs12 -in elasticsearch-metrics.p12 -out cert.pem --nokeys
– openssl pkcs12 -in elasticsearch-metrics.p12 -out keys.pem -nocerts --nodes
– curl --insecure -E cert.pem --key keys.pem
https://<service_host>:<service_port>/_cat/indices?v
36