SOA Database Connectivity Issues Showing DatabaseException Internal Exception
1. Imaging activity is causing performance issues in Production
When running SOA composites and applications we are getting multiple errors of the following
types:
1. [TopLink Warning]: 2011.02.19 10:19:56.413--UnitOfWork(1606199273)--Exception
[TOPLINK-4002] (Oracle TopLink - 11g Release 1 (11.1.1.3.0) (Build 100323)):
oracle.toplink.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Unexpected exception while enlisting
XAConnection java.sql.SQLException: XA error: XAResource.XAER_RMERR start() failed on
resource 'SOADataSource_SOA': XAER_RMERR : A resource manager error has occured in the
transaction branch
javax.transaction.xa.XAException: Unexpected error during start for XAResource
'SOADataSource': Transaction timed out after 31 seconds
2. java.sql.SQLException: Connection has been administratively disabled. Try later.
at weblogic.jdbc.common.internal.ConnectionEnv.checkIfEnabled(ConnectionEnv.java:895)
at weblogic.jdbc.wrapper.PoolConnection.checkConnection(PoolConnection.java:58)
at weblogic.jdbc.wrapper.Connection.preInvocationHandler(Connection.java:93)
at
weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection.getAutoCommit(Un
known Source)
at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler
.setAutoCommit(AttributeRestoringConnectionInvocationHandler.java:83)
at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler
.invoke(AttributeRestoringConnectionInvocationHandler.java:65).
3. ####<Mar 15, 2015 1:46:04 AM PDT> <Info> <EJB> <crete.sdsu.edu> <soa_server1>
<[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'>
<SenaSupportAdmin> <BEA1-06D26210EA85137D24B4>
<16a6b0192850dcba:1286999d:14c0628f21e:-8000-0000000000004315> <1426409164448> <BEA-
010227> <EJB Exception occurred during invocation from home or business:
weblogic.ejb.container.internal.StatelessEJBHomeImpl@23ce6cf1 threw exception:
java.lang.RuntimeException:
Internal Exception: java.sql.SQLException: Internal error: Cannot obtain XAConnection
weblogic.common.resourcepool.ResourceDisabledException: Pool SOADataSource is
Suspended, cannot allocate resources to applications..
After checking the soa_server log we seen lot of errors like we have seen above.
These types of error are basically slow down the performance of production system. As we saw
there is lot of errors but some errors are related to Soa_infra database. In above error list we saw
some connection pooling errors. All the above internal errors are mostly related to DATABASE.
2. Following are some root causes for this error.
The problem can occur due to several different reasons such as:
1. Database connection timeout
2. Slow database
3. Large message or payload size
4. Network interruption
5. Transaction timeout
Solution:
1. Increase the connect timeout from all data sources such as for example
SOALocalTxDataSource, SOADataSource, OraSDPMDataSource, mds-soa and mds-
owsm.
2. Set inactivity timeout to some positive number.
3. Potential fix for slow database.
4. Potential fix for large payloads.
In our case we can resolve this issue by increasing a capacity from properties of connection
pool of each database. In our case it varies from 0 to 100 in case of some data sources.
Please see below screenshot.
3. So in future to avoid this error we can maximize capacity up to 200 and also
CONNECT_TIMEOUT more than 300000.
REFERENCES:
1. java.sql.SQLException: Cannot obtain XAConnection
weblogic.common.resourcepool.ResourceDisabledException: Connection Pool is Suspended (Doc ID
1446899.1).
2. SOA Database Connectivity Issues Showing DatabaseException Internal Exception:
java.sql.SQLException (Doc ID 1309181.1).
3. https://community.oracle.com/thread/2171885