Overloaded Health State @ weblogic server

One of the issues which I recently faced for a client was with regard to Weblogic Server running on Overload status. It was sort of surprising to see a new health status in the server console other than Ok /Running .

Issue

It was pretty clear that the issue was related with Connection Pool usage.

There were clear pointers in log files also.

Internal error: Cannot obtain XAConnection weblogic.common.resourcepool.ResourceLimitException: No resources currently available in pool AppsAQ to allocate to applications, please increase the size of the pool and retry..[[

        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:555)

        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:332)

        at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:438)

        at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:93)

For troubleshooting the issue, we started monitoring the connection pool usage.

Analysis and Resolution

 As part of design we have an Apps queue which will be queuing up all the messages for outbound flow. Of late the number of processes listening to the queue had gone up, and all were trying to establish connections with the queue and de-queue the messages based on correlation ids. The number of connections in connection pool was set at 15 by default and the number of connection to the queue was more than 20 which resulted in the overloaded state. So by increasing the number of connections to the Apps queue we were able to resolve the issue.

More on Overloaded Health State

WebLogic Server 9.0 has a new health state—OVERLOADED—which is returned by the ServerRuntimeMBean.getHealthState() when a server instance whose life cycle state is RUNNING becomes overloaded. This condition occurs when Work Manager capacity is exceeded or as a result of low memory. 

The server instances health state returns to OK after the overload condition passes. An administrator can suspend or shut down an OVERLOADED server instance.