Overloaded Health State @ weblogic server

One of the issues which I recently faced for a client was with regard to Weblogic Server running on Overload status. It was sort of surprising to see a new health status in the server console other than Ok /Running .

Issue

It was pretty clear that the issue was related with Connection Pool usage.

There were clear pointers in log files also.

Internal error: Cannot obtain XAConnection weblogic.common.resourcepool.ResourceLimitException: No resources currently available in pool AppsAQ to allocate to applications, please increase the size of the pool and retry..[[

        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:555)

        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:332)

        at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:438)

        at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:93)

For troubleshooting the issue, we started monitoring the connection pool usage.

Analysis and Resolution

 As part of design we have an Apps queue which will be queuing up all the messages for outbound flow. Of late the number of processes listening to the queue had gone up, and all were trying to establish connections with the queue and de-queue the messages based on correlation ids. The number of connections in connection pool was set at 15 by default and the number of connection to the queue was more than 20 which resulted in the overloaded state. So by increasing the number of connections to the Apps queue we were able to resolve the issue.

More on Overloaded Health State

WebLogic Server 9.0 has a new health state—OVERLOADED—which is returned by the ServerRuntimeMBean.getHealthState() when a server instance whose life cycle state is RUNNING becomes overloaded. This condition occurs when Work Manager capacity is exceeded or as a result of low memory. 

The server instances health state returns to OK after the overload condition passes. An administrator can suspend or shut down an OVERLOADED server instance. 

more on Callout feature @B2B

Callout is one of the features given in B2B to achieve custom functionality which is not supported out of the box by Oracle B2B.The custom functionality can be written in java where you will get access to the payload and can be worked upon based on the requirements.

There are two ways in which we can use the callout.

1. Transport Callouts
2.  Agreement Callouts

In this blog I will touch upon some of the major differences/similarities between these 2 callouts.

Transport Callout

1.  Configure a Channel at Host or trading partner level, depending on whether the document flow is Inbound and outbound.
2. Select the java callout for Transport callout attribute under Channel Attributes.

·         For the inbound message, B2B invokes the transport callout immediately after it receives a message from the transport.

·         For the outbound message, B2B invokes the transport callout immediately before it sends a message to the transport.

3. When you call it from Transport level, java callout will get following metadata information as message parameters to the java callout along with message payload.

Filename , callout.class, DATE, callout.timeout, file.sender.type, callout.library,

TO, callout.param.BASE_LOCATION, callout.name, FROM, MESSAGE-ID

Eg sample data:

filename=Geo_2011Feb23_06_20_02_014_308_284.dat

callout.class=com.geo.b2b.callout.ArchiverTest

DATE=Wed, 23 Feb 2011 11:20:01 GMT

callout.timeout=30

file.sender.type=BINARY

callout.library=TestCallout.jar

TO=OracleINC

callout.param.BASE_LOCATION= /tmp/ARCHIVE

callout.name=TestCallout

FROM=Geo

MESSAGE-ID=34323438343736363736353

Agreement Callout

In case we decide to go callout at Agreement level,

1. Create an agreement at partner level
2. While defining callout, we will declare all the callout parameters that need to be passed to the Callout function. These values can be modified while defining the callout at agreement level for each TP.

points to note:

1. In the case of Agreement level callout, the transport level metadata information will not be available to the java callout. In case we require similar information these will have to be first configured as parameters while defining a callout and these values will have to be passed as callout parameters while defining the agreement.
2. In case of a outbound flow, when we are archiving a file from transport level using a callout will store the payload as a Edi document where as the payload archived at agreement level will be in XML format. So in case the business requirement is to archive EDI format documents, then always use Transport level callouts.(This is applicable in cases we are not using the out-of the box archiving feature.)
3. Transport callout takes precedence over Agreement callout, and so in case we enable the same callout at both levels, and try to update parameters of callout, parameter value updating at agreement level will not affect the callout behavior.
4. Use DiagnosticService for logging audit information (explained in one of my previous blogs) which will help you in debugging.

Hope this helps. J