Tuesday, June 22, 2010

Implementing Correlation sets - issues and workarounds

Correlation is one interesting area which I always wanted to write on. Recently we got to work extensively on implementing correlations. I will list down some of the issues we had while implementing the correlation for complex workflows.

Why we need correlation?

Interaction patterns can be mainly grouped into Synchronous and Asynchronous. In case of Synchronous the request and reply happens using the same port and the pipe is open till the reply is back. But in case for Asynchronous there are separate ports for both request and reply and because of this the message needs to be tracked so that reply goes to the same request. The correlation mechanism helps to track and route messages to appropriate process instances.

So in case of normal BPEL asynchronous calls BPEL Engine takes care of correlating the message to the instance by assigning a unique correlation id. In case of scenarios where process AàBàCàA or A->B->C->D->A (-> = calls) the correlation has to be taken care by the developer by using correlation sets.

Issues /workarounds while using correlations

Scenario: The example scenario I will be taking will be Agile-SAP PIP which has BPEL1(EBF) -> ESB1(ReqEBS1) -> BPEL2(ProvABCS) -> ESB2(RespEBS1) à BPEL1(EBF)

Here correlation sets are created and assigned while invoking the partner services.

In this flow the out of the box PIP supports only processing of Parts which are of type Item. One of the Clients had a requirement to process Parts which are of type documents. For this we added one more ProvABCS and we had 2 parallel flows which will look like below.

1. -> BPEL1(EBF) -> ESB1(ReqEBS1) -> BPEL2(ProvABCS) -> ESB2(RespEBS1) à BPEL1(EBF)

2. -> BPEL1(EBF) -> ESB1(ReqEBS1) -> BPEL3 (New ProvABCS) -> ESB2(RespEBS1) à BPEL1(EBF)

So for correlating this new flow we created one more correlation set in the same scope and tested the flow.

Issue #1

The response message was not able to properly identify the unique correlation set and was failing.

Staktrace snippet

"ORABPEL-03813 Failed to evaluate correlation query. Failed to evaluate the correlationAlias query "/geo:ResponseMessage/header:Header/header:RequestEBMID" on the element "oracle.xml.parser.v2.XMLElement@1473827" Please check your BPEL/WSDL source to make sure that property alias is defined correctly. at com.collaxa.cube.engine.delivery.CorrelationProperty.setMessagePart(CorrelationProperty.java:85) at com.collaxa.cube.engine.delivery.DeliveryHelper.createCorrelationSet(DeliveryHelper.java:148)

Workaround

  1. Create separate scopes for both the flows.
  2. Create Correlation sets for each of the flow in their specific scopes.

Test it and it should work properly.

Life is never easy. The new requirement was to call the new Prov ABCS process as many times as there are documents in the payload and at a time there will be multiple items and documents. So while implementing this we ran into our second issue.

Issue#2

While invoking the ProvABCS process in a loop BPEL engine threw the error.

“cannot initiate correlation set more than once”. It meant we cannot initiate the same instance of correlation set again and again.

Stack Trace snippet

Correlation Violation - cannot initiate correlation set more than once

Workaround

The issue is with initiate attribute where we have mentioned it “initiate=yes”. When we look at the BPEL specifications there are 3 values possible for the

Initiate attribute of correlation set. They are

· yes. to initiate the correlation set.

· no. This is the default option. Activity should not initiate the correlation set.

· join. If the correlation set is not yet initiated, initialize it.

Join attribute was the apt choice here, but the attribute join is not supported by Oracle BPEL PM. This is because the implementation of initiate attribute is declared as a Boolean value, it supports only yes/no.

So next option was to create a separate inner scope inside the while loop and declare the correlation specific inside the inner scope. By doing that, our intention is to isolate the initialized value and will get nullified once we come out of the inner scope. So in each loop, a new instance of correlation set will get initialized when flow enters the inner scope and dies when the scope gets over. This will eliminate the issue due to initiation of the same correlation set more than once.

Special thanks to Haresh who worked on trying out different options to make correlations work and helping me this blog.

That’s it on correlation, now back to football J

19 comments:

Nagaraju said...

This is one finest post i ever read,
weare facing the issue while invoking on BPEL porcess where we are usign correlation set adn after creating scope our issue is resolved.

Thanks for your post :)

kiran said...

I am doing a similar kind of requirement in Oracle 11g where A(async bpel) -> b(mediator1) -> c(mediator2) -> d(async bpel process)....In this flow the interaction from bpel to mediator and between mediators is one way interaction. In my process A, I have invoke to mediaotr1 and after that I have a receive pointing to callback response from process d.....Here I have used correlation sets in invoke and receive...but the callback response is not at all coming from process d to process a....Can you please suggest

George Thomas said...

Kiran,
You will need to send me the error that you get while sending back the reply. On design side, Is there any specific reason for you to have 2 mediators in between the bpel processes.

Anonymous said...

George Thomas,

Thanks for the response. There is no error message as such, but the
receive activity in caller process A is in pending state with message as:
"Waiting for "processResponse" from "CallAsyncResponseReference. Asynchronous callback." Can you please suggest on this.

Anonymous said...

George,

Can you give me your details so that I can contact you on correlation sets issue

George Thomas said...

send mail to georgethomasm@gmail.com

Veera said...

hi Geroge,

can you please assist me with below correlation issue in AIA implemnetation.Please find the below my integration flow.
Req APP <-> Req ABCS -> Req EBS -> Prov ABCS -> Resp EBS - > Req ABCS.
here my problem is every time new instance(req ABCS) is creating if the response EBS sends the response back req ABCS.

Can you please assist me with this.
Regards,
Veera

George Thomas said...

i am not clear on your issue. The REQ-ABCS will create a new instance only if you have enabled the Create New Instance checkbox in the receive that is invoked by the RESP-EBS. Check the setting of the intermediate Receive activity,it should fix the issue. It doesn't look related with Correlation

SJ said...

George,
I am trying a demo on correlation. My scenario is somewhat similar to Kiran. The composite has 3 Asynch Process (say A,B,C all 2-way).
Clients calls Process A then Process A invokes Process B & then Process B invokes Process C. Now to return the response I am using correlation. Process A has receive activity in addition to the initial receiveInput activity. I have created a correlationSet & set two properties one at initial receiveInput activity & other at receive activity waiting for response from Process C. Now Process B & C completes and Process A is Running state saying "waiting for response" and after sometime this Running state becomes stale.
Can you please suggest, where i might be going wrong.

Thanks.

George Thomas said...

if the scenario is A->B->C and A<-B<-C then there is no need of correlation issue.Since callback clients will take care of returning the response to the corresponding instance.The BPEL uses ws-addressing to correlate these requests. Only if the sequence is broken then only there is a correlation issue where we need to pitch in with correlation sets

SJ said...

George,
Thanks for your response.
I think, you din't get the scenario. The flow is A->B->C and response is C->A.

George Thomas said...

ok.. can u send me the error. You had mentioned A, B,C has having callbacks.thats why i told there is no need of co-sets.

SJ said...

There is no error message as such. If I see the flow trace Process B&C are in completed state while Process A is in Running state stating "waiting for the processResponse from PELProcess3.bpelprcoess3_client" Asynchronous callback. And while Process C is in running state there is BPEL Recovery message shown at the EM Console and also this running state changes to Stale state after sometime both in Em Console & Flow trace.

Lisan said...

Hi Thomas,

I'm farely new with BPEL. I'm having similar Correlation initialize issue. I didn't really understand your explanation. You wrote "declare the correlation specific inside the inner scope". Can you please share a screenshot? It would be a great help!!
BR, Lisanul

rajesh said...

Hi Thomas i have a similar issue with my implementation.I am implementing a Async BPEL<---->Async BPEL correlation and getting error:

Error(100): correlation set "sampleCorrelationSet1" cannot be used in this , because property "{http://xmlns.oracle.com/Ramesh_Correlation_Task/Async_Async_callerCorrelation/Async_CallerBPELProcess/correlationset}property_2 of this correlation set is not associated with the message "{http://xmlns.oracle.com/Ramesh_Correlation_Task/Asysnc_Async_ThirdpartyCorrelation/ThirdpartyData_BPEL}ThirdpartyData_BPELRequestMessage"


could you please suggest a work around.

And also can you please suggest me a setting for this below requirement:

A syncBPEL is requesting and had to wait a certain time for ASync BPEL response, i need to use correlation for this?.Can you please help.

Regards,
Rajesh

Ayyadurai M said...

Hi Georgie,

I am trying to use correlation sets in BPEL. My requirement is an async BPEL sends a request to a JMS queue(ReqQueue) and gets back the response from a JMS queue(RespQueue). Here I have used correlation sets in Invoke activity (while producing the msg in ReqQueue) based on the field Msg_Id in the payload and another correlation set in the Receive activty (while consuming the msg from the RespQueue).


Now the problem is, the BPEL process is executing till consumption of msgs from the (RespQueue), JMS adapter is consuming the msg from RespQueue, but the response msg is not reaching the BPEL process. In the instance of the BPEL process, I am seeing the 2nd Receive activity as pending and it is saying "Waiting for "Consume_Message" from "Consume_JMS". Asynchronous callback."


Ranjani said...

Hi George

Good document on a specific issue faced while implementing correlation set and a workaround for it.

I am learning the concept of correlation set and did a simple POC. built 3 simple service A,B,C. A -> B -> C -> A. Services A and C are asynchronous and B is one way. Service A has Invoke to B and a Receive from C. Defined Correlation set with appropriate property and Aliases (one with service B's Request Message type and another with service C's Response Message Type). Initiated the correlation set in Invoke from A -> B and set patter as 'IN' as the request is sent. Used the same correlation set for receive from A <- C with initiate set to 'NO'. When I tested this flow, I got an error "Correlation set no initialized.It cannot be used in receiving activity". Blindly going by the error message, initiated the correlation set at the first receive of service A. Now got no error but the process A is running for long time, pending receive response from C. I know this is not conceptually correct solution to overcome the initiate error in receive of A <- C. Can you help me with a solution to complete this POC successfully?

Thanks in advance
Ranjani

Ranjani said...

Good post on correlation set issues and work around

I was trying to learn the concept of correlation set and did a simple POC. built 3 simple service A,B,C. A -> B -> C -> A. Services A and C are asynchronous and B is one way. Service A has Invoke to B and a Receive from C. Defined Correlation set with appropriate property and Aliases (one with service B's Request Message type and another with service C's Response Message Type). Initiated the correlation set in Invoke from A -> B and set patter as 'IN' as the request is sent. Used the same correlation set for receive from A <- C with initiate set to 'NO'. When I tested this flow, I got an error "Correlation set no initialized.It cannot be used in receiving activity". Blindly going by the error message, initiated the correlation set at the first receive of service A. Now got no error but the process A is running for long time, pending receive response from C. I know this is not conceptually correct solution to overcome the initiate error in receive of A <- C. Can you please help me out with a solution to complete this POC successfully?

George Thomas said...

Can you check if you have created aliases for the same correlation set which has one alias with unique invoke input variable property and unique receive input variable property(like a primary key).So that same correlation set identifies the 2 values using the alias property.