How to use OSB Reporting Service with BAM -Part1

            The Reporting Service will be a generic service which will listen to Reporting queue of OSB, log or record transaction or exception data, or to raise a notification or alert to an end user or technical support group. It will persist report event data to a database or other similar sink so that it will be available for reporting and auditing purposes.

OSB Requesters will access the Reporting service via a non-blocking, asynchronous call, passing a simple XML log event object containing various service identification and log description data. On the service provider side, log events will be processed sequentially.

 An event can be defined as an action capable of causing more than one action to fire – so a serious exception may result in the data being persisted to database and an email notification being sent to the appropriate support team. This can be designed and developed by leveraging the functionalities of Oracle BAM tool.

Requirements ::   Reporting / Analytics

• Ability to report service consumption per service per application
• Calls per sec/min/hour/day
•  Error Counts
•   Performance Metrics
• Performance monitoring on service level.

  Approach

                        OSB defines an extensible Reporting Framework that permits adding custom providers. This OSB/Oracle BAM integration approach consists of developing a custom reporting provider that enables report information to be sent out to Oracle BAM. This is accomplished by configuring the reporting action inside an OSB service flow, and Oracle BAM picking up the message using Enterprise Message Sources.

Oracle BAM defines a basic component called a data object. A data object represents a data entity whose information is to be stored in Oracle BAM. Each new message to be integrated into Oracle BAM must be validated against an underlying XML schema specific for the selected data object

Data Model

      Define a Reporting data structure. This will hold all the data that is defined to hold information that can help in monitoring as well as reporting.

Sample data structure

Element	Description
PROJECT_NAME	Project Name
MODULE_NAME	Module name
SERVICE_NAME	getEmployee
SERVICE_TYPE	SOAP or REST
HTTP_METHOD	GET or POST
USER_NAME	User related name
USER_ID	User related id
START_TIME	Start time of the request
SYSTEM_ID	External system Id
CORRELATION_ID	Uniquely identify the record ,so that status updates can be related to the same record
MSG_ID	Uniquely identify the record
REPOSITORY_NAME
QUERYSTRING	For rest calls to store query string
PAYLOAD	XML payloads
STATUS	Success /Error
PRIMARY_ID	Any Ids that we want to track like EmployeeID
TOTAL_TIME_SEC	Time difference between Start /End time
END_TIME	End time of the request
PROXY_ID
ERROR_TEXT	Error Messages
PROCESS_FLOW_STAGE	Request ,Response, service stage-Transcoding
TECH_COMPONENT	OSB, BPEL, Java

Sample XSD

<?xml version="1.0" encoding="windows-1252" ?>

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"

            xmlns:report="http://geo.com/report/header"

            targetNamespace="http://geo.com/report/header"

            elementFormDefault="qualified">

  <xsd:element name="REPORT_DATA">

    <xsd:complexType>

      <xsd:sequence>

        <xsd:element name="PROJECT_NAME" type="xsd:string"/>

        <xsd:element name="MODULE_NAME" type="xsd:string"/>

        <xsd:element name="SERVICE_NAME" type="xsd:string"/>

        <xsd:element name="SERVICE_TYPE" type="xsd:string"/>

        <xsd:element name="HTTP_METHOD" type="xsd:string"/>

        <xsd:element name="USER_NAME" type="xsd:string"/>

        <xsd:element name="USER_ID" type="xsd:string"/>

        <xsd:element name="START_TIME" type="xsd:string"/>

        <xsd:element name="SYSTEM_ID" type="xsd:string"/>

        <xsd:element name="CORRELATION_ID" type="xsd:string"/>

        <xsd:element name="MSG_ID" type="xsd:string"/>

        <xsd:element name="REPOSITORY_NAME" type="xsd:string"/>

        <xsd:element name="QUERYSTRING" type="xsd:string"/>

        <xsd:element name="PAYLOAD" type="xsd:string"/>

        <xsd:element name="STATUS" type="xsd:string"/>

        <xsd:element name="QUERY_KEY" type="xsd:string"/>

        <xsd:element name="PRIMARY_ID" type="xsd:string"/>

        <xsd:element name="TOTAL_TIME_SEC" type="xsd:string"/>

        <xsd:element name="END_TIME" type="xsd:string"/>

        <xsd:element name="PROXY_ID" type="xsd:string"/>

        <xsd:element name="ERROR_TEXT" type="xsd:string"/>

       <xsd:element name="PROCESS_FLOW_STAGE" type="xsd:string"/>

        <xsd:element name="TECH_COMPONENT" type="xsd:string"/>

      </xsd:sequence>

    </xsd:complexType>

  </xsd:element>

</xsd:schema>

Implementation

These options are useful in case you have custom OSB Reporting Data Handler which will be listening to the OSB JMS Provider QUEUE. OSB will perform reporting using the OSB JMS Provider QUEUE and message payload will contain ReportData and ReportMetadata. These variables can be XML elements or Strings. These will need to be retrieved in the custom OSB reporting handler and processed further.

More on implementation in jvoggels blog

BAM Dashboards

OSB has inbuilt features to monitor service’s health using metrics and alerts; however, this monitoring is mostly oriented toward technical users.    Oracle Business Activity Monitoring is the preferred product to give business users the functional monitoring on their business services and processes in the enterprise

·         BAM can be leveraged to build reports based on the data in the Reporting table.

·         The BAM dashboards are customizable and can define new dashboards as required.

·         It will be constrained by the way we define the data model, so will need to be sure we capture all the information required into the data model.

Notes

•  These can be implemented at Gateway level to capture the request and response so all processes can be monitored.
•  All process developers must be following naming standards in endpoint URI creation, so that we can identify the Module /service.
•  The correlation Id needs to propagate across the proxies so that in case of errors the same record can be updated with the Error status /error message.
•  Correlation Id can be transmitted using custom user headers in transport headers or as part of SOAP headers.

Tips on reporting in blog

References :

• http://jvzoggel.wordpress.com/2012/02/15/oracle-service-bus-logging-tracing-iii-create-custom-report-provider/
• http://jvzoggel.wordpress.com/2012/01/18/osb_tracing_report_action/
• http://blog.grummt.org/?cat=4
• http://eelzinga.wordpress.com/2009/07/17/oracle-service-bus-xquery-and-optional-parameters/
• http://docs.oracle.com/cd/E21764_01/doc.1111/e15867/reporting.htm
• http://www.oracle.com/technetwork/middleware/bam/osb-bam.pdf

Implementing Correlation sets - issues and workarounds

Correlation is one interesting area which I always wanted to write on. Recently we got to work extensively on implementing correlations. I will list down some of the issues we had while implementing the correlation for complex workflows.

Why we need correlation?

Interaction patterns can be mainly grouped into Synchronous and Asynchronous. In case of Synchronous the request and reply happens using the same port and the pipe is open till the reply is back. But in case for Asynchronous there are separate ports for both request and reply and because of this the message needs to be tracked so that reply goes to the same request. The correlation mechanism helps to track and route messages to appropriate process instances.

So in case of normal BPEL asynchronous calls BPEL Engine takes care of correlating the message to the instance by assigning a unique correlation id. In case of scenarios where process AàBàCàA or A->B->C->D->A (-> = calls) the correlation has to be taken care by the developer by using correlation sets.

Issues /workarounds while using correlations

Scenario: The example scenario I will be taking will be Agile-SAP PIP which has BPEL1(EBF) -> ESB1(ReqEBS1) -> BPEL2(ProvABCS) -> ESB2(RespEBS1) à BPEL1(EBF)

Here correlation sets are created and assigned while invoking the partner services.

In this flow the out of the box PIP supports only processing of Parts which are of type Item. One of the Clients had a requirement to process Parts which are of type documents. For this we added one more ProvABCS and we had 2 parallel flows which will look like below.

1. -> BPEL1(EBF) -> ESB1(ReqEBS1) -> BPEL2(ProvABCS) -> ESB2(RespEBS1) à BPEL1(EBF)

2. -> BPEL1(EBF) -> ESB1(ReqEBS1) -> BPEL3 (New ProvABCS) -> ESB2(RespEBS1) à BPEL1(EBF)

So for correlating this new flow we created one more correlation set in the same scope and tested the flow.

Issue #1

The response message was not able to properly identify the unique correlation set and was failing.

Staktrace snippet

"ORABPEL-03813 Failed to evaluate correlation query. Failed to evaluate the correlationAlias query "/geo:ResponseMessage/header:Header/header:RequestEBMID" on the element "oracle.xml.parser.v2.XMLElement@1473827" Please check your BPEL/WSDL source to make sure that property alias is defined correctly. at com.collaxa.cube.engine.delivery.CorrelationProperty.setMessagePart(CorrelationProperty.java:85) at com.collaxa.cube.engine.delivery.DeliveryHelper.createCorrelationSet(DeliveryHelper.java:148)

Workaround

1. Create separate scopes for both the flows.
2. Create Correlation sets for each of the flow in their specific scopes.

Test it and it should work properly.

Life is never easy. The new requirement was to call the new Prov ABCS process as many times as there are documents in the payload and at a time there will be multiple items and documents. So while implementing this we ran into our second issue.

Issue#2

While invoking the ProvABCS process in a loop BPEL engine threw the error.

“cannot initiate correlation set more than once”. It meant we cannot initiate the same instance of correlation set again and again.

Stack Trace snippet

Correlation Violation - cannot initiate correlation set more than once

Workaround

The issue is with initiate attribute where we have mentioned it “initiate=yes”. When we look at the BPEL specifications there are 3 values possible for the

Initiate attribute of correlation set. They are

· yes. to initiate the correlation set.

· no. This is the default option. Activity should not initiate the correlation set.

· join. If the correlation set is not yet initiated, initialize it.

Join attribute was the apt choice here, but the attribute join is not supported by Oracle BPEL PM. This is because the implementation of initiate attribute is declared as a Boolean value, it supports only yes/no.

So next option was to create a separate inner scope inside the while loop and declare the correlation specific inside the inner scope. By doing that, our intention is to isolate the initialized value and will get nullified once we come out of the inner scope. So in each loop, a new instance of correlation set will get initialized when flow enters the inner scope and dies when the scope gets over. This will eliminate the issue due to initiation of the same correlation set more than once.

Special thanks to Haresh who worked on trying out different options to make correlations work and helping me this blog.

That’s it on correlation, now back to football J