This blog is on the implementation details of how to process huge XML payload files in smaller chunks. As you know out of memory happens because XSL transformation and translation leaves huge memory footprint and it proportionally grows with XML payload size. In order to avoid that what we have done is to parse the XML in smaller chunks. To implement this we brought in indexes to XSL transformation. This is done using a begin Index and end Index which will be dynamically passed into the XSL. It helped in getting control on the no.of records being transformed for processing.
I will list down some important steps that were followed as part of the prototype we had developed before implementing it in real time scenarios.
To start with we created 2 BPEL processes – PayloadSplitter and PayloadProcessor
PayloadSplitter will have the logic for breaking down the XML payload into smaller chunks. PayloadProcessor will have the logic required for processing the payload. I will be focusing only PayloadSplitter as that takes of managing the payload.
PayloadSplitter
- Initialization
a. Create a Variable params which should be of type parameters(Refer Param.xsd)
b. Create a variable totalCount for storing the no of records in the payload.
c. Create a variable incrementSize for storing the no of records that needs to be processed in each loop.
d. Create a variable currentIndex for storing the end index of the records processed in each loop.
For eg:-If we have 100 records to be processed and 5 records needs to be processed in each loop.
totalCount = 100 & incrementSize = 5. currentIndex for first loop will be 5 as it will be initialized as incrementSize.
For 2nd loop currentIndex will be 10(currentIndex + incrementSize) and so on
Code Snippet :: Custom schema - Param.xsd
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:p="http://schemas.oracle.com/service/bpel/common"
xmlns="http://schemas.oracle.com/service/bpel/common"
targetNamespace="http://schemas.oracle.com/service/bpel/common"
elementFormDefault="qualified">
<xsd:element name="parameters">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="item" minOccurs="1" maxOccurs="unbounded">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="name" type="xsd:string"/>
<xsd:element name="value" type="xsd:string"/>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:schema>
- Assign XML fragment mentioned below to the parameters
Code Snippet :: XML fragment in Assign
<bpelx:assign name="Assign_ParamNames">
<copy>
<from>
<parameters xmlns="http://schemas.oracle.com/service/bpel/common">
<item>
<name>beginIndex</name>
<value/>
</item>
<item>
<name>endIndex</name>
<value/>
</item>
</parameters>
</from>
<to variable="params" query="/ns2:parameters"/>
</copy>
3. Assign 0 as the begin index to /ns2:parameters/ns2:item[1]/ns2:value .
(You will have to manually edit the index value[i] in the XPath tab)
<copy>
<from expression="'0'"/>
<to variable="params" query="/ns2:parameters/ns2:item[1]/ns2:value"/>
</copy>
4. Assign incrementSize as the end index to /ns2:parameters/ns2:item[2]/ns2:value
(You will have to manually edit the index value [i] in the XPath tab)
<copy>
<from variable="incrementSize"/>
<to variable="params" query="/ns2:parameters/ns2:item[2]/ns2:value"/>
</copy>
- Initialize currentIndex as incrementSize (only for first loop)
- Now starts the while loop which will process records in smaller chunks.
Condition check : totalCount > 0
Inside the while loop we will be having 3 main activities.
a) Transform the payload in smaller chunks to the format expected by PayloadProcessor
b) Invoke the payload processor (invoke activity along with a partner link to PayloadProcessor)
c) Assign activity to update the loop variables for the next run of the loop.
- Have a XSLT transform activity where we will be leveraging the feature of Passing Parameters into XSLT.
Refer Ram Kumar’s article :Passing BPEL Variable contents into XSLT as Parameters
We will pass the params variable to the XSL which is populated with beginIndex and endIndex values for first loop.
The transformation from the payload variable will happen to the input variable of the Payload Processor.
Code Snippet :: .bpel file
<assign name="Transform_1">
<bpelx:annotation>
<bpelx:pattern>transformation</bpelx:pattern>
</bpelx:annotation>
<copy>
<from expression="ora:processXSLT('Transformation_1.xsl',
bpws:getVariableData('inputVariable','payload'),
bpws:getVariableData('params') )"/>
<to variable="invokeProcessInputVariable" part="payload"/>
</copy>
</assign>
Code Snippet :: .xsl file
<xsl:stylesheet>
………………………………
…………………………………
<xsl:param name=" beginIndex"/>
<xsl:param name="endIndex "/>
<xsl:template match="/">
<ns2:PayloadProcessorProcessRequest>
<xsl:for-each select="/ns1:PayloadSplitterProcessRequest/ns1:inputData">
<xsl:if test="(position() > $ beginIndex) and (position() <= $ endIndex)">
<ns2:OutputData>
……………………………….
…………………………………………….
</ns2:OutputData>
</xsl:if>
</xsl:for-each>
</ns2:PayloadProcessorProcessRequest>
</xsl:template>
</xsl:stylesheet>
- Have an invoke activity to call the PayloadProcessor .PayloadProcessor BPEL process should contain the logic to process the payload.
- In assign Activity
a) TotalCount = TotalCount - incrementSize
b) Assign endIndex to beginIndex for next loop.
<copy>
<from variable="params"
query="/ns2:parameters/ns2:item[2]/ns2:value"/>
<to variable="params"
query="/ns2:parameters/ns2:item[1]/ns2:value"/>
</copy>
c) currentIndex = currentIndex + incrementSize
d) Assign currentIndex to endIndex.
<copy>
<from expression="bpws:getVariableData('currentIndex')"/>
<to variable="params"
query="/ns2:parameters/ns2:item[2]/ns2:value"/>
</copy>
</assign>
That’s pretty much the logic required for creating Payload Splitter .Special thanks to Srinivas Kommanaboyina who had worked on developing this prototype.
