Tuesday, June 30, 2009

FTP adapter issues with EOL in BPEL/ESB with different OS's

In most of the environments there will be multiple systems having different operating systems, and in case if we need to transfer the file between these different OS’s then we need to be extra careful before evaluating the solutions . First thing will be to see what is the type of file that we are dealing with .If it is a txt or similar format then what is the type of encoding the files are using. Based on OS the EOL characters will be different .Based on the EOLs the way the input xsd gets created in adapters also needs to be scrutinized. This write up compares between EOL’s used and best practices that needs to be followed while designing/developing BPEL /ESB processes for file transfer.

Different OS’s, use different characters to represent end-of-lines (EOL). When transferring files between these operating systems, the transferring program(GUI tools like Filezilla, WinSCP) will usually make compensation for these differences and convert the EOL characters respectively, in order for them to be displayed correctly on the target operating system. In case of BPEL/ESB it doesn’t automatically take care of converting the EOL’s based on the OS. This will create an issue with the processes where business users are very particular with the formats and content of the file

OS

EOL chars

Representation

UNIX

line feed

\n

Windows

carriage return + line feed

\r\n

When transferring text /ASCII files between these operating systems, the EOL characters need to be converted during the transfer process. This is done typically by specifying text or ascii mode when transferring the files using FTP, as opposed to binary mode. If text files are transferred using binary mode, no EOL character conversion is done and these characters are retained during the transfer process. In this case, the files are often not displayed correctly on the target system. So there will be extra unwanted characters in the file after the transfer.

The FTP adapter provides mechanisms to ensure that correct end-of-line character conversion is done, if required.

1. Select Ascii, instead of binary

2. when defining the message schema in the wizard, do not define the schema as opaque. While using FTP adapter's schema, ensure that the records are terminated using the ${eol} designator. Using ${eol} tells the FTP adpater to use the operating system-specific end-of-file character in this position .

In case you have no control over the number of characters in a line, you will not be able to define an input XSD for files to convert it into NON-Opaque format .So the only option left will be to use opaque schema and use some java embedding code to do the conversion . I will be writing more on this topic in my coming blogs.

3 comments:

srikar said...

Hi George,

Srikar here. How are u sir? WHere are now?
Hey i got a small doubt man, when i m trying to file with multiple columns of same type, i am ending up with an extra line at the End of file. I have to ignore it. I am trying to use arrayTerminatedBY, but it only works when u have to end a file with some character or string. I tried writing null, but it got errored out. I am using nxsd:style="array" and nxsd:cellSeparatedBy="${eol}". But it is not working. Please help me yaar.

Regards,
Srikar.Sreeram

Georgie said...

what is the environments that are participating in the transfer.The issue usually comes because of the difference in the EOL used between environments.I believe your EOL is recognized as an enter character in the other environment. Drop me a mail @ georgethomasm@gmail.com with more details on the scenario.

Dasaradh said...

Srikar,

Use below format to avoid extra blank line..

nxsd:arrayTerminatedBy="${eof}"

Regards,
Dasaradh.