Out of memory error is thrown from java.util.zip.ZipFile.open method.
Stacktrace
oracle.tip.esb.server.common.exceptions.BusinessEventRetriableException: An unhandled exception has been thrown in the ESB system. The exception reported is: "java.lang.OutOfMemoryError
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:203)
at java.util.jar.JarFile.<init>(JarFile.java:132)
at java.util.jar.JarFile.<init>(JarFile.java:97)
at oracle.classloader.SharedJar.doOpen(SharedJar.java:208)
at oracle.classloader.SharedCodeSource.open(SharedCodeSource.java:1136)
at oracle.classloader.SharedCodeSource.ensureOpen(SharedCodeSource.java:948)
at oracle.classloader.SharedCodeSource.getResourceBytes(SharedCodeSource.java:967)
This is inherent bug with JDk1.4 , still open in jdk1.5 and is fixed only in Jdk 1.6. So the only option is to get rid of the old JDK and replace it with JDK1.6. JDK1.6 looks like the best bet for all the issues related to java JDK. Even though 10.1.3.3 is not certified with java 1.6 its always better to use java 1.6 since this bug is only fixed in java 1.6.