Performance Test Issues In Weblogic

Overview
The results during Performance Testing of the Oracle Fusion platform showed a steady decline in both response times and throughput as more services were deployed to the platform.

When the first services that called downstream SOAP web services were deployed it was noticed that the call time was taking far longer than expected. This was the first issue that had to be resolved and details can be found in the Port Issue section below.

After the Port Issue was resolved Performance Testing highlighted degrading performance as more services were added. Eventually this was attributed to XML Marshalling and Unmarshalling and details on the specifics can be read in the relevant section below.

Port Issue

The port issue is described in this link. Invoking Client Web Service Slow Under Load

XML Marshalling / Unmarshalling taking time

A combination of the looking in the logs during a performance test run, and analysing thread dumps lead to the finding that XML Marshalling and Unmarshalling was taking far longer than expected.

The log snippet below shows an outgoing response from the XXX service. The interesting point to highlight is the responseTimestamp in the message and the actual log timestamp. You can see that the time the response was set in the code to when it actually made it to the log file is 700 odd ms.

The thread dumps then highlighted that multiple threads were attempting to access the unix file system during XML marshalling / Unmarshalling. Below is snippet from the thread dump.

Thread Dump Expand source
The root cause of the problem was the fact Weblogic was having to look up various XML related factories and parser implementations by searching the classpath and looking in JAR files.

When searching for the parser factory the following search order is followed :

Use the factory specified by the system property.
Use the factory specifed in jre lib properties file
Search the classpath for META-INF/services/file
Use the JDK defined default factory.
When the number of services increases and the load goes up the Weblogic threads are having to wait on the file system and this contention manifests itself with each request having to wait longer to complete.

In order to resolve the issue a series of JVM system properties were added to the setDomainEnv.sh file to tell Weblogic which factories to use and thus prevent it searching the class path. (point one in search order above).

The system properties added were as follows :

System Property
Description
-Djavax.xml.stream.XMLInputFactory=weblogic.xml.stax.XMLStreamInputFactory

Weblogic Stax Input Parser Factory Class
-Djavax.xml.stream.XMLOutputFactory=weblogic.xml.stax.XMLStreamOutputFactory

Weblogic Stax Output Parser Factory Class
-Djavax.xml.parsers.SAXParserFactory=weblogic.xml.babel.jaxp.SAXParserFactoryImpl

Weblogic SAX parser Factory Class
-Djavax.xml.transform.TransformerFactory=com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl

TransformerFactory for XLST transformation.s
-Dcom.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager=com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager