XML Processing Measurements using XPB4J

Author: Pankaj Kumar
e-mail: pankaj_kumar@acm.org
Date: September 30, 2001

Mesurement Environment

Software environment consisted of:

• Java Software: JDK1.3 from Sun Microsystems.
• Crimson Parser: crimson-1.1.2 beta2 from Apache Software Foundation.
• Xerces Parser: xerces-2.0.0 beta2 from Apache Software Foundation.
• Xalan Processor: xalan-2.2.D10 from Apache Software Foundation.
• Cocoon: Cocoon2 beta1 from Apache Software Foundation.
• Ant Software ( for build only ): jakarta-ant-1.4 from Apache Software Foundation.

Measurement Process

      // Code for illustration only. Won't compile.
      for (int i = 0; i < loopCount; i++)
        {
        System.gc();
        long startMem = Runtime.totalMemory() - Runtime.freeMemory();
        long startTime = System.currentTimeMillis();
        result = process();	// Do the processing.
        long endTime = System.currentTimeMillis();
        long endMem = Runtime.totalMemory() - Runtime.freeMemory();
        memUsed[i] = endMem - startMem;
        elapsedTime[i] = endTime - startTime;
        }

Both Crimson and Xerces parsers were used for the measurement. To use Crimson parser, crimson.jar was kept in .\lib directory. Similarly, to use Xerces parser, xerces.jar was kept in .\lib directory. Note that the run.bat, used for kicking-off the execution run, took jar files from .\lib.

Following precautions were taken while making the measurements:

1. No other activity was happening on the machine.
2. Only one processing method was enabled in file args.xml for each execution run to ensure warmup for each processing method.
3. Multiple execution runs were observed to ensure consistency of measurement figures.

Input Data

Five different input files, labelled as TINY, SMALL, MEDIUM, BIG and HUGE, were used. Their characteristics are shown in the table below:

Label	rxgen arg.	Filesize
TINY	20	9KB
SMALL	200	72KB
MEDIUM	2000	719KB
BIG	20000	7.18KB
HUGE	200000	71.6MB

You can generate a MEDIUM input file by issuing:

       >rxgen 2000

Note: There could be slight variation in size of the generated file everytime rxgen is run, even if the argument remains same.

Mesurements

Conclusions from Measurements

Careful analysis of the measurement figures under Mesurements and with few additional experimentation, a number of inferences can be made.

Note: The following conclusions are valid only for XStat processing under the specified Mesurement Environment. Care must be taken to apply these results to other situations.

1. SAX Processing is faster than DOM processing and uses much less memory. The table below shows the average processing time and memory used with SAX and DOM processing using Crimson parser.

   Input Size	Processing Time			Memory Used
   	SAX	DOM		SAX	DOM
   TINY	24ms	36ms		101KB	142KB
   SMALL	58ms	91ms		172KB	385KB
   MEDIUM	310ms	604ms		279KB	2.90MB
   BIG	2.95s	5.97s		352KB	26.5MB

   Interestingly, the method of measurement favours the DOM processing as the garbage collection is forced outside the measurement window. As DOM processing uses more memory objects, more time will be spent in garbage collection. So, in a real scenario, the DOM processing will have much worse performance than SAX.

   Not surprisingly, the amount of memory used by SAX processing remains almost constant with input size whereas the memory used DOM processing increases in proportion to the input size. This could be a real problem for applications that are going to deal with large input or multiples of them simultaneously. DOM just won't scale.

   The slight increase in memory for SAX processing with increase in input size is attributed to the memory objects holding the processing results and other bookkeeping info.

2. Crimson is faster than Xerces and uses less memory. The table below shows the average processing time and memory used with SAX processing using Crimson and Xerces parser.

   Input Size	Processing Time			Memory Used
   	Crimson	Xerces		Crimson	Xerces
   TINY	24ms	24ms		101KB	37KB
   SMALL	58ms	74ms		172KB	86KB
   MEDIUM	310ms	367ms		279KB	411KB
   BIG	2.95s	3.57s		352KB	450KB
   HUGE	28.6s	36.1s		378KB	470KB

   Note that the difference in processing speed is insignificant at small input size but is quite pronounced at very large input size. Another curious thing to note is that Xerces uses less memory for smaller inputs. Both these facts together could mean better performance with Xerces at moderate input size ( < 50 KB ).

3. Pull Parser API is slower than SAX and uses significantly more memory. The table below shows the average processing time and memory used with SAX and Pull Parser processing using Crimson parser.

   Input Size	Processing Time			Memory Used
   	SAX	PULL		SAX	PULL
   TINY	24ms	24ms		101KB	104KB
   SMALL	58ms	80ms		172KB	746KB
   MEDIUM	310ms	610ms		279KB	1.24MB
   BIG	2.95s	5.14s		352KB	1.50MB
   HUGE	28.6s	52.84s		378KB	1.86MB

   This is hard to explain in theory as the Pull Parsing perhaps couldn't be more expensive. One practical explanation could be that the Pull Parser API is meant to be only peeking into first few elements and hence is not optimized for full document scan ( yet !).

4. XStat processing using XSLT is slowest and most demanding on RAM. The table below compares the average processing time and memory usage of XSLT method with SAX method.

   Input Size	Processing Time			Memory Used
   	SAX	XSLT		SAX	XSLT
   TINY	24ms	147ms		101KB	463KB
   SMALL	58ms	352ms		172KB	562KB
   MEDIUM	310ms	2.47ms		279KB	2.53MB
   BIG	2.95s	38.83s		352KB	22.7MB
   HUGE	28.6s	NA		378KB	NA

   It appears that the transformer loads the whole document in memory. This certainly is not required for XStat processing..

   Note: It might be possible to write XSL stylesheet in a way that avoids loading of whole document. If you know any such way, please do let me know.

5. JDOM is slower than DOM. This is obvious from the following table.

   Input Size	Processing Time			Memory Used
   	JDOM	DOM		JDOM	DOM
   TINY	47ms	36ms		162KB	142KB
   SMALL	111ms	91ms		518KB	385KB
   MEDIUM	860ms	604ms		3.17KB	2.90MB
   BIG	16.0s	5.97s		26.9MB	26.5MB

   One might think that JDOM would be better at runtime and memory usage efficiency as it uses native Java classes. In reality, it compares poorly with DOM. But may be because JDOM software is still in beta and necessary optimizations haven't been carried out.

6. Latest version of Xalan (2.2.D10) is significantly better than an eralier version (2.1.0). This is evident from the table below:

   Input Size	Processing Time			Memory Used
   	xalan-2.1.0	xalan-2.2.D10		xalan2.1.0	xalan-2.2.D10
   MEDIUM	8.57s	2.48s		4.7MB	2.53MB
   BIG	756.0s	38.0s		46.6MB	22.7MB

   The good news is that xalan performance is improving. However, as of now, you are better of avoiding XSL transformations in time critical situations.

------------------