[DAS] SOAP/XML: best way to marshall large objects

[Lincoln Stein]

Hi All,

I have a question for Brian, Thomas, and anyone else who has good experience 
with SOAP/XML.

I have just spent some time with the Perl SOAPLite modules, and am wondering 
how best to marshall large data structures in SOAP.  The case of concern is 
the DAS features request, which in SOAPish form becomes the 
getSequenceFeature service.

This request returns a large number of features, typically thousands.  In the 
Perl SOAP API, there are two alternative ways to handle this.  One processes 
the entire response in one big swallow and returns an array of objects.  The 
other creates a getNextSequenceFeature iterator and calls the iterator 
repeatedly to fetch each feature.

Neither method is particularly optimal.  The first wastes a lot of time 
parsing the XML stream and eating up memory before it returns a result that 
the user code can work with.  The second makes a network method call for each 
feature, killing the transaction with latency delays.

Of course, I can also get the XML stream from the SOAP request, pass it to a 
parser, and invoke callbacks, and perhaps this is the best way to do it.  But 
I wonder what experience and advice people who are experienced with the Java 
APIs have on this before I jump to the wrong conclusions.

Lincoln

-- 
========================================================================
Lincoln D. Stein                           Cold Spring Harbor Laboratory
lstein@cshl.org			                  Cold Spring Harbor, NY
========================================================================