[DAS] SOAP/XML: best way to marshall large objects

[Brian Gilman]

We have taken the last approach that you mentioned. The omnidas API has a
DASResponse object which encapsulated an XML stream. This stream is then
passed to another API which forms "DAS Objects" (JAXB see sun web site).
We tried the DOM style and quickly threw that out becuase it ate too much
memory (your first suggestion). It looks like SOAPlite uses SAX in your
description of the second and incrementally (??) parses the document or
does it make a request and repeatedly parses the document keeping a
pointer into it? Anyway, most people I've talked to like the course
grained approach. Get the whole document to the persons computer and then
do with it what you will. This is the approach we are using and are not
seeing too much delay. When Ewan talked about seeing delays that did not
return I forgot to tell him (and everyone else) that we had to turn off
the WICGR web service. 


			HTH,

				-B

-----------------------
Brian Gilman <gilmanb@genome.wi.mit.edu>
Sr. Software Engineer MIT/Whitehead Inst. Center for Genome Research
One Kendall Square, Bldg. 300 / Cambridge, MA 02139-1561 USA
phone +1 617  252 1069 / fax +1 617 252 1902


On Tue, 8 Jan 2002, Lincoln Stein wrote:

> Hi All,
> 
> I have a question for Brian, Thomas, and anyone else who has good experience 
> with SOAP/XML.
> 
> I have just spent some time with the Perl SOAPLite modules, and am wondering 
> how best to marshall large data structures in SOAP.  The case of concern is 
> the DAS features request, which in SOAPish form becomes the 
> getSequenceFeature service.
> 
> This request returns a large number of features, typically thousands.  In the 
> Perl SOAP API, there are two alternative ways to handle this.  One processes 
> the entire response in one big swallow and returns an array of objects.  The 
> other creates a getNextSequenceFeature iterator and calls the iterator 
> repeatedly to fetch each feature.
> 
> Neither method is particularly optimal.  The first wastes a lot of time 
> parsing the XML stream and eating up memory before it returns a result that 
> the user code can work with.  The second makes a network method call for each 
> feature, killing the transaction with latency delays.
> 
> Of course, I can also get the XML stream from the SOAP request, pass it to a 
> parser, and invoke callbacks, and perhaps this is the best way to do it.  But 
> I wonder what experience and advice people who are experienced with the Java 
> APIs have on this before I jump to the wrong conclusions.
> 
> Lincoln
> 
> -- 
> ========================================================================
> Lincoln D. Stein                           Cold Spring Harbor Laboratory
> lstein@cshl.org			                  Cold Spring Harbor, NY
> ========================================================================
> _______________________________________________
> DAS mailing list
> DAS@biodas.org
> http://biodas.org/mailman/listinfo/das
>