[MOBY-l] parsing large XML using java

[Hoekman, Berend]

Hi M. Senger,

The problem occurs before my service can start parsing the input, in the processIt method.

When debugging i get the impression that the error is caused by tomcat or axis. 
This because when I send the xml to an empty service (non-moses/biomoby), I get the same memory error. 

Chunking has been considerd but the xml stucture is in such a format that this is impossible.
  -The main reason is that multiple collections have to be compared and be merged at the correct position.
  -(this refers to other xml then the one in: "simpleXMLWhichCausesOutOfMemoryError.txt")

I included two files 
  -the example xml which causes the outOfMemoryMessage. 
   (it is plain simple and could be chunked but it purpose was only to determine what the xml-limit of the java-moses services is.)
  -the java file which supposably does't parse the input (and is probably therefore not related to the problem). 
   (the service accepts a collection of moby objects, did not specify any namespaces)

If anything more is needed, please let me know.

Thanks for looking into this matter,
Berend



-----Original Message-----
From: Martin Senger [mailto:martin.senger at gmail.com]
Sent: Fri 11/10/2006 10:28 AM
To: Hoekman, Berend
Cc: moby-l at lists.open-bio.org
Subject: Re: [MOBY-l] parsing large XML using java
 
Hi,

I have a problem parsing large xml


This can, unfortunately, happen. Unless you design your web services
interface in a way that it can accept and send large data in chunks. But
BioMoby API is not prepared for it (at least, not yet).

Having said that, I have also admit that many times the problem can be
diminished (or even removed completely) by looking how various involved
software pieces (client, service, etc.) are implemented and improve them. I
can look how it can be done better with the jMoby-Moses software.

Could you tell me who is reporting the outofmemory error - is it Tomcat? (I
guess, it is, reading your email.) The error happens with large data coming
to your service, or created by your service? Is it possible to have that
large input (or output) that causes the problem so I can play with it here?

Thanks and regards,
Martin


-- 
Martin Senger
   email: martin.senger at gmail.com
   skype: martinsenger



-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: simpleXMLWhichCausesOutOfMemoryError.txt
URL: <http://lists.open-bio.org/pipermail/moby-l/attachments/20061110/f65dff59/attachment-0002.txt>