[MOBY-dev] Binary data streaming

[Pieter Neerincx]

Hi,

On 01 Aug 2008, at 02:03, Martin Senger wrote:

>> the correct way to implement streaming
>
>
> What do you mean by "implement streaming"? I concluded that you meant
> "sending binary data from (or to) a moby service". Is is correct?
>
>
>> would be using XOP/MTOM specifications
>
>
> This is again a SOAP-based specification, meaning another way how to  
> make an
> attachment (correct?).

Well theoretically an attachment would be ideal. It makes sure the  
attachment can be parsed separately to provide scalability.  
Furthermore, you always keep the data "together". With "external"  
references the data is separated and has to go over the wire in two  
different sessions. This way data might get lost. If the first session  
uses SOAP over HTTP with a reference to data available only over FTP  
and my client or service doesn't talk FTP, I received only half of the  
package. With an attachment it either arrives completely or it doesn't  
arrive at all.

So, theoretically I'd love attachment support for BioMoby services,  
but my previous questions remained unanswered:
1. Is there any Perl support for such XOP/MTOM attachments. I mean a  
module that actually *works* in contrast to the MIME-tools package.
2. Do XOP/MTOM attachments scale beyond the reach of the casual  
firewall? Most firewalls block MIME/DIME attachments over 5 MB in  
size. Attachments ain't going to improve scalability of BioMoby  
services if 99% of our users scale to a max of 5 MB.

Martin, if you are at the EBI right now, you might ask for some advise  
at the external services group. I know they are one of the few smart  
guys who managed to get their SOAP services working with MIME  
attachments and hit the firewall limits. It was at ISMB in Fortaleza  
2006 when I talked to them about these issues, so the situation might  
have improved by now...

Still, even if we wouldn't need it for scalability, pass-by-reference  
would be very handy so you can reference existing data from others  
(like a reference to a UniProt record on a EBI server.)

So unless the issues above are resolved, attachments are theoretically  
maybe a nicer solution, but I'd say pass-by-reference is practically  
the best solution to improve scalability in a pragmatic way.

Cheers,

Pi


> I have feeling that our current proposal about the data-by-reference  
> (that I
> am going to summarize in the next few hours) has a reasonable tools  
> for
> getting binary data - if sent as a reference. If the protocol for
> dereferencing is HTTP, we can use MIME type (content-type HTTP  
> header) for
> references data.
>
> Martin
>
> -- 
> Martin Senger
> email: martin.senger at gmail.com,m.senger at cgiar.org
> skype: martinsenger
> _______________________________________________
> MOBY-dev mailing list
> MOBY-dev at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/moby-dev

-------------------------------------------------------------
Wageningen University and Research centre (WUR)
Laboratory of Bioinformatics
Transitorium (building 312) room 1034

Dreijenlaan 3
6703 HA Wageningen
The Netherlands

phone:  +31 (0)317-483 060
mobile: +31 (0)6-143 66 783
e-mail: pieter.neerincx at gmail.com
skype:  pieter.online
-------------------------------------------------------------