[MOBY-dev] data by reference - a request for comments

[Dmitry Repchevsky]

Hello!

Here are my comments. Please note that all they are in mode IMHO, 
because I understand the complicity of the problem and just looking for 
possible solution.
> [A] Well, the first purpose (A) is obvious: we want to be able to deliver huge data from a service.
It depends.
If we are going to stay with SOAP (and especially move to doc/lit) the 
right way is to provide streaming through SOAP Attachments (SwA) or even 
better using MTOM.
This way we have to introduce a special Moby Primitive Type something 
called "BinaryAttachment" which will be bind to [type="base64Binary"]

This type will be replaced with:

<BinaryAttachment href="cid: my_long_streamed_file"/>
...
</SOAP:Envelop>
--MIME_boundary
Content Type:image/gif
Content-Transfer-Encoding:binary
Content-ID:<my_long_streamed_file>
... (file content)
--MIME_boudary

In case of using MTOM:

<BinaryAttachment>
  <xop:Include xmlns:xop="http://www.w3.org/2004/08/xop/include" 
href="cid: my_long_streamed_file"/>
</BinaryAttachment>
...
</SOAP:Envelop>

------=_Part_1_4558657.1118953559446
Content-Type: image/jpeg
Content-ID: <my_long_streamed_file>


Obviously JAVA supports both, but considering Perl, I am not sure which 
is better
(I am very pessimistic in SOAP Perl capabilities... :-( )


> The second purpose (B) is to be able to send around already existing references (such as URLs of the EMBL or NCBI records).
Here is the question where ever we continue to invent standards or 
trying to use SOAP capabilities...
Finally we have WS-Addressing  (and even have it implemented in Perl :-) ).
More we have WSRF that is exactly for this purpose - reference for resource.

> Dmitry suggested to use WSRF. We think that he meant something else: It
> could be used instead of the whole Moby message - but that is not what we
> are looking for. 
I am afraid you didn't understand me well... WSRF doesn't mean we have 
to send ALL the message.
It is only the way to reference a resource. Even in moby-asynchrony we 
return the whole message if needn't be so.
We can perfectly return a reference to ANY part of the original request. 
It is up to the service to locate the asked resource and send it back to 
the WSRF request.

We shouldn't confuse "A" and "B" because they are different.
"A" is to stream the data which in many case means "give me a huge file".
"B" is to keep the reference to it (something temporal like a 
calculation that produces a lot of data and we don't want to pass it 
around).
> We are looking for replacing just data part by references,
> and we want still to keep the original Moby message as it is used now. So we
> have concluded: no WSRF.
Don't understand it... How you suppose to replace some object in Moby 
message by refernce without changing it?

> *How can a client tells a service that she is sending a reference instead of
> data?* This could be useful for chaining services. We have not talked about
> it. Ideas welcome.
There is no way to specify that it can be send both by value and by 
reference and the same type...

I suggest - the service expecting the usual "value", but check if there 
is WSRF resource sent in header.
If it is it just calls the service (or in case it is the same server, 
just locate it locally) to get the object (not all Moby message, but 
just serialized Moby object, like AminoacidSequence) and use it instead 
the EMPTY one in MobyMessage.

Something like - We expect two AminoacidSequence(s) - 
articleName="first", articleName="second"
In header we put WS Addressing (WSRF) resource to "server1/second"
The service check headers for resources and call the server1 to obtain 
"second" Aminoacid.
This can be useful to make a workflow too... because we can call the 
server1 to get AminoacidSequence2, obtain reference to it AND send it to 
the server 2, so actually server2 will get the reference to it...
In case server1 and the server2 are the same (this is the main idea of 
[B]?) we do not need to pass around and get it directly... in case of 
server1 != server2 we have a ... Moby "GRID" :D

> I do not like using the RDF for mandatory features. The RDF, at least in the
> current moby, is optional and should not carry anything that is significant
> for the service and its behaviour.
Agree. IMHO I would use WSDL capability to describe it all, but having 
so many types of services this could be very hard...

IMO the main problem is that BioMoby is using SOAP, but use nothing of 
it trying to replace all W3C and WS-I specification by it's own solution.
So the question is why do we use SOAP?

I see (in a long term) two possibilities:

1. Forget about SOAP and do the REST (which is a fancy name to say we 
are going to send what ever we want without any standard)
2. Use SOAP and Web Services architecture.

Best regards,

Dmitry